2026-08-26 · run 2026-08-26-ha-failover-mid-migration

HA: primary crash mid-migration, promoted replica resumes cleanly

Summary

First run of the HA/failover line. A streaming replica was stood up behind the primary, the migration was started, the primary was SIGKILLed mid-migration (changeset 26.6.0-45009-broker-link-identity-provider), the replica was promoted, Keycloak was repointed at it, and it resumed and finished cleanly.

Four findings, in the order they matter:

1. The replica is a consistent, half-migrated snapshot. At the crash the replica had 179 of 211 changesets applied — every committed changeset, with the in-flight one rolled back. Streaming replication does not ship a half-applied transaction, so the failover target is never corrupt.

2. migration_model is the way to see it. At the crash the replica's changelog said 179 changesets but migration_model still said 26.0.0, because the model stamp is written last. The half-migrated state is therefore detectable, not silent — the same "changelog ≠ model" signal S8 taught us to read, now produced by a crash instead of a pooler.

3. The changelog lock is NOT a stuck-lock hazard across failover. After the crash the replica's DATABASECHANGELOGLOCK rows were LOCKED=f — the crashed primary's lock acquisition did not persist to the standby. Keycloak re-acquires the lock and resumes rather than waiting out a lock held by a dead host.

4. Keycloak resumes, it does not restart. Re-pointed at the promoted replica, Keycloak ran changesets 180 → 211, stamped the model 26.7.1, and reached ready with all 202 users intact. Index audit clean (119/119).

Environment

Procedure

# primary + keycloak, then seed + snapshot (replica initialised by hand)
COMPOSE_FILE=docker-compose.yml:docker-compose.ha.yml
docker compose up -d db keycloak
./bin/seed-realm.sh --profile typical --users 200
./bin/snapshot.sh baseline-26.0.0-ha-typical-200
./bin/ha-init-replica.sh          # pg_basebackup + standby.signal + primary_conninfo

# failover: start migration, SIGKILL primary mid-flight, promote, repoint
sed -i 's/^KC_VERSION=.*/KC_VERSION=26.7.1/' .env
docker compose up -d keycloak &    # build + migrate
# poll for "Updating database", sleep 1s, then:
docker compose kill db            # SIGKILL, not a clean stop — a crash
./bin/ha-failover.sh              # pg_ctl promote + KC_DB_URL -> db-replica
./bin/index-audit.sh 26.7.1       # DB_SERVICE=db-replica

Results

The crash point

Keycloak was mid-changeset when the primary died:

ERROR: Migration failed for changeset META-INF/jpa-changelog-26.6.0.xml::
  26.6.0-45009-broker-link-identity-provider::keycloak:
  Reason: ... PSQLException: An I/O error occurred while sending to the backend.

The replica (still a standby) held, at that instant:

After promote + repoint

The old primary, restarted by depends_on, crash-recovered to its stale 179-changeset state and was then stopped — the promoted replica is the only primary.

What this teaches

• Failover mid-migration is safe to rehearse and to expect: the standby is always a committed-prefix snapshot, and Keycloak resumes from that prefix. No corrupt schema, no stuck lock, no data loss.
• The pre-flight for a customer running HA: the check after a failover is still "does migration_model match the target version" — not "did the changelog reach N rows". The crash leaves them divergent by construction.
• This closes the last Compose-reachable part of the failover scenario. What is still untested: synchronous replication and a clean (promoted-by-choice) failover — a DBA promoting the replica for maintenance while Keycloak keeps writing — which is a different failure shape than a crash.

Verification