S1/S2: Liquibase lock failure modes (could not reproduce)
- Upgrade
- 26.0.0 → 26.7.1
- Scale
- 100,002 users, typical profile
- Database
- PostgreSQL 16.15, single node, container
- Topology
- single container (S1); two containers, one compose network (S2)
- Host
- Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30GB RAM, local NVMe
- Condition
- routine — no adverse condition applied
Summary
We tried to reproduce the canonical Keycloak upgrade horror story — a killed migration leaving a stuck Liquibase lock that blocks every subsequent start — and could not, on Postgres 16 single-node. Every failure mode we induced recovered by itself.
This is a negative result and it is stated as one. It does not prove the story is false; it bounds where it applies.
Environment
| Field | Value |
|---|---|
| Keycloak from → to | 26.0.0 → 26.7.1 |
| Database | PostgreSQL 16.15, single node, container |
| Postgres tuning | stock — shared_buffers 128MB, maintenance_work_mem 64MB, statement_timeout 0 |
| Dataset | 100,002 users, typical profile |
| Topology | single container (S1); two containers, one compose network (S2) |
| Host | Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30GB RAM, local NVMe |
| JVM heap | -Xms1g -Xmx4g |
What we could not observe
The lock was never sampled in a held state, in either scenario. Polling ran
through docker compose exec … psql, roughly 200–400ms per sample, against a
migration that completes in ~3.5s. Absence of observation here is a limit of the
instrument, not evidence the lock is not taken. Do not read this record as
saying Keycloak does not lock — it says we never caught it holding one.
Verification
| Claim | Primary source | Checked |
|---|---|---|
| Half-migrated schema self-completes | changelog 165 → 211 across a restart, read from Postgres | ☑ |
| No stuck lock after SIGKILL | databasechangeloglock queried directly, both runs | ☑ |
| Concurrent start ends in a correct schema | changelog 211, model 26.7.1, users intact | ☑ |
| Lock is genuinely held at some point | not observed — sampling too coarse | ☐ |
| Holds on a slow migration | not tested — every migration here was ~3.5s | ☐ |
| Holds on Postgres HA / failover mid-migration | not tested | ☐ |
| Holds on other databases | not tested — Postgres only | ☐ |
| Holds for other version pairs | not tested | ☐ |
Publishable extract
We tried to reproduce Keycloak's most-repeated upgrade horror story and could not.
The story: a Keycloak upgrade dies partway, Liquibase's lock row is never released, and every subsequent start hangs waiting for a lock nobody holds — including the start you attempt in order to roll back.
On Postgres 16.15, single node, with a 100,002-user realm on a Hetzner CCX33 (8 vCPU dedicated, 32 GB, local NVMe), upgrading 26.0.0 → 26.7.1, we killed Keycloak with SIGKILL during the schema migration. Twice. The second time we waited until the migration was genuinely half-applied — 165 of 211 changesets committed — before killing it.
Both times the lock table showed zero held locks afterwards, and both times the next start completed the migration by itself in 13 seconds, ending at the correct schema version with all 100,002 users intact. We also started two instances against an un-migrated schema simultaneously; both came up, and the schema ended correct.
One honest caveat, and it is the interesting part. These migrations take about three and a half seconds — because Keycloak skips index creation on tables over 300,000 rows rather than block startup on them. The window in which a crash can hurt you is narrow because Keycloak works hard to keep it narrow. If your upgrade includes a changeset that genuinely rewrites a large table, or you have manually built the indexes Keycloak declined to build, your window is wider than ours and this result may not carry.
We would rather publish a negative result than repeat folklore. If you have a reproducible stuck lock, we want the configuration — we will run it and publish what happens.