Backup/rollback tooling defect, and verified rollback at 2M
Verification — full loop at 2,000,003 users
| Step | Result |
|---|---|
| Restore baseline to 26.0.0 | schema 26.0.0, no re-migration |
upgrade.sh 26.7.1 | auto-backup pre-upgrade-26.0.0-to-26.7.1-20260825T182154Z (65 MB), READY |
| Canary written post-upgrade | users 2,000,003 → 2,000,004 |
restore.sh <backup> — no --version, inference only | |
| → running image | quay.io/keycloak/keycloak:26.0.0 |
→ migration_model | 26.0.0 |
| → users | 2,000,003 |
| → canary | absent — correctly lost |
Timings
| 1,000 users | 2,000,003 users | |
|---|---|---|
Backup (pg_dump -Fc) | <1s | ~6s (65 MB from a 3.5 GB database) |
| Upgrade overhead from backup | negligible | ~8s on a 16s upgrade |
| Full rollback to serving | 15.9s | 70s |
Rollback time scales with data; migration time does not. At 2M the migration is 3.84s and the rollback is 70s — the rollback is ~18× the migration it undoes. That inverts the usual assumption and is the number that actually matters when deciding whether to roll back mid-window.
Publishable extract
We added an automatic backup to our upgrade tooling, and testing it revealed our rollback had never worked.
Restoring a pre-upgrade snapshot put the old schema back — and then the running container, still the new Keycloak, immediately re-ran the migration. The restore looked successful: every row was there. But the schema was back on the new version and every write since the snapshot was gone. The worst of both.
The cause is one word.
docker compose startreuses the existing container; only--force-recreatepicks up the image you just rolled back to. Any rollback procedure that restores a database without also pinning the binary version has this bug, whatever tooling it is written in.Two things worth taking from it. First, verify a rollback by reading the schema version back out of the database afterwards, not by confirming your data is present — the data is present in the broken case too. Our restore now compares
migration_modelbefore and after restart and fails loudly if the server re-migrated.Second, the timings. On a 2,000,003-user realm (Postgres 16.15, Hetzner CCX33 — 8 vCPU dedicated, 32 GB, local NVMe), the upgrade's schema migration takes 3.8 seconds. The rollback takes 70 seconds — about eighteen times longer than the thing it undoes. Migration duration is flat with realm size; restore duration is not. If you are sizing a maintenance window, the rollback is the expensive half, and it is the half nobody measures.