2026-08-25 · run 2026-08-25-s14-disk-exhaustion

S14: the bigger realm is the safer one, and a half-migrated schema resumes

Summary

Running out of disk during the migration does break it, and it breaks it at one specific place: CREATE INDEX IDX_USER_CREATED_TIMESTAMP, which needs a temporary sort file that will not fit.

Three things make this the most interesting result of the set.

The amount of headroom needed is tiny. A 224 MB database needed somewhere between 4 MB and 8 MB free. At 8 MB the upgrade completed and used 1.58 MB; at 4 MB it died.

It leaves a genuinely half-applied changelog — DATABASECHANGELOG stops at 187 of 211 rows, committed, with migration_model still 26.0.0. Freeing the space and restarting completed it in 17 seconds, to 211 and 26.7.1, with the index built and all 100,002 users intact.

And the failure only happens on the smaller realm.

⚠ QUALIFIED 2026-08-26. True only while Keycloak is declining the index build. Force the build at 2,000,003 users — which a stale pg_class.reltuples does on its own — and the floor is 125–150 MB, roughly 20× the figure below. Scaling up removes the trigger, not the failure mode, and multiplies the damage if anything pulls it. See 2026-08-26-s14-revisited-disk-floor-at-2m. The index that runs out of space is the same one 2026-08-25-index-skip-threshold shows Keycloak skipping above 300,000 rows. At 100k it is built, and needs temp space. At 2M it is declined, and needs none. Scaling the realm up removes this failure mode. That is the opposite of what anyone provisioning a maintenance window would assume.

Environment

The box's root filesystem is 226 GB, so exhaustion is not reachable there. Free space at the moment of migration is set exactly, with a ballast file, so each run states a real number rather than "nearly full".

Results

The floor is between 4 MB and 8 MB free, for a 224 MB database.

Where it dies, and why

ERROR: could not write to file "base/pgsql_tmp/pgsql_tmp705.0.fileset/0.0":
       No space left on device
[Failed SQL: (0) CREATE INDEX IDX_USER_CREATED_TIMESTAMP ON public.USER_ENTITY(REALM_ID, CREATED_TIMESTAMP)]

base/pgsql_tmp is Postgres's temporary-file area. Building an index sorts, and a sort that exceeds maintenance_work_mem (64 MB here, stock) spills to disk. The space that runs out is not the space the finished index occupies — it is the scratch space needed to build it.

That is why the margin is so small and so easy to miss: the finished index is about 2 MB, and the sort needs more room than the result.

Why a larger realm would not hit this

CustomCreateIndexChange skips index creation on tables above 300,000 rows (2026-08-25-index-skip-threshold). This realm has 100,002 users, so the index is built — the run confirms it, with skipped_idx=0 in every leg, and the index present after recovery.

At 2M users Keycloak declines to build it, logs a WARN, marks the changeset EXECUTED, and moves on. No sort, no temp file, no exhaustion. The dangerous size band for this failure is below the skip threshold, and a customer who manually created the indexes Keycloak declined has re-entered it at any size.

Recovery: the half-migrated schema resumes

From the failed state — DATABASECHANGELOG 187, migration_model 26.0.0 — removing the ballast and restarting:

Liquibase resumed from the last committed changeset. No manual intervention, no lock clearing, no restore.

This is the third route this lab has found to a stuck upgrade (after S8's pooler and S12's database floor) and the second to a genuinely inconsistent database — and in both cases the recovery was "fix the environmental cause and start it again". 2026-08-25-s1-s2-lock-failure-modes reached that conclusion without ever producing a half-migrated schema. It has now been produced twice, and the conclusion held both times.

Verification

Publishable extract

Your Keycloak upgrade can run out of disk with 4 MB free and succeed with 8. And the bigger your realm, the less likely it is to happen.

We put a 100,002-user Keycloak database — 224 MB — on a 2 GB filesystem and upgraded 26.0.0 → 26.7.1 with the free space set to an exact figure each time. With 8 MB free it completed in 17 seconds and consumed 1.58 MB. With 4 MB it died:

ERROR: could not write to file "base/pgsql_tmp/...": No space left on device
[Failed SQL: (0) CREATE INDEX IDX_USER_CREATED_TIMESTAMP ON public.USER_ENTITY(REALM_ID, CREATED_TIMESTAMP)]

base/pgsql_tmp is the giveaway. The space that ran out is not the space the index occupies — it is the temporary sort file Postgres needs to build it, which is larger than the result. Free space equal to your index size is not enough.

Now the counter-intuitive part. That index is the one Keycloak refuses to build on tables over 300,000 rows — it logs a warning, marks the changeset executed, and moves on. So a 2-million-user realm never runs that sort, never writes that temp file, and never hits this. The vulnerable realms are the small ones, and anyone who has manually created the indexes Keycloak declined has put themselves back in range at any size.

Finally, the reassuring half. The failure left a genuinely half-applied schema — 187 of 211 changesets committed, migration_model still on the old version. We freed the space and started Keycloak again. Seventeen seconds later: changelog 211, model 26.7.1, index built, all 100,002 users present, admin login returning 200. Liquibase resumed from the last committed changeset.

If you find a Keycloak stopped partway through a migration, the first thing to try is starting it again — after you have fixed whatever stopped it. That advice has now survived every way we have found to break one.