S14 revisited: at 2M the disk floor is 150 MB, not 8 MB, and scaling up no longer saves you
- Upgrade
- 26.0.0 → 26.7.1
- Scale
- 2,000,003 users, database 3,026 MB
- Database
- PostgreSQL 16, single node, container
- Host
- Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
- Condition
- Disk exhaustion mid-migration
Summary
S14 found the migration dies at CREATE INDEX IDX_USER_CREATED_TIMESTAMP when
disk runs out, needing between 4 and 8 MB free on a 224 MB / 100,002-user
database — and concluded:
And the failure only happens on the smaller realm. … At 2M it is declined, and needs none. Scaling the realm up removes this failure mode.
That was true because Keycloak declines the build above 300,000 rows. Run A
showed a stale reltuples makes it build anyway, at any true size. So the
scenario is reachable at 2M, and there it is far worse:
At 2,000,003 users the floor is between 125 MB and 150 MB — roughly 20× the 100k figure. Scaling up does not remove the failure mode. It removes the trigger, and multiplies the damage if anything pulls the trigger.
Environment
| Field | Value |
|---|---|
| Keycloak from → to | 26.0.0 → 26.7.1 |
| Database | PostgreSQL 16, single node, container |
| Postgres tuning | stock — maintenance_work_mem 64MB, max_parallel_maintenance_workers 2 |
| Storage | 6 GB ext4 loopback at /mnt/tinypg, bind-mounted as pgdata (docker-compose.smalldisk.yml) |
| Dataset scale | 2,000,003 users, database 3,026 MB |
| Seeding method | seed-sql, restored from baseline-26.0.0-sqlseed-2m |
| Adverse scenario | S14 — free space at the moment of migration is the variable |
| Host | Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe |
| JVM heap | -Xms1g -Xmx4g |
How the build was forced. With honest statistics at 2M Keycloak declines the
index and never touches the disk, so pg_class.reltuples was set to 1000 with
autovacuum disabled on USER_ENTITY — the mechanism from
2026-08-26-index-precondition-anatomy. That is a deliberate fixture, and it
stands in for the customer state measured in
2026-08-26-stale-statistics-reachability: autovacuum off on a large table,
plus a loader that does not ANALYZE.
Method. A clean copy of the restored data directory is parked once and
restored before each level, so every leg starts byte-identical without paying 90
seconds for a fresh pg_restore. Free space is then set exactly with a ballast
file.
Results
| Free space at migration | Outcome | Index built | DATABASECHANGELOG | migration_model |
|---|---|---|---|---|
| 200 MB | READY | yes | 211 | 26.7.1 |
| 175 MB | READY | yes | 211 | 26.7.1 |
| 150 MB | READY | yes | 211 | 26.7.1 |
| 125 MB | FAILED | no | 187 | 26.0.0 |
| 100 MB | FAILED | no | 187 | 26.0.0 |
| 50 MB | FAILED | no | 187 | 26.0.0 |
| 25 MB | FAILED | no | 187 | 26.0.0 |
| 12 MB | FAILED | no | 187 | 26.0.0 |
Floor: (125 MB, 150 MB]. Against S14's (4 MB, 8 MB] at 100k.
Every failure is identical and stops at 187 of 211 — the same point as S14's own
failure and as the index-name collision in
2026-08-26-index-precondition-anatomy, because all three die on the same
changeset.
Breakage observed
Disk exhaustion at 2M needs 20× the headroom S14 measured
- Symptom: upgrade fails on
CREATE INDEX IDX_USER_CREATED_TIMESTAMP; changelog 187/211,migration_model26.0.0, server does not start. - Evidence: the sweep above; Postgres error quoted.
- Cause: confirmed — parallel external sort spilling to
pgsql_tmp. - Fix: free space, restart. (Not re-tested here; S14 showed the half-applied changelog resumes and completes.)
- Would this hit a customer? Scale-dependent and conditional on the index build being attempted at all — which needs a stale row estimate. Narrow, but the consequence is a failed maintenance window on the largest installations.
Verification
| Claim | Primary source | Checked |
|---|---|---|
Floor at 2,000,003 users is (125 MB, 150 MB] | eight levels, 150 READY / 125 FAILED | ☑ |
| Failures are identical and stop at 187/211 | count(*) and migration_model at every failing level | ☑ |
| The exhausted space is a parallel sort's temp fileset | Postgres error naming pgsql_tmp70.0.fileset, two backends | ☑ |
| Finished index is 14 MB; net build cost 14 MB | pg_relation_size and df before/after an unconstrained build | ☑ |
| Transient requirement is ~10× the finished index | 14 MB index against a >125 MB floor | ☑ |
| S14's "scaling up removes the failure mode" was conditional | it holds only while the build is declined; forcing it re-arms the scenario | ☑ |
| Recovery by freeing space and restarting | not re-tested at 2M — S14 showed it at 100k | ☐ |
The floor with maintenance_work_mem raised | not tested — stock 64MB throughout | ☐ |
| The floor with parallel maintenance workers disabled | not tested | ☐ |
| Whether the floor scales linearly between 100k and 2M | not tested — two points, no intermediate | ☐ |