S3: `statement_timeout` cannot half-migrate this path
- Upgrade
- 26.0.0 → 26.7.1
- Scale
- 2,000,003 users, 2 realms
- Database
- PostgreSQL 16.15 (Debian 16.15-1.pgdg13+2), single node, container
- Topology
- single container
- Host
- Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
- Condition
- statement_timeout mid-flight
Summary
⚠ CONCLUSION OVERTURNED 2026-08-26. It does produce one. At 2,000,003 users a
statement_timeoutof 500 ms, 200 ms or 50 ms leavesDATABASECHANGELOGat 157 of 211 withmigration_modelon 26.0.0 and the server dead — seven times in ten identical runs.The measurements below are correct and the reasoning from them is not. The statement that dies is
ALTER TABLE CREDENTIAL ADD VERSION INTEGER DEFAULT 0, which executes in 0.177 ms and waits about a second for anAccessExclusiveLockheld by an autovacuum worker — Postgres evicts a blocking autovacuum only afterdeadlock_timeout, andstatement_timeoutis charged for the wait. This record measured statement durations on a single run that happened to be uncontended.See
2026-08-26-s3-revisited-autovacuum-lock-wait. Everything below stands as a measurement of an uncontended migration; nothing below should be quoted as a conclusion aboutstatement_timeout.
S3 was written to produce the half-migrated schema our failure-class notes name. It does not produce one. On this path, at
2,000,003 users, the longest single SQL statement the migration issues is
2.142 ms. There is no statement_timeout value that lands between "too short
to start" and "long enough to finish": at 2 ms Keycloak fails before applying
any changeset, at 3 ms it completes the whole migration. The band that would
leave a schema half-applied is empty.
This is the second scenario in a row (after S1/S2) that fails to reproduce a failure class we have been citing.
Environment
| Field | Value |
|---|---|
| Keycloak from → to | 26.0.0 → 26.7.1 |
| Distribution / start mode | quay.io/keycloak/keycloak official image, start w/ external DB |
| Database | PostgreSQL 16.15 (Debian 16.15-1.pgdg13+2), single node, container |
| Postgres tuning | shared_buffers 128MB, maintenance_work_mem 64MB — stock; statement_timeout is the variable, 10s / 3ms / 2ms |
| Adverse scenario | S3 |
| Dataset scale | 2,000,003 users, 2 realms |
| Seeding method | direct SQL COPY (bin/seed-sql.sh) |
| Topology | single container |
| Host | Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe |
| JVM heap | -Xms1g -Xmx4g |
| Instrumentation | log_min_duration_statement = 0 for the measurement run |
Procedure
./bin/restore.sh baseline-26.0.0-sqlseed-2m
docker compose exec -T db psql -U keycloak -c "alter system set log_min_duration_statement=0" -c "select pg_reload_conf()"
./bin/upgrade.sh 26.7.1 # measurement run
docker compose logs db | grep 'LOG: duration:' | sort -gr # rank statements
# then, per timeout value:
./bin/restore.sh baseline-26.0.0-sqlseed-2m
docker compose exec -T db psql -U keycloak -c "alter role keycloak set statement_timeout='<T>'"
./bin/upgrade.sh 26.7.1
What we could not observe
No half-migrated schema, at any timeout value. The empty band is a consequence of every statement in the migration falling inside 1.8–2.2 ms: the parse-phase reads and the DDL writes are the same order of magnitude, so a timeout tight enough to kill a changeset is also tight enough to kill the parse that precedes it. That coincidence is a property of this pair at this scale, not a general law, and we did not test whether it holds elsewhere.
| Claim | Primary source | Checked |
|---|---|---|
| Longest migration statement is 2.142 ms at 2M users | log_min_duration_statement=0 output, ranked | ☑ |
| 10s timeout has no effect on this path | full run, changelog 211, 2M users intact | ☑ |
| 2ms fails before applying any changeset | changelog still 144, model still 26.0.0 | ☑ |
| Failure leaves no held lock | databasechangeloglock read directly, both rows f | ☑ |
| A half-migrated schema is reachable via timeout | no — refuted for this pair at this scale | ☑ |
| Holds on a pair with data-rewriting changesets | not tested — this pair has none | ☐ |
| Holds if the customer built the skipped indexes manually | not tested — the obvious next run | ☐ |
| Holds on other databases | not tested — Postgres only | ☐ |