The Keycloak Upgrade Ledger

Every upgrade we have rehearsed, with the environment stated and the clock running.

2026-08-25 · run 2026-08-25-s3-statement-timeout

S3: `statement_timeout` cannot half-migrate this path

Upgrade
26.0.0 → 26.7.1
Scale
2,000,003 users, 2 realms
Database
PostgreSQL 16.15 (Debian 16.15-1.pgdg13+2), single node, container
Topology
single container
Host
Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
Condition
statement_timeout mid-flight

Summary

⚠ CONCLUSION OVERTURNED 2026-08-26. It does produce one. At 2,000,003 users a statement_timeout of 500 ms, 200 ms or 50 ms leaves DATABASECHANGELOG at 157 of 211 with migration_model on 26.0.0 and the server dead — seven times in ten identical runs.

The measurements below are correct and the reasoning from them is not. The statement that dies is ALTER TABLE CREDENTIAL ADD VERSION INTEGER DEFAULT 0, which executes in 0.177 ms and waits about a second for an AccessExclusiveLock held by an autovacuum worker — Postgres evicts a blocking autovacuum only after deadlock_timeout, and statement_timeout is charged for the wait. This record measured statement durations on a single run that happened to be uncontended.

See 2026-08-26-s3-revisited-autovacuum-lock-wait. Everything below stands as a measurement of an uncontended migration; nothing below should be quoted as a conclusion about statement_timeout.

S3 was written to produce the half-migrated schema our failure-class notes name. It does not produce one. On this path, at 2,000,003 users, the longest single SQL statement the migration issues is 2.142 ms. There is no statement_timeout value that lands between "too short to start" and "long enough to finish": at 2 ms Keycloak fails before applying any changeset, at 3 ms it completes the whole migration. The band that would leave a schema half-applied is empty.

This is the second scenario in a row (after S1/S2) that fails to reproduce a failure class we have been citing.

Environment

FieldValue
Keycloak from → to26.0.0 → 26.7.1
Distribution / start modequay.io/keycloak/keycloak official image, start w/ external DB
DatabasePostgreSQL 16.15 (Debian 16.15-1.pgdg13+2), single node, container
Postgres tuningshared_buffers 128MB, maintenance_work_mem 64MB — stock; statement_timeout is the variable, 10s / 3ms / 2ms
Adverse scenarioS3
Dataset scale2,000,003 users, 2 realms
Seeding methoddirect SQL COPY (bin/seed-sql.sh)
Topologysingle container
HostHetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap-Xms1g -Xmx4g
Instrumentationlog_min_duration_statement = 0 for the measurement run

Procedure

./bin/restore.sh baseline-26.0.0-sqlseed-2m
docker compose exec -T db psql -U keycloak -c "alter system set log_min_duration_statement=0" -c "select pg_reload_conf()"
./bin/upgrade.sh 26.7.1                       # measurement run
docker compose logs db | grep 'LOG:  duration:' | sort -gr   # rank statements

# then, per timeout value:
./bin/restore.sh baseline-26.0.0-sqlseed-2m
docker compose exec -T db psql -U keycloak -c "alter role keycloak set statement_timeout='<T>'"
./bin/upgrade.sh 26.7.1

What we could not observe

No half-migrated schema, at any timeout value. The empty band is a consequence of every statement in the migration falling inside 1.8–2.2 ms: the parse-phase reads and the DDL writes are the same order of magnitude, so a timeout tight enough to kill a changeset is also tight enough to kill the parse that precedes it. That coincidence is a property of this pair at this scale, not a general law, and we did not test whether it holds elsewhere.

ClaimPrimary sourceChecked
Longest migration statement is 2.142 ms at 2M userslog_min_duration_statement=0 output, ranked
10s timeout has no effect on this pathfull run, changelog 211, 2M users intact
2ms fails before applying any changesetchangelog still 144, model still 26.0.0
Failure leaves no held lockdatabasechangeloglock read directly, both rows f
A half-migrated schema is reachable via timeoutno — refuted for this pair at this scale
Holds on a pair with data-rewriting changesetsnot tested — this pair has none
Holds if the customer built the skipped indexes manuallynot tested — the obvious next run
Holds on other databasesnot tested — Postgres only

← All runs