what Keycloak's index check actually checks, and the upgrade it kills
- Upgrade
- 26.0.0 → 26.7.1
- Scale
- 2,000,003 users, baseline-26.0.0-sqlseed-2m, Postgres 16 at stock settings, Hetzner CCX3
- Database
- Postgres 16 at stock settings, Hetzner CCX33 (8 vCPU dedicated, 30GB RAM, local NVMe
- Topology
- single container
- Host
- Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30GB RAM, local NVMe
- Condition
- routine — no adverse condition applied
Why this run existed
Run A established that Keycloak's index changeset is guarded by a name-based
precondition and that the 300,000-row skip reads a planner statistic. It left
four questions open, and each was cheap to answer once reltuples was known to
be a lever. Answering them turned up a failure mode none of the previous
twenty-one records contains: an upgrade that dies mid-migration and leaves the
server unable to start.
Environment
Identical to 2026-08-25-runA-indexes-manually-created — 2,000,003 users,
baseline-26.0.0-sqlseed-2m, Postgres 16 at stock settings, Hetzner CCX33
(8 vCPU dedicated, 30GB RAM, local NVMe), JVM -Xms1g -Xmx4g, official image,
start with an external DB. Every leg begins with a verified restore of that
baseline, so the legs are comparable to each other and to Run A's.
autovacuum is disabled on USER_ENTITY in the legs that write to pg_class,
so an analyze cannot silently undo the setup between the write and the upgrade.
Results
| Leg | Setup | Upgrade | exectype | Index on USER_ENTITY | WARNs |
|---|---|---|---|---|---|
| A6a | reltuples=1000, relpages honest | READY | EXECUTED | built, correct | 0 |
| A6b | reltuples honest, relpages=100 | READY | EXECUTED | absent | 2 |
| A7a | same index name on the realm table | READY | EXECUTED | absent | 1 |
| A7b | same name, same table, UNIQUE, on (id) | READY | MARK_RAN | wrong index kept | 0 |
| A7c | A7a + falsified reltuples | FAILED | — | absent | 0 |
Breakage observed
An index name collision on an unrelated table fails the upgrade and blocks startup
- Symptom:
relation "idx_user_created_timestamp" already exists, server refuses to start, changelog stops at 187/211 withmigration_modelat 26.0.0. - Evidence: A7c, quoted above.
- Cause: confirmed — per-schema index names against a per-table precondition.
- Fix or workaround: rename or drop the colliding index, restart. The half-applied changelog resumes, as it did in S8 and S14.
- Would this hit a customer? Config-dependent and latent. It needs someone to have created an index carrying a Keycloak index name on another table — which our own runbook makes likelier, since it hands operators those exact names to paste. It then waits for a build to be attempted.
DatabaseIndexChecker reports a healthy database when the index is on the wrong table
- Symptom: the checker runs and emits nothing, while the index the changeset defines does not exist on the table it names.
- Evidence: §3 above;
Running database index checkerpresent, noMissing database indexline,pg_indexesshows the index onrealm. - Cause: confirmed — the checker matches on index name alone.
- Fix: compare definitions, per table, as
bin/index-audit.shdoes. - Would this hit a customer? Anyone relying on Keycloak's own index check.
Verification
| Claim | Primary source | Checked |
|---|---|---|
reltuples is the input; relpages is not | A6a built, A6b skipped | ☑ |
| Keycloak reads the raw value, not the planner's estimate | Run A L5 — planner would compute ~533,920 and skip; index was built | ☑ |
| Precondition ignores columns | Run A L3 — MARK_RAN on a single-column index | ☑ |
| Precondition ignores uniqueness | A7b — MARK_RAN on a UNIQUE index of a different column | ☑ |
| Precondition honours the table | A7a — same name on realm, changeset ran | ☑ |
DatabaseIndexChecker does not honour the table | control 2 WARNs vs A7a 1 WARN, message text captured | ☑ |
| A name collision fails the upgrade | A7c — Liquibase exception, quoted | ☑ |
It leaves 187/211 and migration_model 26.0.0 | A7c — queried directly | ☑ |
| Restarting after clearing the collision resumes the changelog | not tested this run — inferred from S8 and S14, which is not evidence | ☐ |
The threshold constant on USER_ENTITY is 300,000, inclusive | bisected here — 300,000 builds, 300,001 skips | ☑ |
| Behaviour on version pairs other than 26.0.0 → 26.7.1 | not tested | ☐ |