The Keycloak Upgrade Ledger

Rehearsal Record — FGAP-v2 activation path pinned (26.7.3)

2026-09-01T00:00:00+00:00

Question

2026-08-31-fgap-v2-token-exchange left this open: the pre-26.7 procedure (adminFineGrainedAuthz realm field + GET /admin/realms/{realm}/fine-grained-permissions) is dead in 26.7.3 — the field is unrecognized and the endpoint 404s — while the ADMIN_FINE_GRAINED_AUTHZ_V2 feature flag reports type=DEFAULT, enabled=true. How does 26.7.3 actually turn FGAP v2 on, and where is the permission catalog?

Answer (pinned from bytecode + verified live)

FGAP v2 is a DEFAULT feature and activates per-realm with the classic adminPermissionsEnabled flag. No --features gate, no adminFineGrainedAuthz field.

Bytecode path (jars extracted from quay.io/keycloak/keycloak:26.7.3):

org.keycloak.services.resources.admin.fgap.AdminPermissions.evaluator(...) branches on Profile.isFeatureEnabled(ADMIN_FINE_GRAINED_AUTHZ_V2) — v2 selects MgmtPermissionsV2, v1 selects MgmtPermissions. The flag is type=DEFAULT in 26.7.3, so v2 is the only line in play.
org.keycloak.authorization.fgap.AdminPermissionsSchema.init(session, realm) (spi-private) runs when the realm has admin permissions enabled. It creates a client with clientId admin-permissions (openid-connect), sets it as the realm's adminPermissionsClient, creates its authorization resource server, and seeds the schema:
- resources = resource types: Users, Groups, Roles, Clients, Organizations (Realms is handled cross-realm);
- scopes = the permission catalogue.

Live verification (26.7.3, Postgres, Compose lab)

POST /admin/realms {"realm":"fgap","adminPermissionsEnabled":true} → then:

GET /admin/realms/fgap            adminPermissionsEnabled: true
                                  adminPermissionsClient: {clientId: "admin-permissions"}
                                  "adminFineGrainedAuthz" NOT in representation

GET /admin/realms/fgap/clients    -> includes "admin-permissions" (authzServicesEnabled)
GET .../authz/resource-server/resource -> Clients, Groups, Organizations, Roles, Users
GET .../authz/resource-server/scope    -> 16 permission scopes (below)
GET /admin/realms/fgap/fine-grained-permissions        -> 404
GET /admin/realms/fgap/admin-fine-grained-permissions  -> 404

The v2 permission catalogue (scopes on the `admin-permissions` client)

view, manage, manage-members, view-members, manage-membership, manage-membership-of-members, manage-group-membership, map-roles, map-roles-client-scope, map-roles-composite, map-role, map-role-client-scope, map-role-composite, impersonate, impersonate-members, reset-password.

What changed vs pre-26.7 (migration-surfacing)

Thing	pre-26.7 (v1)	26.7.3 (v2)
Feature gate	`--features=admin-fine-grained-authz` (preview)	`ADMIN_FINE_GRAINED_AUTHZ_V2`, `type=DEFAULT`
Realm activation	`adminFineGrainedAuthz` field	`adminPermissionsEnabled` flag
Permissions client	`realm-management` (managed)	`admin-permissions` (new, schema-seeded)
Permission catalogue	`GET .../fine-grained-permissions`	resources+scopes on `admin-permissions`'s resource server
Grant a permission	v1 policy API	standard authz scope-permission API on `admin-permissions`

How to audit / drive FGAP v2 in 26.7.3 (the new procedure)

PUT/POST /admin/realms/{realm} with {"adminPermissionsEnabled": true}.
Read adminPermissionsClient from the realm representation to get the admin-permissions client id.
Catalogue: GET /admin/realms/{realm}/clients/{id}/authz/resource-server/resource (resource types) and .../scope (permissions).
Grant a permission to a role/group: create a scope-based permission policy on the admin-permissions client (.../authz/resource-server/permission/scope), assigning the role/group policy — the standard authorization-services API.

What this unblocks

The FGAP-v2 RBAC CVE reproductions no longer need the dead v1 activation:

role-groups bypass — CVE-2026-14613
composite-endpoint authz — CVE-2026-16105
default-group disclosure — CVE-2026-16108

Each can now be set up by (a) enabling adminPermissionsEnabled, (b) building the role/group + scope-permission on admin-permissions, (c) exercising the admin endpoint under test. Still deferred: token-exchange CVEs (CVE-2026-18215/18214, external IdP) and CVE-2026-79652 (jwt-bearer consentRequired).

Verification

Claim	Evidence
v2 is the active line, DEFAULT feature	`AdminPermissions` bytecode + serverinfo flag `enabled=true` ☑
Activation is `adminPermissionsEnabled`	live realm rep + `admin-permissions` client created ☑
Catalogue = resources+scopes on `admin-permissions`	live `resource` (5 types) + `scope` (16) responses ☑
v1 surface removed	`adminFineGrainedAuthz` absent; both catalog endpoints 404 ☑

Rehearsal Record — Postgres HA: sync replication + clean (promote-while-up) failover

2026-09-01T00:00:00+00:00

Summary

This closes the last two untested legs of the Postgres HA line (2026-08-26-ha-failover-mid-migration covered the crash case). Two results:

Synchronous replication works as advertised. With synchronous_commit=remote_apply + synchronous_standby_names=* the replica reports sync_state=sync, idle replay lag ~44 ms right after the clone settling to ~0.7 ms in steady state, and a committed realm write is immediately visible on the hot standby — the primary does not ack until the standby has applied the WAL.
A clean failover has a different failure shape than a crash. Promoting the replica while the primary is alive leaves the old primary running with sync replication but no standby — so its writes block indefinitely (the probe CREATE TABLE never completed). That is a safety property (the old primary cannot accept split-brain writes), but it means the old primary wedges until stopped. The switchover must therefore be: promote → repoint Keycloak → stop the old primary, and an operator who promotes first without quiescing writes will find the old primary hung, not failed.

Environment

Field	Value
Keycloak	26.7.3 (official image), `start` w/ external DB
Database	PostgreSQL 16, primary + synchronous streaming replica, containers
Replication	`wal_level=replica`, `synchronous_commit=remote_apply`, `synchronous_standby_names=*`, streaming via `pg_basebackup -X stream`
Topology	1 primary (`db`) + 1 replica (`db-replica`); Keycloak single container
Dataset	3 realms (`master`, `lab`, `ha-sync`), seeded in-place
Host	laptop (Docker 29, Compose 5.5.0)
JVM heap	`-Xms1g -Xmx4g`

Procedure

COMPOSE_FILE=docker-compose.yml:docker-compose.ha.yml
HA_SYNC_COMMIT=remote_apply HA_SYNC_STANDBYS='*'
docker compose up -d db                       # primary, sync settings
./bin/ha-init-replica.sh                       # clone + start standby
docker compose up -d keycloak                  # writes now ack only via replica
# create a realm to prove write visibility on the standby
POST /admin/realms {"realm":"ha-sync"}

# clean failover: promote while the primary is STILL UP
docker compose exec -u postgres db-replica pg_ctl promote -D /var/lib/postgresql/data
# probe: write to the OLD primary (sync, no standby) -> BLOCKS, CREATE TABLE never commits

# switchover
sed KC_DB_URL -> jdbc:postgresql://db-replica:5432/keycloak
docker compose up -d --force-recreate keycloak  # repoint
docker compose stop db                          # break split brain

Results

Sync replication (steady state)

Signal	Value
`pg_stat_replication.sync_state`	sync (priority 1)
replay lag right after clone	~44 ms
replay lag steady state (after a write)	~0.7 ms
realm write visible on standby	`ha-sync` present on the replica immediately ☑

Clean failover (promote-while-up)

Step	Observation
`pg_ctl promote` on replica	promoted cleanly, out of recovery
Old primary (alive) after promotion	still reports itself a primary (`pg_is_in_recovery()=f`)
Write to old primary	blocked — `CREATE TABLE sync_block_probe` never committed (`pg_tables` count 0); session hung waiting for a sync standby
Repoint Keycloak → replica	ready; realms `ha-sync`, `lab`, `master` all present ☑
Stop old primary	split brain broken; single primary remains

What this teaches

Sync replication is the "no data loss at failover" configuration: a commit is not acknowledged until the standby has applied it, so the failover target is always current (contrast the async crash case, where the standby is a committed-prefix snapshot with possible lag).
The clean failover is operator-hazardous in the other direction. The crash case is bounded data loss; the clean case is a wedge. Promote the replica while the primary lives, and with sync replication the old primary blocks on every subsequent write — silent, indefinite, and easy to mistake for a healthy node that just "went slow". The pre-flight for a maintenance failover is: stop the old primary (or quiesce all writers) before promoting the replica.
The switchover order the lab already uses (promote → repoint → stop old primary) is correct for async; for sync it leaves a brief window where the old primary is wedged rather than split-brained — safer, but the depends_on gotcha from the crash record still applies.

Verification

Claim	Evidence
Replica is a synchronous standby	`pg_stat_replication`: `sync_state=sync`, `sync_priority=1` ☑
Primary does not ack before standby applies	realm write visible on replica, replay lag ~0.7 ms ☑
Old primary blocks writes after promote	`CREATE TABLE` probe never committed; session hung ☑
Keycloak resumes on promoted replica	ready; `ha-sync`/`lab`/`master` realms intact ☑
No split brain at end	old primary stopped; one primary (`db-replica`) ☑

Rehearsal Record — seed-sql.sh ported to MySQL and MariaDB (2M-class seeder)

2026-09-01T00:00:00+00:00

What was done

bin/seed-sql.sh is now engine-aware for postgres (unchanged, generate_series) and mysql/mariadb (new, recursive CTE). Oracle is still rejected with a loud pointer (see Remaining).

Three engine differences had to be handled, each of which would otherwise fail silently-late or corrupt the fixture:

Identifier case. Keycloak creates UPPERCASE identifiers (REALM, USER_ENTITY, CREDENTIAL, USER_ATTRIBUTE, ID, REALM_ID, USERNAME, …). Postgres folds unquoted identifiers to lowercase; MySQL 8 / MariaDB on Linux (lower_case_table_names=0, the lab's setting) preserve them. The first MySQL attempt failed at once with Table 'keycloak.realm' doesn't exist. The shared lookups (realm id, template id, row count) and the INSERT column lists are now written per engine.
Recursion-depth knob. MySQL 8 caps a recursive CTE with cte_max_recursion_depth; MariaDB calls the same thing max_recursive_iterations. Both default to 1000, so a 250k batch fails without a SET SESSION — and the name is engine-specific.
Credential JSON. The argon2 secret_data contains $argon2id$…, so it cannot be carried across the shell boundary (an unquoted heredoc would expand the $). On MySQL/MariaDB the JSON is copied in-database with a scalar subquery against the template row, so it never enters the shell at all.

Result (verified at 1000 users on both engines)

Engine	users	credentials	attributes	credential hash copied from template
MySQL 8.4	1000 (+1 template)	1000	2000	☑ (matched)
MariaDB 11.4	1000 (+1 template)	1000	2000	☑ (matched)

Also exercised on MariaDB: --start extension (append users), --batch (multi-batch loop), and --no-analyze. The Postgres path was re-verified unchanged (100 users).

Observation worth its own note

--no-analyze on MariaDB reports the planner's row estimate from information_schema.tables.table_rows, and that value is inherently fuzzier than Postgres's pg_class.reltuples: after seeding 1101 rows InnoDB reported 1303. Postgres's reltuples is exact after ANALYZE; InnoDB's table_rows is a statistical estimate even when freshly analyzed. So the "stale planner" model the lab uses to reproduce the index-skip threshold (Run A) will not map one-to-one onto MySQL/MariaDB — the pre-condition that Keycloak reads there is a number that is never exact. Worth pinning before claiming the threshold finding generalises.

Remaining

Oracle — still not supported. Two blockers: the credential columns are VARCHAR2/CLOB (the changelog notes the SECRET_DATA/CREDENTIAL_DATA fill is "used on all databases beside Oracle"), and row generation needs CONNECT BY LEVEL instead of a CTE. Separate port.

Verification

Claim	Evidence
MySQL seeds 1000 with correct shape	1000 users / 1000 creds / 2000 attrs, hash copied ☑
MariaDB seeds 1000 with correct shape	same counts, hash copied ☑
Extension + multi-batch + no-analyze work	`--start`/`--batch`/`--no-analyze` runs ☑
Postgres path unchanged	100 users seeded, 101 rows ☑

Rehearsal Record — seed-sql.sh ported to Oracle (CLOB credential + CONNECT BY)

2026-09-01T00:00:00+00:00

What was done

bin/seed-sql.sh now supports oracle. Two Oracle-specific mechanics, both of which are load-bearing rather than cosmetic:

Row generation — CONNECT BY LEVEL. Oracle has no generate_series and no recursive CTE (well, it has recursive WITH, but the idiomatic generator is SELECT lo + LEVEL - 1 FROM dual CONNECT BY LEVEL <= (hi-lo+1)).
Credential copy — a PL/SQL block. SECRET_DATA / CREDENTIAL_DATA are CLOB on Oracle (confirmed from all_tab_columns; the changelog creates them as CLOB and only the fill SQL is Oracle-specific, because Oracle's CONCAT takes exactly two args). A scalar subquery returning a CLOB is rejected (ORA-00932/ORA-22848), so the template credential is read into PL/SQL CLOB variables with SELECT … INTO and then referenced by the INSERT — the value never crosses the shell boundary (important: the argon2 secret_data contains $argon2id$…).

Also engine-specific: EMAIL_VERIFIED/ENABLED are native BOOLEAN (insert TRUE), USERNAME/FIRST_NAME/LAST_NAME/VALUE are NVARCHAR2, CREATED_TIMESTAMP is NUMBER (ms epoch via ROUND((SYSDATE - DATE '1970-01-01') * 86400000)), and identifiers are UPPERCASE (shared with the mysql/mariadb branch). Stats are gathered with DBMS_STATS.GATHER_TABLE_STATS and the planner estimate is all_tables.num_rows; size is user_segments.

Result (verified at 100 users, then extension + multi-batch + --no-analyze)

Check	Value
`--users 100`	100 users + 1 template; 100 creds; 200 attrs
credential CLOB copied	`DBMS_LOB.COMPARE = 0` (match), `secret_data` length 116
`--start` extension	50 appended users
`--batch 25` multi-batch	2 batches clean
`--no-analyze`	reports `all_tables.num_rows` (102, the pre-stats estimate)
clean run	10 users in 2s, no sqlplus feedback noise

Gotcha worth recording

The deterministic sql-<32-digit> ids are a table-wide primary key, not per-realm. Seeding a second realm with the same --start/--users range collides (ORA-00001 … ID already exists) — this is true on every engine, not Oracle-specific, and is the reason --start exists. Re-running an overlapping range is not idempotent; the script assumes each range is seeded once.

Verification

Claim	Evidence
Oracle seeds 100 with correct shape	100 users / 100 creds / 200 attrs ☑
CLOB credential copied from template	`DBMS_LOB.COMPARE = 0` ☑
BOOLEAN + NVARCHAR2 + NUMBER columns accept the seed	rows inserted, no type errors ☑
Extension / multi-batch / no-analyze paths work	3 extra runs ☑

Rehearsal Record — Oracle S4: a non-owner "DML-only" account cannot run the migration at all

2026-09-01T00:00:00+00:00

Summary

The S4 "DML-only account" row is now closed on the fourth engine. On Oracle it does not just differ in fix — the whole failure mode is structurally different, because the user IS the schema. Two runs:

Run A — DML-only (+ synonyms): fails before any Keycloak changeset, at Liquibase's own changelog-table discovery, with ORA-01031. Liquibase does not resolve synonyms when checking for DATABASECHANGELOG (a synonym is not a table in USER_TABLES), so it believes the table is absent and tries to CREATE TABLE KC_DML.DATABASECHANGELOG, which a DML-only account cannot do.
Run B — DML + full ANY DDL: the same CREATE now fails with ORA-00955: name is already used by an existing object — the existing object is the synonym itself. The synonyms that are required to make DML resolve are themselves the obstacle to Liquibase's DDL.

Conclusion: on Oracle there is no privilege-based fix. The MySQL finding ("DDL privileges suffice") does not translate; neither does the Postgres finding ("ownership required, no GRANT confers DROP INDEX"). On Oracle the account Keycloak connects as must simply be the schema owner — a non-owner DML-only account is not a viable migration posture, because Liquibase cannot see the existing schema through synonyms and cannot create its changelog table beside them.

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.3
Database	Oracle Database Free (23ai/26ai), release 23.26.2.0.0, container `gvenzl/oracle-free:23-slim`
JDBC driver	`ojdbc17` + `orai18n` 23.26.0.0.0 in `fixtures/providers/`
Schema owner	`keycloak` (DB_DEVELOPER_ROLE, USERS tablespace) — 87 tables, 194 indexes, 0 sequences
Account under test	`kc_dml` — DML grants + private synonyms on all 87 tables (348 grants)
Topology	single container
Host	laptop (Docker 29, Compose 5.5.0)
JVM heap	`-Xms1g -Xmx4g`

Procedure

# 26.0.0 schema created by Keycloak as the keycloak owner (87 tables)
./bin/snapshot.sh baseline-26.0.0-oracle-s4

# kc_dml: DML-only account + synonyms (user-is-schema: unqualified names need synonyms)
CREATE USER kc_dml IDENTIFIED BY kc_dml DEFAULT TABLESPACE USERS QUOTA UNLIMITED ON USERS;
GRANT CREATE SESSION TO kc_dml;
-- for each of the 87 tables T owned by KEYCLOAK:
GRANT SELECT,INSERT,UPDATE,DELETE ON keycloak.T TO kc_dml;
CREATE OR REPLACE SYNONYM kc_dml.T FOR keycloak.T;

# run A — DML only
KC_DB_USERNAME=kc_dml ./bin/upgrade.sh 26.7.3 --no-backup     # FAILED, 16s

# run B — DML + ANY-* DDL
GRANT CREATE/ALTER/DROP ANY TABLE|INDEX|SEQUENCE|VIEW|SYNONYM|TRIGGER|PROCEDURE TO kc_dml;
docker compose up -d keycloak                                  # FAILED, 6s

Results

Run	Account rights	Outcome	Error
A	DML only (SELECT/INSERT/UPDATE/DELETE) + synonyms	exit 1, refused to start	`ORA-01031` on `CREATE TABLE KC_DML.DATABASECHANGELOG`
B	DML + full `ANY` DDL + synonyms	exit 1, refused to start	`ORA-00955` on the same CREATE — collides with the synonym

Schema after both runs: keycloak still 26.0.0, 87 tables, 144 changelog rows; kc_dml has 0 tables and 87 synonyms. Neither run modified the owner's schema.

Run A failure

liquibase.exception.SetupException: An error occurred while attempting to create
the database changelog table. ... 'KC_DML.DATABASECHANGELOG'
ORA-01031: insufficient privileges
[Failed SQL: (1031) CREATE TABLE KC_DML.DATABASECHANGELOG (ID VARCHAR2(255) ...)]

Liquibase checks for the changelog table and does not find keycloak.DATABASECHANGELOG, because a synonym is not reported as a table by the catalogue. It concludes the table is missing and tries to create it in the connection's own schema (KC_DML) — a DDL operation a DML-only account cannot perform. Note this is earlier than MySQL, where Liquibase found the existing changelog and only failed trying to ALTER it.

Run B failure

ORA-00955: name is already used by an existing object
[Failed SQL: (955) CREATE TABLE KC_DML.DATABASECHANGELOG (...)]

With CREATE ANY TABLE granted, the same CREATE now clears the privilege check and collides with the synonym kc_dml.DATABASECHANGELOG. A schema cannot hold a table and a synonym of the same name, so the very mechanism that made DML resolve is what blocks Liquibase's DDL.

Verification

Claim	Evidence
DML-only non-owner fails on Oracle	run A `ORA-01031` at `CREATE TABLE KC_DML.DATABASECHANGELOG` ☑
DDL grant does not rescue it	run B `ORA-00955` (synonym collision) ☑
Owner schema untouched by both failures	`keycloak` 26.0.0, 87 tables, 144 changelog rows ☑
Synonyms are the obstacle	`kc_dml` 87 synonyms, 0 tables ☑

Rehearsal Record — CVE-2026-16093 spot-check: unsigned JWT client assertion rejected

2026-08-31T00:00:00+00:00

Method

Created a confidential client (jwt-assert-client) with clientAuthenticatorType=client-secret-jwt, then exercised the client_credentials grant with a JWT client assertion in both forms:

HS256 — a properly HMAC-signed assertion over the client secret.
alg:none — the same header/payload with no signature (the bypass).

Result

Assertion	Response
HS256 (signed)	HTTP 200 — access token issued
`alg:none` (unsigned)	HTTP 400 — `{"error":"invalid_client","error_description":"Invalid signature algorithm"}`

Finding

26.7.3 rejects unsigned JWT client assertions (alg:none → "Invalid signature algorithm"), so the CVE-2026-16093 bypass is closed. The signed path is unaffected. This confirms the OIDC client-assertion fix in the 26.7.3 batch.

Still open (need external IdPs / dedicated harness)

CVE-2026-18215 / CVE-2026-18214 — Microsoft/Google token-exchange tenant/hosted-domain bypasses. These need a real (or mocked) external IdP; the lab has no Google/Microsoft fixture.
CVE-2026-79652 — jwt-bearer grant consentRequired enforcement (same client-assertion harness could be extended).

Rehearsal Record — CVE-2026-35563 spot-check: LDAP TLS hostname verification at 26.7.3

2026-08-31T00:00:00+00:00

Environment

Field	Value
Keycloak	26.7.3 (official image)
Directory	`bitnamilegacy/openldap:2.6`, TLS on 1636 (LDAPS) + StartTLS on 1389
Federation	`corp-ldap`, `connectionUrl=ldap://ldap:1389`, `startTls=true`, `editMode=READ_ONLY`
Certificates	private CA (`lab-ldap-ca`); server cert `CN=ldap`, SAN deliberately set to `DNS:wronghost`
Trust	`KC_TRUSTSTORE_PATHS`, `ca.crt` present (so hostname is the only failure)
Database	PostgreSQL 16, single node, container

Result

Full sync via POST /admin/realms/lab/user-storage/{id}/sync?action=triggerFullSync with the CA trusted and the server cert SAN set to DNS:wronghost:

Signal	Value
Sync API response	`{"added":0,...,"status":"0 imported users, 0 updated users"}` HTTP 200
Keycloak log	`SSLPeerUnverifiedException: hostname of the server 'ldap' does not match the hostname in the server's certificate`

Finding

26.7.3 enforces LDAP TLS hostname verification. A directory cert whose SAN does not match the connection hostname is a hard failure (SSLPeerUnverifiedException, via sun.security.util.HostnameChecker). This confirms the CVE-2026-35563 regression — the LDAP client at 2.1.7 that stopped verifying the hostname — is closed in 26.7.3.

The failure signature is unchanged from the 26.7.1 measurement (2026-08-26-starttls-1389): silent at the API (HTTP 200, added:0), loud in the log. The existing guidance therefore still holds — a federation smoke test must assert on the imported count, not the status code.

Rehearsal Record — FGAP-v2 / token-exchange spot-check (26.7.3): activation surface changed

2026-08-31T00:00:00+00:00

What was checked

Feature flags (serverinfo): ADMIN_FINE_GRAINED_AUTHZ_V2 enabled=true, ADMIN_FINE_GRAINED_AUTHZ enabled=false — v2 is the active line in 26.7.3.
Realm activation field: PUT /admin/realms/lab with adminFineGrainedAuthz:"v2" → HTTP 400 Unrecognized field "adminFineGrainedAuthz". The RealmRepresentation no longer carries this field in 26.7.3.
Permission-catalog endpoint: GET /admin/realms/lab/fine-grained-permissions and /admin-fine-grained-permissions both → HTTP 404 on the lab realm.

Findings

FGAP-v2 activation surface changed in 26.7.3

Symptom: the documented way to turn on FGAP v2 (set adminFineGrainedAuthz on the realm) is rejected as an unrecognized field, and the permission-catalog endpoint 404s even though the v2 feature flag reports enabled.
Additional evidence (2026-08-31): ADMIN_FINE_GRAINED_AUTHZ_V2 is type=DEFAULT, enabled=true (not a preview gate). 26.7.3's RealmRepresentation carries adminPermissionsEnabled and adminPermissionsClient but no adminFineGrainedAuthz. Enabling adminPermissionsEnabled=true on a realm does not expose any fine-grained-permissions/admin-fine-grained-permissions endpoint (still HTTP 404). The fgap REST resources exist in the jar (org.keycloak.services.resources.admin.fgap.*) but their registration path is no longer the documented one.
Cause: the realm-level toggle/endpoint was renamed/relocated in the 26.7.x FGAP-v2 rollout; pinning the exact @Path needs the 26.7.3 admin REST / upgrade notes or source.
Impact: an operator migrating to 26.7.x cannot activate or audit FGAP v2 with the pre-26.7 procedure. The CVE-by-CVE RBAC reproductions (role-groups bypass CVE-2026-14613, composite-endpoint authz CVE-2026-16105, default-group disclosure CVE-2026-16108, etc.) are blocked on re-establishing the activation procedure, which is now a first-class lab task rather than a quick flag flip.

Deferred (need external IdPs / dedicated harness)

CVE-2026-18215 / CVE-2026-18214 — Microsoft/Google token-exchange tenant/hosted-domain bypass. Needs a real (or mocked) external IdP; not reproducible in the Compose lab.
CVE-2026-16093 — signed-JWT assertion bypass with unsigned assertion headers. Needs a client-jwt client + keypair harness.
CVE-2026-79652 — jwt-bearer grant consentRequired enforcement.

Rehearsal Record — remaining version-matrix paths at 26.7.3

2026-08-31T00:00:00+00:00

Results

All direct (unstepped) hops to 26.7.3, Postgres 16, typical profile, 1,000 users, single container, laptop.

Path	Time	Index audit	Realm
25.0.6 → 26.7.3	20s	119/119	1,000 users intact
24.0.5 → 26.7.3	20s	119/119	1,000 users intact
22.0.5 → 26.7.3	20s	119/119	1,000 users intact
26.6.4 → 26.7.3	17s	119/119	1,000 users intact

Findings

Every source version goes direct to 26.7.3 — no stepping needed across the 25→26 major boundary, the 24→26 two-major jump, or the 22→26 three-major jump. Liquibase applies the pending changesets in one pass. This is the same shape as the 21.1.2 → 26.7.1 direct hop already recorded.
Migration duration is flat across the span (~17–20s) — it tracks the changeset count that actually runs, not the version distance. Confirms the lab's earlier finding that "duration against a stated dataset size" is the right frame, and the causal variable is the changeset set.
The 26.6.5 / 26.6.6 container images are not published. GitHub carries the 26.6.5 and 26.6.6 tags, but quay.io/keycloak/keycloak stops the 26.6 line at 26.6.4. 26.6.5 is the patch the 2026-08-06 advisory batch names as the 26.6-line fix, so a customer pinned to the 26.6 line cannot pull that fix as a container image — they must go to 26.7.x. This belongs in the runbook's version-availability note (the lab already has one for the 22.0.x line ending at 22.0.5).

Rehearsal Record — Oracle Data Pump port (snapshot/restore)

2026-08-31T00:00:00+00:00

What was done

Ported bin/snapshot.sh and bin/restore.sh to Oracle using Data Pump (expdp/impdp), closing the last engine for the snapshot/restore tooling.

The port

snapshot.sh (oracle) — exports as SYSTEM (ORACLE_PASSWORD) because the schema owner carries only DB_DEVELOPER_ROLE and cannot read the DATA_PUMP_DIR directory object. expdp SCHEMAS=keycloak writes the dump into a GUID-named subdirectory of DATA_PUMP_DIR inside the container; the script locates it (find /opt/oracle/admin) and docker compose cp's it out to snapshots/<name>.dump.
restore.sh (oracle) — DROP USER keycloak CASCADE, recreate it exactly as the gvenzl image does (DB_DEVELOPER_ROLE, USERS tablespace), copy the dump back in, then impdp SCHEMAS=keycloak EXCLUDE=USER TABLE_EXISTS_ACTION=REPLACE. EXCLUDE=USER avoids the benign ORA-31684 user already exists (the user is pre-created so the password/grants survive, which a SCHEMA-only dump does not carry). An Oracle model_version (read keycloak.migration_model via SYSTEM) preserves the no-re-migration gate.

Result

./bin/snapshot.sh baseline-oracle-test    -> snapshots/baseline-oracle-test.dump (4.8M)
./bin/restore.sh baseline-oracle-test
  ==> restored schema reports model 26.7.3
  ==> ready
  ==> restored baseline-oracle-test — running 26.7.3, schema 26.7.3, no re-migration

Schema intact after restore: 100 tables, migration_model 26.7.3, 1 user.

Remaining Oracle item

S4 DML-only — untested; a non-owner user needs synonyms under the user-is-schema model.

Rehearsal Record — Oracle index-audit port

2026-08-31T00:00:00+00:00

What was done

Ported bin/index-audit.sh / bin/index-audit.py to Oracle, closing the last engine for the schema-correctness gate.

The port

index-audit.py
- norm() now strips all double-quotes (Oracle writes expressions with quoted uppercase identifiers: SUBSTR("VALUE",1,250) → substr(value,1,250)) and folds the result to the changelog's <modifySql dbms="oracle"> form.
- col_norm()'s (N) prefix-length strip is now scoped to mysql/mariadb only (Oracle has no prefix-length columns; previously != postgresql also applied it to Oracle).
- Added parse_db_oracle() (index-name → ordered column list) and the --engine oracle choice.
index-audit.sh
- The Oracle catalogue query reads USER_IND_COLUMNS joined to USER_IND_EXPRESSIONS. USER_IND_EXPRESSIONS.COLUMN_EXPRESSION is a LONG, which cannot be concatenated or SUBSTRed in a plain SELECT (ORA-00932/ORA-24856), so the emission is a PL/SQL block that reads the expression into a VARCHAR2(4000) and writes index|column lines via DBMS_OUTPUT. BIN$% recycle-bin ghosts are excluded.

Result

--- 121 index definitions applied to this database: 121 correct, 0 missing, 0 wrong shape, 0 unverified (9 skipped: table dropped in a later version)

The 26.7.3 Oracle schema is correct by definition — including the function-based indexes (IDX_GROUP_ATT_BY_NAME_VALUE → SUBSTR("VALUE",1,250)), which is the Oracle spelling of the same index that produced the postgres false-positive earlier.

Remaining Oracle tooling

snapshot/restore — still needs Data Pump (expdp/impdp); the one Oracle item left unported.

Verification

Claim	Evidence
Function-based expressions fold to the changelog form	`SUBSTR("VALUE",1,250)` vs `substr(value,1,250)` matched, 121/121
LONG column handled without `ORA-00932`	PL/SQL block + DBMS_OUTPUT, no errors
Recycle-bin ghosts excluded	`WHERE index_name NOT LIKE 'BIN$%'`

Rehearsal Record — 21.1.2 → 26.7.3 (stepped ladder, 100k rich)

2026-08-31T00:00:00+00:00

Environment

Field	Value
Keycloak from → to	21.1.2 → 26.7.3
Stepping path	21.1.2 → 22.0.5 → 23.0.7 → 24.0.5 → 25.0.6 → 26.7.3
Distribution	quay.io/keycloak/keycloak
Start mode	`start` w/ external Postgres 16, single node
Dataset	`baseline-21.1.2-rich-100k` — 1 realm, 100,006 users, rich fixture
Seeding method	`partialImport` (baseline pre-built)
Host	laptop (docker 29.7.2)

Timings

Hop	Duration	Index audit
21.1.2 → 22.0.5	47s	84 defs: 1 missing, 1 false-mismatch
22.0.5 → 23.0.7	28s	84/84
23.0.7 → 24.0.5	24s	88/88
24.0.5 → 25.0.6	20s	89/89
25.0.6 → 26.7.3	20s	119/119
Total	~139s

Outcome

All five hops reached ready; final state 26.7.3, 100,006 users intact
Index audits pass end-to-end after the two findings below are handled

Breakage observed

1. `idx_client_att_by_name_value` genuinely missing (pre-existing in the 21.1.2 baseline)

Symptom: audit at 22.0.5: MISSING idx_client_att_by_name_value on client_attributes.
Evidence: index absent from pg_indexes; client_attributes has 30 rows, so the ~300k-row skip heuristic does not explain it. Changeset 14.0.0-KEYCLOAK-18286-supported-dbs is marked EXECUTED.
Cause: pre-existing in the 21.1.2 baseline (a 14.0.0 changeset predates the hop; the 22.0.5 migration does not touch it). Not introduced by the ladder.
Fix: CREATE INDEX CONCURRENTLY idx_client_att_by_name_value ON client_attributes (name, (value::varchar(250)));
Would this hit a customer? Yes — a 21.1.2 install that skipped this index looks migrated (schema stamp advances) while missing an index. This is exactly the class the index-audit exists to catch, and it validates shipping the audit as part of the runbook rather than trusting the schema version.

2. `index-audit.py` false positive: `varchar` vs `character varying`

Symptom: audit flagged idx_group_att_by_name_value as MISMATCH — changelog (name, (value::varchar(250))) vs DB (name, ((value)::character varying(250))).
Cause: norm() only stripped a cast at end-of-token and did not fold the varchar/character varying alias or collapse redundant parens. Same type, different spelling.
Fix: bin/index-audit.py norm() now folds charactervarying→varchar, strips ::type[(n)] casts anywhere, and collapses redundant parens.
Would this hit a customer? No — tool-only. Re-audit passes 84/84.

Rollback

Not exercised on this path this run (exercised on the patch-hop path in run 3).

Rehearsal Record — 26.0.0 → 26.7.3 (within-26 direct)

2026-08-31T00:00:00+00:00

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.3
Stepping path	direct (no stepping)
Distribution	quay.io/keycloak/keycloak
Start mode	`start` w/ external Postgres 16, single node
Dataset	`baseline-26.0.0-typical-1k` — 1,000 users
Host	laptop

Procedure

./bin/restore.sh baseline-26.0.0-typical-1k
./bin/upgrade.sh 26.7.3

Timings

Phase	Duration
Total to ready	20s

Outcome

Reached ready, 26.7.3
Index audit 119/119 correct
Realm intact (1,000 users)

Breakage observed

None.

Rollback

Not exercised this path.

Rehearsal Record — 26.7.1 → 26.7.3

2026-08-31T00:00:00+00:00

Environment

Field	Value
Keycloak from → to	26.7.1 → 26.7.3
Stepping path (if multi-hop)	direct (patch hop, no stepping)
Distribution	quay.io/keycloak/keycloak (official image)
Start mode	`start` w/ external Postgres — never `start-dev`
Database	Postgres 16, single node
Postgres tuning	stock defaults (no shared_buffers / maintenance_work_mem / statement_timeout override)
Adverse scenario	none — routine patch hop, motivated by the 2026-08 advisory batch
Dataset scale	1 realm (`lab`), 1,000 users, 11 clients (5 seeded + 6 default), 1 group (`corp` + 3 children), 2 IdP brokers, 1 LDAP component, 1 organization, 1 fine-grained-authz client
Seeding method	`partialImport` (1,000 users in 69s)
Fixture profile	`kitchen-sink`
Topology	single container
Host	laptop (Arch Linux, docker 29.7.2 / compose 5.5.0)
JVM heap	`-Xms1g -Xmx4g`
Elapsed clock	24s (stop → ready)

Procedure

./bin/lab-up.sh 26.7.1
./bin/seed-realm.sh --profile kitchen-sink --users 1000
./bin/upgrade.sh 26.7.3     # stop → pg_dump backup → image swap → up → wait-ready → index-audit

Timings

Phase	Duration	Notes
Container start → DB connect	—	not isolated this run
Liquibase / schema migration	0 changesets	no migration log lines emitted; see Finding
Cache/realm warm	—	not isolated
First successful token issue	—	admin-cli token issued immediately after ready
Total to ready	24s	includes image pull for 26.7.3

Outcome

Reached ready (wait-ready.sh → READY)
Realm config intact — verified via admin API post-upgrade: 1,000 users, all 11 clients, corp-oidc + corp-saml brokers, corp-ldap component, lab-org, corp group all present
Token endpoint works (admin-cli)
Login flow works (end-user password grant not re-exercised this run)
Custom SPI loaded (none deployed — fixtures/providers empty)
Themes render (theme files not mounted)
Federation/brokering intact end-to-end (components present; no live IdP/LDAP to sync against)

Breakage observed

None attributable to the migration. One pre-existing seed-time issue (not a migration result):

seed-realm `--user-profile` 400 on 26.7.1

Symptom: declarative user-profile PUT rejected with [Validator 'username_prohibited_characters' defined for attribute 'username' doesn't exist].
Evidence: seed-realm.sh output, before the upgrade ran.
Cause: the seed fixture names a validator that no longer exists at 26.7.1.
Fix or workaround: correct the validator name in bin/seed-realm.sh (or drop it for 26.7.x). Not chased this run — out of scope for the patch hop.
Would this hit a customer? No — fixture-only, no bearing on migration.

Rollback

Exercised this run? no
Method: snapshot restore — backup pre-upgrade-26.7.1-to-26.7.3-20260831T212644Z (pg_dump -Fc)
Time to restore: not measured
Data loss / divergence: n/a
Verdict: backup taken; restore path unexercised (must be exercised on one of the remaining two clean runs per launch-plan §6).

Verification

Claim	Primary source	Checked
26.7.1 already carries the 2026-08-06 advisory batch (7 CVEs, all `keycloak-services`)	GHSA-95cx-vmr5-3cmr etc.; affected range `< 26.7.1`, patched `26.7.1`	yes
26.7.3 adds the 26.7.2 + 26.7.3 security batches (~27 fixes) missing from 26.7.1	26.7.2 / 26.7.3 release notes (github.com/keycloak/keycloak/releases)	yes
26.7.1 → 26.7.3 emits no Liquibase changesets	this run's log (no `liquibase`/`changelog` lines)	yes
Schema indexes correct at 26.7.3	`index-audit.sh 26.7.3`: 119/119 correct, 0 missing/wrong	yes
26.7.2 → 26.7.3 removes 4 SPI signatures	`reports/diffs/api-26.7.2-to-26.7.3.txt`	yes

Publishable extract

(empty — internal; needs the two remaining clean runs + rollback before a ledger entry can be derived.)

Rehearsal Record — 26.7.1 → 26.7.3 (run 2)

2026-08-31T00:00:00+00:00

Environment

Identical to run 1 (2026-08-31-26.7.1-to-26.7.3-1) except:

Fixture was re-seeded from scratch after the seed-realm.sh --user-profile fix (username-prohibited-characters), so the declarative user profile is now present in the fixture.
Baseline: baseline-26.7.1-kitchen-sink-1k (pg_dump -Fc, 544K).

Procedure

./bin/lab-down.sh --purge
./bin/lab-up.sh 26.7.1
./bin/seed-realm.sh --profile kitchen-sink --users 1000     # user-profile now seeds
./bin/snapshot.sh baseline-26.7.1-kitchen-sink-1k
./bin/upgrade.sh 26.7.3

Timings

Phase	Duration
Total to ready	16s

Outcome

Reached ready
Schema migration: 0 changesets (no Liquibase lines)
Index audit 119/119 correct, 0 missing/wrong
Realm config intact (1,000 users, 11 clients — verified after restore in run 3)

Breakage observed

None.

Rollback

Not exercised this run (exercised in run 3).

Rehearsal Record — 26.7.1 → 26.7.3 (run 3, rollback exercised)

2026-08-31T00:00:00+00:00

Environment

Identical to run 2 (same baseline-26.7.1-kitchen-sink-1k).

Procedure

./bin/restore.sh baseline-26.7.1-kitchen-sink-1k   # reset to 26.7.1
./bin/upgrade.sh 26.7.3                            # -> 26.7.3, backup taken
./bin/restore.sh pre-upgrade-26.7.1-to-26.7.3-20260831T213925Z   # rollback

Timings

Phase	Duration
Upgrade to ready	25s
Rollback to ready	~30s (drop + pg_restore + recreate + boot)

Outcome

Upgrade reached ready, 0 changesets, index audit 119/119
Rollback restored the pre-upgrade dump and returned to 26.7.1
No re-migration: schema stamp 26.7.1 before and after the recreate
Data intact post-rollback: 1,000 users, 11 clients

Breakage observed

`restore.sh` postgres rollback path was broken (`model_version: command not found`)

Symptom: on the postgres engine, restore.sh aborted at SCHEMA=$(model_version) — the function is defined only in the mysql|mariadb branch of the first case "$ENGINE", but is called unconditionally for all engines.
Evidence: ./bin/restore.sh: line 118: model_version: command not found; the DB was restored (schema at 26.7.1) but the script exited before the --force-recreate and before the re-migration check.
Cause: regression from the multi-engine refactor — the postgres model_version definition was dropped.
Fix: re-added the postgres model_version() (with the S13 search_path care) to the * branch. Fix is in bin/restore.sh.
Would this hit a customer? Only our own lab tooling — but it means the rollback verification gate had been silently non-functional for postgres. This run is the first to actually exercise it since the refactor.

Rollback

Exercised this run? yes
Method: pg_dump -Fc restore (dropdb + createdb + pg_restore --no-owner), then --force-recreate at 26.7.1
Time to restore: ~30s at 1k users
Data loss / divergence: none (1,000 users, 11 clients intact)
Verdict: rollback IS a point-in-time restore; no schema downgrade, no re-migration. Confirmed available on this path.

Rehearsal Record — Kubernetes / Infinispan rolling update at 26.7.3

2026-08-31T00:00:00+00:00

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.3
Distribution / start mode	quay.io/keycloak/keycloak, `start` w/ external DB
Database	PostgreSQL 16, single pod (StatefulSet), in-cluster
Cache	embedded Infinispan, `KC_CACHE_STACK=kubernetes` (DNS_PING)
Cluster	kind `kclab` (k8s v1.37.0, kind v0.33.0) — note: laptop, not the k3d box
Topology	Deployment `replicas: 2`, `maxSurge=1, maxUnavailable=0`
Host	laptop (16 vCPU, 31 GB RAM)
JVM heap	`-Xms256m -Xmx1g`
Dataset	`master` realm + `lab` realm (1 user)

Result — rolling update completes cleanly, cache still partitions

Timeline (from pod logs):

t	event
0s	`set image` → new 26.7.3 pod (Infinispan `v=16.0.14`) starts, old 26.0.0 pods (`v=15.0.8`) still serving
~25s	new pod forms a 1-member cluster (`v=16.0.14`); `no members discovered`
~36s	second new pod starts; first new pod ready; first old pod terminated
~47s	the two new pods form a 2-member cluster (`v=16.0.14`)
~57s	last old pod terminated; rollout complete
end	`migration_model` = 26.7.3, both pods ready

Findings

Schema migration stays additive at 26.7.3 — migration_model reached 26.7.3, old pods logged no schema errors, rollout completed under maxUnavailable=0. The database layer is safe; the cache is the risk, as at 26.7.1.
The cache still partitions during the transition. 26.7.3 ships Infinispan 16.0.14 (26.7.1 was 16.0.12; 26.0.0 is 15.0.8). The new pods never merge with the old v=15.0.8 cluster — they form a separate 1-member cluster and only merge with each other once the old pods are gone. Same operational consequence as the 26.7.1 record: sessions are not shared between old and new pods during the window.
New signal: 26.7.3 enables JGroups encryption (mTLS). The 26.7.3 pod logs JGroups Encryption enabled (mTLS). This is a second axis of inter-version incompatibility in addition to the Infinispan/JGroups version bump, and belongs in the pre-flight as a thing to check when crossing into 26.7.x.
kind vs k3d: a discovery race at first boot. On kind, two pods started simultaneously each became a 1-member cluster (no members discovered after 2.00s) and did not merge on their own; deleting one pod triggered DNS_PING re-discovery and the 2-member cluster formed. The k3d run (original record) clustered on first boot. This is an environment/timing difference, not a Keycloak change — but it means the baseline "2-member cluster" must be verified explicitly before relying on a kind run.
Cross-pod session sharing confirmed at the final 26.7.3 state — a refresh token issued on pod A was accepted on pod B (HTTP 200), proving the healed 2-member cluster shares the session cache.

What is still not tested

Two new pods migrating concurrently (here maxSurge=1 made them sequential).
Functional session stranding inside the transition window (issuing on an old pod and failing on a new pod) — the partition is proven at the cluster-view level, not the HTTP level. The explicit JGRP000006 cookie does not match line from the 26.7.1 record was in the old pods' logs, which the Deployment deletes on termination; reproducing it cleanly needs live log capture during the rollout.
The 26.7.3 default jdbc-ping stack (this run pinned kubernetes to keep the version the only variable).

Rehearsal Record — multi-DB re-runs at 26.7.3 (MySQL, MariaDB)

2026-08-31T00:00:00+00:00

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.3 (direct)
Distribution	quay.io/keycloak/keycloak
Start mode	`start` w/ external DB
Engines	MySQL 8.4, MariaDB 11.4
Fixtures	`baseline-26.0.0-mysql-typical-1k` (1,000 users), `baseline-26.0.0-mariadb-typical-200` (200 users)
Host	laptop

Results

Engine	Hop	Time	Index audit	Realm
MySQL 8.4	26.0.0 → 26.7.3	20s	121/121 correct	1,000 users intact
MariaDB 11.4	26.0.0 → 26.7.3	20s	121/121 correct	200 users intact

Breakage observed

`.env` predates the multi-DB fields (`MYSQL_ROOT_PASSWORD: unbound variable`)

Symptom: restore.sh failed on the mysql engine with MYSQL_ROOT_PASSWORD: unbound variable — the checked-in .env was created from a pre-multi-DB .env.example.
Cause: .env is gitignored and was never regenerated after the multi-DB variables were added to .env.example.
Fix: appended MYSQL_VERSION/MARIADB_VERSION/MYSQL_PORT/ MYSQL_ROOT_PASSWORD/ORACLE_* to .env. A fresh cp .env.example .env would do the same; worth a lab-up.sh guard that .env carries every .env.example key.
Would this hit a customer? No — lab tooling only.

Still deferred (engine work)

Oracle: snapshot.sh/restore.sh/index-audit.sh are not ported to Oracle (need Data Pump expdp/impdp + *_INDEXES/*_IND_COLUMNS), and the Oracle Free container's first init is ~11 min. Needs a dedicated session.
Kubernetes rolling update: kubectl/kind/k3d are not installed on this host. Needs tooling install + a kind cluster before the 26.0.0 → 26.7.3 Infinispan mixed-cluster leg can be re-run.

Rehearsal Record — Oracle: 26.0.0 → 26.7.3 migration re-run

2026-08-31T00:00:00+00:00

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.3
Database	Oracle Database Free (23ai/26ai), release 23.26.2.0.0, single node, `gvenzl/oracle-free:23-slim`
JDBC driver	`ojdbc17` + `orai18n` 23.26.0.0.0 in `fixtures/providers/` (unchanged from the 26.7.1 run)
Schema/user	`keycloak` (APP_USER; user IS schema, `KC_DB_SCHEMA` not used)
Topology	single container
Host	laptop (16 vCPU, 31 GB RAM)

Procedure

docker compose up -d db        # Oracle Free first init (healthy in ~37s on this host)
docker compose up -d keycloak  # 26.0.0 (schema created)
docker compose stop keycloak && sed -i 's/^KC_VERSION=.*/KC_VERSION=26.7.3/' .env
docker compose up -d keycloak  # migrate

Results

Check	Before (26.0.0)	After (26.7.3)
Keycloak version	26.0.0	26.7.3
`migration_model`	26.0.0	26.7.3
Tables	87	100
User indexes	218	283

Master-realm token endpoint ready after migration; migration clean and additive (no DDL errors surfaced in the Keycloak log).

Still not ported (unchanged from the 26.7.1 run)

snapshot/restore — needs Data Pump (expdp/impdp); snapshot.sh/restore.sh still exit 1 on KC_DB=oracle.
index-audit — Oracle's catalogue is USER_INDEXES/USER_IND_COLUMNS/ USER_IND_EXPRESSIONS (no pg_indexes, no information_schema.statistics); index-audit.sh still exits 1 on KC_DB=oracle.
S4 DML-only — untested; Oracle's user-is-schema model needs synonyms for a non-owner account, so the Postgres/MySQL finding does not translate mechanically.

Rehearsal Record — the `http challenge` flow removal, narrowed to 22.0.0

2026-08-26T00:00:00+00:00

Why this run existed

2026-08-26-rich-21-realm-ladder-100k observed that the built-in http challenge authentication flow — present on the 21.1.2 realm — was absent on 26.7.1, and left two rows open: which hop removed it, and what happens to a custom flow that references it. Neither needed a box: the flow definition and the migration logic are in the published artifacts, so this run is pure jar inspection against the .m2 cache.

Environment

Field	Value
Artifacts inspected	`keycloak-server-spi-private` (flow definitions), `keycloak-model-legacy-private` / `keycloak-model-storage-private` (migrators)
Versions	21.1.2, 22.0.0, 22.0.1–22.0.5, 23.0.7, 24.0.5, 25.0.6, 26.0.0–26.7.1
Method	`javap` / `jar tf` / `strings` on `DefaultAuthenticationFlows` and `MigrateTo22_0_0`
Host	local laptop Docker (Maven container); no Keycloak server, no DB

Findings

Finding 1 — the flow definition was deleted at 22.0.0 ⭐

org.keycloak.models.utils.DefaultAuthenticationFlows (keycloak-server-spi-private) carries the built-in flow definitions. At 21.1.2 it has:

public static final String HTTP_CHALLENGE_FLOW;   // = "http challenge"
public static void httpChallengeFlow(RealmModel); // adds the flow + executions

and both addFlows(RealmModel) and migrateFlows(RealmModel) call httpChallengeFlow(realm) — i.e. the flow is created for new realms and re-added to existing realms during migration.

At 22.0.0 the constant and the method are gone from the class, and every version 22.0.0 → 26.7.1 lacks them:

Version	`HTTP_CHALLENGE_FLOW` + `httpChallengeFlow()`
21.1.2	present
22.0.0	absent
22.0.1 … 22.0.5, 23.0.7, 24.0.5, 25.0.6, 26.x	absent

The flow's constituent authenticators were not removed, only re-homed: http-basic-authenticator moved into samlEcpProfile (SAML ECP) and docker-http-basic-authenticator into dockerAuthenticationFlow. The top-level "http challenge" flow container itself is what disappeared.

Finding 2 — a new migrator actively removes the flow from existing realms, at 22.0.0 ⭐⭐

The flow definition removal alone would leave an upgraded realm's existing "http challenge" flow in place. Keycloak closes that gap with a dedicated migration step, org.keycloak.migration.migrators.MigrateTo22_0_0, which is introduced at 22.0.0 (present in keycloak-model-legacy-private 22.0.0, absent at 21.1.2; lives in keycloak-model-storage-private from 24+).

Its removeHttpChallengeFlow(KeycloakSession, RealmModel) method:

looks the flow up by alias — realm.getFlowByAlias("http challenge");
if present, calls KeycloakModelUtils.deepDeleteAuthenticationFlow(session, realm, flow, onRemoved, onInUse, flow.isBuiltIn()).

The two callbacks encode two distinct outcomes, both visible in the migration log:

not in use → the flow is deleted, and a DEBUG line is logged: Removed 'http challenge' authentication flow in realm '<realm>'.
in use (referenced by a client's flow-binding override or another flow) → the flow is left in place and an ERROR is logged: Authentication flow 'http challenge' is in use in realm '<realm>' and cannot be removed. Please update your deployment to avoid using this flow before migration to latest Keycloak.

This resolves the open "custom flow that references it" question: the removal is not silent data loss. A referenced flow survives the migration as realm data, but Keycloak emits an ERROR and stops maintaining the flow — the reference is left dangling against a definition that no longer exists. The operator must re-write the custom flow before (or at) the 22.0.0 hop.

Finding 3 — the release boundary is the 21 → 22 hop, not any later one ⭐

Both halves (definition deleted + migrator added) land at 22.0.0, so the removal is squarely the 21.1.2 → 22.0.0 hop. The rich-ladder run's six hops started at 21.1.2 and could not see where it happened; the answer is the first hop.

Verification

Claim	Primary source	Checked
`httpChallengeFlow` + `HTTP_CHALLENGE_FLOW` present at 21.1.2	`javap` of `DefaultAuthenticationFlows` 21.1.2	☑
Both absent at 22.0.0 and every later cached version	`strings`/`javap` across 22.0.0 … 26.7.1	☑
`addFlows` and `migrateFlows` both called it at 21.1.2	`javap -c` call sites	☑
`MigrateTo22_0_0` introduced at 22.0.0	present in `keycloak-model-legacy-private` 22.0.0, absent at 21.1.2	☑
It removes the flow by alias, two outcomes	`javap -c` of `removeHttpChallengeFlow` + log strings	☑
`http-basic-authenticator` re-homed, not removed	22.0.0 `DefaultAuthenticationFlows`: referenced by `samlEcpProfile` / `dockerAuthenticationFlow`	☑

Publishable extract

One specific Keycloak built-in flow dies on the 21 → 22 upgrade, and it dies loudly only if you are using it.

Keycloak 22.0.0 removed the built-in http challenge authentication flow. The flow definition and the code that created it are gone, and a dedicated migration step (MigrateTo22_0_0) deletes the flow from every realm on the way up. If your realm does not reference it, it is removed silently — a DEBUG line, nothing more. If a custom flow or client still references it, Keycloak leaves it in place and logs an ERROR telling you to update your deployment before migrating.

The constituent HTTP Basic authenticators were not removed — they moved into the SAML ECP and docker-auth flows. What vanished is the standalone "http challenge" flow, and the place that matters is the 21 → 22 hop, not any later upgrade.

invalid_grant status change 401 → 400 — narrowed to 26.6.0

2026-08-26T00:00:00+00:00

Environment

Field	Value
Keycloak versions	26.0.0, 26.1.0, 26.2.0, 26.3.0, 26.4.0, 26.5.0, 26.5.7, 26.6.0, 26.6.4, 26.7.0, 26.7.1
Distribution	quay.io/keycloak/keycloak (official image)
Start mode	`start` w/ external DB
Database	Postgres 16, single node, fresh empty DB per version (`docker compose down -v`)
Postgres tuning	stock (compose `db` service sets none)
Adverse scenario	none — holds environment stock, varies the version
Dataset scale	master realm only (auto-created), no seeded data
Topology	single container
Host	Hetzner ccx33 (8 vCPU / 32 GB), fsn1
JVM heap	container default
Probe	one password grant with a wrong password against `admin-cli`

Procedure

For each version, in order: purge volumes, set KC_VERSION, start db + keycloak, wait-ready.sh, then:

curl -s -w '\n%{http_code}' \
  -d "client_id=admin-cli" -d "username=admin" \
  -d "password=definitely-wrong" -d "grant_type=password" \
  "http://localhost:8080/realms/master/protocol/openid-connect/token"

A fresh DB per version guarantees a clean master realm, ruling out the "accumulated brute-force state" hypothesis left open in 2026-08-25-s7-custom-spi. Script: bin/narrow-invalid-grant.sh (one-off).

Result

Version	HTTP status	Body
26.0.0	401	`{"error":"invalid_grant","error_description":"Invalid user credentials"}`
26.1.0	401	same
26.2.0	401	same
26.3.0	401	same
26.4.0	401	same
26.5.0	401	same
26.5.7 (last 26.5 patch)	401	same
26.6.0	400	same
26.6.4	400	same
26.7.0	400	same
26.7.1	400	same

The change landed in 26.6.0. The entire 26.5 line, including its final patch 26.5.7, returns 401; 26.6.0 onward returns 400. Body text is unchanged.

Verification

Claim	Primary source	Checked
Change is version-caused, not realm state	fresh DB per version, clean master realm	☑
Boundary is 26.6.0, not a 26.5.x patch	26.5.7 (last 26.5 patch) = 401, 26.6.0 = 400	☑
Entire 26.6+ line is 400	26.6.0, 26.6.4, 26.7.0, 26.7.1 all 400	☑
RFC 6749 says 400 for `invalid_grant`	RFC 6749 §5.2 (as noted in `s9-ldap-federation.md`)	☑
Whether the change is in Keycloak release notes	checked 2026-08-26 — issue #45812 "ROPC: invalid_grant Error Response not RFC Compliant" (area `oidc`) in the 26.6.0 release notes	☑

Publishable extract

A failed password grant against a Keycloak token endpoint answers 401 on 26.0.0 through 26.5.7, and 400 from 26.6.0 onward — same request, same wrong password, identical Invalid user credentials body. The change landed in 26.6.0. RFC 6749 specifies 400 for invalid_grant, so the newer behaviour is the correct one; but any client keying on 401 to detect a bad password breaks on upgrade past 26.6.0, and that code is common.

KC-SERVICES0047 (internal eventsListener SPI warning) — present on 25.0.6, not new in 26

2026-08-26T00:00:00+00:00

Environment

Field	Value
Keycloak versions	25.0.6 and 26.0.0, same provider jar on both
Distribution	quay.io/keycloak/keycloak (official image)
Start mode	`start` w/ external DB
Database	Postgres 16, fresh empty DB per version
Provider under test	`lab-event-listener` (fixtures/spi), built against 25.0.6 via `bin/build-spi.sh 25.0.6`
Topology	single container
Host	Hetzner ccx33 (8 vCPU / 32 GB)

Procedure

./bin/build-spi.sh 25.0.6          # lab-event-listener-built-for-25.0.6.jar
# per version: down -v, set KC_VERSION, up db+keycloak, wait-ready
docker compose logs keycloak | grep -i "KC-SERVICES0047"

The jar is mounted at /opt/keycloak/providers (compose default).

Result

Both versions emit the warning at provider load:

WARN [org.keycloak.services] KC-SERVICES0047: lab-event-listener
  (lab.LabEventListenerProviderFactory) is implementing the internal SPI
  eventsListener. This SPI is internal and may change without notice

Version	`KC-SERVICES0047` emitted?
25.0.6	yes
26.0.0	yes

The warning is not new in 26. It was present in 25.0.6. The earlier record (2026-08-25-s7-custom-spi) had left this unchecked and warned against claiming it was new in 26; that caution was correct, and the claim is now refuted.

Verification

Claim	Primary source	Checked
Warning appears on 25.0.6	`docker compose logs keycloak` on 25.0.6	☑
Warning appears on 26.0.0 (same jar)	`docker compose logs keycloak` on 26.0.0	☑
Same provider on both (attributable jar)	jar filename records build version	☑
Earliest release that emits it	not checked — only 25.0.6 and 26.0.0	☐

Rehearsal Record — brittle protocol mapper: the last row of the brittle matrix, loud at use

2026-08-26T00:00:00+00:00

Why this run existed

2026-08-26-provider-types-protocol-mapper-brittle closed three of the four brittle rows (listener = lazy + silent; authenticator = lazy + loud; user-storage = lazy + loud) and left one ☐: the protocol mapper. The stated blocker was that "token-transform methods moved out of the base ProtocolMapper interface in 26, so the removed helper has no natural home there."

That blocker is refuted. The base ProtocolMapper interface (keycloak-server-spi-private) has never carried a token-transform method — its only abstract methods across 21.1.2 → 26.7.1 are getProtocol / getDisplayCategory / getDisplayType (plus a couple of defaults that grow over time, validateConfig from the start and getEffectiveModel from 23.x). The work method lives on the OIDC marker interface org.keycloak.protocol.oidc.mappers.OIDCAccessTokenMapper (transformAccessToken), which sits in keycloak-services, not keycloak-server-spi-private, and whose signature is stable across the 25 → 26 boundary. So there is a natural home; the previous note mistook "the work method is not in the artifact we build against" for "the work method moved in 26".

This run builds fixtures/spi-brittle-pm/ — a mapper that implements ProtocolMapper + OIDCAccessTokenMapper and calls the same removed three-argument runJobInTransactionWithResult overload the other brittle fixtures use, from inside transformAccessToken().

Environment

Field	Value
Keycloak	built against 25.0.6, loaded on 26.7.1 (crosses the 26.0.0 API-removal boundary)
Fixture	`fixtures/spi-brittle-pm/` (new) — `lab-brittle-protocolmapper`
Build	`bin/build-spi.sh 25.0.6 --module spi-brittle-pm` (Maven in Docker)
New dependency	`keycloak-services` (`provided`) — for `OIDCAccessTokenMapper`, the one artifact the other three brittle fixtures do not need
Database	PostgreSQL 16, fresh empty DB
Host	local laptop Docker (not Hetzner); 1k-class, no scale component
JVM heap	`-Xms1g -Xmx4g` (compose default)

The removed API is the same vehicle used across the whole brittle matrix: the three-argument overload KeycloakModelUtils.runJobInTransactionWithResult( KeycloakSessionFactory, KeycloakContext, KeycloakSessionTaskWithResult) — present in 25.0.6, absent in 26.0.0 (replaced by a four-argument overload), living in keycloak-server-spi-private (no compatibility promise).

Procedure

cd lab
# fixture is checked in; build the brittle mapper against the OLD major
./bin/build-spi.sh 25.0.6 --module spi-brittle-pm
./bin/lab-up.sh 26.7.1          # jar is bind-mounted into /opt/keycloak/providers

# verify it LOADED (no ERROR, internal-SPI warning, registered)
docker compose logs keycloak | grep -E 'lab-brittle-pm|KC-SERVICES0047'
#   KC-SERVICES0047: lab-brittle-protocolmapper ... internal SPI protocol-mapper ...
#   [lab-brittle-pm] protocol mapper init -- provider loaded
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8080/admin/serverinfo \
  | jq '.providers["protocol-mapper"]'          # internal:true, "lab-brittle-protocolmapper" present

# trigger token issuance (the mapper's work method) with a service account:
# realm pmtest, confidential client brittle-client w/ the mapper attached, serviceAccountsEnabled
curl -s -o /tmp/tok.out -w '%{http_code}' \
  -d client_id=brittle-client -d client_secret=brittle-secret -d grant_type=client_credentials \
  http://localhost:8080/realms/pmtest/protocol/openid-connect/token
#   -> HTTP 500, {"error":"unknown_error",...}

Attach the mapper by including it in the client's protocolMappers at create time ({"protocolMapper":"lab-brittle-protocolmapper","config":{}}), then enable serviceAccountsEnabled and issue a client-credentials grant. Any token grant exercises transformAccessToken; the client-credentials grant is the cleanest trigger because it does not involve the user required-actions machinery (see note under Outcome).

Findings

Finding 1 — the token-transform work method is in `keycloak-services`, not `server-spi-private`, and it is stable across 25 → 26 ⭐

The base ProtocolMapper interface carries no token-transform method at any of 21.1.2, 23.0.7, 24.0.5, 25.0.6, 26.0.0 or 26.7.1 — its abstract methods are getProtocol, getDisplayCategory, getDisplayType (with getPriority and validateConfig defaults from the start, getEffectiveModel from 23.x). So the earlier "moved out of the base interface in 26" note does not describe a real change.

The work method is OIDCAccessTokenMapper.transformAccessToken (and OIDCIDTokenMapper.transformIDToken), which live in keycloak-services:

public interface org.keycloak.protocol.oidc.mappers.OIDCAccessTokenMapper {
  public abstract AccessToken transformAccessToken(AccessToken,
    ProtocolMapperModel, KeycloakSession, UserSessionModel, ClientSessionContext);
}

That signature is byte-for-byte identical in 25.0.6, 26.0.0 and 26.7.1. The practical consequence: a customer's OIDC mapper that extends AbstractOIDCProtocolMapper (also keycloak-services) or implements OIDCAccessTokenMapper has a stable work-method home to override — which is exactly where a removed-API call would live, and therefore the right place to ask the load-vs-use question.

Finding 2 — a brittle protocol mapper fails at use, loudly (HTTP 500) ⭐

fixtures/spi-brittle-pm calls the removed three-argument overload from inside transformAccessToken().

It loads. On 26.7.1 the server starts with zero ERRORs, the factory init() runs, KC-SERVICES0047 fires, and the provider registers in /admin/serverinfo under protocol-mapper (SPI internal: true):

WARN KC-SERVICES0047: lab-brittle-protocolmapper (lab.LabBrittleProtocolMapper)
  is implementing the internal SPI protocol-mapper. This SPI is internal and may
  change without notice
[lab-brittle-pm] protocol mapper init -- provider loaded

It dies at first use, loudly. A client-credentials grant returns HTTP 500 with the standard unknown_error body, and the log carries the uncaught error — thrown from transformAccessToken and propagating straight up through the token endpoint:

ERROR [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-1)
  Uncaught server error: java.lang.NoSuchMethodError: 'java.lang.Object
  org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(
  org.keycloak.models.KeycloakSessionFactory, org.keycloak.models.KeycloakContext,
  org.keycloak.models.KeycloakSessionTaskWithResult)'
    at lab.LabBrittleProtocolMapper.transformAccessToken(LabBrittleProtocolMapper.java:89)
    at org.keycloak.protocol.oidc.TokenManager.transformAccessToken(TokenManager.java:827)
    at org.keycloak.protocol.oidc.TokenManager.createClientAccessToken(TokenManager.java:429)
    at org.keycloak.protocol.oidc.grants.ClientCredentialsGrantType.process(ClientCredentialsGrantType.java:147)
    at org.keycloak.protocol.oidc.endpoints.TokenEndpoint.processGrantRequest(TokenEndpoint.java:174)

Finding 3 — the brittle matrix is now complete, and it splits two ways ⭐

All four provider types now have a brittle row, and none fails at load:

Type	Fails at	Visibility	Record
event listener	use	silent (request succeeds)	S7
authenticator	use	loud (HTTP 500)	`…-provider-types-protocol-mapper-brittle.md` F2
user-storage	use	loud (HTTP 500)	`…-provider-types-protocol-mapper-brittle.md` F4
protocol mapper	use	loud (HTTP 500)	this run, F2

Keycloak never invokes a provider's work method during startup, so a removed API that only the work method reaches never breaks the upgrade itself. The visibility split is: event dispatch swallows the error (listener); every other path propagates it to HTTP 500. A protocol mapper fails exactly the way an authenticator and a user-federation provider do, not the way a listener does.

Outcome

Reached ready (26.7.1, fresh DB)
Custom SPI loaded (brittle protocol mapper registered, internal:true)
Failure reproduced at use, loudly (HTTP 500 + NoSuchMethodError)
Login flow works — n/a: the mapper is wired to fail on purpose
Realm config intact — n/a: fresh throwaway realm

Operational note — password grant is the wrong trigger for a mapper

A password (ROPC) grant against a hand-rolled realm returned 400 invalid_grant: "Account is not fully set up" (error=resolve_required_actions) even with the user's requiredActions empty and emailVerified=true; clearing required actions and toggling realm flags did not move it. The client-credentials grant sidesteps this and exercises transformAccessToken directly, so it is the canonical trigger for a protocol-mapper fixture. The ROPC quirk is not diagnosed here and is not a finding of this run — it is the auth-session required-actions path, upstream of the mapper, and deserves its own look if the lab ever needs ROPC against a hand-rolled realm.

Verification

Claim	Primary source	Checked
Base `ProtocolMapper` has no token-transform method at any version	`javap` of `org.keycloak.protocol.ProtocolMapper` in `server-spi-private` 21.1.2 / 23.0.7 / 24.0.5 / 25.0.6 / 26.0.0 / 26.7.1 — abstract methods only `getProtocol`/`getDisplayCategory`/`getDisplayType` throughout	☑
`OIDCAccessTokenMapper` lives in `keycloak-services`, not `server-spi-private`	jar listing: class present in `keycloak-services`, absent from the other three artifacts	☑
`transformAccessToken` signature stable 25 → 26	`javap` of `OIDCAccessTokenMapper` in `keycloak-services` 25.0.6 / 26.0.0 / 26.7.1 — identical	☑
Brittle mapper loads on 26.7.1 (not load-failing)	startup log: `init -- provider loaded`, 0 ERRORs, `KC-SERVICES0047`, registered in serverinfo	☑
Brittle mapper dies at use, loudly	HTTP 500 + `unknown_error` body + `NoSuchMethodError` stack quoted above	☑
Throw site is the work method	stack: `LabBrittleProtocolMapper.transformAccessToken` → `TokenManager.transformAccessToken` → `createClientAccessToken` → `TokenEndpoint`	☑
The exact 24.x release that split model/storage	still not narrowed — bracketed to (23.0.7, 24.0.5]	☐

Publishable extract

A custom protocol mapper that outlived its API breaks your next token, not your upgrade.

We built an OIDC protocol mapper against Keycloak 25.0.6 that calls a helper removed in 26.0.0, then loaded it on 26.7.1. The server started cleanly — the mapper registered, /admin/serverinfo listed it, zero errors. The first token request that reached its transformAccessToken method returned HTTP 500 and a full NoSuchMethodError stack as an uncaught server error.

That is the same shape as a broken authenticator and a broken user-federation provider, and the opposite of a broken event listener: event dispatch swallows the exception and the login still succeeds, while the mapper's failure propagates out of the token endpoint. So a "can users still get a token" check catches a broken mapper and misses a broken listener. The mapper's work method lives in Keycloak's keycloak-services artifact, not the SPI artifact — a mapper that registers cleanly can still be sitting on an API that no longer exists.

Rehearsal Record — protocol mappers are internal SPI; brittle providers fail loudly at use, not load

2026-08-26T00:00:00+00:00

Why this run existed

2026-08-26-provider-types established the SPI relocation (UserStorageProvider left keycloak-server-spi for keycloak-model-storage) and that a tame authenticator + user-storage provider cross 25 → 26 cleanly. It left two rows unchecked: protocol mappers, and brittle (removed-API) versions of each type — whether they fail at load or at first use. S7 (2026-08-25-s7-custom-spi) answered that question for event listeners (lazy, silent) but for no other type. This run adds a tame protocol mapper, a brittle authenticator, and a brittle user-storage provider.

Environment

Field	Value
Keycloak	built against 25.0.6, loaded on 26.7.1 (crosses the 26.0.0 API-removal boundary)
Fixtures	`fixtures/spi-types/` (tame: authenticator + user-storage + new protocol mapper); `fixtures/spi-brittle-auth/` (new: brittle authenticator); `fixtures/spi-brittle-us/` (new: brittle user-storage)
Build	`bin/build-spi.sh 25.0.6 --module …` (Maven in Docker)
Database	PostgreSQL 16, fresh empty DB
Host	local laptop Docker (not Hetzner); 1k-class fixture, no scale component
JVM heap	`-Xms1g -Xmx4g` (compose default)

The removed API the brittle fixtures target is the three-argument overload KeycloakModelUtils.runJobInTransactionWithResult(KeycloakSessionFactory, KeycloakContext, KeycloakSessionTaskWithResult) — present in 25.0.6, absent in 26.0.0 (replaced by a four-argument overload carrying an extra String), living in keycloak-server-spi-private (no compatibility promise).

Findings

Finding 1 — a tame protocol mapper registers on 26.7.1, and the SPI is internal ⭐

A custom ProtocolMapper implementing only the stable base interface loads and registers on 26.7.1 alongside the authenticator and user-storage provider. All three appear in /admin/serverinfo:

SPI (serverinfo key)	Provider
`authenticator`	`lab-authenticator`
`protocol-mapper`	`lab-protocol-mapper`
`storage`	`lab-user-storage`

And the "internal SPI" warning now covers a third type. KC-SERVICES0047 fires for the protocol mapper and the authenticator, and not for the user-storage provider:

WARN KC-SERVICES0047: lab-protocol-mapper (lab.LabProtocolMapper) is
  implementing the internal SPI protocol-mapper. This SPI is internal and may
  change without notice
WARN KC-SERVICES0047: lab-authenticator (lab.LabAuthenticatorFactory) is
  implementing the internal SPI authenticator. This SPI is internal and may
  change without notice

The reason is the same one the earlier record found for user-storage, inverted: org.keycloak.protocol.ProtocolMapper lives in keycloak-server-spi-private (verified: present there, absent from keycloak-core and keycloak-server-spi in 26.7.1), the artifact that carries no compatibility promise. So a customer's protocol mappers sit on the same unstable ground as their authenticators and event listeners; only user federation is a public SPI.

The base ProtocolMapper interface itself is stable across the ladder — the same three abstract methods (getProtocol, getDisplayCategory, getDisplayType) from 21.1.2 through 26.7.1 — so a tame mapper crosses majors intact. (What does not survive is in the ☐ rows: token-transform behaviour, which moved out of the base interface — see Not tested.)

Correction (added 2026-08-26, same day): the parenthetical above is imprecise. The base ProtocolMapper interface never carried the token-transform methods — they live on OIDCAccessTokenMapper / OIDCIDTokenMapper in keycloak-services, a separate artifact, and their signatures are stable 25 → 26. The brittle-protocol-mapper row is resolved in 2026-08-26-provider-types-brittle-protocol-mapper; the "moved out of the base interface in 26" blocker does not describe a real change.

Finding 2 — a brittle authenticator fails at use, not at load, and it fails loudly ⭐⭐

fixtures/spi-brittle-auth calls the removed three-argument overload from inside authenticate() — the work method, deliberately not init() — so the question asked is the one S7 asked of listeners: does Keycloak die at startup, or only when a flow reaches the step?

It loads. On 26.7.1 the server starts with zero ERRORs, the factory init() runs, and the provider registers in /admin/serverinfo:

[lab-brittle-auth] authenticator factory init -- provider loaded
WARN KC-SERVICES0047: lab-brittle-authenticator (lab.LabBrittleAuthenticatorFactory)
  is implementing the internal SPI authenticator ...

It dies at first use, loudly. Wiring it as a REQUIRED execution in a browser flow and attempting a login returns HTTP 500 with a JSON error body, and the log carries the full stack as an uncaught server error:

ERROR [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-1)
  Uncaught server error: java.lang.NoSuchMethodError: 'java.lang.Object
  org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(
  org.keycloak.models.KeycloakSessionFactory, org.keycloak.models.KeycloakContext,
  org.keycloak.models.KeycloakSessionTaskWithResult)'
    at lab.LabBrittleAuthenticator.authenticate(LabBrittleAuthenticator.java:25)
    at org.keycloak.authentication.DefaultAuthenticationFlow.processSingleFlowExecutionModel(...)

Response body:

{"error":"unknown_error","error_description":"For more on this error consult the server log."}

This is the answer to the open row, and it splits two ways. The load-vs-use half generalises S7: Keycloak does not invoke an authenticator's authenticate() (or a listener's onEvent()) during startup, so a removed API that only those methods reach does not break the upgrade — the breakage is deferred to first use. The silent-vs-loud half does not generalise: S7's listener failure was swallowed (the triggering request still returned 201/200, nothing at default log level), whereas the authenticator's NoSuchMethodError propagates out of the authentication flow as an uncaught error and the login fails with HTTP 500.

Why that matters for a post-upgrade check: a brittle authenticator is self-announcing — the first login fails loudly and someone investigates. A brittle listener is the dangerous one — logins keep succeeding while audit events silently stop being forwarded. A smoke test ("can users log in?") catches the former and misses the latter.

Finding 3 — the brittle-listener control reproduces S7 in this environment

Re-running the existing fixtures/spi-brittle listener on the same 26.7.1 instance confirmed S7: it registers, factory init() runs, KC-SERVICES0047 fires — and a successful login returns 200 with no error and no [lab-brittle] user event line in the log. The onEvent println sits after the removed-API call, so its absence means the throw happened before it and was swallowed by event dispatch. The same NoSuchMethodError that is loud in an authentication flow is silent in an event listener.

Finding 4 — brittle user-storage also fails lazily, and it fails loudly ⭐

fixtures/spi-brittle-us calls the same removed three-argument overload from getUserByUsername (the lookup work method), with the session captured in the constructor — the natural shape of a real user-federation provider, since lookup methods receive no session argument of their own.

It loads. On 26.7.1 the server starts with zero ERRORs, the factory init() runs, and the provider registers under the storage SPI in /admin/serverinfo with no KC-SERVICES0047 — consistent with user federation being the one public SPI. The component is not even instantiated at startup: its constructor line appears only on first use, when the realm's UserProvider is built.

It dies at first lookup, loudly. Configuring the provider as a user federation component and attempting a password grant for any username returns HTTP 500 with the same unknown_error body and an uncaught NoSuchMethodError:

ERROR [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-1)
  Uncaught server error: java.lang.NoSuchMethodError: 'java.lang.Object
  org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(...)'
    at lab.LabBrittleUserStorageProvider.getUserByUsername(LabBrittleUserStorageProvider.java:37)
    at org.keycloak.storage.UserStorageManager.getUserByUsername(UserStorageManager.java:559)

The exception escapes UserStorageManager.tryResolveFederatedUser rather than being swallowed, so user federation failures announce themselves the way an authenticator's does — not the way a listener's does.

The brittle matrix is now complete across four types, and it splits cleanly:

Type	Fails at	Visibility	Record
event listener	use	silent (request succeeds)	S7
authenticator	use	loud (HTTP 500)	this run, Finding 2
user-storage	use	loud (HTTP 500)	this run, Finding 4
protocol mapper	use	loud (HTTP 500)	`2026-08-26-provider-types-brittle-protocol-mapper` (added later same day)

No brittle type fails at load. Keycloak does not invoke any provider's work method during startup, so a removed API only reachable from the work method never breaks the upgrade itself — it defers the breakage to first use. The difference between the rows is purely how the failure surfaces: event dispatch swallows it; authentication and user-lookup paths let it propagate to HTTP 500.

Verification

Claim	Primary source	Checked
Tame protocol mapper registers on 26.7.1	live `/admin/serverinfo` query — `protocol-mapper → lab-protocol-mapper` (quoted above)	☑
`ProtocolMapper` is in `keycloak-server-spi-private`, not `-spi`/`-core`	jar listing of all three 26.7.1 artifacts (0/0/1 matches)	☑
`protocol-mapper` SPI is internal	`KC-SERVICES0047` line for `lab-protocol-mapper` in startup log	☑
`authenticator` internal, `storage` not	warning fired for authenticator only, consistent with prior record	☑
Brittle authenticator loads (not load-failing)	startup log: `init -- provider loaded`, 0 ERRORs, registered in serverinfo	☑
Brittle authenticator dies at use, loudly	HTTP 500 + `unknown_error` body + `NoSuchMethodError` stack (quoted above)	☑
Removed overload present 25.0.6, absent 26.x	`javap` of `KeycloakModelUtils` in both artifacts	☑
Brittle listener is silent (S7 control)	login 200, no error, no `[lab-brittle] user event` line	☑
Brittle user-storage (load vs use)	lazy + loud — HTTP 500 on first lookup, `NoSuchMethodError` at `getUserByUsername` (Finding 4)	☑
Brittle protocol mapper (load vs use)	done — `2026-08-26-provider-types-brittle-protocol-mapper`: lazy + loud (HTTP 500). That record refutes the "no natural home" blocker below: the work method `OIDCAccessTokenMapper.transformAccessToken` lives in `keycloak-services` and is stable 25 → 26	☑
The exact 24.x release that split model/storage	narrowed — 24.0.0 (+ 25.0.3 anomaly). See `2026-08-26-userstorage-artifact-split-narrowed`	☑

Publishable extract

A custom authenticator that outlived its API does not break your upgrade. It breaks your next login.

We built an authenticator against Keycloak 25.0.6 that calls a helper removed in 26.0.0, then loaded it on 26.7.1. The server started cleanly — the provider registered, /admin/serverinfo listed it, zero errors. The first login that reached it returned HTTP 500 and a full NoSuchMethodError stack as an uncaught server error.

A broken user-federation provider behaves like the authenticator, not the listener: the first login that consults it returns HTTP 500.

Compare the same mistake in an event listener: the server also starts, the listener also registers, but when the first event fires the exception is swallowed and the request still succeeds. Your audit trail stops silently.

All three are the same removed API. The difference is not the provider, it is where Keycloak calls it: authentication and user-lookup paths propagate the error, event dispatch eats it. So your post-upgrade check cannot be "can users log in" — that catches the loud failures and misses the quiet one. Verify the listener actually produced output, not that login works.

Rehearsal Record — provider types beyond event listeners: the SPI relocation, and which SPIs are "internal"

2026-08-26T00:00:00+00:00

Why this run existed

S7 (2026-08-25-s7-custom-spi) established that an event listener built for 21.1.2 loads and registers on 26.7.1, and that a brittle one fails lazily — at first use, silently. It left unchecked whether other provider types (authenticators, user storage, mappers) behave the same way. This run adds an authenticator and a user-storage provider, and the attempt to write them surfaced a finding S7 never could have: the user-storage SPI is not in keycloak-server-spi at all, and its Maven artifact changed between majors.

Environment

Field	Value
Keycloak	built against 25.0.6, loaded on 26.7.1 (crosses the 26.0.0 API-removal boundary)
Fixture	`fixtures/spi-types/` — `lab-authenticator` (Authenticator + Factory), `lab-user-storage` (UserStorageProvider + Factory), stable SPI only
Build	`bin/build-spi.sh 25.0.6 --module spi-types` (Maven in Docker)
Database	PostgreSQL 16, fresh empty DB
Host	Hetzner CCX33 (fsn1) — 8 vCPU, 30 GB RAM
JVM heap	`-Xms1g -Xmx4g`

Verification

Claim	Primary source	Checked
`UserStorageProvider` absent from `keycloak-server-spi` 25.0.6	jar contents listed; compile failure	☑
It lives in `keycloak-model-legacy` (23) then `keycloak-model-storage` (24+)	Keycloak git tree at 23.0.7 / 24.0.5 / 25.0.6	☑
`keycloak-model-legacy` gone by 25.0.6	Maven Central 404 at 25.0.6, 200 at 23/24	☑
`spi-api-diff.sh` misses it (does not diff model artifacts)	script source diffs only core/server-spi/server-spi-private	☑
Tame authenticator loads + registers on 26.7.1	factory init line; `lab-authenticator` in serverinfo	☑
Tame user-storage loads + registers on 26.7.1	factory init line; `lab-user-storage` in serverinfo	☑
`authenticator` SPI is internal, `user-storage` is not	`KC-SERVICES0047` fired for authenticator only	☑
Protocol mappers (third S7 type)	done — `2026-08-26-provider-types-protocol-mapper-brittle`: registers, SPI is internal	☑
Brittle (removed-API) versions of these types, and when they fail	authenticator + user-storage done — both lazy + loud (HTTP 500), `2026-08-26-provider-types-protocol-mapper-brittle`. Protocol-mapper brittle variant still open	◐
The exact 24.x release that split model/storage from model/legacy	narrowed — 24.0.0; `keycloak-model-legacy` dropped at 25.0.0 with a 25.0.3 anomaly. See `2026-08-26-userstorage-artifact-split-narrowed`	☑

Rehearsal Record — Run C: export/import across 23 → 24 keeps unmanaged attributes, and S15 said it would not

2026-08-26T00:00:00+00:00

Why this run existed

S15 found that in-place upgrading 23 → 24 preserves unmanaged user attributes, because the migration sets that realm's unmanagedAttributePolicy to ENABLED, and that a realm created on 24+ silently discards them. It then generalised:

So the danger is not the upgrade. It is every path that produces a new realm on a new version: export/import migration, a rebuilt environment, disaster recovery, or simply a realm created after the upgrade.

Export/import is the most common of those and the one customers choose because it feels safer than an in-place upgrade. We ranked it the untested case most likely to bite a real customer.

It does not bite. The generalisation was wrong, and we would have told a customer otherwise.

Environment

Field	Value
Source	realm `expimp` created on 23.0.7 through the admin API, 3 users each carrying `department` and `employeeId`
Export	`kc.sh export --dir /exp --realm expimp --users realm_file`, run as a one-shot container against the same database
Import	`kc.sh import --dir /exp` into a freshly purged database at each target version
Targets	24.0.5, 25.0.6, 26.0.0, 26.7.1
Database	PostgreSQL 16, single node, container; `docker compose down -v` between legs
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`

Results

Leg	Users	`USER_ATTRIBUTE` rows	Admin API returns them	Accepts a new attribute
23.0.7 as built	3	6	yes	—
23.0.7 → 23.0.7 (control)	3	6	yes	—
23.0.7 → 24.0.5	3	6	yes	204, rows 6 → 7
23.0.7 → 25.0.6	3	6	yes	204, rows 6 → 7
23.0.7 → 26.0.0	3	6	yes	204, rows 6 → 7
23.0.7 → 26.7.1	3	6	yes	204, rows 6 → 7
in-place 23 → 24, then export/import within 24.0.5	3	6	yes	—

Nothing is lost, at any target version, and the imported realm continues to accept unmanaged attributes afterwards — the write test added one and the row count went to 7. That last column matters: S15's realm-creation failure showed HTTP 201 with zero rows reaching the database. Here the write lands.

Verification

Claim	Primary source	Checked
Attributes survive export/import into 24.0.5	6 `USER_ATTRIBUTE` rows, admin API returns both	☑
…and into 25.0.6, 26.0.0, 26.7.1	same query at each version	☑
The imported realm still accepts new unmanaged attributes	PUT → 204, rows 6 → 7, at all four versions	☑
The import sets `unmanagedAttributePolicy=ENABLED`	`component_config` for `declarative-user-profile`, read directly	☑
A 23.0.7 export carries the attributes but not the policy	the export JSON, both greps quoted	☑
Export/import within 23.0.7 is lossless (control)	6 rows before and after	☑
Larger realms behave the same	not tested — 3 users	☐
`--users different_files` / `same_file` strategies	not tested — only `realm_file`	☐
Partial export through the admin API rather than `kc.sh export`	not tested, and it excludes users by default	☐
Whether a realm created by the admin API on 24+ still loses them	not retested here — S15's result stands as measured	☐

Rehearsal Record — Run D: LDAPS works across three majors; what breaks is trust, and it says "SocketReset"

2026-08-26T00:00:00+00:00

Summary

S9 ran federation over plaintext and said TLS is where real deployments break. Two candidate reasons were on the table. One is refuted and one is confirmed, and the confirmed one is confirmed for a reason nobody would guess from the error message.

Refuted: the truststore option rename does not break federation across 23 → 24. A deployment configured the pre-24 way keeps working on 24.0.5 and on 26.7.1, three majors later, with only deprecation warnings.

Confirmed: an untrusted certificate breaks it completely — and Keycloak reports it as SocketReset, with no mention of a certificate anywhere in the response. The server starts perfectly happily; nothing surfaces until something touches LDAP.

Environment

Field	Value
Directory	`bitnamilegacy/openldap:2.6`, TLS on 1636, plaintext 1389 still up
Certificates	private CA, server cert `CN=ldap`, SAN `DNS:ldap, DNS:localhost, IP:127.0.0.1`, generated by `bin/make-ldap-certs.sh`
Federation config	`seed-realm.sh --ldap --ldap-host ldaps://ldap:1636`, `editMode=READ_ONLY`, `importEnabled=true`
Trust (24+)	`KC_TRUSTSTORE_PATHS=/opt/keycloak/conf/truststores`, a bind-mounted directory
Trust (pre-24)	`KC_SPI_TRUSTSTORE_FILE_FILE` + `_PASSWORD`, a PKCS12 built with `keytool`
Database	PostgreSQL 16, single node, container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`
Lab change	`docker-compose.ldaps.yml`, `docker-compose.ldaps-legacy.yml`, `bin/make-ldap-certs.sh`

Breakage observed

A certificate trust failure is reported as `SocketReset`

Symptom: federation returns nothing; sync fails with {"errorMessage":"SocketReset"} and HTTP 400. Server healthy throughout.
Evidence: D1 against D2 — same server, same config, the only difference being one CA certificate in the truststore directory.
Cause: confirmed by the differential.
Fix: put the directory's CA in the truststore — KC_TRUSTSTORE_PATHS on 24+, KC_SPI_TRUSTSTORE_FILE_FILE before that.
Would this hit a customer? Every customer using LDAPS with an internal CA, which is most of them. The failure is undiagnosable from its message.

A malformed truststore fails the same way as no truststore

Symptom: trust configured, path correct, file present, and LDAPS still fails exactly as if nothing had been configured.
Evidence: found by making the mistake. openssl pkcs12 -export -nokeys writes the CA as a plain certBag; Java requires a trustedCertEntry. The store loads, the trust anchor set is empty, and the only trace is PKIXParameters.setTrustAnchors in a stack trace.
Fix: build truststores with keytool -importcert, and verify with keytool -list that the entry type is trustedCertEntry. bin/make-ldap-certs.sh now does both.
Would this hit a customer? Yes, and it is a plausible pre-flight finding — "your truststore is empty" is not visible from the file existing.

Verification

Claim	Primary source	Checked
LDAPS federation works with the CA trusted	D2 — 3 users imported, HTTP 200, lookup returns `alice`	☑
Without trust it fails as `SocketReset`, HTTP 400	D1, response quoted	☑
Keycloak starts and passes readiness either way	both legs reached ready before the sync was attempted	☑
No certificate wording in the response or default log	grepped for `PKIX`, `handshake`, `truststore`, `certification` — nothing in D1	☑
TLS federation survives 26.0.0 → 26.7.1	D3 — READY, 3 users still linked	☑
Pre-24 truststore config still works on 24.0.5 and 26.7.1	D5/D6/D7 — syncs return 200, 3 users at every step	☑
Keycloak warns that the pre-24 options are deprecated	both warnings quoted, from 24.0.5 and 26.7.1	☑
A certBag-only PKCS12 behaves as an empty truststore	reproduced, then fixed; `keytool -list` confirms `trustedCertEntry` after	☑
StartTLS on 1389	tested — works trusted, fails silently (HTTP 200, 0 imported) — `2026-08-26-starttls-1389`	☑
Hostname-verification failures (cert whose SAN does not match)	tested — hard failure, `SSLPeerUnverifiedException`, silent at the API — `2026-08-26-starttls-1389`	☑
Expired or revoked directory certificates	expired: tested (silent, `CertificateExpiredException`); revoked: not tested — no CRL in the lab CA	☑ (expired) / ☐ (revoked)
Behaviour when the truststore path exists but is empty on 24+	not tested separately — D1's empty directory is that case, but it was not distinguished from "no truststore configured"	☐
`editMode=WRITABLE` / `UNSYNCED` over TLS	not tested — still `READ_ONLY`, as S9 left it	☐

Rehearsal Record — StartTLS on 1389: works when trusted, and fails silently (unlike LDAPS)

2026-08-26T00:00:00+00:00

Why this run existed

Run D (2026-08-26-runD-ldaps-starttls) covered implicit LDAPS on 1636 and left three rows unchecked: StartTLS on 1389, hostname-verification failures, and expired certificates. This run closes all three, and the first one changes what Run D concluded about the failure signature.

Environment

Field	Value
Keycloak	26.7.1 (official image)
Directory	`bitnamilegacy/openldap:2.6`, `LDAP_ENABLE_TLS=yes` — TLS on 1636 (LDAPS) and StartTLS on 1389
Federation config	`seed-realm.sh --ldap --ldap-starttls` → `connectionUrl=ldap://ldap:1389`, `startTls=true`, `editMode=READ_ONLY`
Certificates	private CA (`lab-ldap-ca`); server cert `CN=ldap`, SAN `DNS:ldap, DNS:localhost, IP:127.0.0.1` — plus ad-hoc variants for the failure legs
Trust	`KC_TRUSTSTORE_PATHS=/opt/keycloak/conf/truststores` (bind-mounted dir); trusted = `ca.crt` present
Database	PostgreSQL 16, single node, container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM
JVM heap	`-Xms1g -Xmx4g`
Lab change	`seed-realm.sh` gained `--ldap-starttls` (sets `startTls:["true"]` on the LDAP component)

Result — four legs, one sync each

Full sync via POST /admin/realms/lab/user-storage/{id}/sync?action=triggerFullSync.

Leg	Trust	Cert	Sync API response	HTTP	Log cause
StartTLS trusted	CA trusted	SAN matches `ldap`	`{"added":3, ...}` 3 imported	200	—
StartTLS untrusted	empty truststore	SAN matches	`{"added":0, ...}` 0 imported	200	`SunCertPathBuilderException: unable to find valid certification path`
StartTLS hostname mismatch	CA trusted	SAN `DNS:wronghost`	`{"added":0, ...}` 0 imported	200	`SSLPeerUnverifiedException: hostname of the server 'ldap' does not match the hostname in the server's certificate`
StartTLS expired cert	CA trusted	notAfter = issue time	`{"added":0, ...}` 0 imported	200	`CertificateExpiredException: NotAfter: …` + `CertPathValidatorException: validity check failed`

Findings

1. StartTLS failure is silent; LDAPS failure is loud

Run D established that an untrusted certificate on LDAPS (1636) fails with {"errorMessage":"SocketReset"} and HTTP 400 — loud, and the word "certificate" never appears. On StartTLS the equivalent failures all return HTTP 200 with a body that looks like a successful, empty sync:

{"ignored":false,"added":0,"updated":0,"removed":0,"failed":0,
 "status":"0 imported users, 0 updated users"}

The real cause is only in the Keycloak log, as a full stack trace. An operator (or monitoring integration) that checks the HTTP code — or only greps for SocketReset/errorMessage — sees a successful sync that imported nothing, and reaches for "is the directory empty? is the DN wrong?" instead of "is the certificate trusted?". The same misdirection Run D warned about for LDAPS is worse on StartTLS, because there is no error at all at the API level.

The one correct signal in the response is the count: added:0 where you expect users. The lesson for the runbook is check the imported count, not the status code — on StartTLS, "0 imported" is a failure signal, not a success.

2. Keycloak enforces hostname verification on LDAP TLS

A trusted CA with a cert whose SAN is DNS:wronghost fails with SSLPeerUnverifiedException / No subject alternative DNS name matching ldap found. Keycloak does not default to "trust everything, skip hostname" on LDAP connections — the HostnameChecker runs, and a SAN that does not match the connection hostname is a hard failure (silent at the API, loud in the log). This means a directory cert issued for the wrong name — or a wildcard that does not cover the host — breaks federation exactly like an untrusted CA.

3. StartTLS itself works and survives, matching LDAPS

With the CA trusted and the SAN matching, StartTLS on 1389 imports all three users — no functional difference from implicit LDAPS. The mode is not the problem; the failure signature is.

Verification

Claim	Primary source	Checked
StartTLS on 1389 works when trusted	sync returned `added:3`, HTTP 200	☑
`startTls=true` is actually negotiated (not plaintext fallback)	untrusted leg failed at the TLS handshake (`StartTlsResponseImpl.negotiate` → PKIX)	☑
Untrusted CA under StartTLS is silent	HTTP 200, `added:0`, `SunCertPathBuilderException` in log	☑
Hostname mismatch is a hard failure	HTTP 200, `added:0`, `SSLPeerUnverifiedException` in log	☑
Expired cert is silent	HTTP 200, `added:0`, `CertificateExpiredException` in log	☑
Contrast with LDAPS (loud)	Run D D1: `SocketReset`, HTTP 400	☑
Revoked certificates (CRL/OCSP)	not tested — the lab CA has no revocation list	☐
`editMode=WRITABLE` / `UNSYNCED` over StartTLS	not tested — still `READ_ONLY`	☐

Rehearsal Record — user-storage artifact split, narrowed: 24.0.0 (and a 25.0.3 Maven Central anomaly)

2026-08-26T00:00:00+00:00

Why this run existed

2026-08-26-provider-types established that org.keycloak.storage. UserStorageProvider moved out of keycloak-server-spi and changed Maven artifact — keycloak-model-legacy (≤ 23) → keycloak-model-storage (24+) — with keycloak-model-legacy gone by 25.0.6. It left the exact boundary open, bracketed to (23.0.7, 24.0.5]. This run closes the bracket by inspection of Maven Central coordinates and the published jars.

Environment

Field	Value
Artifacts inspected	`keycloak-model-legacy`, `keycloak-model-storage`
Versions	23.0.7, 24.0.0, 24.0.5, 25.0.0–25.0.6
Method	Maven Central directory listing + per-coordinate HTTP status + `jar tf` for `org/keycloak/storage/UserStorageProvider.class`
Host	local laptop; Maven container + `curl`

Findings

Finding 1 — the relocation happens at exactly 24.0.0 ⭐

UserStorageProvider.class moves between artifacts on the 23 → 24 boundary:

Version	`UserStorageProvider.class` in `keycloak-model-legacy`	in `keycloak-model-storage`
23.0.7	yes	artifact not published
24.0.0	no	yes
24.0.5	no	yes

keycloak-model-storage is first published at 24.0.0, and from that release keycloak-model-legacy no longer contains the class. So the customer-relevant break is the 23 → 24 hop: a user-storage provider built against keycloak-model-legacy keeps its imports (org.keycloak.storage.*, unchanged) but must change its Maven coordinate to keycloak-model-storage at 24.0.0. Notably, keycloak-model-legacy is still published at 24.0.x — it just no longer carries UserStorageProvider, so a 24.x build that still declares the old coordinate resolves but fails to link the class.

Finding 2 — `keycloak-model-legacy` is dropped at 25.0.0, but 25.0.3 is a one-off anomaly ⭐

The publish history of keycloak-model-legacy on Maven Central is not a clean cut. Per-coordinate HTTP checks:

Version	`keycloak-model-storage` POM	`keycloak-model-legacy` POM
25.0.2	200	404
25.0.3	404	200
25.0.4	200	404

keycloak-model-legacy is absent for 25.0.0–25.0.2, reappears at 25.0.3 and there carries UserStorageProvider.class again (verified by jar tf), then is gone again from 25.0.4 onward. The mirror image holds for keycloak-model-storage: published for every 25.0.x except 25.0.3, where it is missing.

So the two artifacts invert at exactly 25.0.3 — a one-off publishing anomaly in that release. Practical consequences, both worth a line in the assessment:

a build pinned to keycloak-model-storage:25.0.3 fails with "could not resolve dependency" (404);
a build still declaring keycloak-model-legacy and bumped to 25.0.3 silently works (200, class present), then fails again at 25.0.4 (404) — the stale coordinate masks the migration for exactly one release and breaks the next.

Finding 3 — the stable end state from 25.0.4

From 25.0.4 onward the state is what the earlier record reported: keycloak-model-storage published, keycloak-model-legacy gone. The split is real and permanent at 24.0.0; the 25.0.3 inversion is a transient anomaly, not a reversal of the move.

Verification

Claim	Primary source	Checked
`keycloak-model-storage` first published at 24.0.0	Maven Central directory listing	☑
`UserStorageProvider.class` in model-storage (not legacy) from 24.0.0	`jar tf` of both artifacts at 23.0.7 / 24.0.0 / 24.0.5	☑
model-legacy dropped for 25.0.0–25.0.2	per-coordinate HTTP 404	☑
model-legacy reappears at 25.0.3 carrying the class	HTTP 200 + `jar tf`	☑
model-storage missing at 25.0.3	HTTP 404	☑
stable end state from 25.0.4	model-storage 200, model-legacy 404	☑

Publishable extract

Your user-federation provider does not change its imports between Keycloak 23 and 24 — it changes its Maven coordinate.

org.keycloak.storage.UserStorageProvider moved from the keycloak-model-legacy artifact to keycloak-model-storage at exactly 24.0.0. The import line is identical; the dependency is not. The old artifact kept publishing through the 24.0.x line but stopped containing the class, so a stale build resolves and then dies at runtime — the worst kind of break.

And there is a one-release trap: at Keycloak 25.0.3 the two artifacts inverted on Maven Central — keycloak-model-storage was not published (404) and keycloak-model-legacy was (200, class present), before settling back at 25.0.4. A dependency audit that checks "does the coordinate resolve" rather than "which artifact actually carries the class" misses both halves.

Rehearsal Record — HA: primary crash mid-migration, promoted replica resumes cleanly

2026-08-26T00:00:00+00:00

Summary

First run of the HA/failover line. A streaming replica was stood up behind the primary, the migration was started, the primary was SIGKILLed mid-migration (changeset 26.6.0-45009-broker-link-identity-provider), the replica was promoted, Keycloak was repointed at it, and it resumed and finished cleanly.

Four findings, in the order they matter:

The replica is a consistent, half-migrated snapshot. At the crash the replica had 179 of 211 changesets applied — every committed changeset, with the in-flight one rolled back. Streaming replication does not ship a half-applied transaction, so the failover target is never corrupt.
migration_model is the way to see it. At the crash the replica's changelog said 179 changesets but migration_model still said 26.0.0, because the model stamp is written last. The half-migrated state is therefore detectable, not silent — the same "changelog ≠ model" signal S8 taught us to read, now produced by a crash instead of a pooler.
The changelog lock is NOT a stuck-lock hazard across failover. After the crash the replica's DATABASECHANGELOGLOCK rows were LOCKED=f — the crashed primary's lock acquisition did not persist to the standby. Keycloak re-acquires the lock and resumes rather than waiting out a lock held by a dead host.
Keycloak resumes, it does not restart. Re-pointed at the promoted replica, Keycloak ran changesets 180 → 211, stamped the model 26.7.1, and reached ready with all 202 users intact. Index audit clean (119/119).

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB
Database	PostgreSQL 16, primary + streaming (async) replica, containers
Replication	`wal_level=replica`, streaming via `pg_basebackup -X stream`, `wal_keep_size=64MB`
Postgres tuning	stock (`shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0)
Adverse scenario	#4 — primary crash + failover, mid-migration
Dataset scale	200 users, 2 realms
Seeding method	`partialImport` (`bin/seed-realm.sh --profile typical`)
Topology	1 primary + 1 replica; Keycloak single container
Host	laptop (Docker 29)
JVM heap	`-Xms1g -Xmx4g`
Elapsed clock	migration ~4s to the crash point; recovery after repoint ~17s

Procedure

# primary + keycloak, then seed + snapshot (replica initialised by hand)
COMPOSE_FILE=docker-compose.yml:docker-compose.ha.yml
docker compose up -d db keycloak
./bin/seed-realm.sh --profile typical --users 200
./bin/snapshot.sh baseline-26.0.0-ha-typical-200
./bin/ha-init-replica.sh          # pg_basebackup + standby.signal + primary_conninfo

# failover: start migration, SIGKILL primary mid-flight, promote, repoint
sed -i 's/^KC_VERSION=.*/KC_VERSION=26.7.1/' .env
docker compose up -d keycloak &    # build + migrate
# poll for "Updating database", sleep 1s, then:
docker compose kill db            # SIGKILL, not a clean stop — a crash
./bin/ha-failover.sh              # pg_ctl promote + KC_DB_URL -> db-replica
./bin/index-audit.sh 26.7.1       # DB_SERVICE=db-replica

Results

The crash point

Keycloak was mid-changeset when the primary died:

ERROR: Migration failed for changeset META-INF/jpa-changelog-26.6.0.xml::
  26.6.0-45009-broker-link-identity-provider::keycloak:
  Reason: ... PSQLException: An I/O error occurred while sending to the backend.

The replica (still a standby) held, at that instant:

Replica state	Value
`databasechangelog` rows	179 (of 211)
`migration_model`	26.0.0
`databasechangeloglock`	two rows, both `LOCKED=f`

After promote + repoint

Promoted replica state	Value
`databasechangelog` rows	211
`migration_model`	26.7.1
`user_entity`	202 (unchanged)
index audit	119 correct / 0 missing / 0 wrong shape / 0 unverified

The old primary, restarted by depends_on, crash-recovered to its stale 179-changeset state and was then stopped — the promoted replica is the only primary.

What this teaches

Failover mid-migration is safe to rehearse and to expect: the standby is always a committed-prefix snapshot, and Keycloak resumes from that prefix. No corrupt schema, no stuck lock, no data loss.
The pre-flight for a customer running HA: the check after a failover is still "does migration_model match the target version" — not "did the changelog reach N rows". The crash leaves them divergent by construction.
This closes the last Compose-reachable part of the failover scenario. What is still untested: synchronous replication and a clean (promoted-by-choice) failover — a DBA promoting the replica for maintenance while Keycloak keeps writing — which is a different failure shape than a crash.

Verification

Claim	Evidence
Replica holds a committed-prefix snapshot after crash	179/211 changelog rows, model 26.0.0 ☑
Lock is not stuck after failover	`databasechangeloglock` `LOCKED=f` ☑
Keycloak resumes on the promoted replica	changelog 211, model 26.7.1, ready ☑
Schema correct after recovery	index audit 119/119 ☑
Data intact	`user_entity` 202 before and after ☑
Streaming works end-to-end	`pg_stat_replication` `streaming`, lag ~24µs ☑

Rehearsal Record — what Keycloak's index check actually checks, and the upgrade it kills

2026-08-26T00:00:00+00:00

Why this run existed

Run A established that Keycloak's index changeset is guarded by a name-based precondition and that the 300,000-row skip reads a planner statistic. It left four questions open, and each was cheap to answer once reltuples was known to be a lever. Answering them turned up a failure mode none of the previous twenty-one records contains: an upgrade that dies mid-migration and leaves the server unable to start.

Environment

Identical to 2026-08-25-runA-indexes-manually-created — 2,000,003 users, baseline-26.0.0-sqlseed-2m, Postgres 16 at stock settings, Hetzner CCX33 (8 vCPU dedicated, 30GB RAM, local NVMe), JVM -Xms1g -Xmx4g, official image, start with an external DB. Every leg begins with a verified restore of that baseline, so the legs are comparable to each other and to Run A's.

autovacuum is disabled on USER_ENTITY in the legs that write to pg_class, so an analyze cannot silently undo the setup between the write and the upgrade.

Results

Leg	Setup	Upgrade	`exectype`	Index on `USER_ENTITY`	WARNs
A6a	`reltuples`=1000, `relpages` honest	READY	`EXECUTED`	built, correct	0
A6b	`reltuples` honest, `relpages`=100	READY	`EXECUTED`	absent	2
A7a	same index name on the `realm` table	READY	`EXECUTED`	absent	1
A7b	same name, same table, UNIQUE, on `(id)`	READY	`MARK_RAN`	wrong index kept	0
A7c	A7a + falsified `reltuples`	FAILED	—	absent	0

Breakage observed

An index name collision on an unrelated table fails the upgrade and blocks startup

Symptom: relation "idx_user_created_timestamp" already exists, server refuses to start, changelog stops at 187/211 with migration_model at 26.0.0.
Evidence: A7c, quoted above.
Cause: confirmed — per-schema index names against a per-table precondition.
Fix or workaround: rename or drop the colliding index, restart. The half-applied changelog resumes, as it did in S8 and S14.
Would this hit a customer? Config-dependent and latent. It needs someone to have created an index carrying a Keycloak index name on another table — which our own runbook makes likelier, since it hands operators those exact names to paste. It then waits for a build to be attempted.

`DatabaseIndexChecker` reports a healthy database when the index is on the wrong table

Symptom: the checker runs and emits nothing, while the index the changeset defines does not exist on the table it names.
Evidence: §3 above; Running database index checker present, no Missing database index line, pg_indexes shows the index on realm.
Cause: confirmed — the checker matches on index name alone.
Fix: compare definitions, per table, as bin/index-audit.sh does.
Would this hit a customer? Anyone relying on Keycloak's own index check.

Verification

Claim	Primary source	Checked
`reltuples` is the input; `relpages` is not	A6a built, A6b skipped	☑
Keycloak reads the raw value, not the planner's estimate	Run A L5 — planner would compute ~533,920 and skip; index was built	☑
Precondition ignores columns	Run A L3 — `MARK_RAN` on a single-column index	☑
Precondition ignores uniqueness	A7b — `MARK_RAN` on a UNIQUE index of a different column	☑
Precondition honours the table	A7a — same name on `realm`, changeset ran	☑
`DatabaseIndexChecker` does not honour the table	control 2 WARNs vs A7a 1 WARN, message text captured	☑
A name collision fails the upgrade	A7c — Liquibase exception, quoted	☑
It leaves 187/211 and `migration_model` 26.0.0	A7c — queried directly	☑
Restarting after clearing the collision resumes the changelog	not tested this run — inferred from S8 and S14, which is not evidence	☐
The threshold constant on `USER_ENTITY` is 300,000, inclusive	bisected here — 300,000 builds, 300,001 skips	☑
Behaviour on version pairs other than 26.0.0 → 26.7.1	not tested	☐

Rehearsal Record — Kubernetes / Infinispan: the cache cluster partitions during a rolling update

2026-08-26T00:00:00+00:00

Summary

First Kubernetes run of the lab. A 2-replica Keycloak Deployment on k3d with embedded Infinispan was rolled 26.0.0 → 26.7.1 under maxUnavailable=0. Four findings:

The cache stack changed inside major 26. 26.0.0 hardcodes stack="udp" (multicast) in cache-ispn.xml, which cannot cluster in Kubernetes; 26.7.1 drops the hardcode and defaults to jdbc-ping (database discovery). Asking 26.0.0 for jdbc-ping fails with ISPN000540: No such JGroups stack 'jdbc-ping'. The kubernetes stack (DNS_PING) works on both but is deprecated in 26.7.1 — it is what this run used, so the version was the only variable.
Embedded Infinispan clusters and shares sessions across pods (26.0.0): with KC_CACHE_STACK=kubernetes + -Djgroups.dns.query=<headless FQDN>, two pods form a 2-member cluster, and a refresh token issued by pod A is accepted by pod B.
The schema migration is not what breaks a rolling update. During the rollout the new 26.7.1 pod migrated the shared Postgres under the two still- serving 26.0.0 pods, and the old pods logged no schema errors — the 26.0.0 → 26.7.1 changesets are additive. migration_model reached 26.7.1 and the rollout completed cleanly.
The cache cluster partitions during the rollout (the finding). Infinispan 15.0.8 (26.0.0) and 16.0.12 (26.7.1) — JGroups 5.3.10 vs 5.5.5 — refuse to form a mixed cluster. The TCP handshake between an old and a new pod fails with JGRP000006 ... cookie sent by ... does not match own cookie; terminating connection. Old pods stay in their 2-member cluster; each new pod starts a 1-member cluster; they only merge once the old pods are gone. Sessions are therefore not shared between old and new pods during the transition — an in-flight user session lands on whichever pod owns it, and a request routed to the other side of the partition finds nothing.

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB
Database	PostgreSQL 16, single pod (StatefulSet), in-cluster
Cache	embedded Infinispan (`KC_CACHE=ispn`), `KC_CACHE_STACK=kubernetes` (DNS_PING)
Cluster	k3d `kclab` (k3s v1.35.5), 1 server node
Topology	Deployment `replicas: 2`, `strategy.maxSurge=1, maxUnavailable=0`
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 32 GB RAM, 240 GB NVMe
JVM heap	`-Xms256m -Xmx1g`
Dataset	`master` realm + `lab` realm (1 user)
Seeding	admin REST API (create realm + user)

Procedure

# on the box: k3d + kubectl (NOT installed on the laptop)
k3d cluster create kclab --k3s-arg --disable=traefik@server:* --k3s-arg --disable=servicelb@server:*
kubectl apply -f lab/k8s/                 # namespace, postgres StatefulSet, keycloak Deployment

# seed, then rolling update
kubectl -n keycloak set image deploy/keycloak keycloak=quay.io/keycloak/keycloak:26.7.1
kubectl -n keycloak rollout status deploy/keycloak --timeout=180s

Manifests are kept with the lab's Kubernetes fixtures. The initial 26.0.0 run failed with ISPN000540: No such JGroups stack 'jdbc-ping' — the stack 26.7.1 defaults to did not exist in 26.0.0, which is finding #1.

Results

Timeline of the rolling update (from pod logs)

t	event
19:31:34	first new 26.7.1 pod starts, forms a 1-member cluster (`v=16.0.12`)
19:31:44	old 26.0.0 pods still in their 2-member cluster (`v=15.0.8`)
19:32:13–14	repeated `JGRP000006 ... cookie ... does not match own cookie` between old and new pods
19:32:15	second new pod starts, also a 1-member cluster
19:32:36	the two new pods form a 2-member cluster (`v=16.0.12`) — old pods gone
end	`migration_model` 26.7.1, both pods ready, cross-pod session sharing restored

The partition, in the log

WARN [org.jgroups.protocols.TCP] JGRP000006: 10.42.0.13:7800: failed accepting
  connection from peer Socket[addr=/10.42.0.15 ...]:
  BaseServer.TcpConnection.readPeerAddress(): cookie sent by ... does not match
  own cookie; terminating connection

The old cluster (JGroups 5.3.10) and the new pod (JGroups 5.5.5) never merge. DNS_PING hands each new pod the old pods' addresses, the handshake fails, and the new pod continues as a cluster of one until the old pods are terminated.

What this teaches

A within-major Keycloak rolling update is safe at the database layer — additive changesets mean old pods keep serving while the new pod migrates. The risk is not the schema, it is the cache.
Infinispan major/minor bumps do not inter-cluster. 26.0.0 (Infinispan 15, JGroups 5.3) and 26.7.1 (Infinispan 16, JGroups 5.5) form separate clusters, so a rolling update across them has a window where the session/authentication caches are partitioned. For upgrades that cross an Infinispan/JGroups version, the safe posture is a recreate rollout (terminate old, then start new), or accept a short cache-cold window.
The cache stack itself is a migration surface. A customer who pinned KC_CACHE_STACK (or relied on the 26.0.0 default) is changing discovery mechanism when they reach 26.7.1 — jdbc-ping did not exist in 26.0.0, and the kubernetes stack is deprecated in 26.7.1. This belongs in the upgrade runbook's pre-flight, not the schema checklist.

What is still not tested

Two new pods migrating concurrently (S2's exact half, here maxSurge=1 made the new pods sequential) — the DATABASECHANGELOGLOCK serialization was shown in Compose (s1-s2-lock-failure-modes.md) but not re-shown under k8s.
Functional proof of session stranding during the partition (issuing a session on an old pod and failing to use it on a new pod inside the transition window) — the partition is proven at the JGroups level, not the HTTP level.
External (remote) Infinispan, and the 26.7.1 default jdbc-ping stack.

Verification

Claim	Evidence
26.0.0 lacks `jdbc-ping`	`ISPN000540: No such JGroups stack 'jdbc-ping'` ☑
2 pods form a cluster and share sessions (26.0.0)	`ISPN000094 ... (2) [...]`, cross-pod refresh OK ☑
Rolling update completes, no old-pod schema errors	`migration_model` 26.7.1, no `PSQLException` in old logs ☑
Cache partitions during the rollout	`JGRP000006 cookie ... does not match` ☑
Cluster heals after rollout	final view `(2) [... v=16.0.12 ...]`, cross-pod refresh OK ☑

Rehearsal Record — MariaDB: toolchain port + S4 confirmed identical to MySQL

2026-08-26T00:00:00+00:00

Summary

MariaDB 11.4 (LTS) is the second non-Postgres engine. Three results:

The toolchain needed one more port. MariaDB renames the MySQL client tools — mariadb / mariadb-dump instead of mysql / mysqldump, with no compat symlinks in the image. snapshot.sh / restore.sh / index-audit.sh now pick the binary from the engine; the dump flag set also differs (MariaDB has no --set-gtid-purged / --no-tablespaces).
The migration is clean. 26.0.0 → 26.7.1 reaches ready, migration_model 26.7.1, and the index audit (engine mariadb) reports 121 correct / 0 missing / 0 wrong shape.
S4 carries over byte-for-byte from MySQL. A DML-only account fails the migration at Liquibase's own setup — ALTER command denied to user 'kc_dml' for table DATABASECHANGELOG [Failed SQL: (1142) ALTER TABLE keycloak.DATABASECHANGELOG ADD PRIMARY KEY (ID, AUTHOR, FILENAME)] — and granting the DDL set (CREATE, ALTER, DROP, INDEX, REFERENCES, CREATE VIEW, SHOW VIEW) makes it succeed. No ownership concept, same as MySQL.

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Database	MariaDB 11.4.13, single node, container `mariadb:11.4`
DB flags	`utf8mb4` / `utf8mb4_unicode_ci` (no invisible-PK flag — that is MySQL-only)
Cache	local (single node)
Dataset scale	200 users, 2 realms
Seeding method	`partialImport` (`seed-realm.sh --profile typical`)
Host	laptop (Docker 29)

Results

Run	Account's rights	Outcome
smoke	`keycloak` (all on `keycloak.*`)	READY, audit 121/121
A	`kc_dml` DML only	exit 1 — `ALTER command denied ... DATABASECHANGELOG ADD PRIMARY KEY` (error 1142)
B	`kc_dml` DML + DDL	READY, audit 121/121

Verification

Claim	Evidence
Migration clean on MariaDB	`migration_model` 26.7.1, audit 121/121 ☑
DML-only fails, DDL suffices (no ownership)	run A 1142 vs run B READY ☑
Snapshot/restore round-trips on MariaDB	restore "no re-migration", users 200 ☑

Rehearsal Record — Oracle: driver install + migration, the third engine

2026-08-26T00:00:00+00:00

Summary

Oracle is the third non-Postgres engine. Keycloak does not ship the Oracle JDBC driver (licence), so this run is also the driver-install deliverable. Two results:

Keycloak runs and migrates on Oracle. With ojdbc17 + orai18n (Maven Central 23.26.0.0.0) dropped into the base file's providers bind-mount — no custom image needed — Keycloak 26.0.0 starts against Oracle Database Free, creates 87 tables, and the 26.0.0 → 26.7.1 migration reaches migration_model 26.7.1 cleanly.
The Oracle Free image self-identifies as "26ai". gvenzl/oracle-free:23-slim reports Oracle AI Database 26ai Free Release 23.26.2.0.0 — Oracle renamed 23ai → 26ai in 2026. Keycloak's documented support is "23.x (23.5+)", and this release reports a 23.x version string, so it is within the supported line, but the rename is worth flagging in the runbook's pre-flight (a customer on "26ai" is on the 23.x line Keycloak supports).

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Database	Oracle Database Free (23ai/26ai), release 23.26.2.0.0, single node, container `gvenzl/oracle-free:23-slim`
JDBC driver	`com.oracle.database.jdbc:ojdbc17:23.26.0.0.0` + `com.oracle.database.nls:orai18n:23.26.0.0.0` in `fixtures/providers/`
Schema/user	`keycloak` (APP_USER; in Oracle the user IS the schema, so `KC_DB_SCHEMA` is not used)
Topology	single container
Host	laptop (Docker 29, 31 GB RAM)
JVM heap	`-Xms1g -Xmx4g`

Procedure

# drivers (Keycloak server/db docs "Installing the Oracle Database driver")
curl -o fixtures/providers/ojdbc17.jar https://repo1.maven.org/maven2/com/oracle/database/jdbc/ojdbc17/23.26.0.0.0/ojdbc17-23.26.0.0.0.jar
curl -o fixtures/providers/orai18n.jar https://repo1.maven.org/maven2/com/oracle/database/nls/orai18n/23.26.0.0.0/orai18n-23.26.0.0.0.jar

# .env: KC_DB=oracle, COMPOSE_FILE=docker-compose.yml:docker-compose.oracle.yml
docker compose up -d db        # Oracle Free first init ~11 min
docker compose up -d keycloak  # 26.0.0
docker compose stop keycloak && sed -i 's/^KC_VERSION=.*/KC_VERSION=26.7.1/' .env
docker compose up -d keycloak  # migrate

Results

Check	Value
Keycloak 26.0.0 starts on Oracle	ready; 87 tables created
`migration_model` before → after	26.0.0 → 26.7.1
Master-realm token endpoint after migration	ready
Oracle banner	`Oracle AI Database 26ai Free Release 23.26.2.0.0`

Verification

Claim	Evidence
Driver loads from providers, no custom image	Keycloak ready on Oracle ☑
Migration clean on Oracle	`migration_model` 26.7.1 ☑
Schema created	87 tables, 283 user indexes ☑
Oracle Free is the 23.x line under a new name	banner "26ai Free Release 23.26.2.0.0" ☑

Rehearsal Record — a rich 21-era realm survives the six-hop ladder to 26.7.1 at 100k

2026-08-26T00:00:00+00:00

Why this run existed

Run E proved the schema migrates across five majors but ran at small scale with a partially-applied seed, so it explicitly could not say whether features survive. This run fixes both gaps: a genuinely rich 21-era realm, at 100k users, walked the full six-hop ladder, with a feature inventory captured before and after.

Environment

Field	Value
Keycloak from → to	21.1.2 → 26.7.1, stepped 22.0.5 → 23.0.7 → 24.0.5 → 25.0.6
Distribution	quay.io/keycloak/keycloak (official image), `start` w/ external DB
Database	PostgreSQL 16, single node, container, stock tuning
Dataset scale	100,006 users (100,000 `seed-sql` + 5 `partialImport` + 1 template), 1 realm, 10 clients, 11 client scopes, 8 realm roles, 1 group, 10 auth flows, 9 required actions, 2 IDPs, 6 key providers
Seeding method	`seed-realm.sh` (features) + `seed-sql.sh` (100k users, direct SQL) — record both
Fixture profile	rich 21-era: clients, client-scopes, groups-roles, auth-flows, required-actions, events, themes, localization, lifespans, password-policy, key-providers, idp
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU, 30 GB RAM
JVM heap	`-Xms1g -Xmx4g`

Result — the ladder

Hop	Elapsed	Outcome	Index audit
21.1.2 → 22.0.5	21s	READY	82/82
22.0.5 → 23.0.7	20s	READY	82/82
23.0.7 → 24.0.5	21s	READY	86/86
24.0.5 → 25.0.6	15s	READY	86/86
25.0.6 → 26.7.1	17s	READY	118/118

~94s wall-clock for the whole ladder, every hop clean. (Run E's six-hop ladder at small scale was 106s; the 100k run is no slower, consistent with the index-skip threshold keeping the migration flat.)

Result — feature survival

A diffable inventory (bin/feature-inventory.sh) captured at 21.1.2 and 26.7.1. Every custom artifact survived unchanged:

custom themes lab-login / lab-account / lab-email
10 clients incl. app-confidential, spa-public, saml-app, api-bearer-only
custom client scope lab-profile
group corp, roles lab-admin / lab-auditor / lab-composite / lab-readonly / lab-user
custom auth flows lab-browser and lab-registration
all configured required actions (CONFIGURE_TOTP, webauthn-register, …)
identity providers corp-oidc (OIDC) and corp-saml (SAML)
key providers lab-rsa-secondary, lab-hmac, plus the generated set
users: 100,006 before and after — nothing lost
credentials: both seeding paths log in post-ladder (HTTP 200) — sqluser1 (seed-sql shared hash) and labuser1 (partialImport)

The only differences are Keycloak's own built-ins changing across the majors, none of which touched the customer's artifacts:

Change	Direction	Notes
client scopes `AuthnContextClassRef`, `basic`	added	new built-ins
role/client-scope `service_account`	added	new built-in
required actions `delete_credential`, `idp_link`	added	new built-ins
component `declarative-user-profile`	added	new in 24+
built-in auth flow `http challenge`	removed	see below

Breakage observed

The built-in `http challenge` flow is gone by 26.7.1

Symptom: the http challenge|true|basic-flow flow present on 21.1.2 is absent on 26.7.1. Keycloak removed/absorbed it at some point in the ladder (not narrowed to a hop here — narrowed later: 22.0.0, see 2026-08-26-http-challenge-flow-removal-narrowed).
Impact here: none — the lab's custom flows do not reference it, and no customer artifact depended on it.
Would this hit a customer? Only if a custom flow references the built-in http challenge flow (e.g. a hardened browser flow that chains it in for SPNEGO or client-cert auth). That reference would go stale. Worth a targeted pre-flight check for customers with custom flows. Resolved: the 22.0.0 migrator leaves a referenced flow in place and logs an ERROR rather than deleting it — the reference goes stale loudly, not silently.

Verification

Claim	Primary source	Checked
Six-hop ladder reaches READY at 100k	per-hop `upgrade.sh` outcome, elapsed	☑
Index audit clean at every hop	82/82 → 86/86 → 118/118	☑
Every custom feature survives	`feature-inventory.sh` diff, 21.1.2 vs 26.7.1	☑
User count unchanged (100,006)	`/users/count` before and after	☑
Credentials functional post-ladder	password-grant login HTTP 200 for both seed paths	☑
Custom flows and IDPs still resolvable	API reads at 26.7.1	☑
The exact hop that removed `http challenge`	narrowed — 22.0.0 (the 21→22 hop). See `2026-08-26-http-challenge-flow-removal-narrowed`	☑
A custom flow that references `http challenge`	answered — `MigrateTo22_0_0` leaves it in place and logs an ERROR ("is in use … cannot be removed"); not silent data loss	☑
Realm export/import equivalence (vs in-place migration)	not compared — in-place ladder only	☐

Publishable extract

We built a Keycloak realm on 21.1.2 with the full surface a real install has — custom themes, ten clients, custom client scopes, groups, composite roles, two custom authentication flows, required actions, an OIDC and a SAML broker, and six key providers — then seeded it to 100,000 users and walked it up the six-hop ladder to 26.7.1, one major at a time. Every hop reached ready; the whole ladder took about 94 seconds; the index audit was clean at every step; and every custom artifact, all 100,006 users, and both credential paths survived unchanged. The only things that changed were Keycloak's own built-ins — new default scopes and required actions, and one built-in flow (http challenge) that Keycloak removed. If a custom flow references that built-in, check it; ours did not, and nothing else was lost.

Rehearsal Record — Run B at 100k: optimized image, migration timing and heap floor at scale

2026-08-26T00:00:00+00:00

Why this run existed

Previously untested: the stepped ladder and Run B each had only one scale. our lab protocol asks every path for two scales. This is the second scale for Run B, and it clears two ☐ rows in the original Run B record — the heap floor at scale, and the migration timing at scale.

Environment

Field	Value
Keycloak from → to	26.0.0 (stock image) → 26.7.1 (pre-built image)
Image	`kclab/keycloak-optimized:26.7.1` (same Dockerfile as Run B)
Start mode	`start --optimized --http-enabled=true`
Database	PostgreSQL 16, single node, container
Postgres tuning	stock
Dataset scale	100,000 users, `typical`, `baseline-26.0.0-typical-100k`
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g` for the migration; variable in the heap leg
Compose overrides	`docker-compose.yml:docker-compose.optimized.yml:docker-compose.optimized-noprov.yml`

Result

Migration timing at 100k

	1k (Run B)	100k (this run)
`upgrade.sh` elapsed	11s	11s
`migration_model`	26.7.1	26.7.1
Index audit	—	118/118 correct, 0 missing, 0 wrong shape

The migration is flat between 1k and 100k — same 11s. This is the same mechanism the stock-image runs found (index-skip-threshold.md, Run A): the expensive DDL is declined above 300,000 rows, so "time to ready" does not grow with realm size on this pair. The optimized image saves the augmentation step regardless of scale; it does not change the migration.

Heap floor at 100k

Same bisect boundary values as B2, but on the 100k database:

`-Xmx`	Result at 100k
96m	READY
64m	FAILED — `Terminating due to java.lang.OutOfMemoryError: Java heap space`

The floor is (64m, 96m] at 100k, unchanged from 1k. Combined with S5's finding that the stock floor is identical at 1k and 2M, this closes the "scale-dependence" question for the optimized floor too — the heap floor is a function of the build/augmentation step, not of dataset size.

Reproduced: the providers bind-mount trap (B3b-i)

The first attempt at this run failed with exit code 2 before any migration:

A provider JAR was updated since the last build, please rebuild for this to be fully utilized.

Cause: bin/build-spi.sh 25.0.6 (run earlier for the KC-SERVICES0047 probe) had left lab-event-listener-built-for-25.0.6.jar in fixtures/providers/, and the base compose bind-mounts that directory over the optimized image's own empty /opt/keycloak/providers. The runtime jar did not match the image's build. Removing the mount (docker-compose.optimized-noprov.yml) makes the same image start. This is a faithful re-occurrence of B3b-i, not a new finding — worth noting because it means any later run that builds a SPI must clear fixtures/providers/ before an optimized-image run, or add the noprov override.

Tooling observation (minor)

upgrade.sh reported from: 26.7.1 on the successful run, because the failed first attempt had already rewritten .env's KC_VERSION to 26.7.1. The actual migration was 26.0.0 → 26.7.1 (the migration log shows the full 26.1.0 → 26.7.0 realm-migration ladder). upgrade.sh's from is read from .env, not from the database's migration_model — a stale .env after a failed attempt makes the label lie. Not load-bearing for this run; noted for the tooling.

Verification

Claim	Primary source	Checked
Migration at 100k takes the same 11s as at 1k	`upgrade.sh` elapsed, both runs	☑
Index audit clean at 100k	`index-audit.sh` output: 118/118	☑
Heap floor `(64m, 96m]` holds at 100k	96m READY, 64m OOM at 100k	☑
The failure mode is the providers shadowing, not the migration	exit 2 before Liquibase; noprov fixes it	☑
Heap floor at 2M under `--optimized`	not tested — 100k only; S5's stock 2M point + this 100k point imply it, but it is not measured	☐
Migration timing at 2M under `--optimized`	not tested — 100k only	☐

Rehearsal Record — Run B: the pre-built image halves the heap floor and turns S7's silent failure loud

2026-08-26T00:00:00+00:00

Why this run existed

docker-compose.yml says, in a comment, why the lab does not use --optimized: start --optimized is valid, but when Keycloak detects a config change it re-runs the build step with the same argv and build rejects --optimized. The supported route is to run kc.sh build at image build time. Nobody had done it, which left the lab's single biggest divergence from a production deployment untested — and left S5's published heap floor carrying a caveat.

Environment

Field	Value
Keycloak from → to	26.0.0 (stock image) → 26.7.1 (pre-built image)
Image	`kclab/keycloak-optimized:26.7.1`, built by `fixtures/optimized/Dockerfile` — `kc.sh build` with `KC_DB=postgres`, health and metrics on; 772 MB against the stock 766 MB
Start mode	`start --optimized --http-enabled=true`
Database	PostgreSQL 16, single node, container
Postgres tuning	stock
Dataset scale	1,000 users, `typical`, `baseline-26.0.0-typical-1k`
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g` except in B2, where it is the variable
Lab change	`fixtures/optimized/Dockerfile`, `bin/build-optimized.sh`, `docker-compose.optimized.yml`, `docker-compose.optimized-noprov.yml`

Outcome

Migration completes under --optimized: 211 rows, model 26.7.1
Time to ready 17s → 11s at 1k users
Heap floor re-bisected: (64m, 96m]
Provider handling characterised in four configurations

Verification

Claim	Primary source	Checked
The optimized image runs the migration normally	211 changelog rows, `migration_model` 26.7.1	☑
Upgrade elapsed 17s → 11s	`upgrade.sh` output, same baseline both runs	☑
The saving is the build step, not the migration	Quarkus `started in` and Bootstrap identical to within 0.03s	☑
Heap floor is `(64m, 96m]` on the optimized image	96m READY, 64m FAILED with an OOM line	☑
S5's floor of `(128m, 160m]` was the build step	128m FAILED stock, READY optimized	☑
A runtime-only provider jar prevents startup	B3a, message quoted	☑
A providers bind mount shadows the image's own	B3b-i vs B3b-ii; `ls` inside the container empty vs populated	☑
The provider registers when built in and unmounted	`/admin/serverinfo` lists `lab-event-listener`	☑
Heap floor at 2M users under `--optimized`	not tested — B2 ran at 1k. S5 showed the stock floor is scale-independent; that is an assumption here, not a measurement	☐
Whether a broken provider built into an optimized image still fails lazily	not tested — only the working jar was built in	☐
`--optimized` across a version boundary (build at 26.0.0, start at 26.7.1)	not tested — the optimized image existed only for 26.7.1, so the 26.0.0 side of B1 ran stock	☐

Rehearsal Record — Run E: 21.1.2 → 26.7.1 goes direct, and the ladder is one changeset longer

2026-08-26T00:00:00+00:00

Summary

The five-major jump goes direct. 21.1.2 → 26.7.1 in a single hop reaches ready in 18 seconds, stamps migration_model 26.7.1, and issues tokens. No stepping required.

The six-hop ladder also works, every hop clean, and ends one changeset further along: 211 rows against the direct jump's 210. The extra one is 25.0.0-28265-index-cleanup, and the schema it leaves is equivalent.

Environment

Field	Value
Distribution / start mode	quay.io/keycloak/keycloak official images, `start` w/ external DB
Database	PostgreSQL 16, single node, container, purged between the two runs
Postgres tuning	stock
Dataset scale	small — realm `lab` seeded on 21.1.2 via `seed-realm.sh --profile typical`
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`
Source state	21.1.2, `DATABASECHANGELOG` 113 rows

Fixture caveat, stated because it bounds the result. seed-realm.sh targets current admin APIs and only partially applied against 21.1.2 — the realm ended with one user rather than the profile's full set. The migration path is exercised; a rich 21.1.2 realm is not. Treat this as "the schema migrates" and not "every 21.1.2 feature survives".

Outcome

Direct 21.1.2 → 26.7.1 reaches ready, model 26.7.1, tokens issue
All six ladder hops reach ready with no failures
Index audit clean at every hop and on the direct result
The only changelog difference identified by name

Verification

Claim	Primary source	Checked
21.1.2 → 26.7.1 works in one hop	E0 — READY, model 26.7.1, token 200	☑
The full ladder works, hop by hop	E1 — six hops, all READY, model advancing each time	☑
Direct ends at 210 changesets, the ladder at 211	`count(*)` from `DATABASECHANGELOG` on both	☑
The difference is `25.0.0-28265-index-cleanup`	`comm` of the two ordered changeset-id lists	☑
The direct result passes a full index audit	`bin/index-audit.sh` — 118/118 correct	☑
The two schemas are otherwise identical	not proven — the direct DB's table/column/index counts were captured, the ladder DB was gone before its own could be. A column-by-column diff was never run	☐
A rich 21.1.2 realm survives	not tested — see the fixture caveat	☐
A custom provider survives the span at runtime	not tested here; S7 covered the endpoints only	☐
The same at 100k+ users	not tested — README asks for two scales and this is one	☐

Rehearsal Record — S14 revisited: at 2M the disk floor is 150 MB, not 8 MB, and scaling up no longer saves you

2026-08-26T00:00:00+00:00

Summary

S14 found the migration dies at CREATE INDEX IDX_USER_CREATED_TIMESTAMP when disk runs out, needing between 4 and 8 MB free on a 224 MB / 100,002-user database — and concluded:

And the failure only happens on the smaller realm. … At 2M it is declined, and needs none. Scaling the realm up removes this failure mode.

That was true because Keycloak declines the build above 300,000 rows. Run A showed a stale reltuples makes it build anyway, at any true size. So the scenario is reachable at 2M, and there it is far worse:

At 2,000,003 users the floor is between 125 MB and 150 MB — roughly 20× the 100k figure. Scaling up does not remove the failure mode. It removes the trigger, and multiplies the damage if anything pulls the trigger.

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Database	PostgreSQL 16, single node, container
Postgres tuning	stock — `maintenance_work_mem` 64MB, `max_parallel_maintenance_workers` 2
Storage	6 GB ext4 loopback at `/mnt/tinypg`, bind-mounted as `pgdata` (`docker-compose.smalldisk.yml`)
Dataset scale	2,000,003 users, database 3,026 MB
Seeding method	`seed-sql`, restored from `baseline-26.0.0-sqlseed-2m`
Adverse scenario	S14 — free space at the moment of migration is the variable
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`

How the build was forced. With honest statistics at 2M Keycloak declines the index and never touches the disk, so pg_class.reltuples was set to 1000 with autovacuum disabled on USER_ENTITY — the mechanism from 2026-08-26-index-precondition-anatomy. That is a deliberate fixture, and it stands in for the customer state measured in 2026-08-26-stale-statistics-reachability: autovacuum off on a large table, plus a loader that does not ANALYZE.

Method. A clean copy of the restored data directory is parked once and restored before each level, so every leg starts byte-identical without paying 90 seconds for a fresh pg_restore. Free space is then set exactly with a ballast file.

Results

Free space at migration	Outcome	Index built	`DATABASECHANGELOG`	`migration_model`
200 MB	READY	yes	211	26.7.1
175 MB	READY	yes	211	26.7.1
150 MB	READY	yes	211	26.7.1
125 MB	FAILED	no	187	26.0.0
100 MB	FAILED	no	187	26.0.0
50 MB	FAILED	no	187	26.0.0
25 MB	FAILED	no	187	26.0.0
12 MB	FAILED	no	187	26.0.0

Floor: (125 MB, 150 MB]. Against S14's (4 MB, 8 MB] at 100k.

Every failure is identical and stops at 187 of 211 — the same point as S14's own failure and as the index-name collision in 2026-08-26-index-precondition-anatomy, because all three die on the same changeset.

Breakage observed

Disk exhaustion at 2M needs 20× the headroom S14 measured

Symptom: upgrade fails on CREATE INDEX IDX_USER_CREATED_TIMESTAMP; changelog 187/211, migration_model 26.0.0, server does not start.
Evidence: the sweep above; Postgres error quoted.
Cause: confirmed — parallel external sort spilling to pgsql_tmp.
Fix: free space, restart. (Not re-tested here; S14 showed the half-applied changelog resumes and completes.)
Would this hit a customer? Scale-dependent and conditional on the index build being attempted at all — which needs a stale row estimate. Narrow, but the consequence is a failed maintenance window on the largest installations.

Verification

Claim	Primary source	Checked
Floor at 2,000,003 users is `(125 MB, 150 MB]`	eight levels, 150 READY / 125 FAILED	☑
Failures are identical and stop at 187/211	`count(*)` and `migration_model` at every failing level	☑
The exhausted space is a parallel sort's temp fileset	Postgres error naming `pgsql_tmp70.0.fileset`, two backends	☑
Finished index is 14 MB; net build cost 14 MB	`pg_relation_size` and `df` before/after an unconstrained build	☑
Transient requirement is ~10× the finished index	14 MB index against a >125 MB floor	☑
S14's "scaling up removes the failure mode" was conditional	it holds only while the build is declined; forcing it re-arms the scenario	☑
Recovery by freeing space and restarting	not re-tested at 2M — S14 showed it at 100k	☐
The floor with `maintenance_work_mem` raised	not tested — stock 64MB throughout	☐
The floor with parallel maintenance workers disabled	not tested	☐
Whether the floor scales linearly between 100k and 2M	not tested — two points, no intermediate	☐

Rehearsal Record — S3 revisited: `statement_timeout` does half-migrate, and autovacuum is why

2026-08-26T00:00:00+00:00

Summary

S3 measured correctly and concluded wrongly.

Its measurement stands: no statement in this migration takes meaningful time to execute. We re-measured the statement that matters and got 0.177 ms.

Its conclusion does not. S3 reasoned that because the longest statement was 2.142 ms, no statement_timeout value could land between "too short to start" and "long enough to finish". That inference assumes a statement's duration is its execution time. In Postgres it is not: a statement's duration includes the time it spends waiting for its lock, and statement_timeout is charged for that wait.

At 2,000,003 users, statement_timeout values of 500 ms, 200 ms and 50 ms all leave the database half-migrated — DATABASECHANGELOG at 157 of 211, migration_model still 26.0.0, and the server refusing to start. 1 s and 10 s complete. The band S3 called empty is (500 ms, 1 s].

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB
Database	PostgreSQL 16, single node, container
Postgres tuning	stock except where a leg names the variable. `deadlock_timeout` 1s (stock), `autovacuum` on (stock), `statement_timeout` set per leg on the `keycloak` role
Adverse scenario	S3
Dataset scale	2,000,003 users · `CREDENTIAL` 2,000,002 rows / 105,296 pages
Seeding method	`seed-sql`, restored from `baseline-26.0.0-sqlseed-2m`
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`
Instrumentation	`log_min_duration_statement=0`, `log_lock_waits=on` on the diagnostic legs

Breakage observed

A `statement_timeout` below one second half-migrates the database

Symptom: canceling statement due to statement timeout on changeset 26.2.0-26106; server exits; DATABASECHANGELOG 157/211, migration_model 26.0.0.
Evidence: the sweep above; Postgres log quoted in full.
Cause: confirmed — autovacuum holds a conflicting lock on CREDENTIAL and Postgres waits deadlock_timeout before evicting it; the wait is charged to statement_timeout.
Fix or workaround: any one of — raise statement_timeout above deadlock_timeout for the migration role; lower deadlock_timeout; or make sure autovacuum is not mid-scan of the large tables when the window opens (VACUUM them beforehand, or set autovacuum_enabled=false for the window).
Would this hit a customer? A statement_timeout on the database role is ordinary hardening, and this is the fourth route the lab has found to a half-applied changelog. It is likeliest immediately after a restore or bulk load — which is the state of every staging rehearsal, and of a production database that has just been restored for a DR test.

Verification

Claim	Primary source	Checked
500 ms / 200 ms / 50 ms half-migrate at 2M	three runs, `DATABASECHANGELOG` and `migration_model` queried	☑
1 s and 10 s complete	two runs, 211 rows, model 26.7.1	☑
The failing statement is 26.2.0-26106, not the index build	Liquibase exception quoted	☑
That statement executes in well under a millisecond	F1 0.177 ms, F2 0.156 ms, F3 0.210 ms, from Postgres' log	☑
The duration is a lock wait, not execution	`still waiting for AccessExclusiveLock` / `acquired … after 100.161 ms`	☑
The blocker is an autovacuum worker on `CREDENTIAL`	`canceling autovacuum task … automatic vacuum of table "keycloak.public.credential"`	☑
The wait length is `deadlock_timeout`	E2 — 100 ms setting produced a 100.485 ms statement	☑
Turning autovacuum off makes 500 ms survivable	F3 — READY	☑
S3's own measurement was of an uncontended run	inferred from F1 reproducing 2.142 ms's order of magnitude; the original run cannot be re-inspected	☐
The autovacuum worker predicts the outcome	ten repetitions, 10/10 agreement, 7 failures	☑
Failure rate immediately after a restore	70% (7/10) at `statement_timeout=500ms`	☑
Whether other version pairs have an equivalent DDL on a large table	not tested	☐
Whether `lock_timeout` behaves differently from `statement_timeout` here	not tested	☐

Rehearsal Record — S4 on MySQL: the DML-only failure holds, but the fix is privileges, not ownership

2026-08-26T00:00:00+00:00

Summary

This is the first run against an engine other than Postgres, and it closes the ☐ "holds on MySQL/MariaDB/Oracle" row in S4. Two findings, one expected and one that changes what we tell customers:

A DML-only account fails the migration on MySQL, same as Postgres — so the failure mode generalises. It fails even earlier on MySQL than on Postgres: before any Keycloak changeset runs, at Liquibase's own changelog-table setup (ALTER TABLE DATABASECHANGELOG ADD PRIMARY KEY (ID, AUTHOR, FILENAME)), not at the first Keycloak DDL changeset.
The fix is different. On Postgres the S4 record found that granting DDL is not enough — DROP INDEX requires ownership, which no GRANT confers. MySQL has no ownership concept: granting the DDL privileges (CREATE, ALTER, DROP, INDEX, REFERENCES, CREATE VIEW, SHOW VIEW on keycloak.*) makes the same migration succeed cleanly. The pre-flight question is therefore engine-specific: on Postgres, "does the account own the objects"; on MySQL, "does the account hold the DDL privileges".

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB
Database	MySQL 8.4 (LTS), single node, container `mysql:8.4`
DB flags	`sql_generate_invisible_primary_key=OFF` (8.4 enables it by default and it breaks Keycloak migrations with error 1068), `utf8mb4` / `utf8mb4_unicode_ci`
Adverse scenario	S4 — the account Keycloak connects as is the variable
Dataset scale	1,000 users, 2 realms
Seeding method	`partialImport` (`bin/seed-realm.sh --profile typical`)
Topology	single container
Host	laptop (Docker 29)
JVM heap	`-Xms1g -Xmx4g`
Elapsed clock	run A failed at 13s; run B ready in ~15s (Liquibase + realm migrators ~5s)

Procedure

The whole run needed a MySQL toolchain first. Then:

# run A — DML only
CREATE USER 'kc_dml'@'%' IDENTIFIED BY 'kc_dml';
GRANT SELECT, INSERT, UPDATE, DELETE ON keycloak.* TO 'kc_dml'@'%';
# .env: KC_DB=mysql, COMPOSE_FILE=docker-compose.yml:docker-compose.mysql.yml,
#        KC_DB_USERNAME=kc_dml, KC_DB_PASSWORD=kc_dml
./bin/upgrade.sh 26.7.1          # FAILED, 13s

# run B — DML + DDL (no ownership concept to add)
GRANT CREATE, ALTER, DROP, INDEX, REFERENCES, CREATE VIEW, SHOW VIEW
      ON keycloak.* TO 'kc_dml'@'%';
./bin/restore.sh baseline-26.0.0-mysql-typical-1k
./bin/upgrade.sh 26.7.1          # READY

Results

Run	Account's rights	Outcome	Schema after	Users
A	DML only (`SELECT, INSERT, UPDATE, DELETE`)	exit 1, refused to start	model 26.0.0, changelog untouched	1,000
B	DML + DDL (`CREATE, ALTER, DROP, INDEX, REFERENCES, CREATE VIEW, SHOW VIEW`)	READY, clean	model 26.7.1, index audit 121/121	1,000

Run A failure — earlier than Postgres, and not a Keycloak changeset

ERROR: liquibase.exception.ChangeLogParseException:
  liquibase.exception.SetupException: liquibase.exception.DatabaseException:
  ALTER command denied to user 'kc_dml'@'172.19.0.3' for table 'DATABASECHANGELOG'
  [Failed SQL: (1142) ALTER TABLE keycloak.DATABASECHANGELOG ADD PRIMARY KEY (ID, AUTHOR, FILENAME)]

Contrast with Postgres S4, where the first failure was a real Keycloak changeset (20.0.0-12964-supported-dbs-edb-migration, DROP INDEX IDX_GROUP_ATT_BY_NAME_VALUE → "must be owner of index"). On MySQL, Liquibase 26.7.1 wants a primary key on its own DATABASECHANGELOG table that the 26.0.0-era Liquibase did not create, and it tries to add it before touching any Keycloak changeset. A DML-only account is therefore rejected at the door, not mid-migration.

Run B — DDL privileges are sufficient

Granting the DDL set (still no ownership concept — MySQL has none) let the same migration run to completion: Updating database → realm migrators 26.1.0 → 26.7.0 (~5s), migration_model 26.7.1, index audit 121 correct / 0 missing / 0 wrong shape, all 1,000 users intact.

Verification

Claim	Evidence
DML-only account fails on MySQL	run A log above, ☑ reproduced once
DDL-privileged account succeeds on MySQL	run B `migration_model` 26.7.1 ☑
Index audit clean after the MySQL migration	121 correct / 0 missing / 0 wrong shape ☑
Realm survives the MySQL upgrade	`users/count` = 1000 ☑
Toolchain round-trips (snapshot/restore) on MySQL	`restore.sh` reported "schema 26.0.0, no re-migration", users 1000 ☑

Rehearsal Record — how a stale `reltuples` actually happens, and how it mostly does not

2026-08-26T00:00:00+00:00

Why this run existed

Run A showed that Keycloak decides whether to build an index from pg_class.reltuples, and forced the interesting case by writing to the catalogue by hand. That proves a mechanism. It does not prove anyone hits it, and a finding published on the strength of a hand-edited catalogue row deserves the scepticism it would get.

This run is the attempt to reach the same state without touching pg_class. Two of the three attempts failed, and saying so is the point.

Environment

baseline-26.0.0-typical-100k restored (100,002 users, partialImport provenance), then extended with bin/seed-sql.sh. Postgres 16 stock — in particular autovacuum on, autovacuum_naptime 60s. Hetzner CCX33, single container, JVM -Xms1g -Xmx4g, official image, start with an external DB.

Outcome

Both negative attempts reproduced the correct behaviour: index skipped, 211 rows, model 26.7.1
Attempt 3 reached ready with the index built and no warning
bin/seed-sql.sh --no-analyze added, so the field's loader can be modelled

Verification

Claim	Primary source	Checked
`seed-sql.sh` analyzes after loading	source line 141, and `last_analyze` set in the run	☑
With stock autovacuum, a 73s load is corrected before the upgrade	`last_autoanalyze` 01:33:30, mid-load; `reltuples` accurate after	☑
With `autovacuum_enabled=false` and no ANALYZE, the estimate stays stale	`reltuples` 100,002 against 1,100,003 true rows	☑
Keycloak then builds the index above the threshold	index present, `EXECUTED`, 0 warnings, 1,100,003 rows	☑
How long stock autovacuum takes to correct a load of this size	not isolated — it fired during the load, so the correction latency was never measured	☐
Whether a load large enough to outrun autovacuum exists	not tested — 1M rows did not	☐
Whether throttled (rather than disabled) autovacuum is enough	not tested — `autovacuum_vacuum_cost_delay` untouched	☐

Rehearsal Record — Backup/rollback tooling defect, and verified rollback at 2M

2026-08-25T00:00:00+00:00

Verification — full loop at 2,000,003 users

Step	Result
Restore baseline to 26.0.0	schema 26.0.0, `no re-migration`
`upgrade.sh 26.7.1`	auto-backup `pre-upgrade-26.0.0-to-26.7.1-20260825T182154Z` (65 MB), READY
Canary written post-upgrade	users 2,000,003 → 2,000,004
`restore.sh <backup>` — no `--version`, inference only
→ running image	quay.io/keycloak/keycloak:26.0.0
→ `migration_model`	26.0.0
→ users	2,000,003
→ canary	absent — correctly lost

Timings

	1,000 users	2,000,003 users
Backup (`pg_dump -Fc`)	<1s	~6s (65 MB from a 3.5 GB database)
Upgrade overhead from backup	negligible	~8s on a 16s upgrade
Full rollback to serving	15.9s	70s

Rollback time scales with data; migration time does not. At 2M the migration is 3.84s and the rollback is 70s — the rollback is ~18× the migration it undoes. That inverts the usual assumption and is the number that actually matters when deciding whether to roll back mid-window.

Publishable extract

We added an automatic backup to our upgrade tooling, and testing it revealed our rollback had never worked.

Restoring a pre-upgrade snapshot put the old schema back — and then the running container, still the new Keycloak, immediately re-ran the migration. The restore looked successful: every row was there. But the schema was back on the new version and every write since the snapshot was gone. The worst of both.

The cause is one word. docker compose start reuses the existing container; only --force-recreate picks up the image you just rolled back to. Any rollback procedure that restores a database without also pinning the binary version has this bug, whatever tooling it is written in.

Two things worth taking from it. First, verify a rollback by reading the schema version back out of the database afterwards, not by confirming your data is present — the data is present in the broken case too. Our restore now compares migration_model before and after restart and fails loudly if the server re-migrated.

Second, the timings. On a 2,000,003-user realm (Postgres 16.15, Hetzner CCX33 — 8 vCPU dedicated, 32 GB, local NVMe), the upgrade's schema migration takes 3.8 seconds. The rollback takes 70 seconds — about eighteen times longer than the thing it undoes. Migration duration is flat with realm size; restore duration is not. If you are sizing a maintenance window, the rollback is the expensive half, and it is the half nobody measures.

Rehearsal Record — S7: the upgrade succeeds, the extension stops working, and nothing fails

2026-08-25T00:00:00+00:00

Summary

Two findings, and the second is the sellable one.

1. "Has a custom SPI" is not a risk by itself. A provider that uses only stable SPI, compiled against 21.1.2, loads and works correctly on 26.7.1 — five majors and 619 removed public signatures later — with no source change at all. The same source compiles unchanged against 21.1.2, 25.0.6, 26.0.0 and 26.7.1.

2. A provider that touches a removed internal API does not fail the upgrade. It fails silently, afterwards. The server starts clean in 15 seconds. The provider loads, prints its init line, and appears in /admin/serverinfo. Every check an operator runs during the window passes. Then the first event arrives and the listener throws NoSuchMethodError — logged, dropped, and invisible to the request that caused it, which returns 201.

For an event listener forwarding to a SIEM, that is audit forwarding that stops without an outage.

Environment

Field	Value
Keycloak path	25.0.6 → 26.0.0 → 26.7.1 (each hop a real migration, jar mounted throughout)
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB, auto-build on start
Database	PostgreSQL 16.15, single node, container
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0
Adverse scenario	S7
Dataset scale	1,002 users, 2 realms
Seeding method	`partialImport` (`bin/seed-realm.sh --profile typical`)
Fixture profile	`typical`, plus custom provider jars in `/opt/keycloak/providers`
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`
Lab change	`fixtures/spi-brittle/` added; `bin/build-spi.sh` gained `--module` and now reads the artifactId from the pom

Results

Compile-time

Source	21.1.2	25.0.6	26.0.0	26.7.1
tame	builds	builds	builds	builds
brittle	—	builds	—	FAILS

[ERROR] LabBrittleEventListener.java:[31,44] no suitable method found for
        runJobInTransactionWithResult(org.keycloak.models.KeycloakSessionFactory,
        org.keycloak.models.KeycloakContext,(s)->event[...]mId())

The API diff predicted the removal; the compiler confirms it independently.

Run-time

Jar built against	Running on	Loads	In `serverinfo`	Events delivered
tame / 25.0.6	25.0.6	☑	☑	☑ user + admin
tame / 25.0.6	26.0.0	☑	☑	☑ user + admin
tame / 25.0.6	26.7.1	☑	☑	☑ user + admin
tame / 21.1.2	26.7.1	☑	☑	☑ user + admin
tame / 26.7.1	26.7.1	☑	☑	☑
brittle / 25.0.6	26.7.1	☑	☑	☒ every event throws

Migration timings with the jar mounted throughout: 25.0.6 → 26.0.0 in 13s, 26.0.0 → 26.7.1 in 16s. The provider did not slow, block or fail any hop.

What the brittle failure actually looks like

Startup — indistinguishable from success:

[lab-brittle] factory init -- provider loaded

/admin/serverinfo lists lab-brittle-listener among the event listeners. wait-ready.sh returns 0 in 15 seconds. Container running.

First event — and only then:

ERROR [org.keycloak.services] KC-SERVICES0085: Failed to send type to
  lab.LabBrittleEventListener@6012fda7: java.lang.NoSuchMethodError:
  'java.lang.Object org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(
     org.keycloak.models.KeycloakSessionFactory, org.keycloak.models.KeycloakContext,
     org.keycloak.models.KeycloakSessionTaskWithResult)'

And the blast radius is nil, which is the problem:

Check after the failure	Result
Admin request that raised the event	HTTP 201
Group actually created in the database	yes
Token endpoint	HTTP 200
Container	running
Keycloak's own `admin_event_entity`	1,015 rows — still recording
Keycloak's own `event_entity`	8 rows — still recording
The tame listener on the same server	still delivering

Keycloak's internal audit tables stay complete. Anyone verifying the audit trail by querying Keycloak sees nothing wrong. It is only the delivery to the custom listener — the path to whatever is downstream — that has stopped.

A note on `KC-SERVICES0047`

On both 26.0.0 and 26.7.1, every provider load emitted:

WARN [org.keycloak.services] KC-SERVICES0047: lab-event-listener
  (lab.LabEventListenerProviderFactory) is implementing the internal SPI
  eventsListener. This SPI is internal and may change without notice

eventsListener — the most commonly extended SPI in the wild — is classified internal. Whether this warning also appears on 25.0.6 was not checked at the time and the containers were gone; re-run on 2026-08-26 (2026-08-26-kc-services0047-not-new-in-26) shows it is present on 25.0.6 — the warning is not new in 26.

An unrelated observation, deliberately not attributed

A failed password grant against the lab realm returned 401 on 25.0.6 and 26.0.0, and 400 on 26.7.1. We tested whether the brittle listener caused it: with the listener registered, removed, and registered again, the response was 400 every time. It is not the listener. Narrowed on 2026-08-26 (2026-08-26-invalid-grant-401-to-400-narrowed): it is a version change that landed in 26.6.0 — 26.0.0 through 26.5.7 return 401, 26.6.0 onward returns 400.

Verification

Claim	Primary source	Checked
A 21.1.2-built provider works on 26.7.1	built, mounted, both callbacks fired, registered in serverinfo	☑
The tame source compiles unchanged on all four versions	four successful `build-spi.sh` runs	☑
The removed signature is genuinely removed	`spi-api-diff.sh 25.0.6 26.0.0` and an independent compile failure on 26.7.1	☑
The brittle provider loads and registers on 26.7.1	init line in log, present in `/admin/serverinfo`	☑
It throws on every event	`KC-SERVICES0085` + `NoSuchMethodError`, both user and admin events	☑
The failure does not affect the request or the server	201 / 200 / container running / group created	☑
Keycloak's own event tables stay populated	`admin_event_entity` 1,015, `event_entity` 8, read from Postgres	☑
`KC-SERVICES0047` is new in 26	no — present on 25.0.6 (`2026-08-26-kc-services0047-not-new-in-26`)	☑
The 401 → 400 change is version-caused	yes — landed in 26.6.0 (`2026-08-26-invalid-grant-401-to-400-narrowed`)	☑
Behaviour of provider types other than event listeners	not tested — authenticators, user storage and mappers have different lifecycles and may fail at load rather than at use	☐
Behaviour when the removed API is called from `init()`	not tested — would likely fail at startup, which is the safe case	☐

Publishable extract

We upgraded Keycloak across five major versions with a custom extension installed. The upgrade succeeded. The extension stopped working. Nothing failed.

We built two event listeners. One uses only stable SPI. The other calls KeycloakModelUtils.runJobInTransactionWithResult(...) — a helper any listener wanting its own transaction would reach for, and one that our API diff shows present in 25.0.6 and gone in 26.0.0.

The stable one is the good news, and it is worth saying plainly: compiled against 21.1.2, it runs correctly on 26.7.1. Five majors, 619 removed public signatures in between, and the same source compiles unchanged against every version we tried. "The customer has a custom SPI" is not, on its own, a reason to expect trouble.

The other one is why you test. On Keycloak 26.7.1, carrying the jar built for 25.0.6:

the server started in 15 seconds;

the provider loaded and printed its startup line;

it appeared in /admin/serverinfo alongside the built-in listeners;

the readiness check passed.

Every check an operator runs during a maintenance window passed. Then the first event arrived:
ERROR KC-SERVICES0085: Failed to send type to lab.LabBrittleEventListener:
  java.lang.NoSuchMethodError: KeycloakModelUtils.runJobInTransactionWithResult(...)
The admin request that triggered it returned 201. The group was created. The token endpoint returned 200. The container stayed up. And Keycloak's own admin_event_entity table kept recording normally — 1,015 rows — so anyone auditing Keycloak's audit trail sees a complete one.

If that listener was shipping your authentication events to a SIEM, the events stopped and your dashboard says everything is fine.

The check that catches this is not a smoke test. It is: fire one real event after the upgrade and grep the log for KC-SERVICES0085.

Rehearsal Record — 25.0.6 → 26.0.0

2026-08-25T00:00:00+00:00

Environment

Field	Value
Keycloak from → to	25.0.6 → 26.0.0
Stepping path (if multi-hop)	none — direct
Distribution	quay.io/keycloak/keycloak (official image), 680MB → 691MB
Start mode	`start --http-enabled=true` w/ external DB
Database	PostgreSQL 16.15 (Debian pgdg13), single node, container
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0. Added retroactively 2026-08-25 to close a gap in our own records: read from the same unmodified `postgres:16` image on the same box. The compose `db` service sets no tuning, so stock is guaranteed by the file, not remembered.
Dataset scale	1000 users · 2 realms (master + lab) · 10 clients · 1 group · 8 realm roles
Seeding method	`partialImport`, batch size 2000 (one batch), 25.2s wall clock
Fixture profile	`typical` — auth-flows, password-policy, lifespans, clients, client-scopes, themes, groups-roles, events, required-actions
Topology	single container, single replica
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30GB usable RAM, 226GB local NVMe, Debian 13, kernel 6.12.101+deb13-cloud-amd64
JVM heap	`-Xms1g -Xmx4g`
Elapsed clock	14s total, container stop → first token issued

Procedure

./bin/lab-up.sh 25.0.6
./bin/seed-realm.sh --profile typical --users 1000
./bin/snapshot.sh baseline-25.0.6-typical-1k
./bin/upgrade.sh 26.0.0

Timings

Phase	Duration	Notes
Container start → DB connect	~3s	from stop to config load at 15:31:36
Liquibase / schema migration	~0.8s	15:31:39,198 "Updating database" → 15:31:39,992 "Migrating older model to 26.0.0"; runs on `(main)`, i.e. before the server reports ready
Cache/realm warm	not separately measured	unknown
First successful token issue	within 14s of stop	`wait-ready.sh` poll granularity is 3s, so this bounds it rather than measures it
Total to ready	14s

At this scale the migration is not the cost — process start is. The 14s is dominated by JVM/Quarkus startup, not by schema work. This number's only use is as the 1k anchor for a scale comparison; it says nothing about a real realm.

Outcome

Reached ready
Realm config intact — users 1000, clients 10, roles 8, auth flows 9, all unchanged across the upgrade
Login flow works — labuser7 issued a token from the lab realm post-upgrade
Custom SPI loaded — none in fixture, not exercised
Themes render — theme names set, theme files not mounted; not exercised
Federation/brokering intact — not in typical profile

Breakage observed

Hostname v1 option rejected at ERROR level, but startup continues

Symptom: ERROR [org.keycloak.quarkus.runtime.configuration.mappers.PropertyMappers] (main) Hostname v1 options [hostname-strict-https] are still in use, please review your configuration
Evidence: container log 2026-08-25 15:31:36,186. Also WARN ... The following run time options were found, but will be ignored during build time: kc.spi-hostname-v2-hostname-strict
Cause: confirmed — KC_HOSTNAME_STRICT_HTTPS is a hostname-v1 option; Keycloak 26 moved to hostname v2. The lab's compose file sets it for cross-version compatibility.
Fix or workaround: drop KC_HOSTNAME_STRICT_HTTPS for 26+. It is a no-op there.
Would this hit a customer? Yes, and it is config-dependent, not scale-dependent. Anyone carrying a 25-era hostname config into 26 hits this. It logs at ERROR, which will trip log-based alerting and read as a failed upgrade to an operator watching the console — while the server is in fact healthy. That gap between "looks failed" and "is fine" is the operationally expensive part.

Two JDBC resource leak warnings during migration

Symptom: WARN [io.agroal.pool] Datasource '<default>': JDBC resources leaked: 3 ResultSet(s) and 16 ResultSet(s)
Evidence: log 15:31:39,738 and 15:31:40,957, both during the Liquibase phase
Cause: suspected — Liquibase/Agroal interaction during schema update. Not investigated.
Fix or workaround: none applied; no observed effect.
Would this hit a customer? Almost certainly universal on this path, and almost certainly cosmetic. Worth knowing only so it is not mistaken for the cause of a real failure.

Rollback

Exercised this run? no
Method: snapshot taken (baseline-25.0.6-typical-1k, 500K) but not restored
Time to restore: unknown
Data loss / divergence: unknown
Verdict: not yet established for this path. The 3-clean-runs gate requires at least one rollback; this run does not satisfy it.

Verification

Claim	Primary source	Checked
`KC_HOSTNAME_STRICT_HTTPS` removed / v1 in KC26	Keycloak 26 upgrade guide — NOT YET CHECKED	☐
Admin env var rename `KEYCLOAK_ADMIN` → `KC_BOOTSTRAP_ADMIN_`	not exercised — compose sets both, so this run cannot confirm it	☐

This record is not publishable until the two rows above are filled. The compose file deliberately sets both admin env var schemes, which means this run says nothing about the rename that the version matrix lists as the headline breaking change for this path. A run that isolates it is still needed.

Publishable extract

Withheld pending the Verification table above.

Rehearsal Record — 26.0.0 → 26.7.1

2026-08-25T00:00:00+00:00

Environment

Identical to 2026-08-25-25.0.6-to-26.0.0-1 (same box, same fixture, same container) except:

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution	quay.io/keycloak/keycloak 691MB → 766MB
Dataset scale	unchanged: 1000 users, carried through the previous upgrade
Prior state	realm had already been migrated 25.0.6 → 26.0.0 in the run above

Caveat on comparability: this realm reached 26.0.0 by migration, not by a clean 26.0.0 install. A realm that was born at 26.0.0 may migrate differently. Stated because our fixture-comparability rule makes differently-shaped fixtures non-aggregatable, and this is a shape difference.

Procedure

./bin/snapshot.sh baseline-26.0.0-typical-1k
./bin/upgrade.sh 26.7.1

Timings

Phase	Duration	Notes
Container start → server "started"	3.029s	Quarkus self-report, 15:32:28,510
Liquibase / schema migration	~3.3s	15:32:29,111 "Updating database" → 15:32:32,414 reaching 26.7.0
Model migration chain	within the above	stepped internally through 26.3.0 → 26.4.0 → 26.4.3 → 26.6.1 → 26.6.2 → 26.7.0, each migrating both `lab` and `master`
Total to ready	not captured	operator error — the elapsed line was filtered out of the captured output. Re-run required for a clean total.

Outcome

Reached ready
Realm config intact — 1000 users
Login flow works — labuser7 token issued
Custom SPI / themes / federation — not in fixture

Breakage observed

Schema migration ran AFTER the server reported "started"

Symptom: Quarkus logged started in 3.029s. Listening on: http://0.0.0.0:8080 at 15:32:28,510 — then Updating database began at 15:32:29,111 on executor-thread-1, finishing ~15:32:32,4. On the 25.0.6 → 26.0.0 run the same work ran on (main) before ready.
Evidence: container log, both runs, timestamps above.
Cause: suspected, NOT confirmed. Two candidate explanations and this run cannot distinguish them: (a) 26.7.x defers schema migration out of the startup path, or (b) migration is lazily triggered by the first inbound request, and the trigger here was wait-ready.sh polling the token endpoint. The executor-thread-1 context is consistent with (b).
Fix or workaround: none — behaviour, not a fault.
Would this hit a customer? Potentially serious if (a) holds. A server that reports started and opens its port while the schema is still migrating will pass a naive TCP or /health/started check, so a load balancer or Kubernetes readiness probe could route production traffic mid-migration. That is precisely the Infinispan/rolling-update failure class the ledger already lists.
RESOLVED 2026-08-25 by 2026-08-25-26.0.0-to-26.7.1-2: explanation (a) holds — migration runs unprompted, confirmed from Postgres with the server left untouched for 120s. But the consequence stated here was wrong: /health/ready correctly reports 503 throughout, so a readiness probe is safe. The exposure is /health/started and TCP port checks. See that record.
~~NEXT ACTION — this is the highest-value follow-up from today.~~ Re-run with no client polling at all, watching the log passively, to establish which explanation holds. It is cheap, and if (a) is true it is publishable ledger content that nobody else has measured.

Hostname v1 warning changed emission path between 26.0.0 and 26.7.1

Symptom: same message, different form. 26.0.0: ERROR [org.keycloak.quarkus.runtime.configuration.mappers.PropertyMappers] (main) Hostname v1 options [hostname-strict-https] are still in use. 26.7.1: WARNING: Hostname v1 options [hostname-strict-https] are still in use — no timestamp, no logger name, i.e. emitted by the CLI layer rather than the logging subsystem.
Evidence: logs from both runs, quoted above.
Cause: suspected — the message moved from the logger to CLI-level output between these versions. Not confirmed against release notes.
Would this hit a customer? Only as an alerting artifact: anyone who wrote a log-scraping rule against the 26.0.0 ERROR form will silently stop matching on 26.7.x. Minor, but it is exactly the kind of thing that erodes trust in an alerting pipeline.
Additionally on 26.7.1: WARNING: With HTTPS not enabled, proxy-headers unset, and hostname-strict=false, the server is running in an insecure context. Expected for this lab config; would be a real finding on a customer system.

Rollback

Exercised this run? no
Method: snapshot baseline-26.0.0-typical-1k exists, not restored
Verdict: not established.

Verification

Claim	Primary source	Checked
26.7.x defers schema migration out of the startup path	UNRESOLVED — see NEXT ACTION above	☐
Hostname v1 message moved from logger to CLI output	Keycloak release notes 26.1–26.7 — not checked	☐

Publishable extract

Withheld — the headline observation is explicitly unconfirmed.

Rehearsal Record — 26.0.0 → 26.7.1 at 100k

2026-08-25T00:00:00+00:00

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution	quay.io/keycloak/keycloak official image
Start mode	`start --http-enabled=true` w/ external DB
Database	PostgreSQL 16.15, single node, container
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0
Dataset scale	100,002 users · 2 realms · 10 clients · 8 roles · 100,001 credentials · 200,001 user attributes · 100,004 admin events
DB size	264 MB before, 265 MB after
Seeding method	`partialImport`, batch 2000 — 5,235s (87 min) for 100k, i.e. 19.1 users/s
Fixture profile	`typical`
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`
Adverse scenario	none

Timings — and the finding

Phase	1k (`…-26.0.0-to-26.7.1-2`)	100k (this run)
Quarkus `started`	11.4s	10.5s
Liquibase migration	12.2s → 16.2s = 3.97s	13.0s → 16.8s = 3.79s
Total to ready	17.0s	15.4s

A 100× increase in users produced no increase in migration time. The 100k run was marginally faster, which is run-to-run noise. This is not a small difference from expectation — it is the opposite of what the lab was built to find.

Why — CORRECTED 2026-08-25 by `2026-08-25-index-skip-threshold`

The explanation below is wrong and is kept for the record. It was built by reading the descriptions of the last 22 of 67 changesets and generalising from the sample. Reading all 67 from source shows nine operations that do touch user-scaled tables, including createIndex on USER_ENTITY and on EVENT_ENTITY.

The real mechanism: Keycloak deliberately skips index creation on tables above 300,000 rows, logs a WARN, and records the changeset as EXECUTED. The migration is flat because the expensive work does not happen — not because there is no expensive work. That behaviour is far more valuable than the finding it replaces, and it was only found because the prediction tool disagreed with the measurement.

Original (incorrect) explanation: migration cost is O(schema change), not O(rows)

Reading the 67 changesets that ran, not one rewrites user data. They are:

createTable on new, empty tables — CLUSTER_EVENT, LOGIN_FAILURE, SINGLE_USE_OBJECT, AUTH_SESSION, ROOT_AUTH_SESSION, ISSUED_VER_CREDENTIAL, OUTBOX_ENTRY, USER_VER_CREDENTIAL, FED_USER_VER_CREDENTIAL
createIndex on those same empty tables
addColumn on CLIENT (10 rows) and REALM (2 rows)
addColumn on USER_CONSENT_CLIENT_SCOPE (0 rows)
one delete on REALM_ATTRIBUTE (tiny)

The only reference to a large table is addForeignKeyConstraint ... referencedTableName=USER_ENTITY on a new empty table — validating an empty child, not scanning the parent.

So the duration is set by how many DDL statements run and what they touch, not by how many users exist. For this version pair the answer is "nothing large", and it would take ~4 seconds on a realm of any size.

Outcome

Reached ready · [x] 100,002 users intact · [x] MIGRATION_MODEL → 26.7.1
DATABASECHANGELOG 144 → 211 · [x] end-user login OK
/health/ready behaved exactly as at 1k: 503 until migration finished

Breakage observed

None. Confirms the started-before-ready timing at 100k: /health/started returned 200 at 10.5s while /health/ready stayed 503 until 15.4s — a 4.9s window, essentially identical to the 5.6s measured at 1k, for the same reason the migration did not grow.

Verification

Claim	Primary source	Checked
No changeset rewrites user data on this path	`DATABASECHANGELOG.description` for all 67, read directly	☑
Migration time flat 1k → 100k	Two instrumented runs, same box, same fixture profile	☑
Holds at 2M	not tested — 100k may simply be too small to expose an effect	☐
Holds for pairs that DO touch user tables	not tested — this is the open question	☐

Publishable extract

We upgraded a 1,000-user Keycloak and a 100,000-user Keycloak on identical hardware. The schema migration took the same four seconds.

26.0.0 → 26.7.1, official image, Postgres 16.15 at stock settings, Hetzner CCX33 (8 vCPU dedicated, 32 GB, local NVMe), JVM -Xms1g -Xmx4g. At 1,000 users the Liquibase phase ran 3.97s; at 100,002 users, 3.79s. Total time to serving traffic was 17.0s and 15.4s respectively — the larger realm was marginally faster, which is noise.

We expected the opposite, so we read all 67 changesets. None of them rewrite user data. They create new empty tables, index those new tables, and add columns to CLIENT (10 rows) and REALM (2 rows). The only mention of USER_ENTITY is a foreign key from a new empty table.

SUPERSEDED. The real reason is in 2026-08-25-index-skip-threshold: Keycloak skips index creation above 300,000 rows and records the changeset as executed anyway. Migration is flat because the expensive work is declined, not because it is absent. Use that extract, not this one.

Rehearsal Record — 26.0.0 → 26.7.1 (instrumented)

2026-08-25T00:00:00+00:00

Environment

Identical to attempt 1, except the management port (9000) is now published to loopback so the health endpoints can be observed. The realm reached 26.0.0 by restoring baseline-26.0.0-typical-1k, so the fixture is byte-identical to attempt 1's starting point rather than merely similar.

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Database	PostgreSQL 16.15, single node
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0. Added retroactively 2026-08-25 to close a gap in our own records: read from the same unmodified `postgres:16` image on the same box. The compose `db` service sets no tuning, so stock is guaranteed by the file, not remembered.
Dataset scale	1000 users · 2 realms · 10 clients · 8 roles
Fixture profile	`typical`
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`

Procedure

Two experiments.

A — does migration require a client request? Restore 26.0.0, swap the image to 26.7.1, start it, and issue zero requests for 120 seconds. Measure the schema from Postgres directly, not from the application.

B — what does the server serve during migration? Same restore and swap, then poll /health/started, /health/ready (port 9000) and the master-realm token endpoint (port 8080) continuously from T0, printing only state transitions.

Findings

1. Schema migration runs unprompted — CONFIRMED

With no client contact whatsoever for 120s:

	before	after
`DATABASECHANGELOG` rows	144	211
latest changeset	`26.0.0-33201-org-redirect-url`	`26.7.0-cluster-event`
`MIGRATION_MODEL`	26.0.0	26.7.1

This settles attempt 1's open question. Migration is performed on executor-thread-1 after startup completes, and is not triggered by a first inbound request — the lazy-trigger hypothesis is refuted. Measured from Postgres, so it does not depend on trusting the application's own logging.

2. Timeline (T0 = container create, 15:48:19 UTC)

Elapsed	Event
0.8s	port 8080 not yet listening — connection refused
11.4s	Quarkus: `started in 3.499s. Listening on: http://0.0.0.0:8080`
11.4s	`/health/started` 000 · `/health/ready` 503 · token 503
11.5s	`/health/started` → 200 · ready 503 · token 503
12.2s	`Updating database. Using changelog META-INF/jpa-changelog-master.xml`
16.2s	`Migrating older model to 26.7.0` (last of six model steps)
16.6s	started 200 · ready 503 · token → 200
17.0s	started 200 · ready → 200 · token 200 — fully serving

T0 includes container recreation, which is why 11.4s elapses before Quarkus's own self-reported 3.499s startup.

3. `/health/ready` is honest. `/health/started` is not a sufficient gate.

/health/ready returned 503 for the entire migration window and only flipped to 200 at 17.0s, after migration completed. A readiness probe pointed at it behaves correctly.
/health/started returned 200 at 11.5s — during migration, ~5.5s before the server was actually able to serve.
Port 8080 accepted TCP connections from 11.4s, ~5.6s before service.
Requests arriving during migration received 503, not timeouts, connection resets, or partially-migrated data. It fails closed.

4. Correction to attempt 1

Attempt 1 flagged this as "a readiness probe could route production traffic mid-migration." That is wrong and this run refutes it. /health/ready reports unready throughout. The real exposure is narrower and different: any gate that is not /health/ready — a TCP port check, an L4 load balancer health check, or a Kubernetes startupProbe on /health/started — goes green while the schema is still migrating.

Recorded prominently rather than quietly amended, per our no-unverified-claims rule and the ledger's publishing rule 4.

Outcome

Reached ready · [x] 1000 users intact · [x] login works (verified attempt 1)
Custom SPI / themes / federation — not in fixture

Rollback

Exercised this run? no. Restore-from-snapshot was used twice to rebuild the starting state, which exercises the mechanism but is not a rollback of a failed upgrade and does not satisfy the gate.

Verification

Claim	Primary source	Checked
`/health/ready` = "only direct traffic once the readiness probe succeeds"	keycloak.org/observability/health, fetched 2026-08-25	☑
`/health/started` = startup probe, precedes liveness	same	☑
Docs describe behaviour during schema migration	They do not — checked 2026-08-25. This gap is why the measurement is worth publishing	☑
Behaviour identical on other major pairs	not tested — only 26.0.0 → 26.7.1	☐

Publishable extract

Keycloak 26.7.1 opens its HTTP port about 5.6 seconds before it can serve a request, and migrates your schema in between.

On a 1,000-user realm (Postgres 16.15, single node, Hetzner CCX33 — 8 vCPU dedicated, 32 GB, local NVMe, JVM -Xms1g -Xmx4g), upgrading 26.0.0 → 26.7.1: Quarkus logged started at T+11.4s and began the Liquibase schema migration at T+12.2s, on an executor thread, after startup had completed. Migration ran through six model steps and finished at T+16.2s. The server first issued a token at T+16.6s.

We verified the migration is not triggered by an inbound request: with the server left completely untouched for 120 seconds, DATABASECHANGELOG still went from 144 rows to 211 and MIGRATION_MODEL from 26.0.0 to 26.7.1.

The good news is that /health/ready is honest — it returned 503 for the whole window and only went green after migration finished. The trap is everything else: /health/started returned 200 at T+11.5s, mid-migration, and the TCP port accepted connections from T+11.4s. If your load balancer health check is a port probe, or your Kubernetes startupProbe gates on /health/started, your infrastructure believes the instance is up roughly five seconds before it is. Requests in that window get a clean 503 rather than corrupt data — it fails closed — but they do fail.

Five seconds does not sound like much, and on this path it stays about five seconds no matter how big your realm is — see the correction below.

CORRECTED 2026-08-25 by 2026-08-25-26.0.0-to-26.7.1-100k. This extract originally continued: "The migration phase is the part that scales with your data … so on a large realm this window is not five seconds." That was an extrapolation from a single scale point and it is false for this version pair. At 100,002 users the migration took 3.79s versus 3.97s at 1k, and the unready window was 4.9s versus 5.6s. None of the 67 changesets on this path rewrite user data. The window is real and worth designing around; its size does not grow with your realm on this path.

Rehearsal Record — 26.0.0 → 26.7.1 at 2,000,003 users

2026-08-25T00:00:00+00:00

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution	quay.io/keycloak/keycloak official image
Start mode	`start --http-enabled=true` w/ external DB
Database	PostgreSQL 16.15, single node, container
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0
Dataset scale	2,000,003 users · 2,000,002 credentials · 4,000,001 user attributes · 2 realms · 10 clients · 8 roles
DB size	3,512 MB
Seeding method	`seed-sql` — direct SQL, not `partialImport`. All users share one argon2id hash. See `bin/seed-sql.sh`
Fixture profile	`typical` (structure) + `seed-sql` (users)
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`
Adverse scenario	none

Comparability warning. Per our fixture-comparability rule, this fixture is not comparable to the 1k and 100k records in every respect: those were seeded through Keycloak's own write path via partialImport, this one bypasses it. Row counts and column contents match; provenance does not.

Timings

Phase	Value
Quarkus `started`	3.32s (self-reported)
`/health/started` → 200	10.83s
Liquibase migration	18:06:58,654 → 18:07:02,490 = 3.84s
`/health/ready` → 200	15.60s

Against every other scale measured

Users	DB size	Liquibase phase	Time to ready
1,000	~13 MB	3.97s	17.0s
100,002	264 MB	3.79s	15.4s
2,000,003	3,512 MB	3.84s	15.6s

Migration duration is flat across a 2,000× range in user count and a 270× range in database size. This is not a weak effect; there is no effect.

Outcome

Reached ready in 15.6s · [x] 2,000,003 users intact · [x] MIGRATION_MODEL → 26.7.1
DATABASECHANGELOG 211 rows · [x] login as a SQL-seeded user OK
IDX_USER_CREATED_TIMESTAMP — absent, silently

Breakage observed

The upgrade completes with a missing index on USER_ENTITY

Symptom: IDX_USER_CREATED_TIMESTAMP absent after a successful upgrade.
Evidence: the two WARN lines above; pg_indexes returns 0.
Cause: confirmed — CustomCreateIndexChange skips index creation above 300,000 rows. Deliberate Keycloak behaviour.
Fix: create it manually, ideally CREATE INDEX CONCURRENTLY so production is not locked — Keycloak's plain CREATE INDEX is exactly what it avoided.
Would this hit a customer? Every customer above 300,000 rows in any affected table, on every upgrade that adds an index. Scale-dependent by definition, and it hits the largest installs hardest.

Verification

Claim	Primary source	Checked
2,000,003 users present and migrated	`count(*)` from Postgres, before and after	☑
Migration flat 1k → 2M	three instrumented runs, same box, same version pair	☑
`USER_ENTITY` index skipped at 2M	`pg_indexes` = 0, plus both WARN lines	☑
Same index created at 1k and 100k	earlier runs in this series	☑
Threshold is 300,000 for `USER_ENTITY` too	not bisected — only 100k (created) and 2M (skipped) tested on this table	☐
Holds for other version pairs	not tested	☐
`seed-sql` fixture behaves like a `partialImport` one under migration	not tested — different provenance, see warning above	☐

Publishable extract

We upgraded a two-million-user Keycloak. The schema migration took 3.8 seconds — the same as a thousand-user realm. Here is why that is bad news.

26.0.0 → 26.7.1, official image, Postgres 16.15 at stock settings, Hetzner CCX33 (8 vCPU dedicated, 32 GB, local NVMe), JVM -Xms1g -Xmx4g.

Users Database Migration Time to serving

1,000 13 MB 3.97s 17.0s

100,002 264 MB 3.79s 15.4s

2,000,003 3,512 MB 3.84s 15.6s

A 2,000-fold increase in users changes nothing. We went looking for the reason and found it: Keycloak skips creating indexes on tables larger than 300,000 rows. It logs a WARN and moves on, and DATABASECHANGELOG records the changeset as EXECUTED — so a schema-version check shows everything applied.

At two million users, this upgrade quietly declined to create IDX_USER_CREATED_TIMESTAMP on USER_ENTITY. That same index is created automatically on a hundred-thousand-user realm. The bigger your Keycloak, the more indexes it does not build.

The design choice is defensible — locking USER_ENTITY on a two-million-row table during startup would be far worse than a late index, and it is why your upgrade window is short. But the consequence is rarely stated: a fast Keycloak upgrade at scale is fast partly because work was deferred onto you, and the only notice is a warning in the noisiest log of your month.

After any Keycloak upgrade, list the indexes its changelogs define and diff them against your database. If any are missing, create them yourself with CREATE INDEX CONCURRENTLY. Then keep doing it, because every future release that indexes a large table will behave the same way.

One note on method, because it changes what these numbers mean: the two-million-user realm was seeded with direct SQL rather than through Keycloak's API. partialImport runs at about 19 users/second — Keycloak 26 hashes every password with argon2id on a single thread — which puts two million users at roughly 29 hours. Direct SQL does it in 167 seconds. Every user shares one password hash, which is fine for migration timing and useless for anything about authentication performance.

Rehearsal Record — 26.0.0 → 26.7.1 (rollback exercised)

2026-08-25T00:00:00+00:00

Environment

Identical to attempts 1 and 2: Hetzner CCX33 (fsn1), 8 vCPU dedicated, 30GB RAM, local NVMe; Postgres 16.15 single node; 1000 users, typical profile, 2 realms, 10 clients, 8 roles; JVM -Xms1g -Xmx4g; single container.

Procedure

restore baseline-26.0.0-typical-1k          # identical starting state
upgrade 26.0.0 -> 26.7.1, wait ready
POST /admin/realms/lab/users {canary-postupgrade}   # data written AFTER the snapshot
# --- roll back ---
stop keycloak; KC_VERSION=26.0.0
dropdb / createdb / pg_restore baseline-26.0.0-typical-1k
start keycloak 26.0.0, wait ready

The canary is the point of the run. Anything written between the snapshot and the rollback decision is what a customer actually loses, and asserting that it disappears is the only way to know the rollback is real rather than nominal.

Timings

Phase	Duration
DB drop + create + `pg_restore`	2.475s
Total, rollback decision → serving traffic again	15.9s

The restore is 2.5s of the 15.9s. The rest is Keycloak process start, which is the same fixed cost seen everywhere else in these runs.

Outcome

Reached ready on 26.0.0 after rollback
serverinfo reports 26.0.0
MIGRATION_MODEL back to 26.0.0 — schema genuinely reverted, not just the binary
1000 users present
End-user login works (labuser7 issued a token)
Canary absent — the post-upgrade user is gone, as it must be

Breakage observed

None. The rollback was clean.

Rollback

NOTE added 2026-08-25: this run performed the restore steps by hand with docker compose up -d --force-recreate. bin/restore.sh as shipped at the time used docker compose start, which reuses the existing container and silently re-migrates — see 2026-08-25-backup-and-rollback-tooling. The finding below holds for the procedure tested; the script did not implement it.

Exercised this run? yes — this is the gate run
Method: stop Keycloak, pg_restore the pre-upgrade snapshot, start the old image
Time to restore: 2.475s DB, 15.9s to serving (1k users)
Data loss / divergence: Total, for everything written after the snapshot. Confirmed by the canary. This is not a downgrade — Keycloak has no schema downgrade path — it is a point-in-time restore, and it takes the data with it.
Verdict: Rollback is genuinely available on 26.0.0 ↔ 26.7.1, with a hard caveat. It is available as restore, not as downgrade. Availability is therefore a property of the customer's snapshot discipline and their tolerance for losing the write window, not a property of Keycloak.

Verification

Claim	Primary source	Checked
Schema reverted, not just the binary	`MIGRATION_MODEL` read from Postgres directly = 26.0.0	☑
Post-snapshot writes are lost	canary user, asserted absent after rollback	☑
Keycloak has no supported schema downgrade	not verified against docs — inferred from the restore being required at all	☐
Holds at scale	not tested — 2.5s restore is a 500K dump	☐

Publishable extract

Rolling back a Keycloak upgrade is a database restore, and it costs you every write since the snapshot.

On a 1,000-user realm (Postgres 16.15, Hetzner CCX33 — 8 vCPU dedicated, 32 GB, local NVMe), rolling 26.7.1 back to 26.0.0 took 15.9 seconds end to end. Only 2.5s of that was the pg_restore; the rest was Keycloak starting.

We verified it is a real rollback and not a cosmetic one: MIGRATION_MODEL in Postgres went back to 26.0.0, so the schema itself reverted. We also created a user after the upgrade and confirmed it was gone afterwards — because that is the part people miss. There is no schema downgrade. You are restoring a point-in-time backup, so every registration, password change, session and admin edit between your snapshot and your rollback decision is discarded.

The practical consequence: your rollback window is not bounded by how long the restore takes. It is bounded by how much write activity you can afford to throw away. Decide that number before the maintenance window, because during one you will not want to.

Publishable. The 15.9s figure is honest but must always be quoted with its 1,000-user scale attached — the restore half scales with the dump, the startup half does not.

Rehearsal Record — Keycloak silently skips index creation above 300,000 rows

2026-08-25T00:00:00+00:00

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Database	PostgreSQL 16.15, single node, container
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0
Dataset	100,002 users, `typical` profile; `EVENT_ENTITY` varied per run
Seeding	users via `partialImport`; events via direct SQL `INSERT … generate_series`, then `ANALYZE`
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`

Procedure

For each row count: restore baseline-26.0.0-typical-100k, insert N rows into EVENT_ENTITY, ANALYZE, upgrade to 26.7.1, then check pg_indexes for idx_event_entity_user_id_type and grep the log for the skip warning.

Results

`EVENT_ENTITY` rows	Time to ready	Index created	Skip warning
0	15.7s	yes	no
1,000	15.4s	yes	no
100,000	15.6s	yes	no
200,000	15.7s	yes	no
300,000	15.9s	yes	no
301,000	15.5s	NO	yes
350,000	15.5s	NO	yes
400,000	16.0s	NO	yes
450,000	15.3s	NO	yes
500,000	15.4s	NO	yes
5,000,000	14.9s	NO	yes

The threshold is exactly 300,000 rows. Time to ready is flat across a range of 0 to 5,000,000 rows — a 2.1 GB database at the top end — precisely because the expensive work is skipped.

Outcome

Upgrade reached ready in every run · [x] realm intact · [x] login works
Threshold reproduced across 11 runs with a clean boundary at 300,000

Verification

Claim	Primary source	Checked
Index absent above threshold	`pg_indexes`, queried directly	☑
Changeset recorded EXECUTED	`DATABASECHANGELOG.exectype`	☑
Keycloak warns at skip and at startup	container log, both messages quoted	☑
Threshold is 300,000	300,000 creates; 301,000 skips	☑
The 300,000 constant applies to other tables/indexes	confirmed on `USER_ENTITY` — bisected to the same inclusive 300,000 boundary in `2026-08-26-index-precondition-anatomy` §3b	☑
Same behaviour in majors other than 26.7.1	not tested	☐
Whether row count comes from statistics or an exact count	answered — the planner estimate. Falsifying `pg_class` for `USER_ENTITY` at 2M made Keycloak build the index; see `2026-08-25-runA-indexes-manually-created` L5	☑

Publishable extract

Keycloak will skip creating a database index during an upgrade if your table is too big, and the only place it tells you is a WARN in the startup log.

Upgrading 26.0.0 → 26.7.1 adds an index to EVENT_ENTITY. We ran that upgrade eleven times against the same 100,000-user realm, varying only the number of rows in EVENT_ENTITY (Postgres 16.15 at stock settings, Hetzner CCX33 — 8 vCPU dedicated, 32 GB, local NVMe).

At 300,000 event rows the index is created. At 301,000 it is not.

The upgrade still succeeds. The server starts. MIGRATION_MODEL reads 26.7.1. And DATABASECHANGELOG records changeset 26.4.0-51321 as EXECUTED, not as skipped — so a schema-version check, which is what most people verify with, shows everything applied. Only two WARN lines say otherwise:
WARN [CustomCreateIndexChange] Following index should be created:
  CREATE INDEX IDX_EVENT_ENTITY_USER_ID_TYPE ON public.EVENT_ENTITY(USER_ID, TYPE, EVENT_TIME);
WARN [DatabaseIndexChecker] Missing database index IDX_EVENT_ENTITY_USER_ID_TYPE
  on table EVENT_ENTITY. Create the index manually: ...
This is deliberate. CustomCreateIndexChange exists so a large table is not locked during startup, and that is a reasonable design choice — a slow upgrade is worse than a late index. We measured what it is avoiding: at 5,000,000 rows the index takes about 3 seconds to build on this hardware.

The consequence is what nobody mentions. The installations that skip the index are exactly the ones large enough to need it, small deployments get it automatically, and every future release that indexes a large table will behave the same way, so the gap grows with each upgrade.

If you have upgraded Keycloak recently and your event table is bigger than 300,000 rows, check whether IDX_EVENT_ENTITY_USER_ID_TYPE exists. Then add "list indexes Keycloak wanted and did not create" to your post-upgrade checklist permanently, because this will happen again.

Incidentally, this also explains something that surprised us. We had expected migration time to grow with data volume. It does not — 15 seconds whether the database holds 0 or 5,000,000 events — and this is why. Keycloak keeps the upgrade fast by declining to do the expensive part.

Rehearsal Record — the indexes Keycloak declined, created manually

2026-08-25T00:00:00+00:00

Why this run existed

bin/upgrade.sh prints, after every run:

>> create these manually, preferably with CREATE INDEX CONCURRENTLY.

We had never once run an upgrade against a database where that advice had been followed. 2026-08-25-index-skip-threshold and 2026-08-25-26.0.0-to-26.7.1-2m both name this as the largest untested configuration in the lab, and it ranked first on our own untested list.

The question was whether our own advice breaks the next upgrade. It does not. Asking only that question would have missed both of the findings below.

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution	quay.io/keycloak/keycloak official image
Start mode	`start --http-enabled=true` w/ external DB
Database	PostgreSQL 16, single node, container
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0. L4 varies `maintenance_work_mem` deliberately and says so
Dataset scale	2,000,003 users · 2 realms · 10 clients · 8 roles · DB 3,026 MB
Seeding method	`seed-sql` — direct SQL. Baseline `baseline-26.0.0-sqlseed-2m`
Fixture profile	`typical` (structure) + `seed-sql` (users)
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`

Every leg starts from a restore of the same baseline — 71s, verified against re-migration by bin/restore.sh each time — so the legs are comparable to each other. The Liquibase phase is measured identically in every leg: from QuarkusJpaUpdaterProvider: Updating database to DefaultMigrationManager: Migrating older model to 26.1.0.

Results

Leg	Upgrade	`exectype`	Index after	WARNs	Liquibase phase	Bootstrap
L1 correct index present	READY	`MARK_RAN`	correct, 14 MB	0	1.72s	4.304s
L2 control, absent	READY	`EXECUTED`	absent	2	1.69s	4.240s
L3 wrong-shape index present	READY	`MARK_RAN`	wrong shape	0	—	5.291s
L5 stats say the table is small	READY	`EXECUTED`	built, correct	0	3.51s	6.156s

Every leg reached ready, ended at migration_model 26.7.1 with a 211-row DATABASECHANGELOG, and kept 2,000,003 users. L1's admin token endpoint returned 200.

L1 — following our own advice is safe

CREATE INDEX CONCURRENTLY idx_user_created_timestamp
  ON user_entity (realm_id, created_timestamp);        -- 970 ms, 14 MB

Liquibase's precondition sees the index, records changeset 26.6.0-43829-user-created-timestamp-index as MARK_RAN, and says nothing — no CustomCreateIndexChange warning, no DatabaseIndexChecker warning. The checker runs (Running database index checker) and reports a clean database.

No "already exists" error, no failure, and no measurable change in migration time. The advice we have been giving is sound.

L2 — the control, and the reason L1 means anything

Same restore, nothing created. Changeset EXECUTED, index absent, both WARNs present — reproducing 2026-08-25-26.0.0-to-26.7.1-2m exactly, this time on a restored rather than freshly seeded database.

1.69s against L1's 1.72s: pre-creating the index neither costs nor saves migration time, because the work being avoided is under a second (L4).

L3 — a wrong index with the right name is accepted, silently ⭐

An operator creates the index Keycloak asked for, with the name Keycloak logged, and gets the columns wrong — one instead of two:

CREATE INDEX CONCURRENTLY idx_user_created_timestamp
  ON user_entity (created_timestamp);                   -- 628 ms

changeset → MARK_RAN
pg_indexes still shows the single-column index; Keycloak never replaced it
warn_count=0 — CustomCreateIndexChange silent
DatabaseIndexChecker silent too, declaring the database correct

The precondition matches on index name only. That is visible in Keycloak's own changelog, not merely inferred from the behaviour — META-INF/jpa-changelog-26.6.0.xml:

<changeSet author="keycloak" id="26.6.0-43829-user-created-timestamp-index">
    <preConditions onSqlOutput="TEST" onFail="MARK_RAN">
        <not>
            <indexExists tableName="USER_ENTITY" indexName="IDX_USER_CREATED_TIMESTAMP" />
        </not>
    </preConditions>
    <createIndex tableName="USER_ENTITY" indexName="IDX_USER_CREATED_TIMESTAMP">
        <column name="REALM_ID" type="VARCHAR(255)" />
        <column name="CREATED_TIMESTAMP" type="BIGINT" />
    </createIndex>
</changeSet>

indexExists takes a table and a name. It does not take columns. Anything with the right name satisfies it, whatever it indexes — no check of columns, order or uniqueness — and Keycloak's own post-upgrade index verifier accepts it as well.

The database ends in a state where the index Keycloak wanted does not exist, the index that does exist does not serve the query it was added for, every Keycloak-provided diagnostic reports success, and — unlike the skip case — not even a WARN is left in the log. The skip leaves evidence. This leaves none.

The route into that state is our own runbook telling the customer to create the index by hand.

L4 — what the deferred DDL costs at 2,000,003 rows

Index (realm_id, created_timestamp) on USER_ENTITY, measured directly:

`maintenance_work_mem`	`CREATE INDEX`	`CREATE INDEX CONCURRENTLY`	Index size
64MB (stock)	754 ms	969 ms	14 MB
1GB	765 ms	1000 ms	14 MB

The work Keycloak declines is under one second at two million users. The index-skip finding is not "Keycloak avoids an expensive rebuild"; it is "Keycloak avoids 750 ms and a brief ACCESS EXCLUSIVE lock on USER_ENTITY". The lock is the real reason. The duration is not.

maintenance_work_mem does not matter for this index. Our own scenario notes claimed that "a 2M-row index build is dominated by" those settings. Raising it 16× changed the build by 1.5%: the index is 14 MB and the sort fits inside the stock 64 MB either way. The field is still worth recording — the reason given for it is wrong at this scale.

L5 — the threshold is a planner estimate, not a row count ⭐⭐

2026-08-25-index-skip-threshold left open "whether row count comes from statistics or an exact count". L2 could not separate them: a freshly restored 2M database carries an accurate reltuples (2000003, relpages 53392) even with no last_analyze, because the index builds during pg_restore set it.

So we lied to the planner instead, changing nothing else:

UPDATE pg_class SET reltuples = 1000, relpages = 100 WHERE relname = 'user_entity';
-- actual rows: 2000003

Keycloak built the index. Correct two-column definition, changeset EXECUTED, zero warnings — a genuine CREATE INDEX against a 2,000,003-row table, during the migration, because a catalogue row said the table was small.

An exact SELECT count(*) is unaffected by pg_class, so that hypothesis is refuted. CustomCreateIndexChange reads the planner's estimate.

And it produced the first non-flat migration this lab has measured:

	Liquibase phase
L2 — index skipped (honest statistics)	1.69s
L5 — index built (statistics say 1,000)	3.51s

+1.82s, at identical true scale, from the same baseline. Everything this lab has said about migration duration being flat holds only while Keycloak is declining the DDL, and what makes it decline is a number that is stale by design.

Breakage observed

A manually created index with the wrong columns is accepted silently

Symptom: upgrade succeeds, changeset MARK_RAN, zero warnings, and the database holds an index that does not match what the changeset defines.
Evidence: L3. pg_indexes.indexdef reads btree (created_timestamp) where the changeset wants btree (realm_id, created_timestamp); warn_count=0 from both CustomCreateIndexChange and DatabaseIndexChecker.
Cause: confirmed — the Liquibase precondition and the index checker both key on index name.
Fix: verify by definition, not by existence. Compare pg_indexes.indexdef against the changeset's column list; name equality is not evidence.
Would this hit a customer? Any customer who followed the "create these manually" advice and mistyped, and nothing in Keycloak will ever tell them.

Whether the index is built depends on when autovacuum last looked

Symptom: the same database, at the same true size, either skips the index or builds it, according to pg_class.reltuples.
Evidence: L2 (accurate statistics → skipped) against L5 (statistics say 1,000 → built, +1.82s of migration).
Cause: confirmed — CustomCreateIndexChange consults the planner estimate. Which of reltuples / relpages it reads was not isolated; both were changed together.
Fix: none available to the operator, and that is the point. reltuples is only refreshed by ANALYZE, VACUUM and index builds, so a table that has grown since its last autovacuum carries a stale, low estimate.
Would this hit a customer? Scale-dependent and timing-dependent. Two identical production databases can diverge on autovacuum history alone — and a staging rehearsal is not evidence about production unless the estimates match. It is also the mechanism by which a customer gets a slow migration they have no way to predict, which re-arms S3 (statement_timeout) and S14 (disk) at any realm size.

`EXECUTED` and `MARK_RAN` do not mean what they look like

Symptom: the row that reads as "we did not run this" is the healthy one.

Evidence: four combinations observed across the legs:

`exectype`	Index	What actually happened
`EXECUTED`	present	Keycloak built it — L5, and every realm under the threshold
`EXECUTED`	absent	Keycloak skipped it — L2, the dangerous case
`MARK_RAN`	present, correct	someone created it — L1, healthy
`MARK_RAN`	present, wrong shape	someone created the wrong thing — L3, silent

Fix: exectype is only usable together with the index's presence and its definition. On its own it distinguishes nothing.
Would this hit a customer? Anyone auditing the changelog rather than the schema — which is exactly what the changelog invites.

Rollback

Exercised this run? L1 took a pre-upgrade backup through the default path (pre-upgrade-26.0.0-to-26.7.1-20260825T225316Z, 65 MB). L2, L3 and L5 ran --no-backup deliberately: each was preceded by a verified restore of the same baseline, which is the case that flag documents.
Verdict: unchanged from 2026-08-25-26.0.0-to-26.7.1-3 — rollback on this path is a point-in-time restore.

Verification

Claim	Primary source	Checked
A pre-existing correct index does not break the upgrade	L1 — READY, model 26.7.1, 211 changelog rows	☑
It is recorded `MARK_RAN`, not `EXECUTED`	L1 `databasechangelog.exectype`	☑
No warning is emitted when the index is present	L1 — grep of both WARN strings, 0 hits	☑
Control reproduces the skip on a restored DB	L2 — `EXECUTED`, index absent, 2 WARNs	☑
Migration time unchanged by the index being present	L1 1.72s vs L2 1.69s, same boundary	☑
A wrong-shape index with the right name is accepted	L3 — `MARK_RAN`, `warn_count=0`, `indexdef` unchanged	☑
The precondition is name-only by construction	`jpa-changelog-26.6.0.xml` — `<indexExists tableName= indexName=>`, no columns	☑
`DatabaseIndexChecker` accepts it too	L3 — checker ran, emitted nothing	☑
Index build at 2M is sub-second	L4 — 754 ms plain, 969 ms concurrent	☑
`maintenance_work_mem` does not change that build	L4 — 64MB vs 1GB, 1.5% apart	☑
The threshold reads a planner estimate, not `count(*)`	L5 — falsified `pg_class`, index built at 2M	☑
A restored database carries accurate `reltuples`	L2 pre — 2000003, `relpages` 53392, no `last_analyze`	☑
Building the index makes the migration non-flat	L5 3.51s vs L2 1.69s, same true scale	☑
Whether `reltuples` or `relpages` is the input	not isolated — both were changed together	☐
Whether the precondition accepts a unique index of the same name	not tested	☐
Whether other `CustomCreateIndexChange` changesets behave identically	not tested — only `26.6.0-43829`	☐
Behaviour on version pairs other than 26.0.0 → 26.7.1	not tested	☐
Whether a stale estimate arises naturally at customer scale	answered, and narrower than this run implies — stock autovacuum corrects a bulk load mid-flight; it needs `autovacuum_enabled=false` on the table. See `2026-08-26-stale-statistics-reachability`	☑

Publishable extract

Keycloak decides whether to build a database index during an upgrade by asking Postgres how big the table is — and Postgres answers from a cached estimate that can be wrong.

Some background. Keycloak skips creating an index during an upgrade when the target table has more than 300,000 rows: it logs two warnings, records the changeset as EXECUTED, and leaves the index uncreated. We measured that boundary earlier and it is exact.

What we had not established is where the row count comes from. It is pg_class — the planner's estimate — not a SELECT count(*).

On a 2,000,003-user realm (Keycloak 26.0.0 → 26.7.1, Postgres 16 at stock settings, Hetzner CCX33: 8 vCPU dedicated, 32 GB, local NVMe), we changed one catalogue row to claim USER_ENTITY held 1,000 rows, and changed nothing else. Keycloak built the index — a real CREATE INDEX on two million rows — and the schema migration went from 1.69 seconds to 3.51 seconds.

That matters because reltuples is refreshed only by ANALYZE, VACUUM and index builds. A table that has grown since autovacuum last visited carries a stale, low estimate. So two databases of identical size can take different paths through the same upgrade, and neither operator has any way to know which they will get. Your staging rehearsal tells you about production only if the statistics match, and nobody checks that.

Second finding, and the one we did not go looking for. Keycloak's warning tells you to create the index yourself:
Missing database index IDX_USER_CREATED_TIMESTAMP on table USER_ENTITY.
Create the index manually: CREATE INDEX IDX_USER_CREATED_TIMESTAMP ON
  public.USER_ENTITY(REALM_ID, CREATED_TIMESTAMP);
Doing exactly that is safe: the next upgrade sees the index, marks the changeset MARK_RAN, and warns about nothing. We checked, because "the changeset will try to create an index that already exists" was a real risk and we had never tested it.

But the check Keycloak performs is on the index's name, and nothing else. We created an index called IDX_USER_CREATED_TIMESTAMP on the wrong columns — one column instead of two, the kind of mistake a hurried copy-paste makes — and upgraded. Keycloak marked the changeset satisfied, emitted no warning, and its own DatabaseIndexChecker reported the database correct. The index Keycloak wanted does not exist. The one that does will not serve the query it was added for. Nothing in Keycloak will ever say so again — and unlike the skip, this state does not even leave a warning in the log.

A last number, because it reframes the trade-off. That index takes 754 ms to build on two million rows, or 969 ms with CONCURRENTLY, and occupies 14 MB. Raising maintenance_work_mem from the stock 64 MB to 1 GB changes it by 1.5%. Keycloak is not avoiding an expensive rebuild — it is avoiding a sub-second ACCESS EXCLUSIVE lock. That is a defensible thing to avoid during startup. It is worth knowing that the price of avoiding it is an index your database may never get.

If you have upgraded Keycloak at any real scale: list the indexes its changelogs define, and compare them against pg_indexes.indexdef — the definition, not just the name. Presence is not correctness here.

Rehearsal Record — S1/S2: Liquibase lock failure modes (could not reproduce)

2026-08-25T00:00:00+00:00

Summary

We tried to reproduce the canonical Keycloak upgrade horror story — a killed migration leaving a stuck Liquibase lock that blocks every subsequent start — and could not, on Postgres 16 single-node. Every failure mode we induced recovered by itself.

This is a negative result and it is stated as one. It does not prove the story is false; it bounds where it applies.

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Database	PostgreSQL 16.15, single node, container
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0
Dataset	100,002 users, `typical` profile
Topology	single container (S1); two containers, one compose network (S2)
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`

What we could not observe

The lock was never sampled in a held state, in either scenario. Polling ran through docker compose exec … psql, roughly 200–400ms per sample, against a migration that completes in ~3.5s. Absence of observation here is a limit of the instrument, not evidence the lock is not taken. Do not read this record as saying Keycloak does not lock — it says we never caught it holding one.

Verification

Claim	Primary source	Checked
Half-migrated schema self-completes	changelog 165 → 211 across a restart, read from Postgres	☑
No stuck lock after SIGKILL	`databasechangeloglock` queried directly, both runs	☑
Concurrent start ends in a correct schema	changelog 211, model 26.7.1, users intact	☑
Lock is genuinely held at some point	not observed — sampling too coarse	☐
Holds on a slow migration	not tested — every migration here was ~3.5s	☐
Holds on Postgres HA / failover mid-migration	not tested	☐
Holds on other databases	not tested — Postgres only	☐
Holds for other version pairs	not tested	☐

Publishable extract

We tried to reproduce Keycloak's most-repeated upgrade horror story and could not.

The story: a Keycloak upgrade dies partway, Liquibase's lock row is never released, and every subsequent start hangs waiting for a lock nobody holds — including the start you attempt in order to roll back.

On Postgres 16.15, single node, with a 100,002-user realm on a Hetzner CCX33 (8 vCPU dedicated, 32 GB, local NVMe), upgrading 26.0.0 → 26.7.1, we killed Keycloak with SIGKILL during the schema migration. Twice. The second time we waited until the migration was genuinely half-applied — 165 of 211 changesets committed — before killing it.

Both times the lock table showed zero held locks afterwards, and both times the next start completed the migration by itself in 13 seconds, ending at the correct schema version with all 100,002 users intact. We also started two instances against an un-migrated schema simultaneously; both came up, and the schema ended correct.

One honest caveat, and it is the interesting part. These migrations take about three and a half seconds — because Keycloak skips index creation on tables over 300,000 rows rather than block startup on them. The window in which a crash can hurt you is narrow because Keycloak works hard to keep it narrow. If your upgrade includes a changeset that genuinely rewrites a large table, or you have manually built the indexes Keycloak declined to build, your window is wider than ours and this result may not carry.

We would rather publish a negative result than repeat folklore. If you have a reproducible stuck lock, we want the configuration — we will run it and publish what happens.

Rehearsal Record — S10: a missing theme falls back silently, and the error arrives after the window closes

2026-08-25T00:00:00+00:00

Summary

The lab realm has named custom themes since the first run — seed-realm.sh's typical profile sets loginTheme: lab-login, accountTheme: lab-account, emailTheme: lab-email — and no theme files have ever been mounted. So S10 has silently been in force in every record this lab has produced.

The answer: Keycloak falls back to the built-in theme and serves a working login page. It logs the problem at ERROR level — but lazily, on the first request that needs the theme, not at startup. An operator who upgrades, watches the startup log, sees a clean start and closes the maintenance window has had no opportunity to see it.

Environment

Field	Value
Keycloak version	26.7.1 (arrived via 26.0.0 → 26.7.1)
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB
Database	PostgreSQL 16.15, single node, container
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0
Adverse scenario	S10
Dataset scale	2,000,003 users, 2 realms
Seeding method	direct SQL COPY, over a `partialImport` realm skeleton
Realm theme config	`login_theme=lab-login`, `account_theme=lab-account`, `email_theme=lab-email`, `admin_theme=keycloak` — read from the `realm` table
Themes mounted	none — `/opt/keycloak/themes/` is empty; only `fixtures/providers` is bind-mounted
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe

Procedure

docker compose exec -T db psql -U keycloak -c \
  "select name, login_theme, account_theme, email_theme, admin_theme from realm"

curl -sL "http://localhost:8080/realms/lab/protocol/openid-connect/auth\
?client_id=spa-public&response_type=code&scope=openid\
&redirect_uri=https%3A%2F%2Fspa.lab.invalid%2Fcb\
&code_challenge_method=S256&code_challenge=<S256 of a verifier>"

curl -s "http://localhost:8080/realms/lab/account/"
docker compose logs keycloak | grep -i theme

Results

Surface	HTTP	Body	Theme actually served
`lab` realm login page	200	8,094 bytes, full `kc-form-login` form, `<title>Sign in to Lab realm lab</title>`	built-in `login/keycloak.v2`
`lab` realm account console	200	4,213 bytes	built-in `account/keycloak.v3`
`master` realm login (no custom theme configured)	302 → 200	—	built-in, as expected

The login page is fully functional. Asset URLs in the served HTML resolve to resources/he5u5/login/keycloak.v2/… and resources/he5u5/account/keycloak.v3/… — the built-in themes, not the configured names.

The logging is real but late

19:03:19,962 ERROR [org.keycloak.theme.DefaultThemeManager] Failed to find ACCOUNT theme lab-account, using built-in themes
19:03:42,949 ERROR [org.keycloak.theme.DefaultThemeManager] Failed to find LOGIN  theme lab-login,   using built-in themes

Three theme-related lines in the entire container log, and all three carry the timestamp of the request that triggered them, not of startup. The ACCOUNT line appeared when the account console was first fetched; the LOGIN line 23 seconds later, when the login page was first fetched. Before those requests, the log contained nothing about themes at all.

That is the finding. The severity is right — ERROR — but the timing is wrong for an upgrade: the signal is emitted by the first user, not by the upgrade.

Outcome

Reached ready
Login flow renders — with the wrong branding
Themes render — no, silently substituted
Logged, at ERROR, but only on first use

Verification

Claim	Primary source	Checked
Realm names themes that do not exist	`realm` table read directly	☑
Login page still returns 200 and a usable form	`curl`, 8,094 bytes, `kc-form-login` present	☑
Built-in theme is substituted	asset paths `login/keycloak.v2`, `account/keycloak.v3` in the served HTML	☑
The ERROR is emitted on first request, not at startup	log timestamps vs. request times; zero theme lines before the requests	☑
Email theme behaviour	not tested — needs an actual send; a broken `emailTheme` may not be as forgiving	☐
A partially present theme (dir exists, files missing)	not tested — likely a different and worse failure than a wholly absent one	☐
Admin theme	not tested — `admin_theme` was set to the valid built-in `keycloak`	☐

Publishable extract

If your Keycloak upgrade loses its custom theme, Keycloak will not tell you during the upgrade. It will tell the first person who tries to log in.

We ran a realm configured with loginTheme: lab-login and no such theme mounted, on Keycloak 26.7.1 / Postgres 16.15. The login page returned HTTP 200 with a complete, working sign-in form — rendered in the built-in theme. The only signal was one line, at ERROR:
ERROR [org.keycloak.theme.DefaultThemeManager] Failed to find LOGIN theme lab-login, using built-in themes
and it was written at the moment of the first login request, not at startup. Our container log contained no theme-related line at all until somebody asked for a page.

This is a graceful fallback, and as engineering it is the right call — nobody wants an outage because a CSS file is missing. But it means the standard upgrade verification, "the server came up clean, close the window", cannot catch it. Custom themes are mounted from outside the image, so they are exactly the kind of thing an image bump, a chart change, or a moved volume quietly drops.

Add one line to your post-upgrade checks: fetch the login page and confirm the asset URLs contain your theme name, not keycloak.v2.

Rehearsal Record — S12: a Keycloak minor upgrade raised the minimum Postgres major

2026-08-25T00:00:00+00:00

Summary

Keycloak 26.0.0 runs on PostgreSQL 12. Keycloak 26.7.1 refuses to. The minimum supported Postgres major moved from 12 to 13 inside major version 26 — across what looks, from the version number, like a routine minor bump.

ERROR: Persistence unit 'keycloak-default' was configured to run with a database
version of at least '13.0.0', but the actual version is '12.22.0'.

The upgrade fails before the migration. Nothing is applied, no data is lost, and the fix is a database upgrade — which is a different project, a different change window and usually a different team.

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB
Database	PostgreSQL 13.23 / 12.22 / 11.16, single node, container — the variable
Postgres tuning	stock defaults for each major
Adverse scenario	S12
Dataset scale	202 users, 2 realms (fresh install + seed per major)
Seeding method	`partialImport` (`bin/seed-realm.sh --profile typical --users 200`)
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`

Each major started from an empty pgdata volume: fresh install at 26.0.0, seed, then the upgrade under test.

Results

Postgres	Keycloak 26.0.0	Keycloak 26.7.1
13.23	installs, ready 19s	upgrade READY in 16s — model 26.7.1, changelog 211, 202 users, login 200
12.22	installs, ready 19s, changelog 144, 202 users seeded	REFUSES TO START — requires ≥ 13.0.0
11.16	REFUSES TO START — requires ≥ 12.0.0	not reached

So the floors are:

Keycloak	Minimum Postgres major
26.0.0	12
26.7.1	13

The failure is clean

After 26.7.1 refused to start on Postgres 12.22:


`migration_model`	26.0.0 — unchanged
`DATABASECHANGELOG`	144 — unchanged
Users	202 — intact
Container	exited 1

Fail-closed, like S3, S4 and S5 before it. The database is exactly as it was and rolling back means putting the old image tag back.

The escape hatch, and its wording

Both versions offer one, and they do not name the same property:

Refusing version	Suggested override
26.7.1 on PG 12	`jakarta.persistence.database-product-version=12.22.0`
26.0.0 on PG 11	`quarkus.datasource.db-version=11.16.0`

Both append the same warning: "but this may disable some features and/or impact performance negatively". We did not test either override. It is a documented way to start on an unsupported database; it is not a supported configuration, and a rescue engagement should treat it as a bridge to a database upgrade rather than a destination.

Verification

Claim	Primary source	Checked
26.7.1 upgrades cleanly on PG 13.23	run: READY 16s, model 26.7.1, changelog 211, login 200	☑
26.0.0 installs and serves on PG 12.22	run: ready 19s, changelog 144, 202 users seeded	☑
26.7.1 refuses on PG 12.22	`at least '13.0.0', but the actual version is '12.22.0'`	☑
26.0.0 refuses on PG 11.16	`at least '12.0.0', but the actual version is '11.16.0'`	☑
The refusal applies nothing	changelog 144, model 26.0.0, 202 users after the failure	☑
Which 26.x release moved the floor	not bracketed — only known to be in (26.0.0, 26.7.1]	☐
Whether the override flags work	not tested — deliberately	☐
Whether PG 13 is the floor for later Keycloak	not tested — the floor evidently moves, so this expires	☐
Managed-service equivalents (RDS/Cloud SQL/Azure majors)	not tested — same engine versions, different upgrade mechanics	☐

Publishable extract

Keycloak 26.0.0 runs on PostgreSQL 12. Keycloak 26.7.1 does not. Same major version of Keycloak.

We installed Keycloak 26.0.0 on Postgres 12.22, seeded a realm, and upgraded to 26.7.1 — the kind of bump most change advisory boards would wave through as a minor. It stopped before it started:
ERROR: Persistence unit 'keycloak-default' was configured to run with a database
version of at least '13.0.0', but the actual version is '12.22.0'.
The floors we measured:

Keycloak Minimum Postgres

26.0.0 12

26.7.1 13

On Postgres 13.23 the same upgrade took sixteen seconds and worked perfectly.

The good news is that the refusal is clean: 144 changelog rows before, 144 after, all 202 users present, container exited 1. Nothing was half-done. The bad news is what the fix is. You cannot resolve this inside the Keycloak change window — you need a database major upgrade, which is a different project with a different risk profile and usually a different team. If you discovered it during the window, the window is over.

Keycloak does offer an override (jakarta.persistence.database-product-version) and warns in the same breath that it "may disable some features and/or impact performance negatively". We deliberately did not test it. Treat it as a bridge to a database upgrade, not a destination.

The pre-flight question is one line, and almost nobody asks it, because the Keycloak version number does not suggest it needs asking:
select version();

Rehearsal Record — S13: a non-default schema works, and it broke our rollback tool

2026-08-25T00:00:00+00:00

Summary

Keycloak handles a non-default schema correctly: 87 tables created in kc, zero in public, migration clean in 16 seconds, nothing leaked.

Two things around it are less clean.

Keycloak will not create the schema. If KC_DB_SCHEMA names a schema that does not exist, startup fails — and the message it fails with is Cannot invoke "CustomLockService.waitForLock(...)" because "this.lockService" is null, the exact error S8 produced from an entirely different cause.

Our own restore.sh was broken by it, in the specific way that matters: it read migration_model unqualified, got relation "migration_model" does not exist, and would have reported an empty schema version rather than a failed rollback.

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB
Database	PostgreSQL 16.15, single node, container
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0
Adverse scenario	S13 — `KC_DB_SCHEMA=kc`
Dataset scale	202 users, 2 realms
Seeding method	`partialImport` (`bin/seed-realm.sh --profile typical --users 200`)
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`

Started from an empty pgdata volume so no public schema objects existed to confuse attribution.

Results

Step	Outcome
Start with `KC_DB_SCHEMA=kc`, schema absent	exit 1 — `ERROR: schema "kc" does not exist`
`create schema kc`, start again	ready in 16s
Tables in `kc`	87
Tables in `public`	0
Fresh install state	model 26.0.0, changelog 144
Seed 200 users	202 users in `kc.user_entity`
Upgrade to 26.7.1	READY in 16s — model 26.7.1, changelog 211, 202 users
Tables leaked into `public` after the upgrade	0
Admin login after the upgrade	200

The misleading error, and what it means for S8

Starting against a missing schema produces this, in this order:

Caused by: org.postgresql.util.PSQLException: ERROR: schema "kc" does not exist
        at ...QuarkusJpaConnectionProviderFactory.createOrUpdateSchema(...:247)
ERROR: Failed to start server in (production) mode
ERROR: Cannot invoke "org.keycloak.connections.jpa.updater.liquibase.lock.CustomLockService
       .waitForLock(org.keycloak.models.dblock.DBLockProvider$Namespace)"
       because "this.lockService" is null

The last line is the same one 2026-08-25-s8-pgbouncer-transaction-mode recorded, where the cause was a transaction-mode connection pooler and no schema was missing at all.

That reclassifies it. lockService is null is not a diagnosis — it is what Keycloak prints whenever schema initialisation failed for any reason, because the error path dereferences a lock service that was never constructed. The real cause is always further up the log, in a Caused by: that the summary lines at the bottom do not repeat.

This matters operationally because the summary block is what an operator sees first, and it is the least informative part of the failure.

Outcome

Reached ready
Realm config intact
Login flow works
No object leakage into public

Verification

Claim	Primary source	Checked
Keycloak does not create a missing schema	run with `kc` absent: exit 1, `schema "kc" does not exist`	☑
It works once the schema exists	87 tables in `kc`, 0 in `public`, ready 16s	☑
The upgrade is clean on a non-default schema	model 26.7.1, changelog 211, 202 users, login 200	☑
Nothing leaks into `public`	`information_schema.tables` count = 0 for `public`, before and after	☑
`lockService is null` also arises from a missing schema	this run's log, with `schema "kc" does not exist` as the `Caused by`	☑
`snapshot.sh` copes	ran clean, 300K dump	☑
`restore.sh` coped	no — it failed, fixed here, re-verified	☑
Behaviour with a non-default schema and a `search_path` set on the role	not tested — a plausible customer configuration that could mask this	☐
Multiple Keycloak instances in separate schemas of one database	not tested	☐

Publishable extract

Keycloak in a non-default Postgres schema upgrades cleanly. The interesting part is what it says when the schema is missing.

With KC_DB_SCHEMA=kc and the schema created in advance, Keycloak put all 87 of its tables in kc, none in public, and the 26.0.0 → 26.7.1 migration ran in sixteen seconds with 202 users intact and nothing leaking into the default schema.

Keycloak will not create the schema for you. Point it at one that does not exist and it exits — with this at the bottom of the log:
ERROR: Cannot invoke "CustomLockService.waitForLock(DBLockProvider$Namespace)"
       because "this.lockService" is null
We had seen that exact line earlier the same day, from a completely different cause: PgBouncer in transaction pooling mode, with every schema present and correct. lockService is null is not a diagnosis. It is what Keycloak prints whenever it failed to initialise its database connection, because the error path then dereferences a lock service that was never built. The actual reason — ERROR: schema "kc" does not exist — is fifteen lines further up, in a Caused by: that the summary at the bottom never repeats.

If you are staring at that null, scroll up. The summary block at the end of a Keycloak startup failure is the least informative part of it.

Rehearsal Record — S14: the bigger realm is the safer one, and a half-migrated schema resumes

2026-08-25T00:00:00+00:00

Summary

Running out of disk during the migration does break it, and it breaks it at one specific place: CREATE INDEX IDX_USER_CREATED_TIMESTAMP, which needs a temporary sort file that will not fit.

Three things make this the most interesting result of the set.

The amount of headroom needed is tiny. A 224 MB database needed somewhere between 4 MB and 8 MB free. At 8 MB the upgrade completed and used 1.58 MB; at 4 MB it died.

It leaves a genuinely half-applied changelog — DATABASECHANGELOG stops at 187 of 211 rows, committed, with migration_model still 26.0.0. Freeing the space and restarting completed it in 17 seconds, to 211 and 26.7.1, with the index built and all 100,002 users intact.

And the failure only happens on the smaller realm.

⚠ QUALIFIED 2026-08-26. True only while Keycloak is declining the index build. Force the build at 2,000,003 users — which a stale pg_class.reltuples does on its own — and the floor is 125–150 MB, roughly 20× the figure below. Scaling up removes the trigger, not the failure mode, and multiplies the damage if anything pulls it. See 2026-08-26-s14-revisited-disk-floor-at-2m. The index that runs out of space is the same one 2026-08-25-index-skip-threshold shows Keycloak skipping above 300,000 rows. At 100k it is built, and needs temp space. At 2M it is declined, and needs none. Scaling the realm up removes this failure mode. That is the opposite of what anyone provisioning a maintenance window would assume.

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB
Database	PostgreSQL 16.15, single node, container
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0
Adverse scenario	S14 — free space on the Postgres data directory is the variable
Storage	2 GB ext4 loopback filesystem mounted at `/mnt/tinypg`, bind-mounted as `pgdata`
Dataset scale	100,002 users, 2 realms; database 223–225 MB
Seeding method	`partialImport` (restored from `baseline-26.0.0-typical-100k`)
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`
Lab change	`docker-compose.smalldisk.yml` added

The box's root filesystem is 226 GB, so exhaustion is not reachable there. Free space at the moment of migration is set exactly, with a ballast file, so each run states a real number rather than "nearly full".

Results

Free space	Outcome	Elapsed	Space used	`migration_model`	`DATABASECHANGELOG`	Users
25 MB	ready	17s	~1 MB	26.7.1	211	100,002
8 MB	ready	17s	1,580 KB	26.7.1	211	100,002
4 MB	exit 1	14s	512 KB	26.0.0	187	100,002
2 MB	exit 1	14s	504 KB	26.0.0	187	100,002
1 MB	exit 1	14s	344 KB	26.0.0	187	100,002

The floor is between 4 MB and 8 MB free, for a 224 MB database.

Where it dies, and why

ERROR: could not write to file "base/pgsql_tmp/pgsql_tmp705.0.fileset/0.0":
       No space left on device
[Failed SQL: (0) CREATE INDEX IDX_USER_CREATED_TIMESTAMP ON public.USER_ENTITY(REALM_ID, CREATED_TIMESTAMP)]

base/pgsql_tmp is Postgres's temporary-file area. Building an index sorts, and a sort that exceeds maintenance_work_mem (64 MB here, stock) spills to disk. The space that runs out is not the space the finished index occupies — it is the scratch space needed to build it.

That is why the margin is so small and so easy to miss: the finished index is about 2 MB, and the sort needs more room than the result.

Why a larger realm would not hit this

CustomCreateIndexChange skips index creation on tables above 300,000 rows (2026-08-25-index-skip-threshold). This realm has 100,002 users, so the index is built — the run confirms it, with skipped_idx=0 in every leg, and the index present after recovery.

At 2M users Keycloak declines to build it, logs a WARN, marks the changeset EXECUTED, and moves on. No sort, no temp file, no exhaustion. The dangerous size band for this failure is below the skip threshold, and a customer who manually created the indexes Keycloak declined has re-entered it at any size.

Recovery: the half-migrated schema resumes

From the failed state — DATABASECHANGELOG 187, migration_model 26.0.0 — removing the ballast and restarting:


Restart	ready in 17s
`DATABASECHANGELOG`	187 → 211
`migration_model`	26.0.0 → 26.7.1
`idx_user_created_timestamp`	present
Users	100,002
Admin login	200

Liquibase resumed from the last committed changeset. No manual intervention, no lock clearing, no restore.

This is the third route this lab has found to a stuck upgrade (after S8's pooler and S12's database floor) and the second to a genuinely inconsistent database — and in both cases the recovery was "fix the environmental cause and start it again". 2026-08-25-s1-s2-lock-failure-modes reached that conclusion without ever producing a half-migrated schema. It has now been produced twice, and the conclusion held both times.

Verification

Claim	Primary source	Checked
8 MB free is sufficient for a 224 MB database	run: ready 17s, changelog 211, 1,580 KB consumed	☑
4 MB free is not	run: exit 1, changelog 187	☑
The failing statement is the index build	`Failed SQL: CREATE INDEX IDX_USER_CREATED_TIMESTAMP` in three runs	☑
The space that runs out is temp sort space	Postgres names `base/pgsql_tmp/...`	☑
The changelog is genuinely half applied	187 of 211, read from Postgres, across three runs	☑
Recovery is a restart once space exists	changelog 187 → 211, model → 26.7.1, index present, login 200	☑
The index is skipped above 300k rows	`2026-08-25-index-skip-threshold`; `skipped_idx=0` at 100k here	☑
That a 2M realm therefore avoids this	not directly tested — inferred from the two records; worth one run to close	☐
Behaviour with a larger `maintenance_work_mem`	not tested — a bigger setting keeps the sort in RAM and may remove the failure entirely	☐
Behaviour when WAL, not the data directory, fills	not tested — separate filesystem in most real deployments	☐
Whether a full disk (0 bytes) corrupts rather than errors	not tested — the tightest run still had 676 KB	☐

Publishable extract

Your Keycloak upgrade can run out of disk with 4 MB free and succeed with 8. And the bigger your realm, the less likely it is to happen.

We put a 100,002-user Keycloak database — 224 MB — on a 2 GB filesystem and upgraded 26.0.0 → 26.7.1 with the free space set to an exact figure each time. With 8 MB free it completed in 17 seconds and consumed 1.58 MB. With 4 MB it died:
ERROR: could not write to file "base/pgsql_tmp/...": No space left on device
[Failed SQL: (0) CREATE INDEX IDX_USER_CREATED_TIMESTAMP ON public.USER_ENTITY(REALM_ID, CREATED_TIMESTAMP)]
base/pgsql_tmp is the giveaway. The space that ran out is not the space the index occupies — it is the temporary sort file Postgres needs to build it, which is larger than the result. Free space equal to your index size is not enough.

Now the counter-intuitive part. That index is the one Keycloak refuses to build on tables over 300,000 rows — it logs a warning, marks the changeset executed, and moves on. So a 2-million-user realm never runs that sort, never writes that temp file, and never hits this. The vulnerable realms are the small ones, and anyone who has manually created the indexes Keycloak declined has put themselves back in range at any size.

Finally, the reassuring half. The failure left a genuinely half-applied schema — 187 of 211 changesets committed, migration_model still on the old version. We freed the space and started Keycloak again. Seventeen seconds later: changelog 211, model 26.7.1, index built, all 100,002 users present, admin login returning 200. Liquibase resumed from the last committed changeset.

If you find a Keycloak stopped partway through a migration, the first thing to try is starting it again — after you have fixed whatever stopped it. That advice has now survived every way we have found to break one.

Rehearsal Record — S15: upgrading does not hide custom attributes; re-creating the realm discards them

2026-08-25T00:00:00+00:00

Summary

S15's hypothesis was wrong, and the truth is worse.

The hypothesis: upgrading across 23 → 24, where the declarative user profile became the default, makes existing custom attributes invisible to the admin API — a data-visibility regression triggered purely by upgrading.

It does not. A realm created on 23.0.7 with users carrying department and employeeId shows those attributes through the admin API on 23.0.7, on 24.0.5 and on 26.7.1, unchanged at every hop. An in-place upgrade preserves them, because the migration sets that realm's unmanagedAttributePolicy to ENABLED.

What actually loses attributes is creating a realm on 24+. Such a realm has no unmanagedAttributePolicy, and then the admin API:

silently discards custom attributes on write — HTTP 201, and zero rows reach the database; and
hides any that reach the database by another route — 2 rows present, attributes: null returned.

So the danger is not the upgrade. It is a realm created on a new version.

⚠ CORRECTED 2026-08-26. This paragraph originally read "every path that produces a new realm on a new version: export/import migration, a rebuilt environment, disaster recovery, or simply a realm created after the upgrade." That generalisation is wrong. Run C exported a 23.0.7 realm and imported it into 24.0.5, 25.0.6, 26.0.0 and 26.7.1: all six USER_ATTRIBUTE rows survive, the admin API returns them, and the realm still accepts new ones — because the import sets unmanagedAttributePolicy=ENABLED, exactly as the in-place migration does. See 2026-08-26-runC-realm-export-import-23-to-24.

The path measured below — a realm created through the admin API — is the one that loses attributes. The measurements in this record stand; the inference from them to a class of untested paths did not.

Environment

Field	Value
Keycloak path	23.0.7 → 24.0.5 → 26.7.1 (in-place, real migrations)
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB
Database	PostgreSQL 16.15, single node, container
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0
Adverse scenario	S15
Dataset scale	small — 3 attribute-carrying users, plus SQL-inserted controls; 2 realms, later 3
Seeding method	`partialImport` (`--profile minimal`), admin API user creation, and direct SQL inserts as a control
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`

Results

The upgrade path: no regression at any hop

Users created on 23.0.7 with attributes: {department, employeeId}:

	`USER_ATTRIBUTE` rows	Admin API returns
23.0.7 (before)	6 (2 per user × 3)	`{"department":["engineering"],"employeeId":["E001"]}`
24.0.5 (after the hop that matters)	6	`{"department":["engineering"],"employeeId":["E001"]}`
26.7.1	6	`{"employeeId":["E001"],"department":["engineering"]}`

Migration timings: 23.0.7 → 24.0.5 in 18s, 24.0.5 → 26.7.1 in 17s.

Two controls ruled out the other candidate explanations. Users inserted by direct SQL — the way seed-sql.sh does it, with long_value_hash left NULL — returned their attributes through the admin API on 26.7.1 just as admin-API-created users did, both before and after a restart. The seeding method is not the variable, and neither is the hash column.

The realm's birth version is the variable

On the same 26.7.1 server, side by side:

	realm `lab` (created on 23.0.7, upgraded in place)	realm `born26` (created on 26.7.1)
`unmanagedAttributePolicy`	`ENABLED`	UNSET
Create user with attributes	—	HTTP 201
`USER_ATTRIBUTE` rows written	10	0
Admin API returns	`{"employeeId":…,"department":…}`	`attributes: null`

The user creation succeeded. It returned 201. The attributes were simply not persisted, and nothing said so.

The policy governs both sides, independently

Toggling unmanagedAttributePolicy on born26 and re-testing:

Policy	Write through the admin API	Rows in DB after	Read back
UNSET	`HTTP 201` — accepted	0	`attributes: null`
`ENABLED`	`HTTP 204` — accepted	2	`{"employeeId":["E001"],"department":["engineering"]}`
back to UNSET, rows left in place	—	2	`attributes: null`

The last row is the S15 scenario as originally described: the data is present and invisible. The middle row shows the fix. And the first row shows the part nobody warns about — with the policy unset, the write is a no-op that reports success.

Setting the policy to ENABLED does not recover attributes that were discarded earlier. There is nothing to recover.

Verification

Claim	Primary source	Checked
Attributes survive 23.0.7 → 24.0.5	admin API returns them on both sides; 6 rows in DB throughout	☑
And survive on to 26.7.1	same	☑
An upgraded realm has `unmanagedAttributePolicy=ENABLED`	`/admin/realms/lab/users/profile`	☑
A realm created on 26.7.1 has it unset	`/admin/realms/born26/users/profile`	☑
With it unset, an admin API write silently discards	HTTP 201, `select count(*) from user_attribute` = 0	☑
With it unset, existing rows are hidden	2 rows present, `attributes: null`	☑
With it `ENABLED`, both write and read work	HTTP 204, 2 rows, attributes returned	☑
Seeding method is not the cause	SQL-inserted users returned attributes on 26.7.1, hash column NULL and populated alike	☑
Keycloak's migration is what sets `ENABLED`	inferred from the two realms' policies, not read from source or release notes	☐
Behaviour of realm export/import across the boundary	not tested — the most likely real-world path to this, and the obvious next run	☐
Whether tokens/mappers see unmanaged attributes	not tested — this record covers the admin API only	☐
`ADMIN_VIEW` / `ADMIN_EDIT` policy values	not tested — only UNSET and `ENABLED`	☐

Publishable extract

Upgrading Keycloak does not hide your custom user attributes. Rebuilding the realm deletes them, and tells you it worked.

Keycloak 24 made the declarative user profile the default, and the widely repeated warning is that custom attributes go invisible when you cross that boundary. We tested it: a realm created on 23.0.7 with users carrying department and employeeId, upgraded in place to 24.0.5 and on to 26.7.1. The attributes were returned by the admin API at every hop, unchanged. The in-place upgrade sets that realm's unmanagedAttributePolicy to ENABLED, and everything keeps working.

Then we created a new realm on 26.7.1 and posted the identical user:
POST /admin/realms/born26/users   →  HTTP 201
select count(*) from user_attribute where user_id = ...  →  0
GET  /admin/realms/born26/users?username=...   →  "attributes": null
The request succeeded. The attributes were never written. Nothing warned us. A realm created on 24 or later has no unmanagedAttributePolicy, and with it unset the admin API accepts custom attributes and drops them on the floor.

The same policy hides rows that got there another way. We inserted attributes straight into Postgres, confirmed two rows, and the admin API still answered attributes: null. Turning the policy on makes both problems go away — but it does not bring back what was discarded, because that was never stored.

So the risk is not the upgrade. It is every path that produces a new realm on a new version: export-and-reimport migrations, environment rebuilds, disaster recovery, and any realm created after the upgrade. Those are precisely the paths people take because they consider them safer than an in-place upgrade.

One query tells you which kind of realm you have:
GET /admin/realms/{realm}/users/profile   →   .unmanagedAttributePolicy
If it is absent, your custom attributes are not being stored.

We nearly published the opposite of this. Our own large test realms were built on 26.0.0 and showed exactly the attributes: null symptom the folklore predicts — and we had written it down as an upgrade regression before checking where the realm came from.

Rehearsal Record — S3: `statement_timeout` cannot half-migrate this path

2026-08-25T00:00:00+00:00

Summary

⚠ CONCLUSION OVERTURNED 2026-08-26. It does produce one. At 2,000,003 users a statement_timeout of 500 ms, 200 ms or 50 ms leaves DATABASECHANGELOG at 157 of 211 with migration_model on 26.0.0 and the server dead — seven times in ten identical runs.

The measurements below are correct and the reasoning from them is not. The statement that dies is ALTER TABLE CREDENTIAL ADD VERSION INTEGER DEFAULT 0, which executes in 0.177 ms and waits about a second for an AccessExclusiveLock held by an autovacuum worker — Postgres evicts a blocking autovacuum only after deadlock_timeout, and statement_timeout is charged for the wait. This record measured statement durations on a single run that happened to be uncontended.

See 2026-08-26-s3-revisited-autovacuum-lock-wait. Everything below stands as a measurement of an uncontended migration; nothing below should be quoted as a conclusion about statement_timeout.

S3 was written to produce the half-migrated schema our failure-class notes name. It does not produce one. On this path, at 2,000,003 users, the longest single SQL statement the migration issues is 2.142 ms. There is no statement_timeout value that lands between "too short to start" and "long enough to finish": at 2 ms Keycloak fails before applying any changeset, at 3 ms it completes the whole migration. The band that would leave a schema half-applied is empty.

This is the second scenario in a row (after S1/S2) that fails to reproduce a failure class we have been citing.

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB
Database	PostgreSQL 16.15 (Debian 16.15-1.pgdg13+2), single node, container
Postgres tuning	`shared_buffers` 128MB, `maintenance_work_mem` 64MB — stock; `statement_timeout` is the variable, 10s / 3ms / 2ms
Adverse scenario	S3
Dataset scale	2,000,003 users, 2 realms
Seeding method	direct SQL COPY (`bin/seed-sql.sh`)
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`
Instrumentation	`log_min_duration_statement = 0` for the measurement run

Procedure

./bin/restore.sh baseline-26.0.0-sqlseed-2m
docker compose exec -T db psql -U keycloak -c "alter system set log_min_duration_statement=0" -c "select pg_reload_conf()"
./bin/upgrade.sh 26.7.1                       # measurement run
docker compose logs db | grep 'LOG:  duration:' | sort -gr   # rank statements

# then, per timeout value:
./bin/restore.sh baseline-26.0.0-sqlseed-2m
docker compose exec -T db psql -U keycloak -c "alter role keycloak set statement_timeout='<T>'"
./bin/upgrade.sh 26.7.1

What we could not observe

No half-migrated schema, at any timeout value. The empty band is a consequence of every statement in the migration falling inside 1.8–2.2 ms: the parse-phase reads and the DDL writes are the same order of magnitude, so a timeout tight enough to kill a changeset is also tight enough to kill the parse that precedes it. That coincidence is a property of this pair at this scale, not a general law, and we did not test whether it holds elsewhere.

Claim	Primary source	Checked
Longest migration statement is 2.142 ms at 2M users	`log_min_duration_statement=0` output, ranked	☑
10s timeout has no effect on this path	full run, changelog 211, 2M users intact	☑
2ms fails before applying any changeset	changelog still 144, model still 26.0.0	☑
Failure leaves no held lock	`databasechangeloglock` read directly, both rows `f`	☑
A half-migrated schema is reachable via timeout	no — refuted for this pair at this scale	☑
Holds on a pair with data-rewriting changesets	not tested — this pair has none	☐
Holds if the customer built the skipped indexes manually	not tested — the obvious next run	☐
Holds on other databases	not tested — Postgres only	☐

Rehearsal Record — S4: it is not DDL rights that the migration needs, it is ownership

2026-08-25T00:00:00+00:00

Summary

S4 asked whether a DML-only database account fails cleanly. It does. But the run answered a more useful question than the one asked: granting that account full DDL rights does not fix it. The first changeset in this upgrade issues DROP INDEX, and in Postgres DROP INDEX requires ownership of the index — a thing no GRANT confers.

The correct pre-flight check is therefore not "does the Keycloak database user have DDL permissions", which is what everyone asks. It is "does the Keycloak database user own the Keycloak objects".

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB
Database	PostgreSQL 16.15, single node, container
Postgres tuning	`shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0 — stock
Adverse scenario	S4 — the database role Keycloak connects as is the variable
Dataset scale	1,002 users, 2 realms
Seeding method	`partialImport` (`bin/seed-realm.sh --profile typical`)
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`
Lab change	`docker-compose.dbidentity.yml` added so `KC_DB_USERNAME` / `KC_DB_PASSWORD` / `KC_DB_SCHEMA` are env-driven without editing the base compose file

Procedure

./bin/restore.sh baseline-26.0.0-typical-1k

# a locked-down enterprise account: DML on everything, DDL on nothing
create role kc_dml login password 'kc_dml';
grant connect on database keycloak to kc_dml;
grant usage on schema public to kc_dml;
grant select,insert,update,delete on all tables in schema public to kc_dml;
grant usage,select on all sequences in schema public to kc_dml;
revoke create on schema public from kc_dml, public;

# .env
COMPOSE_FILE=docker-compose.yml:docker-compose.dbidentity.yml
KC_DB_USERNAME=kc_dml
KC_DB_PASSWORD=kc_dml

./bin/upgrade.sh 26.7.1

Then two further runs, each changing exactly one thing: grant create on schema public to kc_dml (run B), and alter table … owner to kc_dml for every table and sequence (run C).

Results

Run	Role's rights	Outcome	Elapsed	Schema after	Users
A	DML only, no `CREATE`	exit 1, refused to start	died at 182s	model 26.0.0, changelog 144 (untouched)	1,002
B	DML + `CREATE` on schema (verified: can `CREATE TABLE`)	exit 1, identical failure	died at 151s	model 26.0.0, changelog 144 (untouched)	1,002
C	owner of every table and sequence	READY, clean	17s	model 26.7.1, changelog 211	1,002

The failure, identical in runs A and B

ERROR [liquibase.changelog.ChangeSet] ChangeSet
  META-INF/jpa-changelog-20.0.0.xml::20.0.0-12964-supported-dbs-edb-migration::keycloak
  encountered an exception.: liquibase.exception.DatabaseException:
  ERROR: must be owner of index idx_group_att_by_name_value
  [Failed SQL: (0) DROP INDEX public.IDX_GROUP_ATT_BY_NAME_VALUE]
ERROR: Failed to update database
ERROR: Failed to start server in (production) mode

Run B is the load-bearing one. kc_dml demonstrably had DDL rights — the probe create table ddl_probe(x int); drop table ddl_probe; succeeded as that role immediately before the run — and the migration failed at exactly the same changeset with exactly the same message. must be owner of is not a privilege error and cannot be granted away. Postgres reserves DROP INDEX, and ALTER TABLE, to the object's owner (or a superuser, or a member of the owning role).

Why it fails on the first changeset

The 26.0.0 baseline carries 144 changelog rows. Reading them back after the successful run C:

id	exectype	orderexecuted
`20.0.0-12964-supported-dbs`	EXECUTED	109
`20.0.0-12964-unsupported-dbs`	MARK_RAN	110
`20.0.0-12964-supported-dbs-edb-migration`	EXECUTED	145

Order 145 is the first row past the 144-row baseline. The very first thing a 26.0.0 → 26.7.1 migration does is drop and rebuild IDX_GROUP_ATT_BY_NAME_VALUE, via a changeset that lives in the 20.0.0 changelog file and exists to fix up EDB deployments. So the ownership requirement bites immediately — which is the good news, because nothing else has been applied when it does.

Outcome

Failed cleanly, before touching data — databasechangelog still 144, all 1,002 users present, both databasechangeloglock rows locked=f
Container exited 1 and stayed exited (one attempt, no retry loop)
503 from BootstrapFilter while the failed bootstrap was in flight
Recoverable by fixing ownership alone — no restore needed

Verification

Claim	Primary source	Checked
DML-only account fails the migration	run A, exit 1, changelog 144	☑
Granting `CREATE` does not fix it	run B, same changeset, same message, after a successful `CREATE TABLE` probe as that role	☑
Ownership does fix it	run C, ready in 17s, changelog 211	☑
Failure applies nothing	changelog 144 and 1,002 users after both failed runs	☑
The blocking changeset is the first pending one	`orderexecuted = 145` against a 144-row baseline	☑
Holds on other version pairs	not tested — a pair with no `DROP INDEX`/`ALTER TABLE` in its delta may not hit this	☐
Holds on MySQL / MariaDB / Oracle	not tested — Postgres only; ownership semantics differ per engine	☐

Publishable extract

Everyone asks whether the Keycloak database user has DDL permissions. That is the wrong question.

We gave Keycloak a locked-down Postgres account — select, insert, update, delete on every table, no create — and upgraded a 1,002-user realm from 26.0.0 to 26.7.1 on Postgres 16.15. It failed, cleanly, having applied nothing:
ERROR: must be owner of index idx_group_att_by_name_value
[Failed SQL: (0) DROP INDEX public.IDX_GROUP_ATT_BY_NAME_VALUE]
Then we granted that same account CREATE on the schema and proved it could create and drop a table. The upgrade failed again, at the same changeset, with the same message. Only making the account the owner of the tables let the migration run — after which it completed in 17 seconds.

The reason is plain Postgres, not Keycloak: DROP INDEX and ALTER TABLE require ownership, and ownership is not a privilege you can GRANT. Any shop where a DBA creates the schema and the application connects as a different role — which is most shops with a change-control process — has a Keycloak upgrade that will stop dead on its first changeset.

The check to run before your maintenance window is one query:
select tablename, tableowner from pg_tables
where schemaname = 'public' and tableowner <> '<your keycloak role>';
If that returns rows, your upgrade has not been tested. Ours hadn't either.

Rehearsal Record — S5: the heap floor does not move with realm size

2026-08-25T00:00:00+00:00

Summary

S5 asked us to "establish the heap floor per realm size — a number customers can be given directly." There is no per-realm-size floor on this path. A 1,002-user realm and a 2,000,003-user realm have the same floor, bracketed identically between 128 MB and 160 MB, because the thing that runs out of memory is Quarkus augmentation — the auto-build that happens before the migration starts — and augmentation does not care how many users exist.

The lab's own compose file pins -Xmx4g. The measured floor is 160 MB. We have been provisioning 25× the heap the run needs, and had assumed the multiple was there for the data.

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB, auto-build on start (no `--optimized`)
Database	PostgreSQL 16.15, single node, container
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0
Adverse scenario	S5 — `-Xmx` is the variable
Dataset scale	two runs: 1,002 users / 2 realms, and 2,000,003 users / 2 realms
Seeding method	`partialImport` (1k) and direct SQL COPY (2M)
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms64m -Xmx<variable>`; lab default is `-Xms1g -Xmx4g`
Lab change	`docker-compose.heap.yml` added so `JAVA_OPTS_APPEND` is env-driven, defaulting to the base file's `-Xms1g -Xmx4g`

Procedure

# .env
COMPOSE_FILE=docker-compose.yml:docker-compose.heap.yml

for H in 256m 192m 160m 128m 64m; do
  ./bin/restore.sh <baseline>              # at the lab default heap -- see note
  JAVA_OPTS_APPEND="-Xms64m -Xmx$H"
  docker compose up -d --force-recreate keycloak
  TIMEOUT=240 ./bin/wait-ready.sh
done

Note, and a trap worth recording: the heap must be reset to a working value before restore.sh, not after. restore.sh starts Keycloak as part of its re-migration check, so a heap too small to boot makes the restore fail, and the next iteration then measures the wrong baseline. The first attempt at this ladder was lost to exactly that.

Results

`-Xmx`	1,002 users	2,000,003 users
512m	—	ready, 16s
256m	ready, 17s	ready, 16s
192m	ready, 17s	—
160m	ready, 17s	ready, 17s
128m	OOM, died at 241s	OOM, died at 241s
64m	OOM, died at 242s	OOM, died at 242s

Every successful run: model 26.7.1, changelog 211, full user count intact. Every failed run: model 26.0.0, changelog 144 — untouched — full user count intact, container exited.

The floor is (128m, 160m] at both scales. Time to ready is 16–17 s at both scales too, which is the flat-migration result from 2026-08-25-26.0.0-to-26.7.1-100k holding again at 2M.

Where it actually dies

The last phase reached, from the container log:

`-Xmx`	Last phase before death
128m, 64m	`Updating the server image`
160m and above	`Updating database` (i.e. it got to the migration)

Appending additional Java properties to JAVA_OPTS
Changes detected in configuration. Updating the server image.
Updating the configuration and installing your custom providers, if any. Please wait.
Terminating due to java.lang.OutOfMemoryError: Java heap space

Updating the server image is Quarkus augmentation — the build step Keycloak re-runs on start whenever it detects a configuration change. It happens before a single row of the database is read, which is why the number is identical at 1k and 2M, and why no failed run touched the schema.

The failure is slow

Both OOM cases took just over four minutes to die (241 s, 242 s), not seconds. The JVM thrashes the collector at the ceiling before giving up. In a maintenance window that is four minutes of an operator watching a container that looks like it is working.

Outcome

Failed closed at and below the floor — nothing applied, no data loss, no held lock
Recoverable by raising -Xmx alone; no restore required
Container exits and stays exited (CrashLoopBackOff under Kubernetes)

Verification

Claim	Primary source	Checked
160m is sufficient at 1,002 users	run: ready in 17s, changelog 211	☑
160m is sufficient at 2,000,003 users	run: ready in 17s, changelog 211, 2M users intact	☑
128m fails at both scales	two runs, both OOM, both `cl=144` after	☑
The OOM precedes the migration	last log phase is `Updating the server image`; changelog untouched at 144	☑
Failure applies nothing	changelog 144 and full user count after every failed run	☑
Holds with `--optimized` / a pre-built image	not tested — augmentation is skipped there, so the floor may be lower and may then depend on something else	☐
Holds with custom providers in `/opt/keycloak/providers`	not tested — augmentation installs providers, so a large jar plausibly raises this floor. Direct follow-on from S7	☐
Holds where the migration itself is expensive	not tested — this pair skips index creation above 300k rows	☐
Steady-state serving floor (vs. startup floor)	not measured — this is a startup/migration number only, not a number to run production on	☐

Publishable extract

We tried to find the heap a Keycloak upgrade needs per million users. It does not exist.

On Postgres 16.15, upgrading 26.0.0 → 26.7.1, we walked -Xmx down against two realms — one with 1,002 users, one with 2,000,003 — and the boundary landed in exactly the same place for both: 160 MB works, 128 MB dies with OutOfMemoryError. Not "roughly the same". The same bracket, and the same 16–17 seconds to ready on either side of a 2,000× difference in user count.

The reason is in the log. At 128 MB the last thing Keycloak prints is Updating the server image — Quarkus augmentation, the build step that runs before the database is touched at all. The schema was untouched afterwards: 144 changelog rows before, 144 after, every user present. It never got as far as the data, so the data never mattered.

Two practical consequences. First, if your Keycloak dies on startup with a heap error during an upgrade, stop looking at your user count — you are almost certainly looking at the build step, and the fix is a flat number, not a proportional one. Second, the failure is not fast: both of our OOM runs took just over four minutes to give up, thrashing the collector at the ceiling. If your window budget assumes a heap error announces itself immediately, it does not.

One caveat we cannot yet close: we run without --optimized, so augmentation happens on every start. A deployment with a properly pre-built image skips that phase, and its floor is a different number we have not measured.

Rehearsal Record — S8: transaction pooling breaks the upgrade, and this is what half-migrated looks like

2026-08-25T00:00:00+00:00

Summary

PgBouncer in transaction pooling mode makes the first start after an upgrade fail, deterministically, and it leaves the database in the inconsistent state this lab has spent all day failing to produce by other means.

Three reproductions, three identical failures, in 16–19 seconds each:

ERROR: Failed to start server in (production) mode
ERROR: Cannot invoke "org.keycloak.connections.jpa.updater.liquibase.lock.CustomLockService
       .waitForLock(org.keycloak.models.dblock.DBLockProvider$Namespace)"
       because "this.lockService" is null

The state it leaves behind is the important part. DATABASECHANGELOG is at 211 rows — the migration completed and committed — while migration_model still reads 26.0.0. The schema is on the new version; the model stamp and the model-level migrations are not. A verification that counts changelog rows would call this a successful upgrade.

The same pooler in session mode upgrades cleanly in 18 seconds. The variable is the pooling mode, not the pooler.

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB
Database	PostgreSQL 16.15, single node, container
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0
Pooler	PgBouncer 1.23.1 (`edoburu/pgbouncer:v1.23.1-p2`), `default_pool_size=5`, `max_client_conn=100`, `auth_type=scram-sha-256`
Adverse scenario	S8 — `pool_mode` is the variable
Dataset scale	1,002 users, 2 realms
Seeding method	`partialImport` (`bin/seed-realm.sh --profile typical`)
Fixture profile	`typical`; provider jars removed so S7's fixtures could not confound this
Topology	single container + pgbouncer sidecar
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`
Lab change	`docker-compose.pgbouncer.yml` added; `KC_DB_URL` made env-driven; `restore.sh` hardened

Procedure

./bin/restore.sh baseline-26.0.0-typical-1k

# .env
COMPOSE_FILE=docker-compose.yml:docker-compose.dbidentity.yml:docker-compose.pgbouncer.yml
KC_DB_URL=jdbc:postgresql://pgbouncer:6432/keycloak
POOL_MODE=transaction            # or session, for the control

docker compose up -d pgbouncer
sed -i 's/^KC_VERSION=.*/KC_VERSION=26.7.1/' .env
docker compose up -d --force-recreate keycloak

Connectivity through the pooler was confirmed independently before each run (psql postgresql://keycloak:keycloak@127.0.0.1:6432/keycloak -tAc 'select 1' → 1), so a failure could not be mistaken for the pooler simply not working.

Results

Run	`pool_mode`	Outcome	Time	`migration_model`	`DATABASECHANGELOG`	Users
Control	session	ready	18s	26.7.1	211	1,002
A	transaction	exit 1	—	26.0.0	211	1,002
B (repro)	transaction	exit 1	16s	26.0.0	211	1,002
C (repro)	transaction	exit 1	19s	26.0.0	211	1,002
Restart after A	transaction	ready	14s	26.7.1	211	1,002
Restart again	transaction	ready	33s	26.7.1	211	1,002
Recovery	direct, no pooler	ready	32s	26.7.1	211	1,002

Runs B and C started from a freshly restored 26.0.0 baseline (cl=144) each time, and both reached cl=211 before dying. The failure is not a race — it is what happens every time work has to be done through a transaction-pooled connection.

The order of events

21:05:00  container starts
21:05:10  INFO  QuarkusJpaUpdaterProvider — Updating database
          ... all 67 pending changesets apply and COMMIT (144 → 211) ...
21:05:16  ERROR Cannot invoke CustomLockService.waitForLock(...) because "this.lockService" is null
21:05:16  container exits 1

Liquibase's schema update succeeds. What fails is the step after it, which needs Keycloak's DBLockProvider — a session-scoped lock. Transaction pooling hands out a different backend per transaction, so the session-scoped lock service never gets established, and Keycloak dereferences a null.

It recovers on the second start — through the same pooler

This is the part that makes it dangerous rather than merely broken. The restart succeeded in 14 seconds and stamped migration_model to 26.7.1, still through transaction-mode PgBouncer. Under Kubernetes this presents as one CrashLoopBackOff followed by a healthy pod, which most operators will never look at twice.

For those 16–19 seconds and for however long the restart takes, the database has a 26.7.1 schema and a 26.0.0 model stamp.

No stuck lock

databasechangeloglock showed 0 held locks after every failure. Consistent with 2026-08-25-s1-s2-lock-failure-modes: this lab has still never observed a stuck Liquibase lock. The failure here is the absence of a lock service, not a lock that was taken and never released.

Verification

Claim	Primary source	Checked
Session mode upgrades cleanly through the pooler	control run, ready 18s, model 26.7.1	☑
Transaction mode fails the first start	three runs, three exits with the same NPE	☑
Failure is fast, 16–19s	`docker inspect` StartedAt → FinishedAt on both reproductions	☑
The schema migration commits before the failure	`cl=144` before start, `cl=211` after the failed run	☑
The model stamp does not advance	`migration_model` reads 26.0.0 after each failure	☑
The pooler itself works	`select 1` through port 6432 before every run	☑
Restart recovers, still pooled	ready in 14s, model 26.7.1	☑
No held lock at any point	`databasechangeloglock` read directly after each failure	☑
Data survives	1,002 users after every run	☑
Whether serving (not upgrading) is safe in transaction mode	not tested — this record covers startup and migration only	☐
Whether prepared-statement multiplexing bites separately	not observed — the lock failure happens first and masks anything later	☐
Other poolers (Odyssey, RDS Proxy, Cloud SQL connectors)	not tested	☐
`pool_mode=statement`	not tested — expected worse, not verified	☐

Publishable extract

We finally reproduced a half-migrated Keycloak database. It took a connection pooler, not a crash.

Earlier the same day we tried two textbook routes to the "upgrade died partway and left the schema half-applied" failure. We killed Keycloak with SIGKILL mid-migration; it completed itself on the next start. We aborted its SQL with a statement_timeout; it refused to apply anything at all. Neither produced the state everyone warns about.

Then we put PgBouncer in transaction pooling mode in front of Postgres — the default mode nearly everywhere, because it is the mode that actually saves connections — and upgraded 26.0.0 → 26.7.1 on a 1,002-user realm. Three times out of three, Keycloak died 16 to 19 seconds in:
ERROR: Cannot invoke "CustomLockService.waitForLock(DBLockProvider$Namespace)"
       because "this.lockService" is null
Keycloak's DBLockProvider needs a session-scoped lock. Transaction pooling gives you a different backend for every transaction, so the lock service is never established and the server dereferences a null.

Here is the state it left, and why it matters more than the crash:

DATABASECHANGELOG 211 rows — the full migration, committed

migration_model 26.0.0 — unchanged

The schema moved. The version stamp did not. If your upgrade verification counts changelog rows or diffs the schema, it will tell you this upgrade succeeded. The only column that knows the truth is migration_model.

Two more things worth your time. The same pooler in session mode upgraded cleanly in 18 seconds — the pooler is not the problem, the pooling mode is. And the second start succeeded, in 14 seconds, still through transaction-mode PgBouncer. Under Kubernetes that is one CrashLoopBackOff followed by a healthy pod, which nobody investigates.

Before your window: ask what pool_mode your pooler runs, and point Keycloak at the database directly for the upgrade. Afterwards, check select version from migration_model order by update_time desc limit 1 — not the changelog.

Rehearsal Record — S9: LDAP federation survives the upgrade, and the fixture that claimed to test it did not exist

2026-08-25T00:00:00+00:00

Summary

Two results, and one of them is about this lab rather than about Keycloak.

The Keycloak result: LDAP user federation crosses 26.0.0 → 26.7.1 intact. Component config preserved verbatim, 3 federated users still linked, a fresh full sync still works afterwards, zero LDAP-related errors in the log, and a federated user gets a token from a password validated against LDAP.

The lab result: before this run, seed-realm.sh --ldap pointed at ldap://ldap:389, and no compose file in this repository defined a host called ldap. The kitchen-sink profile therefore created a federation component aimed at nothing, and every record produced with it carried an unticked Federation/brokering intact box that could never have been ticked. That is now fixed, and the fix took four separate corrections — none of which were the port number this scenario predicted.

Bonus finding, which resolves an open question from S7: the token endpoint's status for invalid_grant changed from 401 on 26.0.0 to 400 on 26.7.1.

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB
Database	PostgreSQL 16.15, single node, container
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0
Directory	OpenLDAP 2.6 (`bitnamilegacy/openldap:2.6`), `dc=lab,dc=invalid`, 3 users under `ou=people`
Adverse scenario	S9
Dataset scale	1,002 local users + 3 LDAP-federated, 2 realms
Seeding method	`partialImport`, plus a full LDAP sync
Federation config	`READ_ONLY`, `importEnabled=true`, `usernameLDAPAttribute=uid`, `uuidLDAPAttribute=entryUUID`, `searchScope=2` — byte-for-byte what `seed-realm.sh --ldap` writes
Topology	single container + ldap sidecar
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`

Results

Check	26.0.0 (pre)	26.7.1 (post)
LDAP components in realm	1	1
`connectionUrl` stored	`ldap://ldap:1389`	`ldap://ldap:1389`
`editMode` stored	`READ_ONLY`	`READ_ONLY`
Users with `federation_link`	3	3
`alice` present	yes	yes
Full sync	`3 imported, 0 updated`	`0 imported, 3 updated`
LDAP errors in log	0	0

Upgrade elapsed: 17s. Nothing about federation slowed or blocked it.

Proving the LDAP bind is actually evaluated

A federation component that is merely present proves nothing — that was the whole problem with the old fixture. So each side was probed with a wrong password and a correct one:

Probe	26.0.0	26.7.1
`alice` / wrong password	401 `Invalid user credentials`	400 `Invalid user credentials`
`alice` / correct password	400 `Account is not fully set up`	200 TOKEN ISSUED

The two answers differ, which is the point: Keycloak is genuinely binding to LDAP and evaluating the result. Account is not fully set up is raised after successful credential validation — the realm's typical profile makes CONFIGURE_TOTP a default required action, so every freshly imported user carries one.

The post-upgrade TOKEN ISSUED is the first time this lab has verified a federated login end to end. It also exposes a detail worth keeping: clearing user_required_action by direct SQL did not take effect until the server restarted, because Keycloak caches the user. The pre-upgrade probe still said "not fully set up" after the delete; the post-upgrade one issued a token. Do not edit user state under a running Keycloak and expect it to be seen.

The `invalid_grant` status changed: 401 → 400

Same request, same realm, same wrong password. 26.0.0 answers 401. 26.7.1 answers 400. The body is Invalid user credentials in both.

2026-08-25-s7-custom-spi recorded this shift and could only rule out one cause (a broken event listener). This run isolates it properly: it appears on a clean realm with no custom providers loaded, and it tracks the version. RFC 6749 specifies 400 for invalid_grant, so the newer behaviour is the correct one — but any client keying on 401 to detect a bad password breaks on upgrade, and that code is common. The exact release between 26.0.0 and 26.7.1 that changed it was not identified.

Outcome

Reached ready
Realm config intact — component and all config values preserved
Login flow works — federated login, token issued post-upgrade
Federation/brokering intact — first time this box has been truthfully ticked

Verification

Claim	Primary source	Checked
Component config survives the upgrade	`component` / `component_config` read from Postgres, before and after	☑
Federated users survive	`federation_link is not null` count = 3 on both sides	☑
Sync still works after the upgrade	admin API returns `0 imported users, 3 updated users`	☑
The LDAP bind is genuinely evaluated	wrong vs. correct password give different answers on both versions	☑
A federated user can obtain a token on 26.7.1	`http=200`, `access_token` present	☑
`invalid_grant` status changed 401 → 400	two independent runs (here and S7), clean realm, tracks the version	☑
Which release changed it	26.6.0 — see `2026-08-26-invalid-grant-401-to-400-narrowed`	☑
A federated user can obtain a token on 26.0.0	not shown — blocked by a cached required action, not by federation	☐
`editMode=WRITABLE` / `UNSYNCED`	not tested — only `READ_ONLY`	☐
LDAPS / StartTLS	not tested — plaintext 1389 only, and TLS is where real deployments break	☐
Active Directory specifically	not tested — `vendor: other`, and AD differs in `objectGUID`, paging and referrals	☐
Federation at scale	not tested — 3 users; sync cost against 100k LDAP entries is unknown	☐
Kerberos / SPNEGO	not tested	☐

Publishable extract

We upgraded a Keycloak with LDAP user federation from 26.0.0 to 26.7.1 and checked, specifically, that the federation still worked afterwards — rather than that it still appeared to be configured.

It did. Seventeen seconds, component configuration preserved value for value, all three federated users still linked, a fresh full sync returning 0 imported users, 3 updated users, zero LDAP errors in the log, and a federated user obtaining a token with a password validated against the directory.

We checked it the annoying way, because the easy way proves nothing. A federation component that merely exists after an upgrade tells you the row survived, not that anyone can log in. So we probed with a wrong password and a right one and confirmed Keycloak answered differently — which is the only way to know the bind is being evaluated at all.

That distinction is not academic, and here is our own embarrassment as evidence. Until this run, our lab's LDAP fixture pointed at a hostname that no compose file in the repository defined. Every previous run created a federation component aimed at nothing and reported it as configured. When we finally built the thing it was supposed to point at, we found four independent breakages, not one — a missing service, a container image that had been moved to a different namespace and no longer resolved at all, a deprecated environment variable that killed the server with exit code 68 and no error message, and users with no mail attribute so every import produced a null email.

None of those would have failed loudly. An inert fixture never does. If your upgrade test plan contains a step that has never actually run, it is not a test — and it is worth more than the step it replaced, because it reads like coverage.

One incidental finding for anyone writing client code: a bad password returned HTTP 401 on 26.0.0 and HTTP 400 on 26.7.1, with the same Invalid user credentials body. 400 is what RFC 6749 requires, so the change is a fix — but if your client checks for 401 to detect a bad password, it stops working on upgrade, and nothing in the release notes will look like it is about you.

Users	Database	Migration	Time to serving
1,000	13 MB	3.97s	17.0s
100,002	264 MB	3.79s	15.4s
2,000,003	3,512 MB	3.84s	15.6s


`DATABASECHANGELOG`	211 rows — the full migration, committed
`migration_model`	26.0.0 — unchanged

The Keycloak Upgrade Ledger

Rehearsal Record — FGAP-v2 activation path pinned (26.7.3)

Question

Answer (pinned from bytecode + verified live)

Live verification (26.7.3, Postgres, Compose lab)

The v2 permission catalogue (scopes on the admin-permissions client)

What changed vs pre-26.7 (migration-surfacing)

How to audit / drive FGAP v2 in 26.7.3 (the new procedure)

What this unblocks

Verification

Rehearsal Record — Postgres HA: sync replication + clean (promote-while-up) failover

Summary

Environment

Procedure

Results

Sync replication (steady state)

Clean failover (promote-while-up)

What this teaches

Verification

Rehearsal Record — seed-sql.sh ported to MySQL and MariaDB (2M-class seeder)

What was done

Result (verified at 1000 users on both engines)

Observation worth its own note

Remaining

Verification

Rehearsal Record — seed-sql.sh ported to Oracle (CLOB credential + CONNECT BY)

What was done

Result (verified at 100 users, then extension + multi-batch + --no-analyze)

Gotcha worth recording

Verification

Rehearsal Record — Oracle S4: a non-owner "DML-only" account cannot run the migration at all

Summary

Environment

Procedure

Results

Run A failure

Run B failure

Verification

Rehearsal Record — CVE-2026-16093 spot-check: unsigned JWT client assertion rejected

Method

Result

Finding

Still open (need external IdPs / dedicated harness)

Rehearsal Record — CVE-2026-35563 spot-check: LDAP TLS hostname verification at 26.7.3

Environment

Result

Finding

Rehearsal Record — FGAP-v2 / token-exchange spot-check (26.7.3): activation surface changed

What was checked

Findings

FGAP-v2 activation surface changed in 26.7.3

Deferred (need external IdPs / dedicated harness)

Rehearsal Record — remaining version-matrix paths at 26.7.3

Results

Findings

Rehearsal Record — Oracle Data Pump port (snapshot/restore)

What was done

The port

Result

Remaining Oracle item

Rehearsal Record — Oracle index-audit port

What was done

The port

Result

Remaining Oracle tooling

Verification

Rehearsal Record — 21.1.2 → 26.7.3 (stepped ladder, 100k rich)

Environment

Timings

Outcome

Breakage observed

1. idx_client_att_by_name_value genuinely missing (pre-existing in the 21.1.2 baseline)

2. index-audit.py false positive: varchar vs character varying

Rollback

Rehearsal Record — 26.0.0 → 26.7.3 (within-26 direct)

Environment

Procedure

Timings

Outcome

Breakage observed

The v2 permission catalogue (scopes on the `admin-permissions` client)

1. `idx_client_att_by_name_value` genuinely missing (pre-existing in the 21.1.2 baseline)

2. `index-audit.py` false positive: `varchar` vs `character varying`

seed-realm `--user-profile` 400 on 26.7.1

`restore.sh` postgres rollback path was broken (`model_version: command not found`)

`.env` predates the multi-DB fields (`MYSQL_ROOT_PASSWORD: unbound variable`)

Finding 1 — the token-transform work method is in `keycloak-services`, not `server-spi-private`, and it is stable across 25 → 26 ⭐