a rich 21-era realm survives the six-hop ladder to 26.7.1 at 100k
- Upgrade
- 21.1.2 → 26.7.1
- Scale
- 100,006 users (100,000 seed-sql + 5 partialImport + 1 template), 1 realm, 10 clients, 11 client scopes, 8 realm roles, 1 group, 10 auth flows, 9 required actions, 2 IDPs, 6 key providers
- Database
- PostgreSQL 16, single node, container, stock tuning
- Topology
- single container
- Host
- Hetzner CCX33 (fsn1) — 8 vCPU, 30 GB RAM
- Condition
- routine — no adverse condition applied
Why this run existed
Run E proved the schema migrates across five majors but ran at small scale with a partially-applied seed, so it explicitly could not say whether features survive. This run fixes both gaps: a genuinely rich 21-era realm, at 100k users, walked the full six-hop ladder, with a feature inventory captured before and after.
Environment
| Field | Value |
|---|---|
| Keycloak from → to | 21.1.2 → 26.7.1, stepped 22.0.5 → 23.0.7 → 24.0.5 → 25.0.6 |
| Distribution | quay.io/keycloak/keycloak (official image), start w/ external DB |
| Database | PostgreSQL 16, single node, container, stock tuning |
| Dataset scale | 100,006 users (100,000 seed-sql + 5 partialImport + 1 template), 1 realm, 10 clients, 11 client scopes, 8 realm roles, 1 group, 10 auth flows, 9 required actions, 2 IDPs, 6 key providers |
| Seeding method | seed-realm.sh (features) + seed-sql.sh (100k users, direct SQL) — record both |
| Fixture profile | rich 21-era: clients, client-scopes, groups-roles, auth-flows, required-actions, events, themes, localization, lifespans, password-policy, key-providers, idp |
| Topology | single container |
| Host | Hetzner CCX33 (fsn1) — 8 vCPU, 30 GB RAM |
| JVM heap | -Xms1g -Xmx4g |
Result — the ladder
| Hop | Elapsed | Outcome | Index audit |
|---|---|---|---|
| 21.1.2 → 22.0.5 | 21s | READY | 82/82 |
| 22.0.5 → 23.0.7 | 20s | READY | 82/82 |
| 23.0.7 → 24.0.5 | 21s | READY | 86/86 |
| 24.0.5 → 25.0.6 | 15s | READY | 86/86 |
| 25.0.6 → 26.7.1 | 17s | READY | 118/118 |
~94s wall-clock for the whole ladder, every hop clean. (Run E's six-hop ladder at small scale was 106s; the 100k run is no slower, consistent with the index-skip threshold keeping the migration flat.)
Result — feature survival
A diffable inventory (bin/feature-inventory.sh) captured at 21.1.2 and 26.7.1.
Every custom artifact survived unchanged:
- custom themes
lab-login/lab-account/lab-email - 10 clients incl.
app-confidential,spa-public,saml-app,api-bearer-only - custom client scope
lab-profile - group
corp, roleslab-admin/lab-auditor/lab-composite/lab-readonly/lab-user - custom auth flows
lab-browserandlab-registration - all configured required actions (CONFIGURE_TOTP, webauthn-register, …)
- identity providers
corp-oidc(OIDC) andcorp-saml(SAML) - key providers
lab-rsa-secondary,lab-hmac, plus the generated set - users: 100,006 before and after — nothing lost
- credentials: both seeding paths log in post-ladder (HTTP 200) —
sqluser1(seed-sql shared hash) andlabuser1(partialImport)
The only differences are Keycloak's own built-ins changing across the majors, none of which touched the customer's artifacts:
| Change | Direction | Notes |
|---|---|---|
client scopes AuthnContextClassRef, basic | added | new built-ins |
role/client-scope service_account | added | new built-in |
required actions delete_credential, idp_link | added | new built-ins |
component declarative-user-profile | added | new in 24+ |
built-in auth flow http challenge | removed | see below |
Breakage observed
The built-in http challenge flow is gone by 26.7.1
- Symptom: the
http challenge|true|basic-flowflow present on 21.1.2 is absent on 26.7.1. Keycloak removed/absorbed it at some point in the ladder (not narrowed to a hop here — narrowed later: 22.0.0, see2026-08-26-http-challenge-flow-removal-narrowed). - Impact here: none — the lab's custom flows do not reference it, and no customer artifact depended on it.
- Would this hit a customer? Only if a custom flow references the built-in
http challengeflow (e.g. a hardened browser flow that chains it in for SPNEGO or client-cert auth). That reference would go stale. Worth a targeted pre-flight check for customers with custom flows. Resolved: the 22.0.0 migrator leaves a referenced flow in place and logs an ERROR rather than deleting it — the reference goes stale loudly, not silently.
Verification
| Claim | Primary source | Checked |
|---|---|---|
| Six-hop ladder reaches READY at 100k | per-hop upgrade.sh outcome, elapsed | ☑ |
| Index audit clean at every hop | 82/82 → 86/86 → 118/118 | ☑ |
| Every custom feature survives | feature-inventory.sh diff, 21.1.2 vs 26.7.1 | ☑ |
| User count unchanged (100,006) | /users/count before and after | ☑ |
| Credentials functional post-ladder | password-grant login HTTP 200 for both seed paths | ☑ |
| Custom flows and IDPs still resolvable | API reads at 26.7.1 | ☑ |
The exact hop that removed http challenge | narrowed — 22.0.0 (the 21→22 hop). See 2026-08-26-http-challenge-flow-removal-narrowed | ☑ |
A custom flow that references http challenge | answered — MigrateTo22_0_0 leaves it in place and logs an ERROR ("is in use … cannot be removed"); not silent data loss | ☑ |
| Realm export/import equivalence (vs in-place migration) | not compared — in-place ladder only | ☐ |
Publishable extract
We built a Keycloak realm on 21.1.2 with the full surface a real install has — custom themes, ten clients, custom client scopes, groups, composite roles, two custom authentication flows, required actions, an OIDC and a SAML broker, and six key providers — then seeded it to 100,000 users and walked it up the six-hop ladder to 26.7.1, one major at a time. Every hop reached ready; the whole ladder took about 94 seconds; the index audit was clean at every step; and every custom artifact, all 100,006 users, and both credential paths survived unchanged. The only things that changed were Keycloak's own built-ins — new default scopes and required actions, and one built-in flow (
http challenge) that Keycloak removed. If a custom flow references that built-in, check it; ours did not, and nothing else was lost.