2026-08-25 · run 2026-08-25-26.0.0-to-26.7.1-2

26.0.0 → 26.7.1 (instrumented)

Upgrade: 26.0.0 → 26.7.1
Scale: 1000 users · 2 realms · 10 clients · 8 roles
Database: PostgreSQL 16.15, single node
Topology: single container, single replica
Host: Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30GB RAM, local NVMe
Rehearsal: attempt 2 of 3 — this lab holds a path to three clean runs, one exercising rollback
Condition: routine — no adverse condition applied

Environment

Identical to attempt 1, except the management port (9000) is now published to loopback so the health endpoints can be observed. The realm reached 26.0.0 by restoring baseline-26.0.0-typical-1k, so the fixture is byte-identical to attempt 1's starting point rather than merely similar.

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Database	PostgreSQL 16.15, single node
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0. Added retroactively 2026-08-25 to close a gap in our own records: read from the same unmodified `postgres:16` image on the same box. The compose `db` service sets no tuning, so stock is guaranteed by the file, not remembered.
Dataset scale	1000 users · 2 realms · 10 clients · 8 roles
Fixture profile	`typical`
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30GB RAM, local NVMe
JVM heap	`-Xms1g -Xmx4g`

Procedure

Two experiments.

A — does migration require a client request? Restore 26.0.0, swap the image to 26.7.1, start it, and issue zero requests for 120 seconds. Measure the schema from Postgres directly, not from the application.

B — what does the server serve during migration? Same restore and swap, then poll /health/started, /health/ready (port 9000) and the master-realm token endpoint (port 8080) continuously from T0, printing only state transitions.

Findings

1. Schema migration runs unprompted — CONFIRMED

With no client contact whatsoever for 120s:

	before	after
`DATABASECHANGELOG` rows	144	211
latest changeset	`26.0.0-33201-org-redirect-url`	`26.7.0-cluster-event`
`MIGRATION_MODEL`	26.0.0	26.7.1

This settles attempt 1's open question. Migration is performed on executor-thread-1 after startup completes, and is not triggered by a first inbound request — the lazy-trigger hypothesis is refuted. Measured from Postgres, so it does not depend on trusting the application's own logging.

2. Timeline (T0 = container create, 15:48:19 UTC)

Elapsed	Event
0.8s	port 8080 not yet listening — connection refused
11.4s	Quarkus: `started in 3.499s. Listening on: http://0.0.0.0:8080`
11.4s	`/health/started` 000 · `/health/ready` 503 · token 503
11.5s	`/health/started` → 200 · ready 503 · token 503
12.2s	`Updating database. Using changelog META-INF/jpa-changelog-master.xml`
16.2s	`Migrating older model to 26.7.0` (last of six model steps)
16.6s	started 200 · ready 503 · token → 200
17.0s	started 200 · ready → 200 · token 200 — fully serving

T0 includes container recreation, which is why 11.4s elapses before Quarkus's own self-reported 3.499s startup.

3. `/health/ready` is honest. `/health/started` is not a sufficient gate.

/health/ready returned 503 for the entire migration window and only flipped to 200 at 17.0s, after migration completed. A readiness probe pointed at it behaves correctly.
/health/started returned 200 at 11.5s — during migration, ~5.5s before the server was actually able to serve.
Port 8080 accepted TCP connections from 11.4s, ~5.6s before service.
Requests arriving during migration received 503, not timeouts, connection resets, or partially-migrated data. It fails closed.

4. Correction to attempt 1

Attempt 1 flagged this as "a readiness probe could route production traffic mid-migration." That is wrong and this run refutes it. /health/ready reports unready throughout. The real exposure is narrower and different: any gate that is not /health/ready — a TCP port check, an L4 load balancer health check, or a Kubernetes startupProbe on /health/started — goes green while the schema is still migrating.

Recorded prominently rather than quietly amended, per our no-unverified-claims rule and the ledger's publishing rule 4.

Outcome

Reached ready · [x] 1000 users intact · [x] login works (verified attempt 1)
Custom SPI / themes / federation — not in fixture

Rollback

Exercised this run? no. Restore-from-snapshot was used twice to rebuild the starting state, which exercises the mechanism but is not a rollback of a failed upgrade and does not satisfy the gate.

Verification

Claim	Primary source	Checked
`/health/ready` = "only direct traffic once the readiness probe succeeds"	keycloak.org/observability/health, fetched 2026-08-25	☑
`/health/started` = startup probe, precedes liveness	same	☑
Docs describe behaviour during schema migration	They do not — checked 2026-08-25. This gap is why the measurement is worth publishing	☑
Behaviour identical on other major pairs	not tested — only 26.0.0 → 26.7.1	☐

Publishable extract

Keycloak 26.7.1 opens its HTTP port about 5.6 seconds before it can serve a request, and migrates your schema in between.

On a 1,000-user realm (Postgres 16.15, single node, Hetzner CCX33 — 8 vCPU dedicated, 32 GB, local NVMe, JVM -Xms1g -Xmx4g), upgrading 26.0.0 → 26.7.1: Quarkus logged started at T+11.4s and began the Liquibase schema migration at T+12.2s, on an executor thread, after startup had completed. Migration ran through six model steps and finished at T+16.2s. The server first issued a token at T+16.6s.

We verified the migration is not triggered by an inbound request: with the server left completely untouched for 120 seconds, DATABASECHANGELOG still went from 144 rows to 211 and MIGRATION_MODEL from 26.0.0 to 26.7.1.

The good news is that /health/ready is honest — it returned 503 for the whole window and only went green after migration finished. The trap is everything else: /health/started returned 200 at T+11.5s, mid-migration, and the TCP port accepted connections from T+11.4s. If your load balancer health check is a port probe, or your Kubernetes startupProbe gates on /health/started, your infrastructure believes the instance is up roughly five seconds before it is. Requests in that window get a clean 503 rather than corrupt data — it fails closed — but they do fail.

Five seconds does not sound like much, and on this path it stays about five seconds no matter how big your realm is — see the correction below.

CORRECTED 2026-08-25 by 2026-08-25-26.0.0-to-26.7.1-100k. This extract originally continued: "The migration phase is the part that scales with your data … so on a large realm this window is not five seconds." That was an extrapolation from a single scale point and it is false for this version pair. At 100,002 users the migration took 3.79s versus 3.97s at 1k, and the unready window was 4.9s versus 5.6s. None of the 67 changesets on this path rewrite user data. The window is real and worth designing around; its size does not grow with your realm on this path.