Run B at 100k: optimized image, migration timing and heap floor at scale
- Upgrade
- 26.0.0 → 26.7.1
- Scale
- 100,000 users, typical, baseline-26.0.0-typical-100k
- Database
- PostgreSQL 16, single node, container
- Topology
- single container
- Host
- Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
- Condition
- routine — no adverse condition applied
Why this run existed
Previously untested: the stepped ladder and Run B each had only one scale. our lab protocol asks every path for two scales. This is the second scale for Run B, and it clears two ☐ rows in the original Run B record — the heap floor at scale, and the migration timing at scale.
Environment
| Field | Value |
|---|---|
| Keycloak from → to | 26.0.0 (stock image) → 26.7.1 (pre-built image) |
| Image | kclab/keycloak-optimized:26.7.1 (same Dockerfile as Run B) |
| Start mode | start --optimized --http-enabled=true |
| Database | PostgreSQL 16, single node, container |
| Postgres tuning | stock |
| Dataset scale | 100,000 users, typical, baseline-26.0.0-typical-100k |
| Topology | single container |
| Host | Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe |
| JVM heap | -Xms1g -Xmx4g for the migration; variable in the heap leg |
| Compose overrides | docker-compose.yml:docker-compose.optimized.yml:docker-compose.optimized-noprov.yml |
Result
Migration timing at 100k
| 1k (Run B) | 100k (this run) | |
|---|---|---|
upgrade.sh elapsed | 11s | 11s |
migration_model | 26.7.1 | 26.7.1 |
| Index audit | — | 118/118 correct, 0 missing, 0 wrong shape |
The migration is flat between 1k and 100k — same 11s. This is the same
mechanism the stock-image runs found (index-skip-threshold.md, Run A): the
expensive DDL is declined above 300,000 rows, so "time to ready" does not grow
with realm size on this pair. The optimized image saves the augmentation step
regardless of scale; it does not change the migration.
Heap floor at 100k
Same bisect boundary values as B2, but on the 100k database:
-Xmx | Result at 100k |
|---|---|
| 96m | READY |
| 64m | FAILED — Terminating due to java.lang.OutOfMemoryError: Java heap space |
The floor is (64m, 96m] at 100k, unchanged from 1k. Combined with S5's
finding that the stock floor is identical at 1k and 2M, this closes the
"scale-dependence" question for the optimized floor too — the heap floor is a
function of the build/augmentation step, not of dataset size.
Reproduced: the providers bind-mount trap (B3b-i)
The first attempt at this run failed with exit code 2 before any migration:
A provider JAR was updated since the last build, please rebuild for this to be fully utilized.
Cause: bin/build-spi.sh 25.0.6 (run earlier for the KC-SERVICES0047 probe) had
left lab-event-listener-built-for-25.0.6.jar in fixtures/providers/, and the
base compose bind-mounts that directory over the optimized image's own empty
/opt/keycloak/providers. The runtime jar did not match the image's build.
Removing the mount (docker-compose.optimized-noprov.yml) makes the same image
start. This is a faithful re-occurrence of B3b-i, not a new finding — worth
noting because it means any later run that builds a SPI must clear
fixtures/providers/ before an optimized-image run, or add the noprov
override.
Tooling observation (minor)
upgrade.sh reported from: 26.7.1 on the successful run, because the failed
first attempt had already rewritten .env's KC_VERSION to 26.7.1. The actual
migration was 26.0.0 → 26.7.1 (the migration log shows the full 26.1.0 → 26.7.0
realm-migration ladder). upgrade.sh's from is read from .env, not from the
database's migration_model — a stale .env after a failed attempt makes the
label lie. Not load-bearing for this run; noted for the tooling.
Verification
| Claim | Primary source | Checked |
|---|---|---|
| Migration at 100k takes the same 11s as at 1k | upgrade.sh elapsed, both runs | ☑ |
| Index audit clean at 100k | index-audit.sh output: 118/118 | ☑ |
Heap floor (64m, 96m] holds at 100k | 96m READY, 64m OOM at 100k | ☑ |
| The failure mode is the providers shadowing, not the migration | exit 2 before Liquibase; noprov fixes it | ☑ |
Heap floor at 2M under --optimized | not tested — 100k only; S5's stock 2M point + this 100k point imply it, but it is not measured | ☐ |
Migration timing at 2M under --optimized | not tested — 100k only | ☐ |