2026-08-25 · run 2026-08-25-s5-jvm-heap-floor

S5: the heap floor does not move with realm size

Upgrade: 26.0.0 → 26.7.1
Scale: two runs: 1,002 users / 2 realms, and 2,000,003 users / 2 realms
Database: PostgreSQL 16.15, single node, container
Topology: single container
Host: Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
Condition: JVM heap too small

Summary

S5 asked us to "establish the heap floor per realm size — a number customers can be given directly." There is no per-realm-size floor on this path. A 1,002-user realm and a 2,000,003-user realm have the same floor, bracketed identically between 128 MB and 160 MB, because the thing that runs out of memory is Quarkus augmentation — the auto-build that happens before the migration starts — and augmentation does not care how many users exist.

The lab's own compose file pins -Xmx4g. The measured floor is 160 MB. We have been provisioning 25× the heap the run needs, and had assumed the multiple was there for the data.

Environment

Field	Value
Keycloak from → to	26.0.0 → 26.7.1
Distribution / start mode	quay.io/keycloak/keycloak official image, `start` w/ external DB, auto-build on start (no `--optimized`)
Database	PostgreSQL 16.15, single node, container
Postgres tuning	stock — `shared_buffers` 128MB, `maintenance_work_mem` 64MB, `statement_timeout` 0
Adverse scenario	S5 — `-Xmx` is the variable
Dataset scale	two runs: 1,002 users / 2 realms, and 2,000,003 users / 2 realms
Seeding method	`partialImport` (1k) and direct SQL COPY (2M)
Topology	single container
Host	Hetzner CCX33 (fsn1) — 8 vCPU dedicated, 30 GB RAM, local NVMe
JVM heap	`-Xms64m -Xmx<variable>`; lab default is `-Xms1g -Xmx4g`
Lab change	`docker-compose.heap.yml` added so `JAVA_OPTS_APPEND` is env-driven, defaulting to the base file's `-Xms1g -Xmx4g`

Procedure

# .env
COMPOSE_FILE=docker-compose.yml:docker-compose.heap.yml

for H in 256m 192m 160m 128m 64m; do
  ./bin/restore.sh <baseline>              # at the lab default heap -- see note
  JAVA_OPTS_APPEND="-Xms64m -Xmx$H"
  docker compose up -d --force-recreate keycloak
  TIMEOUT=240 ./bin/wait-ready.sh
done

Note, and a trap worth recording: the heap must be reset to a working value before restore.sh, not after. restore.sh starts Keycloak as part of its re-migration check, so a heap too small to boot makes the restore fail, and the next iteration then measures the wrong baseline. The first attempt at this ladder was lost to exactly that.

Results

`-Xmx`	1,002 users	2,000,003 users
512m	—	ready, 16s
256m	ready, 17s	ready, 16s
192m	ready, 17s	—
160m	ready, 17s	ready, 17s
128m	OOM, died at 241s	OOM, died at 241s
64m	OOM, died at 242s	OOM, died at 242s

Every successful run: model 26.7.1, changelog 211, full user count intact. Every failed run: model 26.0.0, changelog 144 — untouched — full user count intact, container exited.

The floor is (128m, 160m] at both scales. Time to ready is 16–17 s at both scales too, which is the flat-migration result from 2026-08-25-26.0.0-to-26.7.1-100k holding again at 2M.

Where it actually dies

The last phase reached, from the container log:

`-Xmx`	Last phase before death
128m, 64m	`Updating the server image`
160m and above	`Updating database` (i.e. it got to the migration)

Appending additional Java properties to JAVA_OPTS
Changes detected in configuration. Updating the server image.
Updating the configuration and installing your custom providers, if any. Please wait.
Terminating due to java.lang.OutOfMemoryError: Java heap space

Updating the server image is Quarkus augmentation — the build step Keycloak re-runs on start whenever it detects a configuration change. It happens before a single row of the database is read, which is why the number is identical at 1k and 2M, and why no failed run touched the schema.

The failure is slow

Both OOM cases took just over four minutes to die (241 s, 242 s), not seconds. The JVM thrashes the collector at the ceiling before giving up. In a maintenance window that is four minutes of an operator watching a container that looks like it is working.

Outcome

Failed closed at and below the floor — nothing applied, no data loss, no held lock
Recoverable by raising -Xmx alone; no restore required
Container exits and stays exited (CrashLoopBackOff under Kubernetes)

Verification

Claim	Primary source	Checked
160m is sufficient at 1,002 users	run: ready in 17s, changelog 211	☑
160m is sufficient at 2,000,003 users	run: ready in 17s, changelog 211, 2M users intact	☑
128m fails at both scales	two runs, both OOM, both `cl=144` after	☑
The OOM precedes the migration	last log phase is `Updating the server image`; changelog untouched at 144	☑
Failure applies nothing	changelog 144 and full user count after every failed run	☑
Holds with `--optimized` / a pre-built image	not tested — augmentation is skipped there, so the floor may be lower and may then depend on something else	☐
Holds with custom providers in `/opt/keycloak/providers`	not tested — augmentation installs providers, so a large jar plausibly raises this floor. Direct follow-on from S7	☐
Holds where the migration itself is expensive	not tested — this pair skips index creation above 300k rows	☐
Steady-state serving floor (vs. startup floor)	not measured — this is a startup/migration number only, not a number to run production on	☐

Publishable extract

We tried to find the heap a Keycloak upgrade needs per million users. It does not exist.

On Postgres 16.15, upgrading 26.0.0 → 26.7.1, we walked -Xmx down against two realms — one with 1,002 users, one with 2,000,003 — and the boundary landed in exactly the same place for both: 160 MB works, 128 MB dies with OutOfMemoryError. Not "roughly the same". The same bracket, and the same 16–17 seconds to ready on either side of a 2,000× difference in user count.

The reason is in the log. At 128 MB the last thing Keycloak prints is Updating the server image — Quarkus augmentation, the build step that runs before the database is touched at all. The schema was untouched afterwards: 144 changelog rows before, 144 after, every user present. It never got as far as the data, so the data never mattered.

Two practical consequences. First, if your Keycloak dies on startup with a heap error during an upgrade, stop looking at your user count — you are almost certainly looking at the build step, and the fix is a flat number, not a proportional one. Second, the failure is not fast: both of our OOM runs took just over four minutes to give up, thrashing the collector at the ceiling. If your window budget assumes a heap error announces itself immediately, it does not.

One caveat we cannot yet close: we run without --optimized, so augmentation happens on every start. A deployment with a properly pre-built image skips that phase, and its floor is a different number we have not measured.