2026-08-25 · run 2026-08-25-s8-pgbouncer-transaction-mode

S8: transaction pooling breaks the upgrade, and this is what half-migrated looks like

Summary

PgBouncer in transaction pooling mode makes the first start after an upgrade fail, deterministically, and it leaves the database in the inconsistent state this lab has spent all day failing to produce by other means.

Three reproductions, three identical failures, in 16–19 seconds each:

ERROR: Failed to start server in (production) mode
ERROR: Cannot invoke "org.keycloak.connections.jpa.updater.liquibase.lock.CustomLockService
       .waitForLock(org.keycloak.models.dblock.DBLockProvider$Namespace)"
       because "this.lockService" is null

The state it leaves behind is the important part. DATABASECHANGELOG is at 211 rows — the migration completed and committed — while migration_model still reads 26.0.0. The schema is on the new version; the model stamp and the model-level migrations are not. A verification that counts changelog rows would call this a successful upgrade.

The same pooler in session mode upgrades cleanly in 18 seconds. The variable is the pooling mode, not the pooler.

Environment

Procedure

./bin/restore.sh baseline-26.0.0-typical-1k

# .env
COMPOSE_FILE=docker-compose.yml:docker-compose.dbidentity.yml:docker-compose.pgbouncer.yml
KC_DB_URL=jdbc:postgresql://pgbouncer:6432/keycloak
POOL_MODE=transaction            # or session, for the control

docker compose up -d pgbouncer
sed -i 's/^KC_VERSION=.*/KC_VERSION=26.7.1/' .env
docker compose up -d --force-recreate keycloak

Connectivity through the pooler was confirmed independently before each run (psql postgresql://keycloak:keycloak@127.0.0.1:6432/keycloak -tAc 'select 1' → 1), so a failure could not be mistaken for the pooler simply not working.

Results

Runs B and C started from a freshly restored 26.0.0 baseline (cl=144) each time, and both reached cl=211 before dying. The failure is not a race — it is what happens every time work has to be done through a transaction-pooled connection.

The order of events

21:05:00  container starts
21:05:10  INFO  QuarkusJpaUpdaterProvider — Updating database
          ... all 67 pending changesets apply and COMMIT (144 → 211) ...
21:05:16  ERROR Cannot invoke CustomLockService.waitForLock(...) because "this.lockService" is null
21:05:16  container exits 1

Liquibase's schema update succeeds. What fails is the step after it, which needs Keycloak's DBLockProvider — a session-scoped lock. Transaction pooling hands out a different backend per transaction, so the session-scoped lock service never gets established, and Keycloak dereferences a null.

It recovers on the second start — through the same pooler

This is the part that makes it dangerous rather than merely broken. The restart succeeded in 14 seconds and stamped migration_model to 26.7.1, still through transaction-mode PgBouncer. Under Kubernetes this presents as one CrashLoopBackOff followed by a healthy pod, which most operators will never look at twice.

For those 16–19 seconds and for however long the restart takes, the database has a 26.7.1 schema and a 26.0.0 model stamp.

No stuck lock

databasechangeloglock showed 0 held locks after every failure. Consistent with 2026-08-25-s1-s2-lock-failure-modes: this lab has still never observed a stuck Liquibase lock. The failure here is the absence of a lock service, not a lock that was taken and never released.

Verification

Publishable extract

We finally reproduced a half-migrated Keycloak database. It took a connection pooler, not a crash.

Earlier the same day we tried two textbook routes to the "upgrade died partway and left the schema half-applied" failure. We killed Keycloak with SIGKILL mid-migration; it completed itself on the next start. We aborted its SQL with a statement_timeout; it refused to apply anything at all. Neither produced the state everyone warns about.

Then we put PgBouncer in transaction pooling mode in front of Postgres — the default mode nearly everywhere, because it is the mode that actually saves connections — and upgraded 26.0.0 → 26.7.1 on a 1,002-user realm. Three times out of three, Keycloak died 16 to 19 seconds in:

ERROR: Cannot invoke "CustomLockService.waitForLock(DBLockProvider$Namespace)"
       because "this.lockService" is null

Keycloak's DBLockProvider needs a session-scoped lock. Transaction pooling gives you a different backend for every transaction, so the lock service is never established and the server dereferences a null.

Here is the state it left, and why it matters more than the crash:

DATABASECHANGELOG	211 rows — the full migration, committed
migration_model	26.0.0 — unchanged

The schema moved. The version stamp did not. If your upgrade verification counts changelog rows or diffs the schema, it will tell you this upgrade succeeded. The only column that knows the truth is migration_model.

Two more things worth your time. The same pooler in session mode upgraded cleanly in 18 seconds — the pooler is not the problem, the pooling mode is. And the second start succeeded, in 14 seconds, still through transaction-mode PgBouncer. Under Kubernetes that is one CrashLoopBackOff followed by a healthy pod, which nobody investigates.

Before your window: ask what pool_mode your pooler runs, and point Keycloak at the database directly for the upgrade. Afterwards, check select version from migration_model order by update_time desc limit 1 — not the changelog.