2026-09-04 · run 2026-09-04-k8s-concurrent-migration-and-session-stranding

Kubernetes: concurrent migration + functional session stranding

Summary

Two runs on a fresh kind kclab (k8s v1.37.0, kind v0.33.0), 2-replica Keycloak Deployment, embedded Infinispan on KC_CACHE_STACK=kubernetes:

1. Concurrent migration (maxSurge=2) — the changelog lock serializes it. Two new 26.7.3 pods start at the same instant against a 26.0.0 schema. One (jgvgr) wins DATABASECHANGELOGLOCK, runs the Liquibase changesets and the realm migrators (26.1.0 → 26.6.x); the other (ncrj6) logs Updating database and then makes no migration progress — it blocks on the lock and, once the winner releases it, finds 0 changesets and boots. Both reach ready; no corruption. This re-shows, under real k8s concurrency, the lock serialization the Compose record s1-s2-lock-failure-modes.md established.

2. Functional session stranding did NOT reproduce as predicted. A refresh token issued by a 26.0.0 pod (Infinispan 15) was accepted (HTTP 200, same sid) by a 26.7.3 pod (Infinispan 16) after the old pods were gone. This refines the prior record's "a request routed to the other side of the partition finds nothing": the online session survived the cache partition and the termination of its owning pods. There is no USER_SESSION table in 26.7.3 (only auth_session, root_auth_session, offline_*), so the recovery is not an obvious DB fallback — the mechanism is not yet pinned and is the follow-up below.

Environment

Results

Leg 1 — concurrent migration timeline (pod logs)

Leg 2 — session stranding (HTTP-level)

session issued on 26.0.0 pod (Infinispan 15)        → refresh_token captured
rolling update 26.0.0 → 26.7.3, old pods terminated
refresh on 26.7.3 pod (Infinispan 16)               → HTTP 200, new access_token, SAME sid

The refresh succeeded with the same session id, i.e. the session was found despite the cache partition and the death of the pods that held it.

Findings

1. The changelog lock is the concurrency control for parallel migrators. Two pods migrating the same schema at once do not corrupt it; one serialises the other. The loser's Liquibase phase is a no-op (0 changesets) after the lock wait. The lock is held across the realm migrator phase too, not just the changesets (the winner's Migrating older model … lines sit under the lock).

2. The cache partition is real at the JGroups level but its functional, session-loss consequence is NOT as the earlier record stated. Online sessions survive across the Infinispan 15 → 16 boundary in the refresh flow, with no USER_SESSION table to explain a DB fallback. The prior claim "a request routed to the other side of the partition finds nothing" is therefore too strong — it was inferred from the JGRP000006 cookie log line, not measured at the HTTP layer. The exact recovery mechanism (cache-loader persistence into a table that isn't USER_SESSION, or stateless re-hydration of the refresh token) is unpinned.

3. kind discovery race persists. Both pods again came up as 1-member clusters on first boot (no members discovered after 2001 ms); deleting one pod re-triggered DNS_PING and formed the 2-member cluster. Same as the 2026-08-31 record — verify the baseline before relying on a kind run.

What is still not tested

• The mechanism behind finding 2 — where the session lived that the new pod found it. Candidates to rule out: (a) a session cache-loader writing to a table other than USER_SESSION; (b) the refresh flow re-creating the session from the signed refresh token without a cache hit. Pin by: issue a session, kubectl delete pod the ENTIRE cluster (both pods), restart, refresh.
• External (remote) Infinispan — untouched; separate setup.

Verification