2026-08-26 · run 2026-08-26-k8s-infinispan-rolling-update

Kubernetes / Infinispan: the cache cluster partitions during a rolling update

Summary

First Kubernetes run of the lab. A 2-replica Keycloak Deployment on k3d with embedded Infinispan was rolled 26.0.0 → 26.7.1 under maxUnavailable=0. Four findings:

1. The cache stack changed inside major 26. 26.0.0 hardcodes stack="udp" (multicast) in cache-ispn.xml, which cannot cluster in Kubernetes; 26.7.1 drops the hardcode and defaults to jdbc-ping (database discovery). Asking 26.0.0 for jdbc-ping fails with ISPN000540: No such JGroups stack 'jdbc-ping'. The kubernetes stack (DNS_PING) works on both but is deprecated in 26.7.1 — it is what this run used, so the version was the only variable.

2. Embedded Infinispan clusters and shares sessions across pods (26.0.0): with KC_CACHE_STACK=kubernetes + -Djgroups.dns.query=<headless FQDN>, two pods form a 2-member cluster, and a refresh token issued by pod A is accepted by pod B.

3. The schema migration is not what breaks a rolling update. During the rollout the new 26.7.1 pod migrated the shared Postgres under the two still- serving 26.0.0 pods, and the old pods logged no schema errors — the 26.0.0 → 26.7.1 changesets are additive. migration_model reached 26.7.1 and the rollout completed cleanly.

4. The cache cluster partitions during the rollout (the finding). Infinispan 15.0.8 (26.0.0) and 16.0.12 (26.7.1) — JGroups 5.3.10 vs 5.5.5 — refuse to form a mixed cluster. The TCP handshake between an old and a new pod fails with JGRP000006 ... cookie sent by ... does not match own cookie; terminating connection. Old pods stay in their 2-member cluster; each new pod starts a 1-member cluster; they only merge once the old pods are gone. Sessions are therefore not shared between old and new pods during the transition — an in-flight user session lands on whichever pod owns it, and a request routed to the other side of the partition finds nothing.

Environment

Procedure

# on the box: k3d + kubectl (NOT installed on the laptop)
k3d cluster create kclab --k3s-arg --disable=traefik@server:* --k3s-arg --disable=servicelb@server:*
kubectl apply -f lab/k8s/                 # namespace, postgres StatefulSet, keycloak Deployment

# seed, then rolling update
kubectl -n keycloak set image deploy/keycloak keycloak=quay.io/keycloak/keycloak:26.7.1
kubectl -n keycloak rollout status deploy/keycloak --timeout=180s

Manifests are kept with the lab's Kubernetes fixtures. The initial 26.0.0 run failed with ISPN000540: No such JGroups stack 'jdbc-ping' — the stack 26.7.1 defaults to did not exist in 26.0.0, which is finding #1.

Results

Timeline of the rolling update (from pod logs)

The partition, in the log

WARN [org.jgroups.protocols.TCP] JGRP000006: 10.42.0.13:7800: failed accepting
  connection from peer Socket[addr=/10.42.0.15 ...]:
  BaseServer.TcpConnection.readPeerAddress(): cookie sent by ... does not match
  own cookie; terminating connection

The old cluster (JGroups 5.3.10) and the new pod (JGroups 5.5.5) never merge. DNS_PING hands each new pod the old pods' addresses, the handshake fails, and the new pod continues as a cluster of one until the old pods are terminated.

What this teaches

• A within-major Keycloak rolling update is safe at the database layer — additive changesets mean old pods keep serving while the new pod migrates. The risk is not the schema, it is the cache.
• Infinispan major/minor bumps do not inter-cluster. 26.0.0 (Infinispan 15, JGroups 5.3) and 26.7.1 (Infinispan 16, JGroups 5.5) form separate clusters, so a rolling update across them has a window where the session/authentication caches are partitioned. For upgrades that cross an Infinispan/JGroups version, the safe posture is a recreate rollout (terminate old, then start new), or accept a short cache-cold window.
• The cache stack itself is a migration surface. A customer who pinned KC_CACHE_STACK (or relied on the 26.0.0 default) is changing discovery mechanism when they reach 26.7.1 — jdbc-ping did not exist in 26.0.0, and the kubernetes stack is deprecated in 26.7.1. This belongs in the upgrade runbook's pre-flight, not the schema checklist.

What is still not tested

• Two new pods migrating concurrently (S2's exact half, here maxSurge=1 made the new pods sequential) — the DATABASECHANGELOGLOCK serialization was shown in Compose (s1-s2-lock-failure-modes.md) but not re-shown under k8s.
• Functional proof of session stranding during the partition (issuing a session on an old pod and failing to use it on a new pod inside the transition window) — the partition is proven at the JGroups level, not the HTTP level.
• External (remote) Infinispan, and the 26.7.1 default jdbc-ping stack.

Verification