The Keycloak Upgrade Ledger

Every upgrade we have rehearsed, with the environment stated and the clock running.

2026-08-26 · run 2026-08-26-provider-types-protocol-mapper-brittle

protocol mappers are internal SPI; brittle providers fail loudly at use, not load

Database
PostgreSQL 16, fresh empty DB
Host
local laptop Docker (not Hetzner); 1k-class fixture, no scale component
Condition
Custom SPI vs old internals — untested rows: *protocol mappers* · *brittle (removed-API) variants — load vs use*

Why this run existed

2026-08-26-provider-types established the SPI relocation (UserStorageProvider left keycloak-server-spi for keycloak-model-storage) and that a tame authenticator + user-storage provider cross 25 → 26 cleanly. It left two rows unchecked: protocol mappers, and brittle (removed-API) versions of each type — whether they fail at load or at first use. S7 (2026-08-25-s7-custom-spi) answered that question for event listeners (lazy, silent) but for no other type. This run adds a tame protocol mapper, a brittle authenticator, and a brittle user-storage provider.

Environment

FieldValue
Keycloakbuilt against 25.0.6, loaded on 26.7.1 (crosses the 26.0.0 API-removal boundary)
Fixturesfixtures/spi-types/ (tame: authenticator + user-storage + new protocol mapper); fixtures/spi-brittle-auth/ (new: brittle authenticator); fixtures/spi-brittle-us/ (new: brittle user-storage)
Buildbin/build-spi.sh 25.0.6 --module … (Maven in Docker)
DatabasePostgreSQL 16, fresh empty DB
Hostlocal laptop Docker (not Hetzner); 1k-class fixture, no scale component
JVM heap-Xms1g -Xmx4g (compose default)

The removed API the brittle fixtures target is the three-argument overload KeycloakModelUtils.runJobInTransactionWithResult(KeycloakSessionFactory, KeycloakContext, KeycloakSessionTaskWithResult) — present in 25.0.6, absent in 26.0.0 (replaced by a four-argument overload carrying an extra String), living in keycloak-server-spi-private (no compatibility promise).

Findings

Finding 1 — a tame protocol mapper registers on 26.7.1, and the SPI is internal ⭐

A custom ProtocolMapper implementing only the stable base interface loads and registers on 26.7.1 alongside the authenticator and user-storage provider. All three appear in /admin/serverinfo:

SPI (serverinfo key)Provider
authenticatorlab-authenticator
protocol-mapperlab-protocol-mapper
storagelab-user-storage

And the "internal SPI" warning now covers a third type. KC-SERVICES0047 fires for the protocol mapper and the authenticator, and not for the user-storage provider:

WARN KC-SERVICES0047: lab-protocol-mapper (lab.LabProtocolMapper) is
  implementing the internal SPI protocol-mapper. This SPI is internal and may
  change without notice
WARN KC-SERVICES0047: lab-authenticator (lab.LabAuthenticatorFactory) is
  implementing the internal SPI authenticator. This SPI is internal and may
  change without notice

The reason is the same one the earlier record found for user-storage, inverted: org.keycloak.protocol.ProtocolMapper lives in keycloak-server-spi-private (verified: present there, absent from keycloak-core and keycloak-server-spi in 26.7.1), the artifact that carries no compatibility promise. So a customer's protocol mappers sit on the same unstable ground as their authenticators and event listeners; only user federation is a public SPI.

The base ProtocolMapper interface itself is stable across the ladder — the same three abstract methods (getProtocol, getDisplayCategory, getDisplayType) from 21.1.2 through 26.7.1 — so a tame mapper crosses majors intact. (What does not survive is in the ☐ rows: token-transform behaviour, which moved out of the base interface — see Not tested.)

Correction (added 2026-08-26, same day): the parenthetical above is imprecise. The base ProtocolMapper interface never carried the token-transform methods — they live on OIDCAccessTokenMapper / OIDCIDTokenMapper in keycloak-services, a separate artifact, and their signatures are stable 25 → 26. The brittle-protocol-mapper row is resolved in 2026-08-26-provider-types-brittle-protocol-mapper; the "moved out of the base interface in 26" blocker does not describe a real change.

Finding 2 — a brittle authenticator fails at use, not at load, and it fails loudly ⭐⭐

fixtures/spi-brittle-auth calls the removed three-argument overload from inside authenticate() — the work method, deliberately not init() — so the question asked is the one S7 asked of listeners: does Keycloak die at startup, or only when a flow reaches the step?

It loads. On 26.7.1 the server starts with zero ERRORs, the factory init() runs, and the provider registers in /admin/serverinfo:

[lab-brittle-auth] authenticator factory init -- provider loaded
WARN KC-SERVICES0047: lab-brittle-authenticator (lab.LabBrittleAuthenticatorFactory)
  is implementing the internal SPI authenticator ...

It dies at first use, loudly. Wiring it as a REQUIRED execution in a browser flow and attempting a login returns HTTP 500 with a JSON error body, and the log carries the full stack as an uncaught server error:

ERROR [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-1)
  Uncaught server error: java.lang.NoSuchMethodError: 'java.lang.Object
  org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(
  org.keycloak.models.KeycloakSessionFactory, org.keycloak.models.KeycloakContext,
  org.keycloak.models.KeycloakSessionTaskWithResult)'
    at lab.LabBrittleAuthenticator.authenticate(LabBrittleAuthenticator.java:25)
    at org.keycloak.authentication.DefaultAuthenticationFlow.processSingleFlowExecutionModel(...)

Response body:

{"error":"unknown_error","error_description":"For more on this error consult the server log."}

This is the answer to the open row, and it splits two ways. The load-vs-use half generalises S7: Keycloak does not invoke an authenticator's authenticate() (or a listener's onEvent()) during startup, so a removed API that only those methods reach does not break the upgrade — the breakage is deferred to first use. The silent-vs-loud half does not generalise: S7's listener failure was swallowed (the triggering request still returned 201/200, nothing at default log level), whereas the authenticator's NoSuchMethodError propagates out of the authentication flow as an uncaught error and the login fails with HTTP 500.

Why that matters for a post-upgrade check: a brittle authenticator is self-announcing — the first login fails loudly and someone investigates. A brittle listener is the dangerous one — logins keep succeeding while audit events silently stop being forwarded. A smoke test ("can users log in?") catches the former and misses the latter.

Finding 3 — the brittle-listener control reproduces S7 in this environment

Re-running the existing fixtures/spi-brittle listener on the same 26.7.1 instance confirmed S7: it registers, factory init() runs, KC-SERVICES0047 fires — and a successful login returns 200 with no error and no [lab-brittle] user event line in the log. The onEvent println sits after the removed-API call, so its absence means the throw happened before it and was swallowed by event dispatch. The same NoSuchMethodError that is loud in an authentication flow is silent in an event listener.

Finding 4 — brittle user-storage also fails lazily, and it fails loudly ⭐

fixtures/spi-brittle-us calls the same removed three-argument overload from getUserByUsername (the lookup work method), with the session captured in the constructor — the natural shape of a real user-federation provider, since lookup methods receive no session argument of their own.

It loads. On 26.7.1 the server starts with zero ERRORs, the factory init() runs, and the provider registers under the storage SPI in /admin/serverinfo with no KC-SERVICES0047 — consistent with user federation being the one public SPI. The component is not even instantiated at startup: its constructor line appears only on first use, when the realm's UserProvider is built.

It dies at first lookup, loudly. Configuring the provider as a user federation component and attempting a password grant for any username returns HTTP 500 with the same unknown_error body and an uncaught NoSuchMethodError:

ERROR [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-1)
  Uncaught server error: java.lang.NoSuchMethodError: 'java.lang.Object
  org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(...)'
    at lab.LabBrittleUserStorageProvider.getUserByUsername(LabBrittleUserStorageProvider.java:37)
    at org.keycloak.storage.UserStorageManager.getUserByUsername(UserStorageManager.java:559)

The exception escapes UserStorageManager.tryResolveFederatedUser rather than being swallowed, so user federation failures announce themselves the way an authenticator's does — not the way a listener's does.

The brittle matrix is now complete across four types, and it splits cleanly:

TypeFails atVisibilityRecord
event listenerusesilent (request succeeds)S7
authenticatoruseloud (HTTP 500)this run, Finding 2
user-storageuseloud (HTTP 500)this run, Finding 4
protocol mapperuseloud (HTTP 500)2026-08-26-provider-types-brittle-protocol-mapper (added later same day)

No brittle type fails at load. Keycloak does not invoke any provider's work method during startup, so a removed API only reachable from the work method never breaks the upgrade itself — it defers the breakage to first use. The difference between the rows is purely how the failure surfaces: event dispatch swallows it; authentication and user-lookup paths let it propagate to HTTP 500.

Verification

ClaimPrimary sourceChecked
Tame protocol mapper registers on 26.7.1live /admin/serverinfo query — protocol-mapper → lab-protocol-mapper (quoted above)
ProtocolMapper is in keycloak-server-spi-private, not -spi/-corejar listing of all three 26.7.1 artifacts (0/0/1 matches)
protocol-mapper SPI is internalKC-SERVICES0047 line for lab-protocol-mapper in startup log
authenticator internal, storage notwarning fired for authenticator only, consistent with prior record
Brittle authenticator loads (not load-failing)startup log: init -- provider loaded, 0 ERRORs, registered in serverinfo
Brittle authenticator dies at use, loudlyHTTP 500 + unknown_error body + NoSuchMethodError stack (quoted above)
Removed overload present 25.0.6, absent 26.xjavap of KeycloakModelUtils in both artifacts
Brittle listener is silent (S7 control)login 200, no error, no [lab-brittle] user event line
Brittle user-storage (load vs use)lazy + loud — HTTP 500 on first lookup, NoSuchMethodError at getUserByUsername (Finding 4)
Brittle protocol mapper (load vs use)done2026-08-26-provider-types-brittle-protocol-mapper: lazy + loud (HTTP 500). That record refutes the "no natural home" blocker below: the work method OIDCAccessTokenMapper.transformAccessToken lives in keycloak-services and is stable 25 → 26
The exact 24.x release that split model/storagenarrowed — 24.0.0 (+ 25.0.3 anomaly). See 2026-08-26-userstorage-artifact-split-narrowed

Publishable extract

A custom authenticator that outlived its API does not break your upgrade. It breaks your next login.

We built an authenticator against Keycloak 25.0.6 that calls a helper removed in 26.0.0, then loaded it on 26.7.1. The server started cleanly — the provider registered, /admin/serverinfo listed it, zero errors. The first login that reached it returned HTTP 500 and a full NoSuchMethodError stack as an uncaught server error.

A broken user-federation provider behaves like the authenticator, not the listener: the first login that consults it returns HTTP 500.

Compare the same mistake in an event listener: the server also starts, the listener also registers, but when the first event fires the exception is swallowed and the request still succeeds. Your audit trail stops silently.

All three are the same removed API. The difference is not the provider, it is where Keycloak calls it: authentication and user-lookup paths propagate the error, event dispatch eats it. So your post-upgrade check cannot be "can users log in" — that catches the loud failures and misses the quiet one. Verify the listener actually produced output, not that login works.

← Back to the Ledger