2026-08-25 · run 2026-08-25-s7-custom-spi

S7: the upgrade succeeds, the extension stops working, and nothing fails

Summary

Two findings, and the second is the sellable one.

1. "Has a custom SPI" is not a risk by itself. A provider that uses only stable SPI, compiled against 21.1.2, loads and works correctly on 26.7.1 — five majors and 619 removed public signatures later — with no source change at all. The same source compiles unchanged against 21.1.2, 25.0.6, 26.0.0 and 26.7.1.

2. A provider that touches a removed internal API does not fail the upgrade. It fails silently, afterwards. The server starts clean in 15 seconds. The provider loads, prints its init line, and appears in /admin/serverinfo. Every check an operator runs during the window passes. Then the first event arrives and the listener throws NoSuchMethodError — logged, dropped, and invisible to the request that caused it, which returns 201.

For an event listener forwarding to a SIEM, that is audit forwarding that stops without an outage.

Environment

Results

Compile-time

[ERROR] LabBrittleEventListener.java:[31,44] no suitable method found for
        runJobInTransactionWithResult(org.keycloak.models.KeycloakSessionFactory,
        org.keycloak.models.KeycloakContext,(s)->event[...]mId())

The API diff predicted the removal; the compiler confirms it independently.

Run-time

Migration timings with the jar mounted throughout: 25.0.6 → 26.0.0 in 13s, 26.0.0 → 26.7.1 in 16s. The provider did not slow, block or fail any hop.

What the brittle failure actually looks like

Startup — indistinguishable from success:

[lab-brittle] factory init -- provider loaded

/admin/serverinfo lists lab-brittle-listener among the event listeners. wait-ready.sh returns 0 in 15 seconds. Container running.

First event — and only then:

ERROR [org.keycloak.services] KC-SERVICES0085: Failed to send type to
  lab.LabBrittleEventListener@6012fda7: java.lang.NoSuchMethodError:
  'java.lang.Object org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(
     org.keycloak.models.KeycloakSessionFactory, org.keycloak.models.KeycloakContext,
     org.keycloak.models.KeycloakSessionTaskWithResult)'

And the blast radius is nil, which is the problem:

Keycloak's internal audit tables stay complete. Anyone verifying the audit trail by querying Keycloak sees nothing wrong. It is only the delivery to the custom listener — the path to whatever is downstream — that has stopped.

A note on KC-SERVICES0047

On both 26.0.0 and 26.7.1, every provider load emitted:

WARN [org.keycloak.services] KC-SERVICES0047: lab-event-listener
  (lab.LabEventListenerProviderFactory) is implementing the internal SPI
  eventsListener. This SPI is internal and may change without notice

eventsListener — the most commonly extended SPI in the wild — is classified internal. Whether this warning also appears on 25.0.6 was not checked at the time and the containers were gone; re-run on 2026-08-26 (2026-08-26-kc-services0047-not-new-in-26) shows it is present on 25.0.6 — the warning is not new in 26.

An unrelated observation, deliberately not attributed

A failed password grant against the lab realm returned 401 on 25.0.6 and 26.0.0, and 400 on 26.7.1. We tested whether the brittle listener caused it: with the listener registered, removed, and registered again, the response was 400 every time. It is not the listener. Narrowed on 2026-08-26 (2026-08-26-invalid-grant-401-to-400-narrowed): it is a version change that landed in 26.6.0 — 26.0.0 through 26.5.7 return 401, 26.6.0 onward returns 400.

Verification

Publishable extract

We upgraded Keycloak across five major versions with a custom extension installed. The upgrade succeeded. The extension stopped working. Nothing failed.

We built two event listeners. One uses only stable SPI. The other calls KeycloakModelUtils.runJobInTransactionWithResult(...) — a helper any listener wanting its own transaction would reach for, and one that our API diff shows present in 25.0.6 and gone in 26.0.0.

The stable one is the good news, and it is worth saying plainly: compiled against 21.1.2, it runs correctly on 26.7.1. Five majors, 619 removed public signatures in between, and the same source compiles unchanged against every version we tried. "The customer has a custom SPI" is not, on its own, a reason to expect trouble.

The other one is why you test. On Keycloak 26.7.1, carrying the jar built for 25.0.6:

• the server started in 15 seconds;
• the provider loaded and printed its startup line;
• it appeared in /admin/serverinfo alongside the built-in listeners;
• the readiness check passed.

Every check an operator runs during a maintenance window passed. Then the first event arrived:

ERROR KC-SERVICES0085: Failed to send type to lab.LabBrittleEventListener:
  java.lang.NoSuchMethodError: KeycloakModelUtils.runJobInTransactionWithResult(...)

The admin request that triggered it returned 201. The group was created. The token endpoint returned 200. The container stayed up. And Keycloak's own admin_event_entity table kept recording normally — 1,015 rows — so anyone auditing Keycloak's audit trail sees a complete one.

If that listener was shipping your authentication events to a SIEM, the events stopped and your dashboard says everything is fine.

The check that catches this is not a smoke test. It is: fire one real event after the upgrade and grep the log for KC-SERVICES0085.