A Silent JWT Format Change That Took Down Production
by Raymond K. Zhao
I recently watched our product go dark in production for a reason that no diff, no deploy, and no config change could explain. Every request carrying a user’s session token came back 403. Development and staging were perfectly healthy, running the exact same build. Nothing on our side had changed. The code that passed CI earlier was now rejecting every real user.
The cause was a single new field in a JWT header. One letter: m. Our identity provider had started emitting it, and our token verification refused to accept it.
This post is about how one library shrugs at a new header field while another treats it as fatal, why the change was invisible until it hit real users, and why "ship to production, fix it when someone complains" is fine for a hobby project and inexcusable for infrastructure sitting in the authentication path of paying customers.
What a JWT Header Is Allowed to Contain
A JWT is three base64url segments: a header, a payload, and a signature. The header is a small JSON object describing how the token was signed — normally alg (algorithm), kid (key id), and typ. The JOSE specifications (RFC 7515 for JWS) define a set of registered header parameters — alg, kid, typ, cty, crit, and friends — and they also explicitly permit private header parameter names: fields a token’s producer and consumer agree to use between themselves.
Two clauses in that spec decide this entire story.
First, private header parameters are permitted by agreement. RFC 7515 §4.3 says a producer and a consumer may agree to use header names outside the registered set. The operative word is agree. It is a two-party contract.
Second, the spec says what a recipient should do with a field it does not recognise: unless the field is listed in the crit (critical) header, an implementation that does not understand it is meant to ignore it, not fall over. The crit mechanism exists precisely so a producer can say "you MUST understand this or reject the token". A field added without crit is, by design, safe to ignore.
Hold onto that: under the JOSE model, an unknown, non-critical header field is an ignorable field. Our provider added m with no crit. By the spec, nothing should have broken.
Why It Broke Anyway: Strict vs. Lenient Libraries
Whether "nothing broke" or "everything broke" came down entirely to which library does the verifying.
PyJWT, the most widely used JWT library in Python, follows the spec’s lenient baseline. It reads the header fields it cares about — alg, kid, typ — and ignores the rest. It does no schema validation of the header. Hand it a token with a surprise m field and it decodes without a blink.
joserfc, a modern and deliberately strict JOSE implementation, takes the opposite stance. It validates the protected header against a registry of known parameters, and by default — strict_check_header=True — it rejects any header key that is not in that registry. (See joserfc 's registry documentation.) This is not a bug. It is a design decision, and a defensible one: "only accept header fields you explicitly understand" is a safer default in a security-critical path. But it is stricter than the spec requires — it turns an ignorable field into a hard failure.
So the same production token produced two different outcomes:
| Library | Default Header Handling | Token Carrying m |
|---|---|---|
| Ignores unregistered header parameters | Accepted |
| Rejects any header key not in its registry | Rejected |
We verify with the strict one. When the provider started stamping m onto tokens, every real request failed the header check before the signature was even considered.
The lesson is not "strict is bad" or "lenient is good". Both are legitimate. The lesson is that your library’s strictness posture is a decision you own, and most teams have never consciously made it. Strictness here caught a real, unannounced change to our security tokens — arguably the system working. It also caused an outage. That tradeoff deserves a deliberate choice, not a default you inherited.
The Part That Turns a Bug Into Negligence: Production Only
Here is what elevates this from "the Internet is held together with tape" to genuine sloppiness.
The provider rolled the new field out to production instances only. Development instances kept issuing the old header. As best I can tell, m is a token mode marker — a flag describing which kind of instance minted the token — which is exactly why it appeared where it did: the production issuer began stamping its tokens with a mode flag, and the development issuer never did.
Sit with that. Every environment we control was telling us the truth: healthy. Tests passed. Staging passed. The build that shipped was byte-for-byte the build that broke. The only place the new field existed was the one place we cannot reproduce on demand — real production, against the real production issuer. There was no environment in which we could have caught this before customers did, because the vendor had broken environment parity on their side, silently, in a dimension we did not even know was a variable.
We spend enormous effort keeping dev, staging, and prod in parity. It never occurred to us that our identity provider’s token format was an unversioned, environment-dependent surface that could drift underneath us. It is on the list now.
The Fix Was Three Lines
The repair was almost insulting. joserfc lets you register additional header parameters so its strict registry recognises them. We registered the provider’s private fields — m and its couple of siblings — with a permissive validator, and decoding accepted them again:
registry = JWSRegistry(
{
"m": HeaderParameter("token mode", accept_anything),
# ...the provider's other private header fields
}
)
jwt.decode(token, key, algorithms=["RS256"], registry=registry)You can also blunt-force it with JWSRegistry(strict_check_header=False), which disables the header check entirely. Registering the specific fields is better: you keep strict validation for every other header key and only carve out the ones you have knowingly agreed to accept.
Three lines. That is how much code it took to bring the product back, and that asymmetry is the whole point. It took the vendor one line to break us and three lines to recover — but between those two tiny diffs sat a full customer-facing outage, an emergency, and a very bad afternoon.
Break-Then-Fix Is an OSS Mindset, Not an Enterprise One
In open source, "ship it, fix it when someone opens an issue" is a healthy rhythm. Contributors are volunteers, users are co-developers, and the contract everyone signed up for is we are all figuring this out together. If a hobby project adds a header field and something downstream breaks, that is a GitHub issue and a patch, not a betrayal.
A paid identity provider sitting in the critical path of other companies' authentication is not a hobby project, and its customers did not sign that contract. When you are the thing standing between a business and its users, the bar is different. None of the following happened:
-
Token formats are an API. Changes to the header — even "harmless" additive ones — are API changes, and deserve to be announced, documented, and versioned.
-
Roll to non-production first, or in lockstep. Shipping to production-only is the exact inversion of a safe rollout. Customers should be able to see a change coming in an environment where a failure costs nothing.
-
Document your private claims and header fields. If you put
minto a security token that thousands of backends parse, those backends' authors deserve to be able to look up whatmis. -
Say something. A changelog entry, a status-page note, an email to technical contacts. Anything but silence.
There was no changelog entry, no email, no documentation, no notice of any kind. The first notification we received that our provider’s token format had changed was our own product going down.
Recommendations
For Teams Consuming Third-Party JWTs
-
Know your library’s strictness. Find out explicitly whether your JOSE library rejects or ignores unknown header parameters, and choose on purpose. Both answers are defensible; inheriting one by accident is not.
-
Treat the vendor’s token as an evolving, untrusted surface. The header can change without warning. Make a decode failure diagnosable in minutes — log the offending header and alert on header / decode failures distinctly from signature failures.
-
Add your provider to your parity model. Parity is not just your code and config; it is the behaviour of everything you depend on. If a vendor can behave differently in prod than in dev, that difference is part of your risk surface.
-
Assume "additive" is not safe. "We only added a field" are the famous last words of a breaking change. Whether an addition breaks you depends entirely on how strict the other side is — and you do not control the other side.
For Auth Providers
-
Version your token format and treat header changes as API changes. Additive is not the same as backward-compatible when your consumers validate strictly.
-
Never ship to production before development. If anything, do the reverse, so the blast radius of a surprise lands where nobody is paying for uptime.
-
Publish your private header parameters and claims. Undocumented fields in a security token are a liability for everyone who has to parse them.
Final Thoughts
A single letter took down our production. The frustrating part is not that m exists — private header fields are legal and even sensible. The frustrating part is that it arrived with no announcement, no documentation, and in production before anywhere else, from a vendor whose entire job is to be the boring, dependable layer you never have to think about.
Boring and dependable is the product. Silent break-then-fix is the opposite of it.