Measurement
Published
Every article in this series has a paragraph about a caller-supplied guard being downgraded: you hand a library your own validator, the option name carrying it is renamed, the library cannot find it and quietly uses its own weaker one. This article is about the failure one step further in, which is quieter and has a rule of its own. The option name survives. The library calls your function. Your function runs, correctly, and computes the right answer. And then the library reads the answer out of a field whose name has moved.
The shape of the failure
A guard you supply to a library is two contracts, not one. The first is the option name it arrives under. The second is the shape of the value it hands back - typically a small object with a verdict field and a reason field, which the library destructures and acts on.
Renaming the option name is the well-known case, and this series has now measured it in fifteen libraries. The library falls back to a builtin. That builtin is weaker, but it is a working decision procedure: it produces an answer, and the answer is at least defensible.
Renaming the result fields is different in a way that matters. Your guard is still installed, still found, still invoked. It reads its inputs correctly, applies your policy correctly and returns the correct decision. The library then looks for that decision under a name that no longer exists, finds undefined, and proceeds on whatever undefined means in the expression it landed in.
The telemetry stays reassuring throughout. A library that reports which guard is in use answers by checking whether the option is a function, and it still is. "Is our threat-feed check wired in?" returns a true answer to the wrong question.
Six areas were measured this way in one pass: geographic access control, CAPTCHA verification, agent tool permissions, session replay masking, credential verification, and password hashing. Protection alone was clean on all five presets in every area, so nothing here is an engine defect - every result required member renaming pointed at the names.
The one that failed open, and why it was that one
The geographic access middleware calls a reputation check the application supplies, which returns a verdict field. The library acts on it by asking whether the verdict is the string that means refuse.
Renaming that field: the reputation check ran, consulted the feed, matched a known credential-stuffing source, and returned its refusal. The middleware read the verdict as undefined, undefined is not the refusal string, and the request was permitted. The measured line went from known-stuffing-source=refused(reputation-refused(credential-stuffing-source)) to ALLOWED(permitted), with the reported check still reading caller-supplied.
That is the whole failure: a correct refusal, computed on time, thrown away. And the reason it failed open rather than closed is not luck. It is the polarity of the flag. The middleware asks a prohibition question - is this denied? - and a missing answer is not a denial, so the request proceeds.
The four that failed closed, and the same reason
In the CAPTCHA area the application supplies a token check returning a valid field, and the library acts on it by asking whether the token is not valid. Renaming that field made every token invalid, and the measured line reads genuine-login=refused(token-invalid(undefined)) for every request including the real customer's. A loud, immediate, unmistakable outage - and nobody who should not get in got in.
The agent runtime returns a ran field for each tool call, and the application acts on it by asking whether the call ran. Renaming it took every call to refused and the budget line to calls-executed=0/8. The agent stops working entirely.
The credential verifier returns an ok field. Renaming it produced legitimate-login=REJECTED while the wrong password was also rejected - nobody logs in, everybody notices, and the security property holds throughout.
The replay recorder returns a field naming which masking rule fired. Renaming it left every value correctly masked and the reported route reading undefined: no security consequence at all, only a telemetry loss, because nothing acts on that field.
Four different libraries, four flags read as permissions - is it valid, did it run, is it ok - and a missing permission is falsy, so the answer is no. That is a fail-closed answer arrived at by accident, and it is the direction you want the accident to point.
The rule that falls out of it
Write the flag your collaborator hands back as a permission, never as a prohibition. A field meaning "this is allowed" degrades to refusal when its name moves. A field meaning "this is denied" degrades to permission. The cryptography, the policy and the care taken over the guard itself are identical in both cases; the only difference is which way round the boolean is phrased, and that difference decides whether an unreadable answer opens the door or closes it.
This is the same instinct as deny-by-default in policy authoring, applied one layer down, to the interface between your code and somebody else's. It costs nothing at design time and it is close to unfixable afterwards, because by then the field name is in a published API that other callers depend on.
It also generalises past this failure mode. A result field can go missing for reasons that have nothing to do with a build step - a version bump that renames it, a serialisation round trip through a worker or a message channel that drops undefined keys, a partial response, a typo in a hand-written mock. Every one of those turns the flag into undefined, and every one of them is safe if the flag means allow and dangerous if it means deny.
Why a crashing arm is not a safe arm
One measurement in this group needs an explicit caution, because it nearly hid a result. Renaming a group of field names on the credential record crashed outright with a type error, because one of those names is also the method name the module exposes - a pattern matches names, not roles, and a member pattern does not know that one of its targets is a function on an export.
A crash looks like the safe outcome. Nothing was silently permitted; the process stopped. But the crash also means every other name in that group was never exercised, and re-running the group without the colliding name produced the quiet result the crash had been hiding: the record's parameter field read undefined in the migration report while the upgrade itself proceeded correctly.
The same thing happened in the agent area, where a group containing a result field crashed on a length read. Whenever an arm ends in an exception, split the group and run it again. The loud failure is frequently sitting on top of a silent one.
What to do about it
Scope member renaming with MemberRegexp, and make the exclusion cover result field names as well as option names. Most exclusion lists are written from the documentation's options table, which does not list what your callbacks return - the return shape is in a type definition or an example, and it is just as much part of the contract.
Phrase your own returned flags as permissions. If you own both sides of an interface today, you may not own both sides tomorrow, and the phrasing is the cheapest safety margin available.
Do not read a guard's own telemetry as evidence that the guard is effective. "Which validator is installed?" is answered by a type check on the option, and it stays true through every failure in this article. The question worth asking is whether a request that must be refused is refused, and the only way to answer it is to send one.
The area-by-area articles linked below carry the full arm-by-arm captures for each of the systems mentioned here.
Frequently asked questions
What is the difference between this and a downgraded guard?
A downgraded guard is not called: the option name carrying it moved, so the library uses its own builtin. Here the option name survives, your guard is called, it runs correctly, it returns the right decision - and the library reads that decision out of a field whose name has moved.
Which case actually failed open?
The one whose flag meant deny. A reputation check returned its refusal for a known credential-stuffing source, the middleware read the verdict field as undefined, undefined is not the refusal value, and the request was permitted.
Why did the other four fail closed?
Because their flags meant allow - is it valid, did it run, is it ok. A missing value is falsy, so the answer is no, and the result is a loud outage rather than a silent admission. That is a fail-closed answer arrived at by accident.
So what is the actual recommendation?
Phrase the flag your collaborator returns as a permission rather than a prohibition. A field meaning allowed degrades to refusal when its name moves; a field meaning denied degrades to permission. Nothing else about the guard changes.
Does this only happen with obfuscation?
No, and that is the argument for the rule. A result field goes undefined on a version bump that renames it, a serialisation hop that drops undefined keys, a partial response, or a hand-written mock. All of them are safe if the flag means allow.
Why does a crash matter here?
Because it hides things. One group crashed on a name collision with an exported method, which meant every other name in the group went unexercised. Re-running the group without the colliding name produced the quiet result underneath. Split the group whenever an arm ends in an exception.
How do we check our own callbacks?
Send a request the guard must refuse and confirm it is refused, against a build from your real pipeline. Reading the configuration, or asking the library which validator is installed, returns a true answer to a different question and stays true through every failure described here.
Related reading