Measured Behaviour
Published
Every failure we have measured in this series ends in a system that produced a wrong answer, and every one of them is findable in principle by somebody running the system, because the system is what reads the field. A certification is read by an auditor, years later, and by nobody in between. We went looking for what that changes.
A payroll where every number stayed right
We built an effort period the way a real one is shaped: twenty-four salary charges across three awards and eight people, six of the eight earning above the statutory cap, nine-month academic appointments, a cap of 221,900 dollars quoted on a twelve-month basis and therefore available only nine twelfths, a five per cent tolerance between what payroll charged and what was certified, and one certification per person per award signed inside a ninety-day window by somebody entitled to sign.
The correct run charges 353,001.25 to the awards, holds 138,048.75 as institutional cost share because the cap bites on six salaries, declines to charge four salaries whose certifications attest to a percentage nine points from what payroll ran, and reports twenty certifications, all attributable to a named individual. Everybody is within a hundred per cent effort. The institution's own commitment check passes.
Protection alone, on all five profiles, produced identical behaviour. What follows came from member renaming reaching the certification records, and the mechanics of scoping that pattern are in Protect Members.
One rename, and not a single figure moved
The certification record carries six fields. We renamed the one that records who signed it.
Charged to the award: 353,001.25. Held as cost share: 138,048.75. Uncertified: zero. Late certifications: zero. Wrong role: zero. Effort mismatches: four, worth 73,182.08. Over-committed: none. The institution's commitment check: passed, reporting 353,001.25 charged and eight people all within a hundred per cent effort.
Every one of those figures is identical to the correct run. Not close, not within rounding - identical. The payroll journal foots. Nothing throws, nothing warns and no control anywhere in the system has anything to say.
What changed is a string. Twenty evidence lines that read by=STAFF-611 now read by=undefined, and a counter went from twenty certifications attributable and none unattributable, to none attributable and twenty unattributable.
That is the whole difference. There is 353,001.25 dollars of salary on those awards and there is no longer a certification behind any of it - not a wrong certification, not a missing one, but twenty documents that assert somebody verified the work and decline to say who. At audit that is not a certification, and the salary it covers is a questioned cost against work that was really done, by people who were really there, at a price that was always correct.
The quieter version: the counter that measures evidence read perfectly
We renamed a second field on the same record - the certification's own identifier, the thing that ties an evidence line back to a document in the institution's system.
Every figure was identical again, including the attributable counter. It still read twenty attributable and none unattributable, because that counter is computed from the signer's name, and the signer's name was fine.
The only trace anywhere in the run is that SAL-88001<-CERT-7304 became SAL-88001<-undefined. The one metric in the system that exists to measure whether the evidence is any good reported that the evidence was perfect, because it measures a different field of the same record.
A third field, the payroll source document reference, behaved the same way: every figure identical, doc=PR-5101 became doc=undefined, and no counter moved at all.
Three fields on the same record stopped the run dead
This is the part that makes the shape precise rather than anecdotal. The certification record has six fields and we renamed each of them separately.
Renaming the person the certification covers: nothing charged at all. Zero dollars to the awards, twenty-four charges reported as uncertified. Renaming the day it was signed: zero charged, twenty-four late certifications. Renaming the signer's role: zero charged, twenty-four rejected for a role not permitted to certify.
Three of six fields halt the institution's payroll charging completely and loudly, on the first run, in a way nobody could possibly miss. Two of six move nothing at all.
The split is not luck and it is not about how important the field sounds. The three loud ones are consumed by the engine: it matches on the person, it compares the date to a window, it checks the role against a permitted list. The two silent ones are consumed by an auditor and by nobody else, because no library can know whether a given employee has suitable means of verifying that the work was performed. That is a fact about an org chart, not about software.
A field your system reads fails closed and screams. A field only your regulator reads fails open and says nothing, and the loss lands years later against a charge that was always proper.
The fourth field is the one that pays out
The remaining record field is the percentage the certification actually attests to. Renaming it took charges from 353,001.25 to 426,183.33 - the four salaries that payroll ran nine points away from what anybody certified were charged in full, worth 73,182.08.
And the evidence counter improved. It went from twenty attributable to twenty-four, because those four charges are no longer being declined, so four more certifications now appear in the evidence line. The run looks better in the only place it looks at all, while charging seventy-three thousand dollars of salary against attestations that say something different.
For completeness, the option keys behaved as this series has measured repeatedly. Renaming the cap took charges to 491,050.00 and cost share to zero. Renaming the appointment length un-prorated the cap, printing 221,900.00 instead of 166,425.00. Renaming the certification requirement took charges to 426,183.33 and emptied the evidence line entirely. Renaming the permitted roles alone moved no money, because every certification already had a permitted role - and renaming it together with the requirement flag was identical to the flag alone, because the flag is checked first.
What to do about it
Nothing here is an argument against protecting the code, and the default profiles moved nothing. The exposure is a member-renaming pattern reaching records that leave your process, and that is a choice you make.
The general rule this measurement suggests is short. Before pointing a rename at a record, ask who reads each field. Fields your own code reads will tell you loudly when they move. Fields that exist for somebody else - a regulator, an auditor, a counterparty, a court - will not tell you at all, because nothing in your process consumes them.
Concretely: exclude evidence records from the member pattern, and assert their content rather than their count. A test that pins twenty attributable certifications passes on the arm that removed every identifier. A test that asserts the evidence line contains a certification id, a signer and a source document for each charge fails on all three silent arms.
And treat a clean reconciliation as evidence about money and nothing else. In the arm that mattered most here, every figure a reconciliation could compare was identical to the cent.
Where the standard is about demonstrable resilience rather than arithmetic, the verification-level framing in OWASP MASVS and JavaScript Resilience is the closer analogue.
Frequently asked questions
Did obfuscation change what was charged to the awards?
Not on its own. All five profiles we tried - the ES5 target, the modern target, both emit-gate configurations and the string-encoding profile - produced identical behaviour to the original. Every difference in this article came from member renaming reaching the certification records.
What exactly changed when the signer field was renamed?
One string and one counter. Charged to the award stayed 353,001.25, cost share stayed 138,048.75, uncertified and late and wrong-role all stayed zero, and the commitment check passed identically. The evidence lines went from naming a person to reading undefined, and twenty certifications became unattributable.
Why do three fields on the same record fail loudly and two silently?
Because of who reads them. The person covered, the date signed and the signer's role are all consumed by the engine, so losing one stops the match, the window check or the role check and nothing is charged. The certification identifier and the signer's name are consumed only by an auditor, so losing them changes no decision at all.
Would our reconciliation have caught it?
No. Every figure a reconciliation compares was identical to the cent, including the total charged, the cost share held, the effort percentages and the count of exceptions. A reconciliation is evidence about money, and the thing that moved was not money.
Is there a counter that would have flagged it?
The run reports how many certifications are attributable, and that counter did move for the signer field, from twenty to zero. It did not move for the certification identifier or the source document reference, because it is computed from the signer's name. One metric does not cover a record with six fields.
What is the cheapest test that catches all three silent arms?
Assert the content of the evidence line rather than its count: every charge must map to a certification identifier, a named signer and a source document reference, none of them the string undefined. That fails on all three arms where no money moved.
How do we keep renaming away from evidence records?
Scope the member regexp so records that leave your process are excluded, and keep renaming pointed at logic rather than at data you hand to somebody else. The option and regexp mechanics are documented in Protect Members.
Related reading