Measured Behaviour
Published
A lead compliance calculation has obvious failures everybody tests for: a result that will not parse, a missing sample, a concentration that comes back negative. It has one almost nobody tests for, because every result stays genuine and every exceedance stays visible in the report - the POSITION in the ordering that the action level is compared against. We measured what a rename does to it.
The rule is not written on any one tap
The lead action level is 0.015 mg/L. That figure is famous, and almost everything people assume about how it is applied is wrong. It is not a limit that any individual tap has to meet. It is not compared against the average of your samples, and it is not compared against the worst of them. It is compared against the ninetieth percentile of a pool of taps chosen for being the most likely in the system to have a problem.
Every one of those qualifiers is deliberate. The percentile is ninety rather than a hundred because the rule is a judgement about a system, and a system with one bad tap is not a system that has failed. The pool is restricted to high-risk plumbing because a statistic about lead taken over premises with no lead in them is a statistic about nothing. The samples are first-draw after six hours of standing because water that has just run through the tap is a measurement of the main, not the premises.
So there are at least three separate things that have to be true of the ARRANGEMENT of your samples before the comparison means anything, and not one of them is a property of any sample. That is what makes this worth measuring. Everything this series has measured so far ends in a number that is wrong. Here the numbers can all be right.
What we built and what the correct run says
Thirty-six taps on one distribution system. Twenty of them are the pool: tier 1 is a lead service line, tier 2 is copper pipe with lead solder, all valid, all first-draw, all six hours or more of standing. Ten more are tier 3 - plastic service, plastic plumbing, no lead anywhere in them - which were sampled, whose results are real, and which the rule does not pool. Four were invalidated by the laboratory for a documented reason: they were analysed outside the holding time for the method. Two were collected after somebody had already run the tap.
The correct run pools twenty samples, computes a ninetieth percentile of 0.0190 mg/L, counts three taps over the action level, and concludes ACTION REQUIRED. That last word is the whole point of the exercise: it is what triggers public education, notification of the people drinking the water, and a corrosion-control review. Our price counter - the number of people drinking from a tap that is genuinely over, on a run that notified nobody - reads zero, because everybody who needed telling was told.
Three taps are over: 0.019, 0.024 and 0.031 mg/L. One of them is a school. Between them they serve 715 people. Hold on to that number, because it is the only thing that changes in most of what follows.
The percentile is a position, and a position is not a value
The first arm renames one option: the percentile itself. The library it is handed to computes summary statistics over a pool of results. It does not know which regulation is being applied to them, and the ordinary summary of a set of numbers is its median, so that is what it falls back to.
The result: the reported statistic drops from 0.0190 to 0.0070 mg/L, and the determination goes from ACTION REQUIRED to none. 715 people are not notified.
Now read the rest of the same output line, because this is the part that decides whether anybody catches it. The pool is still twenty samples. Every result is printed in the schedule exactly as the laboratory reported it. And SITES-OVER-LEVEL still reads 3. The report says, on one line, that three taps are over the action level and that no action is required. Both halves of that sentence are true statements about the run that produced them. The three high results are not hidden, not suppressed, not rounded away; they are right there, and the compliance determination beside them is a different question with a different answer.
A reviewer checking that the results are real, that the pool is the right size, and that the exceeding taps appear in the report finds nothing wrong, because nothing about any of those is wrong.
Ten premises with no lead in them, and the answer changes
The second arm does not touch the statistic at all. It renames the option carrying the list of eligible tiers. An empty list of eligible tiers is not a strict list - it is indistinguishable from a list that was never supplied, so the library pools everything.
The pool goes from 20 to 30. Every one of the ten new samples is a genuine measurement at a genuine tap on the system. Every one of them is low, because there is no lead in those premises. The ninetieth percentile falls to 0.0140 mg/L - just under the action level - and the system clears. The same 715 people are not notified.
Nothing was hidden and nothing was invented. The three high results are still in the pool and still counted; SITES-OVER-LEVEL still reads 3. What moved was where the ninetieth percentile falls, and that is decided by how many samples there are, not by how large any of them is.
There is a detail here worth knowing on its own, because it shapes how much dilution it takes. An order statistic at the ninetieth percentile is robust to a handful of extra samples: adding four low ones to a pool of twenty moves the index by four as well and lands on the same value. It takes about a tenth of the pool to move it one rank. Ten out of twenty is comfortably past that, and a real system has far more tier-3 premises than tier-1 ones. The dilution is not a contrived edge case; it is the shape of the estate.
The same criterion, written two ways, fails two ways
This is the finding we pre-registered, and it came out sharper than we wrote it down.
The pool is decided by criteria, and criteria come in two shapes. TIER is an inclusion criterion: a sample must match a list to be kept. INVALIDATION is an exclusion criterion: a sample must match to be dropped. Each is an option key plus the record field that option reads, so each has two halves that can be lost.
For the exclusion criterion the two halves are indistinguishable. Rename the option and invalidated samples rejoin the pool; rename the flag and nothing reads as invalidated, so they rejoin the pool. Both arms and their union produce byte-identical counters: pool 24, statistic 0.0350, seven taps over. The run gets STRICTER, because the four invalidated results happened to be high ones.
For the inclusion criterion the two halves are opposites. Renaming the option pools everything, as above. Renaming the TIER FIELD leaves every site uncategorised - and an uncategorised site matches no entry in the eligible list, so the pool is EMPTY. The run refuses outright: pool-too-small, 0 of 20 required, no determination. That is the loudest failure in the whole area, and it comes from the same conceptual place as the quietest one.
And the union tells you which half is load-bearing. Rename both the option and the field and you get the option's result exactly, because once the option is gone the field is never consulted. A wide pattern here is no worse than a narrow one - but only for this pair, and area 3 shows the same structure reversing.
The action level that reverts to a real historical figure
The third route is the most dangerous one to an audit, and it is the least dramatic to read.
Rename the option carrying the action level and the library substitutes its own: 0.05 mg/L. That is not a placeholder and not a corrupt value. It is the figure the standard actually used before 1991. It is defensible-looking, it is documented, and it appears in the configuration dump as a perfectly ordinary number.
The statistic is unchanged at 0.0190 mg/L, because nothing about the pool moved. The determination goes to none. And SITES-OVER-LEVEL, which read 3 in every arm above, now reads 0 - because the count of taps over the level is also taken against the level, and the level is the thing that moved. The one figure that gave the game away in the previous two arms is gone.
715 people are not notified, and a configuration screen reading 'action level 0.0500 mg/L' passes any audit that does not compare it against the regulation.
The guard that catches it, and the guard that turns the guard off
We wrote the utility's own check the way a competent engineer writes one: refuse if no sample was pooled, refuse if the statistic is not a real number, refuse if the pool is larger than the schedule it came from. It works. Renaming the tier field is caught by the engine's minimum pool size before the check even runs.
Take the minimum pool size away as well, and the engine's refusal disappears - but the utility's own rule catches it and prints 'no sample was pooled'. Two independent guards, both doing their job.
Now take away the flag that turns the utility's rule on. The same pattern that lost the tier field can lose it; they are two names on one options object. The result is a compliance determination made on ZERO samples, reported as no action required, with the check line reading 'unchecked', and 715 people not notified. The half that would have screamed was switched off by a half that fails quietly.
That is the reason we keep saying to test the halves rather than the union. Here the union of three names is silent and each of the narrower patterns is loud.
What this means if you ship compliance code to a browser
None of this is an argument against obfuscating a water-quality application. It is an argument about which names you let a member-renaming pass reach, and about what your own checks are defined over.
Three things are worth doing regardless of whether you obfuscate anything, because a hand-edited configuration file and a dropped key over a serialisation hop produce identical symptoms. First, keep the statistic and the level out of the surface member renaming reaches: a member regexp that names the fields you own, rather than one that sweeps an options object, is the whole fix. Second, assert the POOL'S COMPOSITION, not just its size - how many tier-1 sites, out of how many the site-selection plan says exist - because a pool of the right size made of the wrong premises is the failure this article is about. Third, compare the configuration your code is running against the regulation, not against itself; every arm above printed a configuration line that was internally consistent and externally wrong.
And if you take one thing from the measurements: the report that says three taps are over the action level and no action is required is not a contradiction the software will flag. It is two correct answers to two different questions, and only a person who knows which question the rule asks will notice.
Frequently asked questions
Does obfuscating a water quality application change any lead result?
In our measurements, no. Every concentration the laboratory reported was printed unchanged in every arm we ran, on both member-renaming presets. What moved was which samples were pooled and which position in the pool the action level was compared against.
What does renaming the percentile option actually do?
The library we handed the pool to falls back to a median when it is not told which percentile to use. In our fixture that took the reported statistic from 0.0190 to 0.0070 mg/L and the determination from action required to none, while the count of taps over the level still read three on the same output line.
Why does adding low samples change the answer if none of them are high?
An order statistic is a position in an ordering, not a value in the data. The ninetieth percentile of thirty samples sits at a different rank than the ninetieth of twenty, so adding ten low results moved our statistic from 0.0190 to 0.0140 mg/L without changing, removing or hiding a measurement.
How many extra samples does it take to move a ninetieth percentile?
Roughly a tenth of the pool for each rank. Adding four low samples to our pool of twenty left the statistic unchanged, because the index moved by four as well. Ten was enough to shift it one rank and cross the action level.
Why did renaming a tier field fail loudly when renaming the tier option failed silently?
Because the two ask the comparison in opposite directions. The option carries a list a sample must match to be kept, so losing it keeps everything. The field carries the value that is matched, so losing it matches nothing and keeps nothing. Our run refused with an empty pool rather than clearing.
What should a member regexp look like for this kind of code?
Name the members you own rather than sweeping an options object; the MemberRegexp option is what scopes it. Every arm in our measurements that mattered was an option key or a record field on a configuration literal, and a regexp scoped to your own model classes does not reach either.
What check would have caught this?
One defined over the pool's composition rather than its size: how many tier-1 sites were pooled, against how many the site selection plan says exist. Our utility's own check verified that a pool existed and that the statistic was a real number, and both of those were true in every arm that cleared the system wrongly.
Related reading