Measured Behaviour
Published
An average is the statistic nobody thinks can go wrong. It has no percentile to lose and no ordering to disturb. But the limit on disinfection byproducts is not written on 'the average' - it is written on a LOCATIONAL RUNNING ANNUAL average, and both of those adjectives live in an option key. We measured what happens when one of them goes missing.
Two adjectives, and each one is a name
The maximum contaminant level for total trihalomethanes is 80 micrograms per litre and for the five haloacetic acids it is 60. Those numbers are not compared against a sample, and they are not compared against your system's average either. They are compared against an average taken at EACH MONITORING LOCATION SEPARATELY, over FOUR CONSECUTIVE QUARTERS.
Both qualifiers exist for a reason a water engineer can state in a sentence. Per location, because a distribution system is not uniform: the far ends of the network hold water longest and form the most byproduct, so an average over every location is an average the good locations pay for. Four quarters, because formation is seasonal: a single quarter is allowed to be high and a year is not.
Lose either and the library still returns an average. It is a correct arithmetic mean of real laboratory results. It is simply not the statistic the limit is written on, and there is nothing in the number itself that says so.
The fixture, and the location the determination is about
Four monitoring locations, two analytes, four consecutive quarters, thirty-four samples. DBP-01 sits beside the plant and averages 33.3 for trihalomethanes over the year. DBP-02 is mid-network at 50.5. DBP-03 and DBP-04 are at the far ends, where the water has been in the pipe longest.
DBP-03 is the location this determination is about. Its four-quarter average is 81.5 for trihalomethanes against a limit of 80, and 67.0 for haloacetic acids against a limit of 60. It is in violation on both, and 1,850 people are served from it.
Look at the shape of DBP-03's year, because it is the ordinary shape of a system that has changed how it doses and it decides several of the arms below. The two high quarters are the OLDEST two - 95 and 91 - and the most recent quarter is the lowest of the four at 62. A running annual average exists precisely so that a good quarter cannot erase a bad year.
The correct run forms eight averages, raises two violations, and the price counter - people drinking from a location that is genuinely over, on a run that cleared it - reads zero.
Renaming the grouping pools the whole system into one number
Rename the option that says 'per location' and the library falls back to one average for the whole system. Eight buckets become two. The trihalomethane average across the system is 57.0 against a limit of 80, and the haloacetic acid average is 41.0 against 60. Both clear. VIOLATIONS goes from 2 to 0 and 1,850 people are at a location that was cleared and should not have been.
Every result is still in the calculation. Every result is still printed in the schedule. DBP-03's 95 and 91 are averaged in, along with DBP-01's 28 and 34, and the arithmetic is exactly right. What is wrong is which records shared a bucket.
Now rename the LOCATION FIELD on the records instead. The counters are identical: two buckets, no violations, the same 1,850 people. But there is a difference that matters more than the counters do. The configuration line still reads 'averaged at each location separately', because that option is intact and doing exactly what it says. The engine grouped by location. There was one location, called undefined, and every sample joined it.
A populated configuration screen is evidence that a setting exists. It has never been evidence that anything found the data it refers to.
The window, and a union that is identical to the correct run
Rename the option carrying the window length and it falls back to one quarter - the most recent. DBP-03's trihalomethanes for the latest quarter are 62, comfortably under 80, so that violation disappears. Its haloacetic acids for the same quarter are 63, still over 60, so that one survives. VIOLATIONS goes 2 to 1 and the price counter stays at zero, because the location still carries a violation and still gets attention.
That is a genuinely useful half-result: the shortened window cleared one analyte and not the other, from the same records, in the same run. A spot check on the analyte that survived would conclude the software was working.
Renaming the QUARTER FIELD is the loudest arm in the area. Every sample's quarter collapses to one value, so no bucket has four distinct quarters, and the full-window requirement refuses all eight: SHORT=8, no determination, the run reports failure.
And then the cell we pre-registered, which is the reason to run unions at all. Rename the quarter field AND the full-window requirement together - two names on one object, easily reached by one pattern - and the counters are BYTE-IDENTICAL TO THE CORRECT RUN. Eight buckets, two violations, six cleared, price counter zero. With nothing to trim the window to, the averaging falls back to averaging everything in the bucket, which for this data is the right four quarters.
A wider pattern that produces the correct answer while its own narrower half fails loudly is not a comforting result. It means the dangerous cells in a matrix like this are not at the wide end, and a team that tested only the realistic broad pattern would have seen nothing at all.
The two samples that were the investigation, not the evidence
After DBP-03's August results came back high, the utility went out the following week and sampled again with the hydrant flushed first. Both repeat samples read low, for the reason flushing makes a sample read low. They are real measurements and they belong in the file. They are the INVESTIGATION of the high result, not a second opinion about it, and averaging them into the compliance figure averages in the response to the problem.
Rename the option that excludes them and both rejoin. DBP-03's trihalomethane average falls to 73.2 and its haloacetic acid average to 58.0. Both clear. VIOLATIONS goes from 2 to 0 and the 1,850 people appear on the counter.
Rename the SAMPLE KIND FIELD instead and the counters are byte-identical, because a sample whose kind cannot be read does not match 'repeat' and is therefore not dropped. This is the same polarity result area 1 measured on invalidated samples, reproduced independently on a different mechanism in a different area: where a record value must match to be DROPPED, losing either half of the pair fails the same way.
A lookup that misses is a fallback, not an error
Two more arms are worth reporting because they are the shape that appears in every configuration-driven system.
Rename the per-analyte limit table and every analyte falls to the catch-all. In our fixture the catch-all was set to 80, so trihalomethanes were unaffected and haloacetic acids - whose real limit is 60 - were suddenly measured against 80. DBP-03's 67.0 cleared. VIOLATIONS went 2 to 1, and the location kept its other violation, so nobody was left unnotified. Rename the catch-all as well and both analytes are measured against the library's own 250, which nothing in the fixture approaches: VIOLATIONS 0 and 1,850 exposed.
Rename the ANALYTE FIELD and something different happens. The bucket key is location plus analyte, so losing the analyte pools trihalomethanes and haloacetic acids together, per location. Four buckets instead of eight, an average of two unrelated quantities, compared against a limit reached by a lookup that also missed. Everything clears.
In each case the missing key is not an error. A table lookup that finds nothing returns a default, a bucket key that has not been seen before creates a bucket, and neither of those is a condition anybody would raise an exception for.
What to do about it
The mitigations here are design decisions worth making whether or not you obfuscate anything, because a dropped key over a serialisation hop produces the same miss.
Make a missing grouping key an ERROR rather than a new bucket. An average over a bucket called undefined is not a result; it is a bug that reported successfully. Make a missed table lookup an error rather than a permissive fallback, or at minimum make the fallback the strictest cell rather than the loosest. And check the number of buckets you formed against the number of monitoring locations your sampling plan says you have - that one assertion catches both the grouping arms in this article, and it is a single line.
For the obfuscation side specifically: every arm above was an option key on a configuration literal or a field name on a data record. A member-renaming regexp scoped to your own model classes reaches neither. The setting that decides whether your compliance calculation is the one the regulation describes should not be a renameable member of an object literal you hand to a library.
Frequently asked questions
What is a locational running annual average?
An average of the results at one monitoring location over four consecutive quarters, compared against the limit at that location on its own. Both qualifiers matter: per location because a network is not uniform, and four quarters because byproduct formation is seasonal.
Does obfuscation change any laboratory result?
In our measurements, no. Every result was printed unchanged in every arm on both member-renaming presets. What moved was which results shared a bucket and how many quarters that bucket spanned.
What happens if the per-location option is renamed?
The library we used falls back to one average for the whole system. In our fixture that took eight averages down to two, cleared a location whose own annual average was over both limits, and left 1,850 people at a location that should have been in violation.
Why is renaming the location field worse than renaming the option?
The counters are the same, but the configuration line still reads 'averaged at each location separately' - which is true. The engine did group by location. There was one location, and every sample joined it. A configuration screen that reads correctly is not evidence that anything found the data.
Can a wider rename pattern be safer than a narrower one?
We measured a case where it was identical to the correct run. Renaming the quarter field alone refused every location for a short window; renaming it together with the full-window requirement produced counters byte-identical to the correct answer. That is why testing only the broad realistic pattern can miss everything.
Should repeat samples be in the compliance average?
Not in the arrangement we modelled. A repeat sample is taken after a high result, usually after flushing, to find out whether the condition persists. Averaging it into the compliance figure averages in the response to the problem, and in our fixture doing so cleared both violations.
What single assertion would have caught most of this?
Comparing the number of buckets the run formed against the number of monitoring locations the sampling plan says exist. That one check catches both grouping arms in this article, and our own rule - which verified that averages existed and that no window was short - passed in every arm that cleared the system wrongly.
Related reading