Measured Behaviour
Published
Areas that lose a statistic still end in a number you can argue about. A filter breach is not a number. It is an ADJACENCY - two consecutive readings, on the same filter, in time order, both above the ceiling - and three properties of how the readings are ARRANGED have to hold before any comparison happens. We measured what a rename does to each.
Three things that are not properties of any reading
An individual filter is allowed to run at a low turbidity and is not allowed to pass particles. The way that gets expressed in a monitoring system is a ceiling and a run length: a single reading above the ceiling is instrument noise, and two consecutive readings above it is a filter doing something wrong. That definition contains three things that no reading carries.
THE GROUPING decides which readings can be adjacent to each other at all. Pool four filters into one stream and two readings that were fifteen minutes apart on one filter now have three readings from other filters between them.
THE ORDERING decides what consecutive means. Readings do not arrive from a plant historian in time order; they arrive in the order the poll returned them, which in our fixture - and in every historian export we have seen - means two exports concatenated.
THE EXCLUSION decides which readings are eligible. A filter coming back from backwash spikes every time, by design, and the first minutes after a backwash are not compliance readings.
Lose any one and every reading is still present, still correct, still printed above the ceiling, and the breach is not there.
The trace, and the filter the assessment is about
Four filters, eight readings each at fifteen-minute intervals, plus two readings from the combined-effluent analyser. Thirty-four readings, twenty-six of them eligible once the backwash windows and the combined stream are set aside.
F-3 is the filter this is about. It comes back from backwash, settles, and then at minute 30 and minute 45 reads 1.35 and 1.22 NTU against a ceiling of 1.00 - two consecutive readings above it, which is a reportable breach that owes a filter profile. Both readings are printed above the ceiling in every arm below.
F-4 is a control built into the data rather than bolted on beside it. It spikes to 2.40 and 1.45 in the first eleven minutes after its own backwash and then runs clean all shift. That is a filter working correctly, and the backwash exclusion is the only thing standing between it and a breach it did not have.
The correct run: 26 eligible readings, four streams, two readings over the ceiling, one breach, one profile owed, and 3.4 megalitres of water that left a breaching filter and was correctly reported.
Renaming the grouping dissolves the adjacency
Rename the option that says 'per filter' and the library assesses one stream for the plant. Both of F-3's readings are still eligible, still above the ceiling, and OVER-CEILING still reads 2 in the output. But sorted into one plant-wide sequence they are no longer next to each other: F-1's, F-2's and F-4's readings at minutes 30 and 45 sit between them. The longest run of consecutive readings above the ceiling on the pooled stream is one. BREACHES goes from 1 to 0, no profile is owed, and 3.4 megalitres go unreported.
Rename the FILTER FIELD on the records instead and the counters are identical - with the difference that the configuration line still reads 'assessed on each filter separately'. The engine did assess per filter. There was one filter, called undefined, and every reading belonged to it.
The readings do not arrive in time order
This is the arm that is easiest to dismiss on a code review and hardest to find afterwards.
Our fixture holds the readings exactly as the historian returns them: batch A first, carrying the readings on the hour and the half hour, then batch B carrying the quarter hours. That is invisible in a data file and completely ordinary in practice.
The engine sorts by minute before looking for a run, which is correct. Rename the MINUTE FIELD and the comparator reads undefined on both sides, the subtraction produces NaN, V8 leaves the pair alone, and the sort is a no-op. What survives is the poll order. In poll order F-3's 1.35 at minute 30 and its 1.22 at minute 45 are separated by the readings at minutes 60 and 90 that came in the same batch. The longest run is one. BREACHES goes to 0 and 3.4 megalitres go unreported.
Four streams. Twenty-six eligible readings. Two readings above the ceiling, printed. A correct sort - of the wrong sequence. Nothing in the output says the ordering failed, because from the sort's point of view it did not: it was handed a comparator that declared every pair equal and it honoured that faithfully.
Two exclusions, one object, one meaning, opposite directions
This is the finding we pre-registered across three areas, and this file is where it is sharpest, because both halves of the contrast sit on the SAME options object and express the SAME intent: do not assess readings that are not compliance readings.
The backwash exclusion is written as a numeric comparison: skip a reading whose minutes-since-backwash is below the window. Rename the option and nothing is excluded. Rename the FIELD and the comparison is made against NaN, which is false, so nothing is excluded either. Both arms and their union are byte-identical: 32 eligible readings, six above the ceiling, and TWO breaches - because F-4's legitimate backwash spike now counts, and a filter that was correctly returned to service is declared in breach. That direction is expensive but it is loud, and somebody takes a good filter offline.
The individual-stream exclusion sounds like the same kind of thing. In English it is: leave the combined-effluent analyser out of the individual filter assessment. In code it is written the other way up - keep a reading whose stream kind equals 'individual'. Rename the option and the combined readings join in: five streams, four readings over the ceiling, two breaches. Rename the FIELD and every reading's stream kind reads undefined, undefined does not equal 'individual', and NOT ONE READING IS ELIGIBLE. The run refuses outright.
So: two exclusions, one object, the same intent, and their record halves fail in opposite directions. The determinant is not what the criterion means. It is whether the unreadable value is compared by MEMBERSHIP - where matching nothing excludes everything - or by MAGNITUDE, where every comparison against NaN is false and nothing is excluded at all.
That is worth carrying past this article, because it says something about how to write a filter you intend to be robust. A membership test fails closed and a numeric test fails open, and the English description of the rule does not tell you which one you wrote.
The ceiling and the run length
Two more arms, both of the shape this series keeps finding.
Rename the ceiling and the library substitutes 5.0 NTU. That is not a placeholder; it is a real figure from an older standard. Nothing in the trace reaches it - not even F-4's 2.40 backwash spike - so OVER-CEILING drops from 2 to 0 and there is nothing for a run to be made of. This is the most audit-proof of the arms, because the count that gave the game away in the grouping and ordering arms is itself taken against the ceiling that moved.
Rename the run length and the library debounces at three consecutive readings instead of two. F-3's run of two no longer qualifies. OVER-CEILING still reads 2, both readings are still printed above the ceiling, and BREACHES reads 0.
In every one of these the price counter reads 3.4 megalitres: water that left a filter which really was passing particles, on a run that did not report it.
The shift you would have tested against
We ran a negative control beside every arm: the same thirty-four readings at the same minutes on the same four filters, on a shift where every filter ran clean. That is the fixture almost everybody builds, because it is the shift you report.
Across all five areas in this pass we ran 79 arms against their controls and measured which ones the control's own decision caught. It caught eleven. Every one of the eleven is an arm that makes a GOOD trace read as a bad one - a reading that cannot be parsed, a stream kind that excludes everything - and those fail on the first run, loudly, before anything ships.
Every arm that dissolves an adjacency is in the other list. A trace with no two readings above the ceiling has no adjacency to lose, so switching off the grouping, the ordering, the run length or the ceiling changes nothing about it at all. The fixture cannot show you the failure because the failure needs a problem to hide, and the fixture has no problem in it.
That is the argument for keeping one deliberately bad fixture in your test data, and it costs one file.
What to do about it
The obfuscation-side fix is narrow and mechanical. Every arm above is an option key on a configuration literal or a field name on a data record. A member-renaming regexp scoped to the members your own model classes own reaches neither, and that is the whole of it. If you are sweeping an options object you hand to a library, you are renaming the library's contract.
The design-side fixes are worth having regardless, because a dropped column in a historian export produces every symptom in this article. Sort by an explicit key and ASSERT that the sequence is monotonic afterwards - a sort that did nothing is detectable in one line. Make an unreadable grouping key an error rather than a new group; a stream called undefined is not a stream. And check the number of streams you assessed against the number of filters the plant has, which catches both grouping arms here.
Finally: if you write an exclusion as a numeric comparison, remember what it does when the number will not read. Our backwash window and our stream filter meant the same thing to a reader and opposite things to the machine.
Frequently asked questions
What is a filter turbidity breach, exactly?
In the arrangement we modelled, two consecutive readings on the same filter, in time order, both above the ceiling. All three qualifiers matter and none of them is a property of any single reading, which is what makes this different from a simple threshold.
Does obfuscation change any instrument reading?
In our measurements, no. Every NTU value was printed unchanged in every arm on both member-renaming presets, and the two readings that constitute the breach were printed above the ceiling in every arm - including the ones that reported no breach.
How can readings be out of order in a monitoring system?
Historian exports are commonly assembled from more than one pull. Our fixture holds the readings as a two-batch export, which is ordinary and invisible. The engine sorts by minute before looking for a run, so the minute field is load-bearing rather than decorative.
What happens when a sort comparator cannot read its key?
It returns NaN, which V8 treats as leaving the pair alone, so the sort completes without reordering anything. You get a correct sort of the wrong sequence, and nothing in the output reports a problem.
Why did two exclusions on the same object fail in opposite directions?
Because of how each test is written rather than what it means. The backwash exclusion compares a number, and every comparison against NaN is false, so nothing is excluded. The stream filter compares a string against 'individual', and a value that reads undefined matches nothing, so everything is excluded and the run refuses.
Does a clean test fixture catch any of this?
Very little of it. Across the five areas in this pass we ran 79 arms against clean-fixture controls and the controls' own decisions caught eleven, all of them arms that make good data read as bad. A trace with no breach in it has no adjacency to lose.
What is the cheapest check to add?
Assert that your sorted sequence is monotonic in the key you sorted on, and compare the number of streams you assessed against the number of filters the plant has. Between them those two lines catch the ordering arm and both grouping arms in our measurements.
Related reading