Measured Behaviour
Published
Twenty-six measured failure shapes into this series, every one of them concerns a record the software could read. This one does not. A recall has to decide what to do about a vehicle whose build record cannot be established at all, and that decision is one flag - which is exactly as losable as any other name.
A campaign is a scope, and a scope has three answers
We built a parc of thirty vehicles against three open campaigns: two safety recalls and one emissions action, each scoped by a build sequence range, a plant, and in two cases an option code. Twenty-six vehicles carry a complete build record and land correctly inside or outside each campaign's scope.
Four do not. Two came through an acquired dealer group whose plant code never reached the record; two are pre-production units whose build sequence was never assigned. A vehicle like that is not out of scope. It cannot be placed either way, and that is a third answer the software has to have a policy for.
This programme's policy is the one a safety engineer writes: a vehicle whose build cannot be established is treated as affected and held from sale, because inspecting a vehicle that did not need it costs a fraction of missing one that did. The correct run reports 30 examined, 14 affected, 4 undetermined and 4 held.
Protection on its own changed none of that. All five profiles - the ES5 target, the modern target, both emit-gate configurations and the string-encoding profile - produced identical behaviour.
The prediction we wrote down before measuring
The flag carrying that policy is one name. Before running anything we wrote down what losing it should do, because a prediction made afterwards is not a prediction.
Vehicles examined should not move. Undetermined should not move, because whether a build record is readable is a fact about the data and not about the policy for handling it. Every vehicle that can be identified should be classified exactly as before, and the per-campaign counts for those vehicles should be unchanged. Only the four unidentifiable vehicles should move, from affected to not affected. And the programme's own completeness rule should pass.
Measured, all of it landed. Examined: 30, unchanged. Undetermined: 4, unchanged. Held: 4, unchanged. Per-campaign counts for the identifiable vehicles: 5, 4 and 1, identical to the correct run. Affected: 14 to 10. Not affected: 16 to 20.
And the completeness rule passed, printing 30 of 30 vehicle(s) examined, 10 affected, 4 undetermined. Every figure in that sentence is true. Four vehicles nobody can identify were released without a remedy, and the line certifying the run reports the undetermined count correctly while saying nothing about what was done with them.
Why every check we have written in this series is blind to it
The mitigations this series has accumulated are all defined over records: every record examined, every total reconciled, every item on the right side of a partition, every requirement between two records checked. This failure violates none of them.
Every vehicle was examined. Every determination that was made was correct. Nothing was dropped, miscounted, double-counted or attributed to the wrong campaign. The population is complete and the answer for each member of it is right. What moved is the rule for what a vehicle IS when the answer cannot be computed, and there is no record for that rule to be wrong about.
The counter that looks like it should catch it is the undetermined count, and it is precisely the counter that cannot: it reports how many vehicles could not be identified, which is unchanged, rather than what happened to them, which is the whole difference.
The blast radius is the part worth carrying. The vehicles this moves are the ones with incomplete records - acquired dealer groups, pre-production units, imports, anything whose data was always the worst in the parc. They are the population least likely to be spot-checked and least likely to be missed.
The fixture most people would build cannot see it at all
We ran the same thirty vehicles a second time with the four incomplete records completed - every plant code present, every build sequence assigned. That run reports 30 examined, 13 affected, 0 undetermined.
In that fixture the flag decides nothing, and every arm in this article is inert. Not inert because the software is correct: inert because there is no doubt for a doubt-resolution policy to resolve.
That is the fixture almost everybody builds. Test data is usually constructed, and constructed data is complete. The class of failure described here is invisible to a suite built that way, by construction rather than by luck.
It generalises past obfuscation, which is why it is worth the words. A dropped column in a build-data import, a supplier feed that stops sending plant codes, or a serialisation hop that silently omits nulls all produce the same unidentifiable vehicle and the same silent release.
Two flags, and losing both is quieter than losing one
The doubt policy has a second half: an undetermined vehicle is also held from sale. We measured that name on its own and together with the first.
Renaming the stop-sale flag alone moves held from 4 to 0 while affected stays at 14. The four vehicles are still called in; they are simply not held in the meantime. That is a real loss and a comparatively mild one.
Renaming both leaves 30 examined, 10 affected, 4 undetermined and 0 held. Four vehicles that cannot be identified are neither recalled nor held, and the run still reports four of them as undetermined while the completeness check passes. The wider pattern is the quieter one, which this series has now measured in seven separate domains.
Switching the completeness rule off as well changes the output by exactly one word - the review line reads unchecked - and not a single vehicle's disposition. The rule was contributing nothing to remove, because it was never able to see this.
What to do about it
None of this is an argument against protecting the code. The engine's default profiles moved nothing in this file. The damage came from a member-renaming pattern reaching a configuration object, and that pattern is something you choose - the scoping mechanics are in Protect Members.
Make the disposition of the unknown explicit in the output, not just in the configuration. The correct run prints a pending-determination bucket with four vehicles in it; the failing arm prints no such bucket. A test that asserts the bucket exists and has the expected size fails on every arm here. A test that asserts the undetermined count fails on none of them.
Put unidentifiable records in the fixture on purpose. If every row in your test data is complete, no assertion you write can distinguish a system that resolves doubt safely from one that resolves it away.
And where a decision cannot be computed, prefer a policy that makes the software stop rather than one that makes it choose. A vehicle that cannot be identified is a question for a person. Guidance on scoping protection for software that ships inside devices is in Protect JavaScript on Kiosks, Digital Signage and Embedded Devices.
Frequently asked questions
Did obfuscation change which vehicles were recalled on its own?
No. On all five profiles we tried - the ES5 target, the modern target, both emit-gate configurations and the string-encoding profile - the protected bundle produced identical behaviour to the original. The differences in this article came from member renaming reaching a configuration object, which is a pattern you choose.
What exactly moved when the flag was renamed?
Affected went from 14 to 10. Examined stayed at 30, undetermined stayed at 4, held stayed at 4, and the per-campaign counts for every vehicle that could be identified were unchanged at 5, 4 and 1. Only the four vehicles whose build record could not be established changed side.
Did the completeness check catch it?
No, and it could not. It passed, printing 30 of 30 vehicles examined, 10 affected, 4 undetermined - every figure true. It reports how many vehicles could not be identified, which did not change, rather than what the software decided to do about them, which is the entire difference.
Why is this different from the failures you have measured before?
Every earlier shape concerns a record the software could read: a value, a unit, a population, a split, a row, a partition, a moment. Here every determination that was made was correct and nothing was dropped. What moved is the rule for what a record IS when no determination is possible.
Which vehicles does this affect in practice?
The ones with incomplete build data - acquired dealer groups, pre-production units, imports, anything whose records were already the worst in the parc. They are the population least likely to be spot-checked, which is what makes the failure durable.
Would our test suite have found this?
Almost certainly not. We ran the same parc with all four incomplete records completed and every arm in this article is inert in it, because there is no doubt left to resolve. Constructed test data is usually complete, so a suite built from it is blind to this class by construction.
What should we assert instead?
The disposition of the unknown, as a visible bucket in the output. The correct run puts four vehicles in a pending-determination bucket and the failing arm produces no such bucket at all. Asserting that bucket fails on every arm here; asserting the undetermined count fails on none. Keeping the pattern away from the configuration is covered in Protect Members.
Related reading