Observability
Published
Every guard we have measured in this series so far protects something inside your system. Telemetry is the exception: its options decide what leaves. A redaction list, an attribute length cap and a query-string scrubber are the difference between exporting the shape of your traffic and exporting its contents, and the exporter's defaults are permissive because a general-purpose SDK cannot know what your attributes contain.
What the sample actually does
The file configures a tracing exporter with seven settings: a five percent sampling rate, a redaction list naming four attribute keys, a 64-character attribute length cap, query-string scrubbing on, a cardinality cap of 50 series per metric, one setting pinned to the value the SDK already uses, and an internal collector endpoint.
Two spans carry the interesting data. A checkout span carries an authorization header, a card number, a URL with a token in the query string, a large request body and a user identifier. A database span carries a SQL statement containing a customer email address, the database system name and a session cookie. A metric carries 400 distinct user identifiers as labels, which is what a cardinality cap is for.
A second profile exports at full rate, because at five percent the database span is never drawn and two of the four redaction keys would have reported blind. That is the same fixture problem this pass hit three times: an arm whose guard never runs is not clean, it is uninformative. With the full-rate profile in place every redaction key fires exactly once somewhere in the unprotected run.
The exporter is copied unprotected. Its defaults are the permissive ones: sample everything, redact nothing, no length cap, no query scrubbing, unbounded cardinality, and a localhost endpoint. Protection alone, on all five presets, behaved identically to the unprotected file.
The redaction list is the whole article
Renaming redactKeys alone took four attributes from [redacted] to their actual values. The authorization header exported as a complete bearer token. The card number exported as sixteen digits. The session cookie exported with its session value. The SQL statement exported with a customer email address in the WHERE clause. Our credential detector went from false to true on both profiles.
Nothing about this is an error condition. The exporter did precisely what it was configured to do, which after the rename was to redact nothing. Every span was still sampled, still serialised, still sent. Latency did not change. No log line was produced. The application is healthy and its traces are complete, and they contain credentials and personal data.
The destination is what makes this different from every other quiet failure in this series. Telemetry endpoints are frequently third-party services, frequently in a different jurisdiction, and frequently retained for months with broad internal read access - because trace data is not supposed to be sensitive. A redaction list is the control that makes that assumption true, and it is one property name.
It is also the failure least likely to be found by testing, because the data is correct at every point you would look. The span is right. The trace is right. The dashboard is right. The only thing that is wrong is who can read it, and that is not a property any assertion in your test suite is written to check.
Length, query strings, and the two things that are only expensive
Renaming maxAttributeLength removed the 64-character cap and exported a 1,239-character request body attribute in full. That is the same failure as the redaction list arriving through a different door: the request body was never on the redaction list because it was never supposed to arrive intact.
Renaming scrubQueryString exported the checkout URL with its query string, including the reservation token we planted there. Tokens in query strings are common enough to be worth naming as a category, and query-string scrubbing exists precisely because URLs end up in more places than headers do.
Two arms cost money rather than data. Renaming sampleRate took the exporter from sampling one span in our two-span set to sampling both - in the real configuration that is a twenty-fold increase, and the bill and the collector's ingest limits are where it shows up. Renaming maxCardinality took a metric from 50 series with 350 label values dropped to 400 series with none dropped, which is how observability platforms produce a surprising invoice and, past a threshold, start rejecting writes.
Neither of those is a security failure and both are worth separating out, because the arms an audit should prioritise are the ones where the wrong outcome is somebody else holding your data rather than you holding a larger bill.
The endpoint arm inverts the whole failure mode
Renaming endpoint moved the destination from the internal collector to the SDK's default of localhost. In every other arm in this pass a lost option sends more than intended; this one sends it precisely nowhere. There is no collector on that port in production, so the exports fail silently in the background the way telemetry exports are designed to.
The result is an application that looks completely healthy and produces no traces. That is the observability equivalent of a smoke detector with the battery out, and the thing about it is that the absence of alerts is indistinguishable from the absence of problems until you go looking. The detection is a heartbeat: a synthetic span emitted on a schedule and alerted on when it does not arrive.
It is a useful counterweight to the redaction arm. One rename in this area exports everything to the wrong people; a neighbouring rename exports nothing to anybody. An audit that checks one proves nothing about the other.
The control, and both directions
One option was pinned to the SDK's own default and renaming it changed nothing on either target, which is the sixth area in a row where that prediction held exactly. As always, it is a fact about this configuration: an application that had turned request-body capture on would find the same rename turning it off.
Renaming the span attribute container crashed the exporter with a type error on the first span, which is the loud in-direction shape we have now seen in eleven areas out of eleven.
The out direction needed the fixture repaired before it could be measured at all. Our first version never read the exporter's result fields, so the arm renamed names that appear nowhere in the file and reported no change for a reason that had nothing to do with the engine. Once the sample read the counts the exporter hands back, the arm reported undefined/undefined for sampled-versus-offered - a dashboard tile that silently stops having a number in it while the exporter continues working correctly.
What to do about it
Exclude the exporter's option names, and put the redaction list at the top of that list. Of everything measured across five areas this pass, it is the one whose failure sends data outside your control while leaving no trace anywhere in your system.
Independently of protection, redaction is stronger when it is not the only layer. Attributes that are never attached cannot be exported, so filtering at the point where spans are created - rather than only at the point where they are exported - means a lost option changes the size of your traces rather than their sensitivity.
Test the decision. A test that creates a span with an authorization attribute, runs it through your exporter pipeline, and asserts the exported value is the redaction marker catches the redaction arm, the length arm and the query-string arm. A test that asserts telemetry is initialised catches none of them, and would also pass in the arm where every export goes to localhost and disappears.
Frequently asked questions
Does obfuscation break telemetry or tracing?
Not by itself. Protection alone, on all five presets we tested, behaved identically to the unprotected file. The results here required member renaming pointed at the exporter's option names.
What is the most serious result you measured in this area?
Renaming the redaction list. A bearer token, a card number, a session cookie and a SQL statement containing a customer email address were all exported verbatim to the configured endpoint. Every span still arrived, nothing failed, and no log line recorded it.
Why is a telemetry leak different from the other failures in this series?
Because the data leaves your system. Other quiet failures leave you in a weaker state internally; this one puts credentials and personal data into an endpoint that is frequently a third-party service with long retention and broad internal read access, on the assumption that trace data is not sensitive.
Would anything in our monitoring detect it?
Not readily, because the observable outputs are all correct. The spans are complete, the traces are well formed, the dashboards render. The only thing that changed is who can read the contents, which is not a property test assertions are usually written to check.
Which telemetry options are expensive rather than dangerous?
The sampling rate and the cardinality cap. Losing the sampling rate multiplied the exported span count, and losing the cardinality cap took a metric from 50 series with 350 label values dropped to 400 series with none. Those produce invoices and ingest-limit rejections rather than disclosure.
What happens if the exporter endpoint is renamed?
It reverts to the SDK's localhost default, so exports silently go nowhere while the application looks healthy. It is the inverse of the redaction failure, and the practical detection is a synthetic span emitted on a schedule with an alert for when it fails to arrive.
What should I exclude from member renaming here?
The exporter's option names, the attribute containers on the spans you build, and the count fields on whatever the exporter hands back. In our sample that was the sampling rate, redaction list, length cap, query scrubber, cardinality cap and endpoint, plus the span attributes object.
Related reading