Tracing a Request You're Not Allowed to Log

Four services. Four dashboards, all green. One user watching a run fail.

That's the situation that got me interested in tracing properly, on a platform I worked on as one contributor across a few repositories. The shape was ordinary enough:

TypeScript application service
    │   enqueue evaluation job
    ▼
queue + cloud job layer
    │   invoke worker
    ▼
Python evaluation service
    │
    ▼
result  or  exception

A user submits work. The TypeScript side accepts it and enqueues a job. A queue and cloud job layer picks it up and invokes a worker. Python does the actual evaluation and returns a result — or throws.

Every layer logged. Every layer logged well, in isolation. Each one could tell you that it had received something, done something, and passed something on. What none of them could tell you was whether the thing they handled was the same thing the user was complaining about, because no identifier survived the arrows between those boxes.

So when a run failed, you had four healthy services and four sets of logs that couldn't be joined.

The standard fix is a correlation identifier that travels with the work, and that's most of what I ended up building. But there was a second constraint that made it more interesting than the usual version of this problem.

I wasn't allowed to log the things that would have made debugging easy.

Logging isn't observing, it's copying

The tempting shortcut when a distributed system is opaque is to log everything at every boundary. Dump the request, dump the payload, dump the response, sort it out later in the search bar.

Think about what that actually does. The job in question carried submitted code, model inputs, runtime output, signed resource links, stack traces and authentication material. Logging all of it doesn't observe that data — it copies it, into a completely different system with its own access controls, its own retention policy, its own vendor, and its own backup regime.

That copy is almost always governed more loosely than the original. Far more people can read production logs than can read the production database. Log retention is frequently longer than the retention you promised for the underlying records, and it's rarely covered by the deletion path you built. Your telemetry vendor is a third party you've now given a second copy to.

So "just log it" isn't a shortcut with a small privacy cost attached. It's a decision to create a second, less-governed replica of your most sensitive data, taken casually, usually by whoever was debugging something at the time.

Which leaves the real problem: make one run reconstructable across languages and async boundaries, while reducing how much sensitive content routine telemetry can receive at all. Those pull in opposite directions, and the whole design is about where you let them meet.

Correlation belongs in the message, not in the logger

The first half is the boring half, and it's boring because it's well understood — which doesn't stop people getting it wrong.

The contract is a single canonical field carried in job metadata: metadata.traceId. TypeScript-originated work stamps it and forwards it across the queue and cloud execution boundaries, on both success and failure paths. Python evaluation receives the same identifier, makes it available to tracing and reporting, and keeps it across both completion and exception paths.

The important word is carried. Trace identity has to travel inside the message contract, because everything else you might use to reconstruct it is a guess. Correlating by timestamp falls apart the moment two jobs overlap. Correlating by logger context falls apart the moment work crosses a process boundary, which for a queue is immediately — the producer's context is gone by the time the consumer picks the job up.

If the identifier isn't in the payload, it doesn't survive the queue. That's the whole rule.

Both sides need it on the failure path too, and this is where implementations tend to be half-finished. The success path gets the trace ID because that's the path people test. The exception path is exactly where you need correlation most, and it's the one where somebody has usually written a bare catch that logs a message and moves on.

Adding context must not destroy context

Here's the one I'd flag hardest to anyone building this, because it fails silently and it fails in production.

The natural way to add trace context to a job is to attach a metadata object with the trace ID in it. Whatever was in metadata before — job identity, caller configuration, routing state — is now gone, and nothing errors. The job runs. The trace works beautifully. Something unrelated three services downstream starts behaving oddly a week later, because the routing hint it depended on quietly stopped arriving.

In code the difference is one character of intent and no characters of error handling:

// Destroys whatever else was in metadata — job identity, caller config,
// routing hints. Nothing throws. The trace works perfectly.
await queue.add(jobName, payload, {
  metadata: { traceId },
})

// Additive: keep what is there, forward an existing id, only mint a new
// one when nothing upstream supplied it.
await queue.add(jobName, payload, {
  metadata: {
    ...existingMetadata,
    traceId: existingMetadata?.traceId ?? traceId,
  },
})

Note the second detail in there, which is separate from the merge: prefer the incoming trace ID over a fresh one. If you always mint, every hop starts a new trace and you end up with four one-span traces instead of one four-span trace — technically instrumented, practically useless.

Trace context has to be an additive contract. Preserve what's there, forward the identifier, never replace the envelope. The TypeScript work has targeted tests specifically around metadata preservation and forwarding, which sounds like over-testing for a field assignment until you realise the failure mode produces no error at all.

I'd generalise it: any time you're enriching a message that other people's code also reads, merge semantics are the feature and you should test them like one.

Named events, not object dumps

The privacy half starts with a decision about what a logging helper is for.

The easy design is a helper that takes any object and serialises it. That's a convenience wrapper around your data, and it means the shape of your logs is determined by whatever the caller happened to have in scope at the time — which is to say, unpredictable, unreviewable, and quietly growing.

The design I ended up with records named boundary transitions with constrained metadata. It answers a fixed set of operational questions:

• Which boundary did this job reach?
• Which trace and job identifiers correlate this event?
• Did the transition succeed or fail?
• What safe structural information describes the resource involved?
• Did the queue handoff and the worker execution begin and end?

Concretely, the difference is between these two call sites:

# The convenience wrapper. What ends up in your log index is decided by
# whatever `job` happens to hold today — and by whatever someone adds to
# it next quarter without thinking about logs at all.
logger.info("evaluation finished", extra={"job": job, "result": result})

# The contract. A named transition, a fixed set of fields, nothing that
# carries content.
log_boundary_event(
    event="evaluation.worker.finished",
    trace_id=trace_id,
    job_id=job_id,
    outcome="failed",
    failure_kind="runtime_error",
    resource={"kind": "submission_archive", "host": "storage.internal", "count": 1},
    duration_ms=8412,
)

The second one is answerable and reviewable. You can look at that event definition and say what it can and cannot leak, forever. The first one's blast radius is whatever the object graph grows into.

And it produces something you can actually query. Filter one trace_id and you get an ordered story:

evaluation.request.accepted     trace=7f3a…  outcome=ok
evaluation.queue.enqueued       trace=7f3a…  outcome=ok
evaluation.queue.dequeued       trace=7f3a…  outcome=ok      delay_ms=213
evaluation.worker.started       trace=7f3a…  outcome=ok
evaluation.worker.finished      trace=7f3a…  outcome=failed  failure_kind=runtime_error

That's the whole run, across three services and two languages, and it contains none of the submitted content. You know where it died and what class of failure it was. If you need more than that, you have a specific question to go and answer — which is a much better position than a search bar full of payloads.

It deliberately isn't an arbitrary object serialiser wearing a logging API, and the constraint is the point: you can review a fixed vocabulary of events for what they might leak. You cannot review "whatever anyone passes in".

It reframes what logs are. They have consumers, a schema, access rules and a retention policy — which makes the event contract a data product, and it should get the same review any other API surface gets. Nobody would ship an endpoint that returns arbitrary internal objects. People ship that logger constantly.

Redaction has to go all the way down

Redacting sensitive keys at the top level of an object is close to useless, and it's the default in a lot of codebases.

Real request objects, evaluation payloads and connector configs nest. Credentials sit three or four levels deep, inside a config object inside a client object inside the thing you're actually logging:

{
  "jobId": "e41c…",
  "submittedAt": "2026-03-04T09:12:44Z",
  "runtime": {
    "image": "python:3.12",
    "client": {
      "endpoint": "https://internal.example/eval",
      "config": {"apiKey": "sk-live-9f2c…"}      # <- four levels down
    }
  }
}

A filter that redacts apiKey on the top-level keys never reaches that. It walks jobId, submittedAt, runtime, finds nothing called apiKey, and hands you a log line that looks sanitised and isn't. That's worse than no redaction, because now you trust it.

So redaction descends:

SENSITIVE_KEYS = {"apikey", "authorization", "password", "token", "secret", "cookie"}

def redact(value):
    if isinstance(value, dict):
        return {
            k: "[redacted]" if k.lower() in SENSITIVE_KEYS else redact(v)
            for k, v in value.items()
        }
    if isinstance(value, list):
        return [redact(item) for item in value]
    return value

Same object, after:

"config": {"apiKey": "[redacted]"}

And the honest limit, which I'd rather state than let someone infer: this catches sensitive values under the key names you anticipated. A credential stored under an innocuous name still gets through. Recursive redaction reduces the blast radius of a mistake — it doesn't prove the mistake can't happen, and anyone claiming their redaction layer makes leakage impossible hasn't thought about it for long enough.

A signed URL in a log is a credential in a log

This one deserves separating out, because it's the leak people don't picture as a leak.

Signed URLs carry their own authorisation in the query string. Anyone holding the URL can fetch the resource, without any of your auth. Writing one to a log puts a working credential into a system with broad read access, and it stays valid until an expiry somebody else picked — which may be days, and which you do not control from the logging side.

So resources get summarised rather than emitted:

# before — a working credential, valid for hours, now sitting in a log index
https://storage.example.com/bucket/submissions/sub_9f2c.zip
    ?X-Amz-Algorithm=AWS4-HMAC-SHA256
    &X-Amz-Expires=86400
    &X-Amz-Signature=6c1f0b…

# after — everything you actually use while debugging
{"kind": "submission_archive", "host": "storage.example.com", "count": 1, "resolved": true}

You keep the resource class, the host, a count, whether it resolved. You drop the part that grants access.

In practice this loses you almost nothing, and that's the argument for it. When you're diagnosing a failure at 2am you need to know which kind of resource the job was working on, whether there was one or forty of them, and whether the fetch succeeded. I have never once needed the signing parameters to answer a question. They're in the log because somebody logged the whole URL object, not because anyone decided they should be there.

The general shape: log the structure, not the contents. Whether a field was present, how many items came back, what class of thing it was, whether it parsed. Structure is what you reason with anyway.

Telemetry is secondary. Always.

This is the part I'd keep above everything else in this article, and it's the one that came from an actual bug.

The rule: the evaluation result or exception is primary, telemetry is secondary.

A first implementation can let an error inside a telemetry or reporting call mask the original failure. It looks completely reasonable when you write it:

try:
    return run_evaluation(job)
except EvaluationError as error:
    report_to_tracing(error, trace_id=trace_id)   # if THIS throws...
    raise                                         # ...we never get here

Follow the sequence. The evaluation fails. The except block fires. The reporting call runs, the tracing vendor is having a bad day, and it throws. That new exception propagates out of the except block and replaces the one you were about to re-raise.

What the operator sees is a telemetry error. The evaluation failure — the thing that triggered the whole diagnostic path — has been eaten by the machinery that existed to show it to them.

The correction is small and it needs to be deliberate:

try:
    return run_evaluation(job)
except EvaluationError as error:
    try:
        report_to_tracing(error, trace_id=trace_id)
    except Exception:
        telemetry_dropped.increment()    # count it; never re-raise it
    raise                                # the evaluation failure always wins

The counter matters as much as the swallow. Silently discarding a telemetry failure and not recording that you discarded it is how you end up with a system that looks calm because nothing is reporting.

It's a nasty class of bug because it only surfaces when two things go wrong at once — which is exactly when you're depending on your diagnostics. The Python path was corrected to preserve the primary exception, and it's the merged behaviour I'd port to any system I work on next.

The same principle applies one level down. Breadcrumb emission has to be failure-safe in the same way: the helper catches its own serialisation and reporting failures and hands control straight back to application logic. A logging call that can take down the job it's describing is worse than no logging call, because you've added a failure mode in exchange for visibility you now can't trust.

Over-redaction is also a failure

The counterweight, because a post that only argues for less logging is giving you half an answer.

Redact aggressively enough and you arrive at a trace that proves a job existed, crossed four boundaries, and failed. Congratulations — you have reconstructed the dashboard you already had. Some incidents genuinely need to look at the actual input, and no amount of structural summarising substitutes for that.

The answer isn't to loosen routine logging until it's useful. Routine logging is the thing that runs on every request, forever, into a system with broad access — it should stay boring and structural. The answer is a separately governed break-glass path for deep diagnosis: explicit access, a named reason, a short window, an audit record, and ideally somebody else's approval.

Most teams have neither. They have permissive routine logging and no break-glass process, which is the worst configuration of the two: sensitive data everywhere it isn't needed, and no sanctioned way to get at it when it genuinely is.

I'd call defining that path the largest gap in what I built.

What this proves, and what it doesn't

Being precise here, because observability work attracts overclaiming.

What the implementation supports: a trace identifier can be preserved from TypeScript-originated work into Python execution; success and failure events can share one correlation identity; the logging path emits constrained structural information rather than raw payloads; and a telemetry failure is prevented from replacing the original evaluator exception in the corrected path.

What it does not support, and I want these stated as plainly as the wins:

Not every path emits a complete trace. Producers and consumers I didn't touch may not forward the field. The contract exists; universal adoption is a different claim.

Sensitive data cannot be proven absent from all logs. The helper constrains what it emits. Ad hoc logging elsewhere in a large codebase can bypass it entirely, and inventorying those paths is unfinished work.

No measured operational improvement. I can't tell you this reduced time-to-diagnosis by some percentage, because I don't have the operational evidence, and a number I can't defend is worse than no number.

No compliance claim. Privacy-aware is not privacy-proven. Assurance would need an inventory of every logging path, tested schemas, controlled access and retention, and monitoring for policy regressions. Redaction plus safe summaries is risk reduction, not a certificate.

The trace propagation work and the scoped privacy-aware boundary logging are merged. A broader expansion of the logging work remains branch-only, and I'd rather label that than let it blur into the shipped part. Policy tests exist in the source; I haven't verified a passing CI run, so I'm not claiming one.

Things I'd do next, in order: put a version field on the event and metadata contract, run the policy tests in CI so a new unsafe field fails the build, add end-to-end fixtures that actually cross the TypeScript-to-Python boundary, find a way to measure trace completeness that doesn't involve recording the inputs, and inventory the ad hoc logs living outside the helper so each one is either migrated or deliberately justified.

The takeaway

Two ideas, and they're less about tracing than about how you treat a category of code that usually gets treated as harmless.

Correlation has to live in the message contract. Everything else — timestamps, logger context, ambient state — evaporates at the first async boundary, and a queue is an async boundary.

And your log schema is an API. It has consumers, a shape, access rules, retention, and a blast radius when it's wrong. Design it like one: a fixed vocabulary of events, structure instead of contents, recursive redaction, resources summarised rather than copied, and a hard rule that diagnostics never get to change the truth about whether the operation succeeded.

You can build a system that's fully traceable and doesn't hold a second copy of everything it touched. It just doesn't happen by accident, and it definitely doesn't happen by passing objects to a logger.