Distributed tracing across queues, cron, and workers: propagation patterns that hold

A trace shows a 14-ms HTTP request that disappears into a queue and reappears 40 minutes later as four orphan spans. The orphan spans have a trace ID that matches nothing in your backend. The root span that produced the job — the one that called queue.enqueue() — is a healthy closed span showing a 14-ms wall time and no children. From the perspective of your tracing backend, nothing unusual happened. Your p99 latency looks clean. Your error rate is flat. The worker that processed the job took 3.8 seconds across four downstream calls, and none of that time is visible anywhere in the trace waterfall for the original request.

This is not a bug in your tracing library. This is the async boundary problem, and it affects every team that runs queued workers, background jobs, cron tasks, or event-driven pipelines alongside a traced HTTP layer.

The root cause is structural. HTTP tracing works because the context travels with the request: the W3C traceparent header arrives in the incoming HTTP request, the SDK extracts it, the new span is parented to the remote span, and the trace is connected. When you call queue.enqueue(), the job payload goes to Redis or SQS or Kafka. The HTTP response closes. The queue worker wakes up sometime later — maybe milliseconds, maybe forty minutes — reads the payload, and starts processing. Nothing in that payload carries the trace context unless you put it there explicitly. The worker starts a fresh root span with a new trace ID, because there is no parent to inherit.

This post is about the mechanics of why that happens and the patterns that fix it — for BullMQ, Sidekiq, Celery, SQS, and Kafka, for cron jobs that have no parent at all, and for the fan-in scenarios where multiple producer spans legitimately converge on a single consumer. It covers manual propagation via job metadata, custom propagators, span links, and the sampling complications that arise when a trace spans 40 minutes rather than 40 milliseconds.

What's actually happening

The W3C Trace Context wire format

The W3C Trace Context specification defines two HTTP headers that carry distributed trace state: traceparent and tracestate.

The traceparent header has a fixed format: version-trace-id-parent-id-trace-flags. For version 00, a concrete example looks like this:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Breaking that down:

• 00 — version byte, always 00 in the current spec
• 4bf92f3577b34da6a3ce929d0e0e4736 — 32 hex characters, a 16-byte trace ID uniquely identifying the entire distributed trace
• 00f067aa0ba902b7 — 16 hex characters, an 8-byte span ID identifying the calling span (the parent)
• 01 — trace flags: the rightmost bit is the sampled flag (01 = sampled, 00 = not sampled)

The tracestate header carries vendor-specific key-value pairs that ride alongside the traceparent:

tracestate: dd=s:1;t.tid:4bf92f3500000000,rojo=00f067aa0ba902b7

Multiple vendors in a request chain can each append their own key to tracestate, up to a maximum of 32 key-value pairs. A service receiving this header must propagate all pairs it does not understand, and may update or add pairs it owns. This lets DataDog, Honeycomb, and a custom in-house system all annotate the same trace without destroying each other's context.

The specification's core propagation rule is straightforward: a service that receives a traceparent header must forward it (or a mutated form of it, if the service is creating a new child span) in all outgoing requests. The parent-id field is updated to the current span's span ID on each hop, so the chain of causality is preserved.

What the specification does not define is how to propagate trace context across non-HTTP transports. That gap is the source of every orphan span problem in async architectures.

The OpenTelemetry Context API

OpenTelemetry's Context specification defines Context as an immutable key-value store that carries execution-scoped values — including the active span — across API boundaries within a single process. When you call tracer.startActiveSpan(), the SDK stores the new span in the current Context and makes that Context the "active" context for the duration of the callback. Any code running within that callback can call trace.getActiveSpan() and receive the correct span without any explicit passing.

The Propagators API defines how Context crosses process boundaries. A TextMapPropagator has two operations:

• Inject: reads the active span from Context and writes its trace ID, span ID, and flags as string key-value pairs into a carrier (for HTTP, the carrier is the request headers map)
• Extract: reads string key-value pairs from a carrier and constructs a new Context containing the remote span context as a parent

For HTTP requests, this happens automatically in auto-instrumented services. The @opentelemetry/instrumentation-http package patches the Node.js http module so that every outgoing request has context injected into its headers and every incoming request has context extracted before the handler runs.

For a queue message, there is no automatic injection or extraction. The message payload is application-defined. The SDK has no way to know that job.data is a carrier for trace context unless you tell it explicitly.

Why the span closes before the work is done

This is the part that surprises engineers encountering the problem for the first time. When your HTTP handler calls queue.add('sendEmail', { userId }), control returns immediately. The span for the HTTP handler closes. Its duration is 14ms. From the tracer's perspective, the work is complete.

The work is not complete. A worker will pick up that job and do the actual work — maybe making three API calls and a database write — somewhere between a few milliseconds and several hours later. If the trace context is not serialised into the job payload at enqueue time, the worker has no information connecting its activity to the request that triggered it.

The OpenTelemetry Trace API specification describes span links precisely for this scenario. A link connects two spans that have a causal relationship but cannot be expressed as a simple parent-child relationship — either because they live in different traces, or because the timing is such that the parent span has already closed by the time the child span opens. Span links carry a SpanContext (trace ID + span ID + flags) and an optional attributes map describing the nature of the link.

How the sampling flag travels (and why it matters)

The trace flags byte in traceparent carries the sampled bit. When a head-based sampler decides to sample a trace at the entry point — say, at a rate of 10% — it sets trace-flags to 01. Downstream services that extract this header are expected to honour the sampling decision from the upstream caller rather than making their own independent decision. The W3C spec is explicit: "If the service that receives the traceparent header decides to participate in the trace, it MUST propagate the same sampled flag" unless the service is explicitly permitted to make its own sampling decisions.

This creates a specific failure mode in async systems: if you extract the trace context from a job payload and create a new child span, but you do not propagate the sampled flag, your worker may create spans that the SDK immediately discards because the sampling decision is absent or negative. The result looks identical to a context propagation failure — orphan spans or no spans — even though the trace ID is being carried correctly.

The fix is to propagate the full SpanContext, not just the trace ID and span ID. The SpanContext object in the OTel API carries the trace flags, and the sampling decision rides in those flags.

For tail-based sampling, the problem is more complex. The OTel Collector tail sampling processor makes sampling decisions after the full trace is assembled. But a trace that spans 40 minutes may be partially exported before the processor has seen all spans, because the default decision_wait is 30 seconds. For long-running async traces, you either need to extend decision_wait to match your maximum job execution time, or design your async spans to link rather than parent, so the HTTP trace and the worker trace can be sampled independently.

The messaging semantic conventions

The OpenTelemetry semantic conventions for messaging define five operation types for message-based spans:

The critical convention is that a producer SHOULD attach the message creation context to each message. For a single-message send, the producer span's context becomes the link target in the consumer span. For batch sends where one producer span covers multiple messages, the convention recommends span links rather than parent-child relationships, because a single parent span would not correctly represent the fan-out.

The semantic conventions also specify required attributes: messaging.system (kafka, aws_sqs, bullmq, etc.), messaging.operation.name, messaging.operation.type, and messaging.destination.name. These attributes make your messaging spans queryable and understandable in backends like Grafana Tempo and Honeycomb without custom parsing logic.

The pattern

Async context loss manifests in three distinct patterns, each with a different propagation strategy.

Pattern 1: Fire-and-forget queues

The most common pattern. An HTTP request handler enqueues a job and returns immediately. The worker processes the job later. The job is in BullMQ backed by Redis, or Sidekiq backed by Redis, or Celery backed by Redis or RabbitMQ, or an SQS queue.

The trace context that exists at enqueue time is the context of the HTTP handler span. That context must be serialised into the job payload as a propagation carrier — a plain key-value object — before the HTTP span closes.

At worker startup, the worker reads the propagation carrier from the job payload, extracts the context, and creates either a child span (with the HTTP handler span as remote parent) or a new root span with a span link to the HTTP handler span. The choice depends on how you want the trace to look in your backend:

• Child span: the worker span appears nested under the HTTP request in the trace waterfall, and the total elapsed time includes the queue wait. This is often surprising and misleading — the HTTP request appears to have taken 40 minutes, and the 14ms wall time is invisible.
• New root span with link: the worker creates its own trace, and the link connects it back to the originating HTTP trace. The HTTP trace shows 14ms accurately. The worker trace shows 3.8s accurately. Your backend (Tempo, Jaeger, Honeycomb) renders the link as a navigable connection between the two traces.

For most teams, new root span with link is the better choice for jobs with a meaningful delay (anything over a few seconds). For jobs that execute within milliseconds of being enqueued — essentially synchronous but async in implementation — child span is acceptable.

Pattern 2: Cron and scheduled jobs

A scheduled job has no incoming trace context. There is no HTTP request, no producer span, no traceparent header. The job starts because a clock fired.

The correct pattern is a new root span with meaningful attributes that make the span self-describing. The OTel semantic conventions for FaaS timer triggers specify:

• faas.trigger = "timer"
• faas.cron — the cron expression, e.g. "0 2 * * *"
• faas.time — the scheduled invocation time in ISO 8601 UTC
• faas.invocation_id — a unique identifier for this specific invocation

For cron jobs that are triggered by a deployment or configuration change — say, a nightly report that was reconfigured to run hourly — span links can connect the job trace to the deployment event's trace, if the deployment pipeline creates a trace. This is an edge case, but it is the right model when you need to answer "which config change caused this cron job to start running more frequently?"

Pattern 3: Fan-in and fan-out

Fan-out: one producer publishes N messages, each consumed by a separate worker. The producer span is a single span. Each consumer creates its own processing span. If each consumer span is parented to the producer span, the trace waterfall shows N parallel children under one parent — which is correct. If each consumer creates a new root span and links to the producer span, the producer span becomes the target of N links — which is also navigable in most backends.

Fan-in: N producers each send a message to a queue. One consumer reads all N messages and processes them together (for example, a batch aggregation job that waits for all shards to complete before producing an output). The consumer span cannot have N parents. The OTel specification is explicit that a span has exactly one parent. For fan-in, span links are the only correct model: the consumer creates one span with N links, one per producer span.

This is the distinguishing use case for span links. Any time the causal relationship is many-to-one, parent-child is structurally wrong, and span links are the right primitive.

Pattern 4: Edge functions and serverless

Cloudflare Workers and Vercel Edge Functions run JavaScript at the edge and receive HTTP requests with headers. The traceparent header arrives in the request. The challenge is that these environments do not have the same Node.js AsyncLocalStorage context propagation that the OTel Node.js SDK relies on. The SDK's automatic context propagation may not work out of the box; manual context extraction and injection are required.

For Cloudflare Workers, the OTel Cloudflare Workers SDK provides a compatibility layer. For Vercel Edge Functions, the Vercel trace headers (x-vercel-trace, x-vercel-id) are separate from the W3C traceparent and need explicit bridging if you want your traces to connect across the Vercel edge layer.

The general principle applies regardless of runtime: extract the traceparent header explicitly using propagation.extract(ROOT_CONTEXT, headers, getter), create your span with the extracted context as parent, and inject the updated context into any outgoing requests.

The fix / approach

The following patterns are presented with TypeScript and Python code that compiles against current library versions. All TypeScript samples use @opentelemetry/api v1.x and @opentelemetry/sdk-node v0.x (the SDK version tracks separately from the API version). Python samples use opentelemetry-sdk v1.x and opentelemetry-instrumentation-celery v0.x.

1. Serialise context into job metadata at enqueue time

   At the point where you call queue.add() or client.send_task(), extract the current context into a plain object and embed it in the job payload.

   The key insight is that you call propagation.inject() — not propagation.extract() — at enqueue time. You are injecting the current context (the active HTTP span's context) into a carrier that will travel with the job.

   // lib/queue.ts
   import { Queue } from 'bullmq';
   import { context, propagation } from '@opentelemetry/api';
   import { connection } from './redis';
    
   const emailQueue = new Queue('email', { connection });
    
   interface TraceCarrier {
     traceparent?: string;
     tracestate?: string;
   }
    
   export async function enqueueEmail(userId: string, templateId: string): Promise<void> {
     // Capture the current trace context into a plain object
     const carrier: TraceCarrier = {};
     propagation.inject(context.active(), carrier);
    
     await emailQueue.add('sendEmail', {
       userId,
       templateId,
       // Embed the carrier alongside application data
       _otelCarrier: carrier,
     });
   }

   The _otelCarrier field is a convention, not a standard. You can name it anything. What matters is that it is a plain key-value object containing the traceparent (and optionally tracestate) string, and that it survives serialisation to JSON and deserialisation on the worker side.

   For Python with Celery, the pattern uses task headers rather than the payload body, which is the approach the official opentelemetry-instrumentation-celery package uses:

   # tasks/email_tasks.py
   from celery import Celery
   from opentelemetry import context, propagation
    
   app = Celery('tasks', broker='redis://localhost:6379/0')
    
   def enqueue_email(user_id: str, template_id: str) -> None:
       # Inject current context into a dict that becomes task headers
       carrier: dict[str, str] = {}
       propagation.inject(carrier)
    
       # Pass carrier as headers — Celery task headers survive serialisation
       send_email_task.apply_async(
           args=[user_id, template_id],
           headers={'_otel': carrier},
       )
    
   @app.task(bind=True)
   def send_email_task(self, user_id: str, template_id: str) -> None:
       # Extraction happens in the instrumentation layer (see Step 2)
       pass

   For AWS SQS, the same carrier object becomes a message attribute. The AWS X-Ray documentation demonstrates the equivalent pattern with the X-Ray SDK; for OpenTelemetry, the principle is identical — inject context into a message attribute map and extract it in the consumer.

   // lib/sqs-producer.ts
   import { SQSClient, SendMessageCommand } from '@aws-sdk/client-sqs';
   import { context, propagation } from '@opentelemetry/api';
    
   const sqs = new SQSClient({ region: 'us-east-1' });
    
   export async function sendToQueue(
     queueUrl: string,
     body: Record<string, unknown>
   ): Promise<void> {
     const carrier: Record<string, string> = {};
     propagation.inject(context.active(), carrier);
    
     // SQS message attributes have a specific structure
     const messageAttributes: Record<string, { DataType: string; StringValue: string }> = {};
     for (const [key, value] of Object.entries(carrier)) {
       messageAttributes[key] = { DataType: 'String', StringValue: value };
     }
    
     await sqs.send(new SendMessageCommand({
       QueueUrl: queueUrl,
       MessageBody: JSON.stringify(body),
       MessageAttributes: messageAttributes,
     }));
   }

2. Extract context and create the worker span

   On the worker side, read the carrier from the job payload, extract the context, and create your worker span with the extracted context as the remote parent or as a link.

   // workers/email-worker.ts
   import { Worker, Job } from 'bullmq';
   import { context, propagation, trace, SpanKind, SpanStatusCode, ROOT_CONTEXT } from '@opentelemetry/api';
   import { connection } from '../lib/redis';
    
   const tracer = trace.getTracer('email-worker', '1.0.0');
    
   const worker = new Worker(
     'email',
     async (job: Job) => {
       const carrier = (job.data._otelCarrier ?? {}) as Record<string, string>;
    
       // Extract the producer's context from the carrier
       const parentContext = propagation.extract(ROOT_CONTEXT, carrier);
    
       // Retrieve the remote SpanContext (if one was propagated)
       const remoteSpan = trace.getSpan(parentContext);
       const remoteSpanContext = remoteSpan?.spanContext();
    
       // Create the worker span.
       // Option A — child span (appropriate for fast jobs):
       // const workerSpan = tracer.startSpan(
       //   'email.process',
       //   { kind: SpanKind.CONSUMER },
       //   parentContext
       // );
    
       // Option B — new root span with link (appropriate for deferred jobs):
       const workerSpan = tracer.startSpan(
         'email.process',
         {
           kind: SpanKind.CONSUMER,
           // No parent — this is a new root span
           // But link back to the producer span so backends can navigate
           links: remoteSpanContext
             ? [{ context: remoteSpanContext, attributes: { 'link.type': 'enqueued_by' } }]
             : [],
           attributes: {
             'messaging.system': 'bullmq',
             'messaging.operation.type': 'process',
             'messaging.destination.name': 'email',
             'job.id': job.id ?? 'unknown',
             'job.name': job.name,
             'job.attempt': job.attemptsMade,
           },
         }
       );
    
       // Make this span the active span for downstream instrumentation
       return context.with(trace.setSpan(context.active(), workerSpan), async () => {
         try {
           await sendEmail(job.data.userId, job.data.templateId);
           workerSpan.setStatus({ code: SpanStatusCode.OK });
         } catch (err) {
           workerSpan.recordException(err as Error);
           workerSpan.setStatus({
             code: SpanStatusCode.ERROR,
             message: (err as Error).message,
           });
           throw err;
         } finally {
           workerSpan.end();
         }
       });
     },
     { connection }
   );

   For Kafka consumers, the propagation carrier lives in the message headers. The OTel messaging semantic conventions specify that the traceparent and tracestate values should be carried in Kafka message headers using the same key names as the HTTP headers. The @opentelemetry/instrumentation-kafkajs package handles this automatically when auto-instrumentation is enabled; for manual control the pattern is the same as the BullMQ example above with message.headers as the carrier.

   For Python Celery workers, the opentelemetry-instrumentation-celery package uses a CeleryGetter class to extract context from task headers automatically, provided the instrumentor is initialised after the Celery worker process is initialised (using the worker_process_init signal, as the Celery OTel instrumentation docs specify). Manual extraction follows the same propagation.extract(carrier) pattern as TypeScript.

   # workers/email_worker.py
   from opentelemetry import context, propagation, trace
   from opentelemetry.trace import SpanKind, StatusCode, NonRecordingSpan
   from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
    
   tracer = trace.get_tracer('email-worker', '1.0.0')
    
   def process_email_job(job_data: dict) -> None:
       carrier: dict[str, str] = job_data.get('_otel', {})
    
       # Extract the producer context
       parent_ctx = propagation.extract(carrier)
    
       # Get the remote span context for linking
       parent_span = trace.get_current_span(parent_ctx)
       remote_span_ctx = parent_span.get_span_context()
    
       links = []
       if remote_span_ctx and remote_span_ctx.is_valid:
           from opentelemetry.trace import Link
           links = [Link(context=remote_span_ctx, attributes={'link.type': 'enqueued_by'})]
    
       with tracer.start_as_current_span(
           'email.process',
           kind=SpanKind.CONSUMER,
           links=links,
           attributes={
               'messaging.system': 'celery',
               'messaging.operation.type': 'process',
               'messaging.destination.name': 'email',
           },
       ) as span:
           try:
               _send_email(job_data['user_id'], job_data['template_id'])
               span.set_status(StatusCode.OK)
           except Exception as exc:
               span.record_exception(exc)
               span.set_status(StatusCode.ERROR, str(exc))
               raise

3. Implement a custom propagator for job payloads

   If you have multiple queue systems or want to standardise propagation across your platform, a custom propagator is cleaner than repeating the inject/extract logic in every producer and consumer.

   A TextMapPropagator needs two methods: inject and extract. For job payloads, the carrier is the job data object rather than HTTP headers.

   // lib/job-propagator.ts
   import {
     TextMapPropagator,
     TextMapGetter,
     TextMapSetter,
     Context,
     propagation,
     trace,
     ROOT_CONTEXT,
   } from '@opentelemetry/api';
    
   // The key we use to nest trace context inside job data
   const CARRIER_KEY = '_otelCarrier';
    
   /**
    * Getter that reads from a nested _otelCarrier object within a job payload.
    */
   const jobGetter: TextMapGetter<Record<string, unknown>> = {
     get(carrier, key) {
       const nested = carrier[CARRIER_KEY] as Record<string, string> | undefined;
       return nested?.[key];
     },
     keys(carrier) {
       const nested = carrier[CARRIER_KEY] as Record<string, string> | undefined;
       return nested ? Object.keys(nested) : [];
     },
   };
    
   /**
    * Setter that writes to a nested _otelCarrier object within a job payload.
    */
   const jobSetter: TextMapSetter<Record<string, unknown>> = {
     set(carrier, key, value) {
       if (!carrier[CARRIER_KEY]) {
         carrier[CARRIER_KEY] = {};
       }
       (carrier[CARRIER_KEY] as Record<string, string>)[key] = value;
     },
   };
    
   export function injectJobContext(jobData: Record<string, unknown>): void {
     propagation.inject(trace.context(trace.getActiveSpan()?.spanContext() ?? {} as any), jobData, jobSetter);
   }
    
   export function extractJobContext(jobData: Record<string, unknown>): Context {
     return propagation.extract(ROOT_CONTEXT, jobData, jobGetter);
   }

   This propagator can be dropped into any queue producer/consumer without changing the job payload structure more than adding the _otelCarrier key.

4. Instrument cron and scheduled jobs with meaningful root spans

   A cron job that starts at 02:00 with a span named span and no attributes is invisible in a trace backend. A cron job with a well-attributed root span is queryable, alertable, and debuggable.

   // jobs/nightly-report.ts
   import { trace, SpanKind, SpanStatusCode, context } from '@opentelemetry/api';
    
   const tracer = trace.getTracer('report-service', '1.0.0');
    
   export async function runNightlyReport(scheduledTime: Date): Promise<void> {
     // Cron jobs have no incoming context — they always start a new trace
     const span = tracer.startSpan('nightly_report.run', {
       kind: SpanKind.SERVER, // SERVER is the correct kind for timer-triggered jobs (OTel FaaS conventions)
       attributes: {
         'faas.trigger': 'timer',
         'faas.cron': '0 2 * * *',
         'faas.time': scheduledTime.toISOString(),
         'faas.invocation_id': crypto.randomUUID(),
         'job.name': 'nightly-report',
         'job.environment': process.env.NODE_ENV ?? 'production',
       },
     });
    
     return context.with(trace.setSpan(context.active(), span), async () => {
       try {
         await generateReport();
         await uploadToStorage();
         await notifyStakeholders();
         span.setStatus({ code: SpanStatusCode.OK });
       } catch (err) {
         span.recordException(err as Error);
         span.setStatus({
           code: SpanStatusCode.ERROR,
           message: (err as Error).message,
         });
         throw err;
       } finally {
         span.end();
       }
     });
   }

   The faas.cron attribute in the root span is what makes the span queryable. In Grafana Tempo you can filter by faas.cron = "0 2 * * *" to see all invocations of this specific schedule. In Honeycomb you can GROUP BY faas.cron to see how different scheduled jobs compare in duration. Without this attribute, all cron jobs look alike in a trace list.

   For scheduled jobs that process data from a specific time window — a job that processes events from the previous hour, for example — add job.window.start and job.window.end as span attributes. These let you correlate a slow job invocation with the data volume in its window.

5. Use span links for fan-in aggregation jobs

   A fan-in job receives messages from multiple producers and aggregates them. The correct model is one consumer span with N links.

   // workers/aggregation-worker.ts
   import { Worker, Job } from 'bullmq';
   import {
     context,
     propagation,
     trace,
     SpanKind,
     SpanStatusCode,
     ROOT_CONTEXT,
     Link,
   } from '@opentelemetry/api';
    
   const tracer = trace.getTracer('aggregation-worker', '1.0.0');
    
   interface ShardJob {
     shardId: string;
     _otelCarrier?: Record<string, string>;
   }
    
   const aggregationWorker = new Worker<{ shards: ShardJob[]; batchId: string }>(
     'aggregate',
     async (job: Job) => {
       // Build a link for each shard's producer span
       const links: Link[] = [];
    
       for (const shard of job.data.shards) {
         const carrier = shard._otelCarrier ?? {};
         const shardContext = propagation.extract(ROOT_CONTEXT, carrier);
         const remoteSpan = trace.getSpan(shardContext);
         const remoteSpanCtx = remoteSpan?.spanContext();
    
         if (remoteSpanCtx?.isValid) {
           links.push({
             context: remoteSpanCtx,
             attributes: {
               'link.type': 'aggregated_from',
               'shard.id': shard.shardId,
             },
           });
         }
       }
    
       // One span, N links — the correct model for fan-in
       const aggregationSpan = tracer.startSpan('batch.aggregate', {
         kind: SpanKind.CONSUMER,
         links,
         attributes: {
           'messaging.system': 'bullmq',
           'messaging.operation.type': 'process',
           'messaging.destination.name': 'aggregate',
           'batch.id': job.data.batchId,
           'batch.shard_count': job.data.shards.length,
         },
       });
    
       return context.with(trace.setSpan(context.active(), aggregationSpan), async () => {
         try {
           await performAggregation(job.data.batchId, job.data.shards);
           aggregationSpan.setStatus({ code: SpanStatusCode.OK });
         } catch (err) {
           aggregationSpan.recordException(err as Error);
           aggregationSpan.setStatus({ code: SpanStatusCode.ERROR });
           throw err;
         } finally {
           aggregationSpan.end();
         }
       });
     },
     { connection }
   );

   The links array on the aggregation span contains one entry per shard. In Grafana Tempo, clicking the "Linked Traces" panel on the aggregation span shows you all the producer traces that fed into this batch. In Honeycomb, you can query for all traces that link to a given aggregation span by its trace ID, giving you a complete picture of the fanin.

6. Configure tail sampling for long-running async traces

   Head-based sampling works well for HTTP traffic. For async jobs that run for minutes or hours, the sampling decision made at the HTTP handler (or at the worker root span) cannot be revisited. This creates a problem: a job that eventually fails after 40 minutes was sampled at 01 (sampled) or 00 (not sampled) based on information available at the start of the job, not the outcome.

   Tail-based sampling in the OTel Collector solves this, but requires that all spans for a given trace arrive at the same collector instance within the decision_wait window.

   # otel-collector-config.yaml
   processors:
     tail_sampling:
       # For async jobs that can run up to 10 minutes, extend decision_wait
       decision_wait: 12m
       # In-memory trace limit — size this for your expected concurrent jobs
       num_traces: 10000
       # Cache decisions for traces that overflow in-memory storage
       decision_cache:
         sampled_cache_size: 50000
       policies:
         # Always keep errors, regardless of job duration
         - name: errors
           type: status_code
           status_code:
             status_codes: [ERROR]
         # Keep slow jobs (over 5 seconds total)
         - name: slow-jobs
           type: latency
           latency:
             threshold_ms: 5000
         # Keep jobs that link to HTTP traces (so you can navigate from request to job)
         - name: has-links
           type: boolean_attribute
           boolean_attribute:
             key: 'has_producer_link'
             value: true
         # Sample 5% of healthy fast jobs
         - name: baseline
           type: probabilistic
           probabilistic:
             sampling_percentage: 5
    
   exporters:
     otlp:
       endpoint: "tempo:4317"
    
   service:
     pipelines:
       traces:
         receivers: [otlp]
         processors: [memory_limiter, batch, tail_sampling]
         exporters: [otlp]

   The decision_wait of 12 minutes gives a 2-minute buffer over a 10-minute maximum job duration. For jobs that can run longer, increase this value proportionally. Be aware that the in-memory footprint of the tail sampling processor scales with num_traces × average_trace_size, so a 12-minute window for high-throughput job queues may require significant collector memory.

   For async traces with span links (the worker is a separate trace from the HTTP request), the two traces are sampled independently by the tail sampling processor — the HTTP trace based on its outcome, the worker trace based on its outcome. This is correct behaviour. The link connects them in the backend UI, but the sampling decision for each is made on the merits of that individual trace.

Anti-patterns

Passing only the trace ID, not the full span context

// Wrong — loses parent-id and trace-flags
await queue.add('job', {
  userId,
  traceId: span.spanContext().traceId, // trace ID only
});
 
// Right — serialises the full context including parent-id and sampled flag
const carrier: Record<string, string> = {};
propagation.inject(context.active(), carrier);
await queue.add('job', { userId, _otelCarrier: carrier });

Passing the trace ID alone lets you link traces by trace ID in your backend, but it does not propagate the sampling flag. If the originating request was not sampled, the worker will inherit no information to make a matching decision. More importantly, passing the trace ID without the parent span ID breaks the parent-child chain — the worker has no way to know which span within the originating trace was the direct producer.

The propagation.inject() API serialises the complete SpanContext including version, trace ID, parent span ID, and trace flags into the W3C traceparent string. Always use it instead of building the carrier manually.

Creating a child span that outlives the parent's recorded duration

// Problematic — the HTTP handler span closes at 14ms
// but the worker span opens 40 minutes later as a child
// The trace waterfall shows the HTTP request "took" 40 minutes
async function httpHandler(req, res) {
  const span = trace.getActiveSpan(); // HTTP handler span
 
  await queue.add('job', {
    _carrier: getCarrierFromActive(), // serialises HTTP span as parent
  });
 
  res.json({ status: 'queued' }); // HTTP span closes here at 14ms
}
 
// Worker creates a child of the HTTP span that opens 40 minutes later
// The backend computes trace duration as: max(child end) - min(parent start)
// Result: the trace appears to have duration of 40 minutes + 3.8 seconds

Most trace backends compute the root span's effective duration by taking the earliest start time and the latest end time across all spans in the trace. If a child span starts 40 minutes after the root span ended, the root span's recorded duration (14ms) is inconsistent with the computed trace duration (40+ minutes), and the waterfall display is broken or misleading.

Use span links for deferred jobs. Use child spans only for jobs that complete within a time window where the visual inconsistency is not confusing — roughly speaking, within the time the HTTP handler span could plausibly still be "open" in the reader's mental model (a few seconds at most).

Using the same trace ID for both the HTTP trace and the worker trace

Some teams try to "connect" the traces by passing the HTTP trace ID to the worker and creating worker spans that use that same trace ID. This is not a valid OTel pattern. Trace IDs are immutable identifiers assigned to a trace at root span creation time. Creating a new span with an artificially assigned trace ID bypasses the SDK's span creation logic, breaks sampling flag inheritance, and produces invalid telemetry that most backends reject or display incorrectly.

The correct model is: separate trace IDs, connected by a span link.

Forgetting to propagate context through sub-workers or downstream calls

When a worker spawns additional work — a job that enqueues another job, or a worker that makes HTTP calls — the propagation must continue. The child job needs the worker's span context in its carrier, not the original HTTP span's context. If the worker does not call propagation.inject(context.active(), ...) (where context.active() returns the context with the worker span as active) before enqueuing the next job, the downstream job will either have no context or will link back to the original HTTP request, skipping the worker span in the chain.

// Inside a worker callback, context.active() returns the context with
// the worker span as active (because we used context.with(...) in Step 2).
// So this inject correctly captures the worker span as the new "parent":
const childCarrier: Record<string, string> = {};
propagation.inject(context.active(), childCarrier);
await childQueue.add('downstream-job', { _otelCarrier: childCarrier });

Not setting kind: SpanKind.CONSUMER on worker spans

Span kind is not cosmetic. It affects how backends group and display spans. A CONSUMER span represents a span that processes a message from a producer. Setting kind: SpanKind.SERVER on a queue worker span (a common mistake, because workers "serve" jobs) is semantically wrong and causes backends to misclassify the span in service maps and dependency graphs. The OTel messaging semantic conventions are explicit: use CONSUMER for message processing, PRODUCER for message sending.

Not handling the case where the carrier is absent or malformed

A job that was enqueued before your propagation code was deployed will have no _otelCarrier field. A job from an external system will have no OTel context. Your worker must handle this gracefully:

const carrier = (job.data._otelCarrier ?? {}) as Record<string, string>;
const parentContext = propagation.extract(ROOT_CONTEXT, carrier);
// If carrier is empty, parentContext === ROOT_CONTEXT, and the worker span
// will be a root span with no links — correct, not an error.

The propagation.extract() API is safe to call with an empty carrier. It returns ROOT_CONTEXT unchanged, which is the correct result: no context to inherit means the worker starts a fresh trace.

Case studies / numbers

The 40-minute gap in a payment processing pipeline

A fintech team was running a payments service where the HTTP API acknowledged payment requests immediately and handed them to a BullMQ worker for actual processing. The HTTP response was 200ms. The worker ran for anywhere between 800ms and 3 minutes depending on the payment provider.

Their tracing setup — OTel with Grafana Tempo — showed clean HTTP traces with no issues. Worker errors were tracked in logs. When a payment failed, the on-call engineer would cross-reference HTTP trace IDs from the API logs with worker job IDs from the BullMQ dashboard and Celery logs — a manual process that took 8–15 minutes per incident.

After implementing the propagation pattern described in this post (manual injection into job metadata, span links in workers), the same investigation took under 2 minutes: navigate to the HTTP trace, click the "Linked Traces" panel, see the worker trace, see the failed span with the provider SDK's error message and the request payload. The 8–15 minute gap became a single click.

They chose span links over child spans specifically because payment workers could take 3 minutes, and the HTTP traces showed 200ms response times that needed to stay accurate. Showing the HTTP trace as "taking 3 minutes" would have confused their p99 latency dashboards.

Their tail sampling configuration keeps 100% of traces for worker spans with status ERROR, 20% of traces for worker spans with duration above 60 seconds, and 2% of healthy fast worker traces. On a processing volume of approximately 4,000 jobs per hour, this keeps storage costs roughly the same as before instrumentation while dramatically improving error trace coverage.

Orphan spans from a Sidekiq deployment migration

A Ruby on Rails team migrated from Sidekiq to a Kubernetes-based job runner over three months. During the migration, jobs could be processed by either system. Their OTel setup was based on the opentelemetry-ruby SDK with opentelemetry-instrumentation-sidekiq.

The instrumentation worked correctly for pure-Sidekiq jobs and pure-new-worker jobs. For jobs that were enqueued by the Rails application and processed by the new worker (which happened during the transition), the carrier format was incompatible: Sidekiq carried context in a specific header key (_otel_ctx), while the new worker expected the standard traceparent key.

The fix was a carrier adapter in the new worker that checked for both key formats:

# config/initializers/otel_carrier_compat.rb
module JobCarrierCompat
  SIDEKIQ_KEY = '_otel_ctx'
  W3C_KEY = 'traceparent'
 
  def self.extract_from_job(job_hash)
    # Try W3C format first
    carrier = { W3C_KEY => job_hash[W3C_KEY], 'tracestate' => job_hash['tracestate'] }
    return carrier if carrier[W3C_KEY]
 
    # Fall back to Sidekiq OTel instrumentation format
    sidekiq_ctx = job_hash[SIDEKIQ_KEY]
    return {} unless sidekiq_ctx
 
    # Sidekiq instrumentation stores context as a base64 JSON blob
    JSON.parse(Base64.decode64(sidekiq_ctx))
  rescue StandardError
    {}
  end
end

This illustrates a general principle: when you have multiple queue producers or consumers that may use different propagation formats, a compatibility adapter is cheaper than requiring simultaneous migration of all producers and consumers. The W3C traceparent format should be the long-term standard, but migrations are rarely simultaneous.

Cron job attribution in a data pipeline

A data engineering team ran twelve cron jobs processing time-windowed batches of events. All twelve created spans with job.name = 'batch' and no other identifying attributes. In Tempo, searching for slow traces returned a list of twelve indistinguishable traces.

After adding faas.cron, faas.time, job.window.start, job.window.end, and job.record_count as span attributes, the same search returned traces labelled with their schedule, their data window, and the number of records processed. A trace showing job.record_count = 0 for a window that should have had events was immediately identifiable as a data pipeline gap — something that previously required log cross-referencing to diagnose.

The team also added job.upstream_lag_seconds — the difference between the scheduled start time and the oldest event timestamp in the batch — as a span attribute. Querying for traces where job.upstream_lag_seconds > 300 identified jobs that were processing significantly delayed data, a leading indicator for upstream data source problems that previously appeared as unexplained batch failures two hours later.

In their experience, adding 4–6 business-domain attributes to each cron job's root span is the single highest return-on-investment observability investment for a data pipeline. The cost is trivial. The query surface it opens up is substantial.

Kafka fan-in latency investigation

A team running event-driven aggregation on Kafka had an aggregation job that consumed from eight partition topics and produced a single output event. Each topic had one producer. The aggregation span had eight links, one per partition producer span.

When the output latency spiked, they queried in Honeycomb for aggregation spans with high duration and used the link navigation to identify which partition's producer span was the outlier. In four out of five incidents, one specific partition's producer was slow — the root cause was a database query on a table with skewed cardinality that only manifested in one partition's data range.

Without span links, the investigation would have required manually correlating Kafka consumer offset lags, producer timestamps, and application logs across eight separate service traces. With span links, the correlation was visual and took under 90 seconds.

The team's observation: for fan-in jobs specifically, the link attributes matter almost as much as the links themselves. Adding 'partition.id': partitionId and 'message.offset': offset to each link made the outlier partition immediately visible in the link list rather than requiring them to click into each linked trace individually.

Limits + caveats

This approach does not automatically fix traces that cross vendor boundaries. If your HTTP request goes through a Cloudflare CDN, is processed by a service instrumented with DataDog APM, passes to a queue processed by a service instrumented with Honeycomb's OTel SDK, the context will propagate correctly because all three use the W3C traceparent format. However, the trace may be stored across three separate backends, and navigating the links requires switching between vendor UIs. Unified backends (sending all telemetry to a single collector and single storage backend) are the clean solution; span links across vendor-siloed backends require additional work in the form of deep links or external correlation IDs.

Tail sampling with long decision_wait has memory implications. A decision_wait of 12 minutes means the collector holds in-memory spans for all active traces for 12 minutes. At high job throughput, this can require substantial collector memory. Calculate: num_concurrent_jobs × avg_spans_per_job × avg_span_size_bytes. For a system running 5,000 concurrent jobs with 20 spans each at 1KB each, that is 100MB of spans in-memory at any moment — manageable but non-trivial. Scale the collector horizontally if needed, and use the load-balancing exporter to route all spans for a given trace to the same collector instance, which the tail sampling processor requires.

Span links are not universally supported in all UI features. Jaeger renders span links as navigable connections in the trace detail view. Grafana Tempo renders linked traces in a side panel. Honeycomb shows links as queryable attributes. DataDog APM's support for OTel span links is partial as of mid-2024 — links may not render as clickable UI elements in all contexts. Check your specific backend's link support before designing your architecture around link navigation as the primary debugging UX.

Context propagation is a contract between teams. If a producer injects context and a consumer does not extract it, the link is silently lost. If a consumer extracts context but the producer never injected it, the extraction returns ROOT_CONTEXT (not an error). The failure mode is invisible: traces are missing links, and there is no error in logs or metrics to indicate that propagation failed. Automated contract testing — a test that injects a known trace ID into a test job and asserts that the worker span carries a link to that trace — is the only reliable way to catch propagation regressions before production.

Sampling flag propagation can cause unexpected trace gaps. If your HTTP layer samples at 10% and your job worker inherits the sampling flag from the carrier, 90% of worker spans will be discarded. This is correct behaviour — you sampled 10% of HTTP requests, and the corresponding worker spans follow the same decision. But if you are debugging a specific failure that only occurs in the worker (independent of the HTTP request quality), you may find that the specific job traces you need were not sampled. Tail-based sampling on the worker side — creating new root spans and sampling based on worker outcome rather than HTTP request outcome — is the correct architectural answer for worker traces that need independent coverage.

This post covers the OTel SDK patterns, not the collector's behaviour with every backend. Backends differ in how they index and query span links, in how they handle spans that arrive after the trace has been partially assembled, and in their pricing models for high-cardinality span attributes. Test the patterns in this post against your specific backend before deploying to production at scale.

Sources & further reading

W3C Trace Context Recommendation https://www.w3.org/TR/trace-context/ The normative specification for the traceparent and tracestate headers: format, field semantics, propagation rules, and mutation constraints. The primary reference for all header-level propagation behaviour described in this post.

OpenTelemetry Context Specification https://opentelemetry.io/docs/specs/otel/context/ The OTel specification for the in-process Context API: immutability guarantees, key-value operations, and the implicit vs explicit context handling distinction. Defines the contract that propagation.inject and propagation.extract operate against.

OpenTelemetry Propagators API Specification https://opentelemetry.io/docs/specs/otel/context/api-propagators/ The normative specification for TextMapPropagator, including the Inject and Extract operation contracts, the Getter and Setter interfaces, and the composite propagator requirement. The reference for implementing custom propagators.

OpenTelemetry Trace API Specification — Span Links https://opentelemetry.io/docs/specs/otel/trace/api/ The specification for span links including their use for fan-in and async causality. Defines that links may be added at span creation time (preferred, so samplers can consider them) or after creation.

OpenTelemetry Trace SDK Specification — Sampling https://opentelemetry.io/docs/specs/otel/trace/sdk/ The SDK-level specification for sampling: IsRecording, the Sampled flag, ParentBased sampler, TraceIdRatioBased sampler, and the prohibition on the Sampled=true, IsRecording=false combination. Critical for understanding sampling flag propagation in async traces.

OpenTelemetry Semantic Conventions — Messaging Spans https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/ The canonical attribute definitions for producer, consumer, and broker spans. Defines PRODUCER and CONSUMER span kinds, required attributes (messaging.system, messaging.operation.type, messaging.destination.name), and the recommendation to use span links for batch fan-out scenarios.

OpenTelemetry Semantic Conventions — FaaS Spans https://opentelemetry.io/docs/specs/semconv/faas/faas-spans/ The attribute definitions for timer-triggered spans: faas.trigger = "timer", faas.cron, faas.time, faas.invocation_id. The normative reference for instrumenting cron jobs and scheduled workers.

OpenTelemetry Collector Contrib — Tail Sampling Processor https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor The reference implementation and configuration documentation for tail-based sampling in the OTel Collector. Covers available policies (status code, latency, probabilistic, composite), decision_wait, decision_cache, and the requirement that all spans for a trace arrive at the same collector instance.

OpenTelemetry Instrumentation — Celery (Python) https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/celery/celery.html The CeleryGetter implementation, the worker_process_init initialisation requirement, and the span lifecycle for Celery task instrumentation. The primary reference for Python async workers.

Amazon SQS and AWS X-Ray — Trace Context Propagation https://docs.aws.amazon.com/xray/latest/devguide/xray-services-sqs.html AWS's documentation on propagating trace headers through SQS via the AWSTraceHeader message system attribute. Demonstrates the receive-extract-recover pattern and confirms that the trace header does not count against SQS message size quotas.

Grafana Tempo — Overview and Architecture https://grafana.com/docs/tempo/latest/ Tempo's architecture documentation: object-storage-only design, supported protocols (Jaeger, Zipkin, OTLP), and integration with Loki (derived fields for log-to-trace navigation) and Prometheus (exemplars for metric-to-trace navigation). Reference for teams choosing an OSS trace backend.

Jaeger — Architecture and Deployment https://www.jaegertracing.io/docs/1.57/architecture/ Jaeger's architectural overview including the direct-to-storage and Kafka-buffered deployment modes, context propagation semantics (only IDs and baggage propagate in-band; span data is exported out of band), and the relationship between the Jaeger components.

as required. */}