Head and Tail Sampling

• 🎯 Where the decision is made
• 🎲 Head sampling in the SDK
• 🔢 Consistent probability sampling
• 🧠 Tail sampling in the Collector
• 🔀 Combining head and tail
• 📊 Sampling and derived data
• 🚨 Common failure modes
• Related lessons

Keeping every span is the most expensive option in observability, and nobody reads 99% of them. Sampling decides which traces you keep.

The decision can be made at the start of a trace (head), after the trace is complete (tail), or both. Each option loses something different.

🎯 Where the decision is made

🎲 Head sampling in the SDK

The root service decides. The decision travels downstream in the last byte of traceparent:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
                                                                  └─ 01 = sampled, 00 = not sampled

Built-in samplers, set with OTEL_TRACES_SAMPLER:

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1

• Always use a parentbased_* sampler in non-root services. A plain traceidratio in the middle of a call chain makes its own decision and cuts traces in half.
• Non-recording spans still propagate. A dropped span keeps its trace ID and sends -00 downstream, so the whole trace is dropped consistently.
• The ratio applies to traces, not spans. Setting 10% at the frontend removes 90% of the traces from every service behind it.

🔢 Consistent probability sampling

Different SDKs used to implement traceidratio with different algorithms. Two services with the same trace ID and different ratios could make contradictory decisions, and nothing recorded what probability produced a span.

The newer spec fixes both with two fields in the ot entry of tracestate:

tracestate: ot=th:c;rv:9b8233f7e3a151

What this gives you:

• Consistency across services. Every stage compares the same randomness against its own threshold. A 1% stage keeps a subset of what a 10% stage keeps.
• Adjusted counts. A backend or connector can multiply by 1 / probability and estimate real request counts from sampled spans.
• Collector support. The probabilistic_sampler processor has mode: proportional (and equalizing), which reads and writes th instead of hashing the trace ID on its own.

SDK support is rolling out language by language. Check whether your SDK version writes th before you rely on adjusted counts.

🧠 Tail sampling in the Collector

The tail_sampling processor (otelcol.processor.tail_sampling in Alloy) holds spans in memory for decision_wait, then evaluates policies against the whole trace. A trace is kept if any policy says so.

processors:
  tail_sampling:
    decision_wait: 10s                 # how long to wait for late spans
    num_traces: 100000                 # traces held in memory at once
    expected_new_traces_per_sec: 2000  # sizes internal structures
    decision_cache:
      sampled_cache_size: 100000       # remember "keep" for spans that arrive after the decision
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow
        type: latency
        latency: {threshold_ms: 1000}
      - name: checkout-always
        type: string_attribute
        string_attribute: {key: service.name, values: [checkout, payment]}
      - name: slow-and-premium
        type: and
        and:
          and_sub_policy:
            - name: slow
              type: latency
              latency: {threshold_ms: 300}
            - name: premium
              type: string_attribute
              string_attribute: {key: app.customer.tier, values: [premium]}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

Operational facts:

• All spans of a trace must reach the same replica. Put a stateless tier with the loadbalancing exporter (routing_key: traceID) in front of the sampling tier. See Topologies on Kubernetes.
• Memory = trace rate × decision_wait × spans per trace. When num_traces is exceeded, the oldest traces are dropped before a decision — watch otelcol_processor_tail_sampling_sampling_trace_dropped_too_early.
• Late spans. A span arriving after the decision is kept only if decision_cache remembers the trace. Long async flows (queues, batch jobs) exceed any sensible decision_wait.
• Latency is measured on the whole trace, from the earliest start to the latest end among the buffered spans.

🔀 Combining head and tail

Tail sampling can only keep what head sampling sent. With 10% head sampling, the errors policy sees 10% of the errors.

Health checks are easier to drop in the Collector’s filter processor than with a custom SDK sampler. See Cost Optimization.

📊 Sampling and derived data

Everything computed from spans after the sampler sees only the kept traces.

Order in a two-tier setup:

# Tier 1 (stateless): metrics from 100% of spans, then route by trace ID
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes]
      exporters: [spanmetrics, loadbalancing]
    metrics/spanmetrics:
      receivers: [spanmetrics]
      exporters: [prometheusremotewrite]
# Tier 2 (stateful): tail_sampling → Tempo

🚨 Common failure modes

Related lessons

• Traces
• Cost Optimization
• Exemplars
• Pipeline Model
• Topologies on Kubernetes