Exemplars from the SDK

• 🎯 What an exemplar is
• 🧰 How the SDK picks exemplars
• 🔗 The path from SDK to Grafana
• 🔍 Checking the workshop stack
• 📉 Where exemplars get lost
• 🚨 Common failure modes
• Related lessons

A p99 latency panel says that something is slow. An exemplar is one real request behind that point, with its trace ID, so one click opens the slow trace.

Exemplars are produced by the metrics SDK at the moment a measurement is recorded. Every hop between the SDK and Grafana can drop them without an error.

🎯 What an exemplar is

An exemplar is a sample measurement attached to a metric data point:

In Prometheus terms it becomes:

http_server_request_duration_seconds_bucket{le="2.5",http_route="/api/checkout"} 1043 # {trace_id="4bf92f35…",span_id="00f067aa…"} 2.43 1758700000.123

Exemplars cost almost nothing: a few per series per export interval, no new series.

🧰 How the SDK picks exemplars

Two components decide:

1. Exemplar filter — which measurements are candidates. Set with OTEL_METRICS_EXEMPLAR_FILTER:

2. Exemplar reservoir — which candidates are kept per data point:

The trace link comes from the context at record time. The measurement must be recorded inside the span:

with tracer.start_as_current_span("checkout"):
    process_order()
    duration.record(time.monotonic() - start, {"http.route": "/api/checkout"})  # ✅ exemplar gets this span
duration.record(...)  # ❌ span already ended → no trace link

// Go passes context explicitly — context.Background() means no exemplar
hist.Record(ctx, elapsed.Seconds(), metric.WithAttributes(attribute.String("http.route", route)))

Defaults differ by SDK. Java, Go and Python use trace_based in current versions. .NET needs an opt-in (OTEL_METRICS_EXEMPLAR_FILTER=trace_based or SetExemplarFilter(ExemplarFilterType.TraceBased) on the MeterProvider). Check the release notes of the version you run.

Auto-instrumented HTTP server metrics (http.server.request.duration) record inside the server span, so they get exemplars without code changes once the filter allows it.

🔗 The path from SDK to Grafana

Each hop has its own switch. Most are off by default.

Label names differ by producer:

Grafana needs one exemplarTraceIdDestinations entry per label name:

jsonData:
  exemplarTraceIdDestinations:
    - name: traceID      # Tempo metrics-generator
      datasourceUid: tempo
    - name: trace_id     # OTel SDKs
      datasourceUid: tempo

In Explore or a panel, enable Exemplars in the query options. They show up on histogram queries (histogram_quantile(... _bucket ...)) as dots on the time series.

🔍 Checking the workshop stack

The workshop stack configures the Grafana end, but not the storage chain:

To light up the whole path: add exemplar-storage to enableFeatures, sendExemplars: true to the Mimir remoteWrite, max_global_exemplars_per_user to the Mimir limits, and a trace_id destination to both Prometheus data sources.

Verify at the storage layer before blaming Grafana:

# Does Prometheus hold any exemplars for this metric?
curl -s "http://localhost:9090/api/v1/query_exemplars?query=http_server_request_duration_seconds_bucket&start=$(date -d '-1 hour' +%s)&end=$(date +%s)" \
  | jq '[.data[].exemplars[]] | length'

📉 Where exemplars get lost

Tail sampling and exemplars partly align: “keep slow” and “keep errors” policies keep the traces behind the high-latency buckets, which are the dots you click.

🚨 Common failure modes

Related lessons

• Metrics
• Traces
• Head and Tail Sampling
• Remote Write
• Tempo Integrations & TraceQL