💪Exercise💪 — Audit of links between signals
An inherited environment has “configured” links between logs, traces, metrics and profiles. Some of them don’t work and nobody reports it: a broken link simply doesn’t appear, or it opens an empty chart. You audit five transitions in the workshop lab and for each one name the cause in the datasource or backend configuration. Variant with an AI agent: Audit of links between signals with an AI agent. Do one of the variants.
Goal
- Check five transitions: log → trace, trace → logs, trace → metric, trace → profile, exemplar → trace.
- For each one decide: works / doesn’t work / works, but the result is useless.
- Justify the cause with a specific key from the datasource or backend configuration.
Environment: workshop lab, your own account (org role Admin).
Prerequisites
- Your lab login (
user<N>) and password from the lab handout. - Active
paymentFailurescenario (check described in the prerequisites of exercise 02).
Read the configuration through the API, in the browser, in the same session as Grafana:
https://grafana.workshop2.indexoutofrange.com/api/datasources/uid/loki
https://grafana.workshop2.indexoutofrange.com/api/datasources/uid/tempo
https://grafana.workshop2.indexoutofrange.com/api/datasources/uid/prometheus
https://grafana.workshop2.indexoutofrange.com/api/datasources/uid/mimir
You care about the jsonData object. The API doesn’t return passwords and tokens (secureJsonData).
⚠️ As Admin you also see Connections → Data sources and can save changes to the shared datasources. Don’t: a fix changes the audit result for the whole group. You apply fixes on your own datasource copies in exercise 05.
💪Exercise💪 — steps
Record results in the table:
| # | Transition | Result | Cause (configuration key) | Fix |
|---|---|---|---|---|
| 1 | log → trace (derived field) | |||
| 2 | trace → logs (tracesToLogsV2) |
|||
| 3 | trace → metric (tracesToMetrics) |
|||
| 4 | trace → profile (tracesToProfiles) |
|||
| 5 | exemplar → trace (exemplarTraceIdDestinations), Prometheus and Mimir |
Step 1 — log → trace
-
Explore → Loki, range Last 15 minutes:
{service_name="payment", namespace="otel", stream="otel"} - Expand any line. In log details look for a Links section with a TraceID link.
- Find the trace ID in the line content. Append
| jsonto the query, run it again and find the same field among the parsed fields. - In
/api/datasources/uid/lokireadjsonData.derivedFields.
Check that: log details have no Links section, and the ID in the line has the form "traceid":"<32 hex chars>". You can say why matcherRegex doesn’t match it: compare letter case and format (key=value vs JSON).
Step 2 — trace → logs
-
Explore → Tempo → Search → TraceQL mode:
{ resource.service.name = "payment" && resource.k8s.namespace.name = "otel" && status = error }The filter
resource.k8s.namespace.name = "otel"skips the shop copies from the SRE agent lab (namespacesuserN), which have the same service names. - Open any trace, expand the
grpc.oteldemo.PaymentService/Chargespan of thepaymentservice. - In the span details find the link to logs. Its label depends on configuration: Logs for this span (with
filterBySpanID), Logs for this trace (withfilterByTraceIDonly) or Related logs. - Click the link. In the right pane open Query inspector → Query and note the generated LogQL.
- In
/api/datasources/uid/temporeadjsonData.tracesToLogsV2. ComparedatasourceUidwith theuidfield in/api/datasources/uid/loki.
Check that: the LogQL selector has two labels (service_name and service_namespace), contains label_format with the trace ID and returns lines. You know which labels went into the selector and where they came from (tracesToLogsV2 has no tags key, so the default set applies), how Grafana filters by trace ID (label_format … contains …, not |=) and whether datasourceUid is a uid or a datasource name.
Step 3 — trace → metric
- On the same span click the metrics link. Its label is the query name from the configuration (Latency buckets).
- Note the generated PromQL and the time range in the right pane.
-
Check which labels exist in span metrics:
group by (service, service_name, span_name, span_kind, k8s_namespace_name) (traces_spanmetrics_calls_total{service_name="payment", k8s_namespace_name="otel"}) - In
/api/datasources/uid/temporeadjsonData.tracesToMetrics(tags,queries,spanStartTimeShift,spanEndTimeShift).
Check that: the PromQL is sum(rate(traces_spanmetrics_latency_bucket{k8s_namespace_name="otel", service="payment"}[5m])) and the time range is about 2 h (span time ±1 h). You can answer three questions. Does the query return data? Does the value make sense as latency? Hint: _bucket series are cumulative counters per le. What does the chart cover: the operation of the clicked span or all operations of the service? Hint: compare tags with the labels from item 3.
Step 4 — trace → profile
-
Open a trace of the
checkoutservice (Go, profiled by the Pyroscope SDK):{ resource.service.name = "checkout" && resource.k8s.namespace.name = "otel" && name = "oteldemo.CheckoutService/PlaceOrder" && kind = server } - On the
PlaceOrderserver span expand Links → Profiles for this span. The span has thepyroscope.profile.idattribute, so Grafana gives two profile links (Profiles for this span and Open in Profiles Drilldown) and, with four links, folds them into a Links menu. Note the Pyroscope query, profile type, time range (Query inspector) and the sample count under the flame graph. - On the
paymentserver span from step 2 click Related profiles and note the same. - In
/api/datasources/uid/temporeadjsonData.tracesToProfiles(customQuery,query,profileTypeId,spanStartTimeShift). - Drilldown → Profiles → All services: check whether the services
otel.checkoutandotel.paymentexist and which profile types each has (profile type switch above the list).
Check that:
checkout: query{service_name="otel.checkout"}, profile typeprocess_cpu, range = span time ±60 s, filter by span ID, and0 samplesunder the flame graph,payment: query{service_name="otel.payment"}, typeprocess_cpu, range = the span time alone (about 1 ms), and instead of a flame graph the errorinvalid span id length: the span has nopyroscope.profile.id, so the span ID filter gets the unexpanded${__span.tags["pyroscope.profile.id"]},- you can explain where the
otel.prefix in the query comes from (comparequerywith the span’s resource attributes), why neither service shows samples (separately forcheckoutandpayment— the causes differ) and whether thespanStartTimeShiftfrom the configuration had any effect.
Step 5 — exemplar → trace
-
Explore → Prometheus:
histogram_quantile(0.95, sum by (le) (rate(traces_spanmetrics_latency_bucket{service_name="payment", k8s_namespace_name="otel"}[5m]))) - Options under the query → Exemplars on → Run query. Hover an exemplar point and click the trace link.
- Switch the datasource to Mimir, same query and options. Look for exemplar points.
- Compare
jsonData.exemplarTraceIdDestinationsin/api/datasources/uid/prometheusand/api/datasources/uid/mimir. - Open
https://prometheus.workshop2.indexoutofrange.com→ Status → Command-line flags and findenable-feature.
Check that: Prometheus has exemplar points with a trace link, Mimir has none, and exemplarTraceIdDestinations of both datasources are identical. You can say on which side (Grafana or backend) the difference lies.
Success criteria
- The table has five filled rows, each with the cause named by a configuration key (
matcherRegex,datasourceUid,tags,queries,profileTypeId,enable-feature…) or by a missing key. - At least two rows have the result “works, but the result is useless”, with justification.
- Every row with the result “doesn’t work” has a fix written as a concrete value (regex, query, flag, limit).
You apply the fixes on your own datasource copies in exercise 05.
⭐Stretch: dead keys in the configuration
/api/datasources/uid/tempo has keys that Grafana reads but ignores. Find at least two and justify them: the tracesToLogs block (without V2) next to tracesToLogsV2, and spanStartTimeShift / spanEndTimeShift in tracesToProfiles.
Check that: you can show the evidence from steps 2 and 4 — the LogQL doesn’t contain the tags from the tracesToLogs block, and the profile time range has no -5m / 5m shift.
Related lessons
- Logs, traces and datasource links
- Links between signals in datasource configuration
- Integrations & TraceQL — Tempo links to logs, metrics and profiles
- Exemplars from the SDK — how exemplars get from the SDK to the backend
- Correlations (Grafana docs) — the remaining links in the span menu