💪Exercise💪 — Audit of links between signals

An inherited environment has “configured” links between logs, traces, metrics and profiles. Some of them don’t work and nobody reports it: a broken link simply doesn’t appear, or it opens an empty chart. You audit five transitions in the workshop lab and for each one name the cause in the datasource or backend configuration. Variant with an AI agent: Audit of links between signals with an AI agent. Do one of the variants.

Goal

  • Check five transitions: log → trace, trace → logs, trace → metric, trace → profile, exemplar → trace.
  • For each one decide: works / doesn’t work / works, but the result is useless.
  • Justify the cause with a specific key from the datasource or backend configuration.

Environment: workshop lab, your own account (org role Admin).

Prerequisites

  • Your lab login (user<N>) and password from the lab handout.
  • Active paymentFailure scenario (check described in the prerequisites of exercise 02).

Read the configuration through the API, in the browser, in the same session as Grafana:

https://grafana.workshop2.indexoutofrange.com/api/datasources/uid/loki
https://grafana.workshop2.indexoutofrange.com/api/datasources/uid/tempo
https://grafana.workshop2.indexoutofrange.com/api/datasources/uid/prometheus
https://grafana.workshop2.indexoutofrange.com/api/datasources/uid/mimir

You care about the jsonData object. The API doesn’t return passwords and tokens (secureJsonData).

⚠️ As Admin you also see Connections → Data sources and can save changes to the shared datasources. Don’t: a fix changes the audit result for the whole group. You apply fixes on your own datasource copies in exercise 05.

💪Exercise💪 — steps

Record results in the table:

# Transition Result Cause (configuration key) Fix
1 log → trace (derived field)      
2 trace → logs (tracesToLogsV2)      
3 trace → metric (tracesToMetrics)      
4 trace → profile (tracesToProfiles)      
5 exemplar → trace (exemplarTraceIdDestinations), Prometheus and Mimir      

Step 1 — log → trace

  1. Explore → Loki, range Last 15 minutes:

    {service_name="payment", namespace="otel", stream="otel"}
    
  2. Expand any line. In log details look for a Links section with a TraceID link.
  3. Find the trace ID in the line content. Append | json to the query, run it again and find the same field among the parsed fields.
  4. In /api/datasources/uid/loki read jsonData.derivedFields.

Check that: log details have no Links section, and the ID in the line has the form "traceid":"<32 hex chars>". You can say why matcherRegex doesn’t match it: compare letter case and format (key=value vs JSON).

Step 2 — trace → logs

  1. Explore → Tempo → Search → TraceQL mode:

    { resource.service.name = "payment" && resource.k8s.namespace.name = "otel" && status = error }
    

    The filter resource.k8s.namespace.name = "otel" skips the shop copies from the SRE agent lab (namespaces userN), which have the same service names.

  2. Open any trace, expand the grpc.oteldemo.PaymentService/Charge span of the payment service.
  3. In the span details find the link to logs. Its label depends on configuration: Logs for this span (with filterBySpanID), Logs for this trace (with filterByTraceID only) or Related logs.
  4. Click the link. In the right pane open Query inspector → Query and note the generated LogQL.
  5. In /api/datasources/uid/tempo read jsonData.tracesToLogsV2. Compare datasourceUid with the uid field in /api/datasources/uid/loki.

Check that: the LogQL selector has two labels (service_name and service_namespace), contains label_format with the trace ID and returns lines. You know which labels went into the selector and where they came from (tracesToLogsV2 has no tags key, so the default set applies), how Grafana filters by trace ID (label_format … contains …, not |=) and whether datasourceUid is a uid or a datasource name.

Step 3 — trace → metric

  1. On the same span click the metrics link. Its label is the query name from the configuration (Latency buckets).
  2. Note the generated PromQL and the time range in the right pane.
  3. Check which labels exist in span metrics:

    group by (service, service_name, span_name, span_kind, k8s_namespace_name) (traces_spanmetrics_calls_total{service_name="payment", k8s_namespace_name="otel"})
    
  4. In /api/datasources/uid/tempo read jsonData.tracesToMetrics (tags, queries, spanStartTimeShift, spanEndTimeShift).

Check that: the PromQL is sum(rate(traces_spanmetrics_latency_bucket{k8s_namespace_name="otel", service="payment"}[5m])) and the time range is about 2 h (span time ±1 h). You can answer three questions. Does the query return data? Does the value make sense as latency? Hint: _bucket series are cumulative counters per le. What does the chart cover: the operation of the clicked span or all operations of the service? Hint: compare tags with the labels from item 3.

Step 4 — trace → profile

  1. Open a trace of the checkout service (Go, profiled by the Pyroscope SDK):

    { resource.service.name = "checkout" && resource.k8s.namespace.name = "otel" && name = "oteldemo.CheckoutService/PlaceOrder" && kind = server }
    
  2. On the PlaceOrder server span expand Links → Profiles for this span. The span has the pyroscope.profile.id attribute, so Grafana gives two profile links (Profiles for this span and Open in Profiles Drilldown) and, with four links, folds them into a Links menu. Note the Pyroscope query, profile type, time range (Query inspector) and the sample count under the flame graph.
  3. On the payment server span from step 2 click Related profiles and note the same.
  4. In /api/datasources/uid/tempo read jsonData.tracesToProfiles (customQuery, query, profileTypeId, spanStartTimeShift).
  5. Drilldown → Profiles → All services: check whether the services otel.checkout and otel.payment exist and which profile types each has (profile type switch above the list).

Check that:

  • checkout: query {service_name="otel.checkout"}, profile type process_cpu, range = span time ±60 s, filter by span ID, and 0 samples under the flame graph,
  • payment: query {service_name="otel.payment"}, type process_cpu, range = the span time alone (about 1 ms), and instead of a flame graph the error invalid span id length: the span has no pyroscope.profile.id, so the span ID filter gets the unexpanded ${__span.tags["pyroscope.profile.id"]},
  • you can explain where the otel. prefix in the query comes from (compare query with the span’s resource attributes), why neither service shows samples (separately for checkout and payment — the causes differ) and whether the spanStartTimeShift from the configuration had any effect.

Step 5 — exemplar → trace

  1. Explore → Prometheus:

    histogram_quantile(0.95, sum by (le) (rate(traces_spanmetrics_latency_bucket{service_name="payment", k8s_namespace_name="otel"}[5m])))
    
  2. Options under the query → Exemplars on → Run query. Hover an exemplar point and click the trace link.
  3. Switch the datasource to Mimir, same query and options. Look for exemplar points.
  4. Compare jsonData.exemplarTraceIdDestinations in /api/datasources/uid/prometheus and /api/datasources/uid/mimir.
  5. Open https://prometheus.workshop2.indexoutofrange.com → Status → Command-line flags and find enable-feature.

Check that: Prometheus has exemplar points with a trace link, Mimir has none, and exemplarTraceIdDestinations of both datasources are identical. You can say on which side (Grafana or backend) the difference lies.

Success criteria

  • The table has five filled rows, each with the cause named by a configuration key (matcherRegex, datasourceUid, tags, queries, profileTypeId, enable-feature…) or by a missing key.
  • At least two rows have the result “works, but the result is useless”, with justification.
  • Every row with the result “doesn’t work” has a fix written as a concrete value (regex, query, flag, limit).

You apply the fixes on your own datasource copies in exercise 05.

⭐Stretch: dead keys in the configuration

/api/datasources/uid/tempo has keys that Grafana reads but ignores. Find at least two and justify them: the tracesToLogs block (without V2) next to tracesToLogsV2, and spanStartTimeShift / spanEndTimeShift in tracesToProfiles.

Check that: you can show the evidence from steps 2 and 4 — the LogQL doesn’t contain the tags from the tracesToLogs block, and the profile time range has no -5m / 5m shift.

results matching ""

    No results matching ""