💪Exercise💪 — Payment incident in Drilldown

A business report: “customers can’t pay, every other attempt fails”. You run the diagnosis in the Drilldown apps without writing queries, from the symptom in metrics to a specific trace. Several other faults run in the lab at the same time, as in production, so part of the work is filtering out the noise. You finish with three ready queries (PromQL, LogQL, TraceQL) for a dashboard or an alert rule. Variant with an AI agent: Payment incident with an AI agent. Do one of the variants.

Goal

  • You go Metrics → Logs → Traces Drilldown for the paymentFailure scenario.
  • You establish the source service, the error rate, the error message, the attribute that distinguishes errors and an example trace ID.
  • You export the generated queries to Explore and save them as dashboard panels.
  • You recognize where Drilldown shows an empty or misleading view because of configuration or filters, not missing data.

Environment: the workshop lab, your personal account (Admin role in the organization).

Prerequisites

  • Your lab login (user<N>) and Grafana password from the lab handout.
  • The paymentFailure scenario in its 50% variant — the error-rate check is in the prerequisites of exercise 02. The lab baseline is 10%: with it the rate in step 1 comes out at ~10%.
  • The folder Uczestnicy/<login> for saving the dashboard.
  • Set the time range to Last 30 minutes in every app. If the variant was switched recently, narrow the range to the time since the switch, otherwise the rate mixes both variants. Drilldown runs dozens of parallel queries per view, and the workshop lab serves the whole group at once — don’t widen the range without need.
  • The lab also runs copies of the shop from the SRE agent lab (namespaces userN) with the same service names and their own, deliberately injected faults. The shop in the report is namespace otel. The namespace label has a different name in every signal: k8s_namespace_name in span metrics, namespace in Loki, resource.k8s.namespace.name in Tempo.

Prepare an incident note and fill it in as you go:

Field Value
Services with errors (metrics)  
Source service (traces)  
Second fault visible in the same chain  
Error rate of payment server spans  
Error message and log level  
Attribute with the biggest difference in Comparison  
Slowest operation (service, operation, p90) and its trace ID  
Trace ID with an error  
PromQL query  
LogQL query  
TraceQL query  

💪Exercise💪 — steps

Step 1 — Metrics Drilldown: scale and reach

  1. Drilldown → Metrics → datasource Prometheus.
  2. Search for traces_spanmetrics_calls → on traces_spanmetrics_calls_total click Select.
  3. Add the label filters k8s_namespace_name = otel and status_code = STATUS_CODE_ERROR.
  4. Breakdown tab → By label = service_name.

Check that: several services have errors, not just payment. Write them in the note. Metrics show the reach of the failure but not where it started — an error propagates up the call chain, and other faults run in the lab too.

  1. Remove both filters. Add the filter service_name = payment.
  2. Related logs tab.

Check that: the tab shows payment logs from Loki (the counter next to the tab name is above zero).

  1. Add the filter span_kind = SPAN_KIND_SERVER and go back to Related logs.

Check that: the counter dropped to 0. Related logs searches for logs only when every filter label exists in Loki. service_name exists, span_kind does not. The view is empty because of the filter, not because there are no logs. k8s_namespace_name zeroes it the same way, because in Loki the namespace is the namespace label.

  1. Add the filter k8s_namespace_name = otel. Breakdown → By label = status_code. Read the rates of the STATUS_CODE_ERROR and STATUS_CODE_UNSET series and compute the error rate.
  2. Menu (⋮) of the traces_spanmetrics_calls_total graph at the top → Explore (opens in a new tab). Explore gets sum(rate(traces_spanmetrics_calls_total{…}[$__rate_interval])) with your filters. Replace sum( with sum by (status_code) (, run it and copy the PromQL into the note.

Check that: the error rate of payment server spans is about 50%.

💡 Without the span_kind filter the rate comes out clearly lower. On failure, payment ends the internal charge span without an error status and only then throws the exception, so only the gRPC server span gets status ERROR. Internal spans dilute the denominator.

Step 2 — Logs Drilldown: the error message

  1. Drilldown → Logs → datasource Loki → find payment in the service list → Show logs.
  2. Labels tab → on namespace click Select → on otel Include. Then Select on stream. The service has two streams: otel (OTLP, one JSON line per record) and stdout (the same record from the container console, split into lines). On otel click Include.
  3. Labels tab → detected_level → Select. Note the level at which the service logs payment errors.
  4. Fields tab → on attributes_err_message click Select.

Check that: the field has one value — the payment error message — detected from the line’s JSON without writing a parser. Select on attributes_loyalty_level shows a bronze / silver / gold / platinum distribution, but only from logs of successful transactions: the error log does not carry the loyalty level, the charge span does (step 3).

  1. Patterns tab.

Check that: for stream="otel" the Patterns tab is empty. Remove the stream filter and return to Patterns: there are patterns, but they are fragments of the stdout stream (traceId:'<_>',, message:'Payment request failed…'). The pattern ingester is enabled in the lab, but it tokenizes a JSON line only from log, message, msg and a few similar fields. OTLP content sits in body, so without pattern_ingester_tokenizable_json_fields_append: body in Loki there are no patterns for otel.

  1. Restore the filter stream = otel. Logs tab → in the Line filter field (Filter logs by string) type Invalid token and press Enter.
  2. Expand a line and find the full error message (the body field). Write it in the note.
  3. Menu (⋮) of the Logs panel → Explore. Copy the LogQL into the note.

Check that: the generated LogQL has a selector with service_name="payment", namespace="otel" and stream="otel", the filter |~ "(?i)Invalid token", and the | json | logfmt parsers that Logs Drilldown appends for the Fields tab.

Step 3 — Traces Drilldown: source and evidence

  1. Drilldown → Traces → datasource Tempo.
  2. Next to the Root spans / All spans switch, expand the menu (arrow) and choose Server spans. The payment server span (Charge) is never the root of a trace. With the default Root spans, payment shows only its own background spans (tcp.connect, dns.lookup), without payment errors. All spans is the heaviest query variant.
  3. Errors rate signal → Breakdown tab → attribute resource.service.name. You see the same services as in the metrics.
  4. In the filter bar add resource.k8s.namespace.name = otel, then the filter resource.service.name = checkout (Include on checkout or the filter bar) → Root cause errors tab.

Check that: the structure for checkout (PlaceOrder) has two error branches below it: PaymentService/Charge → payment and ProductCatalogService/GetProduct → product-catalog. For the “can’t pay” report the source is payment. The second branch is an independent fault from the lab baseline (productCatalogFailure). Write both in the note. Without the checkout filter, Root cause errors shows separate structures for every entry service and the payment chain gets lost in the noise.

  1. Change the service filter to resource.service.name = payment.
  2. Exceptions tab. Compare the exception message with the one from the logs.
  3. Comparison tab. Note the attribute with the biggest difference between error spans (selection) and the rest (baseline).
  4. Errored traces tab → open any trace. Write down the trace ID.
  5. In the trace view expand the payment server span with error status, then its child span charge.

Check that: the charge span has the attribute app.loyalty.level = gold and has no error status. The same gold is in the exception message. In a real incident this is the lead to a business rule that rejects a specific customer group.

  1. On the payment server span use the Logs for this trace link.

Check that: the right pane shows payment logs with the same Invalid token message.

  1. Return to Traces Drilldown, Breakdown tab → menu (⋮) of the only panel → Explore. Explore gets a TraceQL metrics query: a { … } selector and | rate() by (…). Copy only the selector into the note, without the part after |.

Check that: the selector contains resource.service.name="payment", status=error and kind=server, plus your namespace filter (order may differ). With a service filter, Drilldown breaks the result down by the next attribute, e.g. resource.service.namespace, and adds it to the selector.

Step 4 — Traces Drilldown: the slowest operation

  1. Remove the service filter (keep the namespace filter). Stay on Server spans and switch the signal to Duration.
  2. Breakdown tab → attribute resource.service.name. Services are sorted by average p90. The highest histogram buckets (10 min and more) are flagd (operation EventStream) — a long gRPC stream, not a slow request. If flagd is high on the list, exclude it (Exclude).
  3. Root cause latency tab — note the service and operation that add the most time.
  4. Slow traces tab → open the slowest trace. In the trace view, in the Filters section, click Critical path (the docs call this switch Show critical path only in Span filters: Timeline).

Check that: the note has the service, operation and p90 of the slowest operation (in the lab baseline a few seconds, e.g. ad GetAds, checkout PlaceOrder or cart EmptyCart; values depend on lab load) and a trace ID from Slow traces whose critical path goes through that operation. Without the namespace filter the ranking mixes in the SRE agent lab’s shop copies, and without excluding flagd the “slowest operation” is often a stream lasting 10 minutes or more — a typical false lead in Duration.

Step 5 — queries on a dashboard

  1. In Explore paste the PromQL from step 1 (datasource Prometheus) → Add → Add to dashboard → New dashboard → Open dashboard.
  2. Save: title Incydent payment <login>, folder Uczestnicy/<login> → Save.
  3. Return to Explore, paste the LogQL from step 2 (datasource Loki) → Add → Add to dashboard → Existing dashboard → Incydent payment <login> → Open dashboard → Save.

Check that: the dashboard Incydent payment <login> has two panels: the error metric and logs with the Invalid token message.

Step 6 — cleanup

export GRAFANA_URL=https://grafana.workshop2.indexoutofrange.com
export LOGIN=<login>
printf 'Password: ' && read -rs P && GRAFANA_AUTH="$LOGIN:$P" && echo
for u in $(curl -sS -u "$GRAFANA_AUTH" "$GRAFANA_URL/api/search?type=dash-db&folderUIDs=uczestnicy-$LOGIN&query=incydent" | jq -j '.[] | .uid, " "'); do
  curl -sS -u "$GRAFANA_AUTH" -X DELETE "$GRAFANA_URL/api/dashboards/uid/$u" | jq -r .message
done

Check that: the loop prints Dashboard Incydent payment <login> deleted.

Success criteria

  • Every field of the incident note is filled in.
  • The source service is payment, established in Root cause errors with the checkout filter, not from the list of services with errors in metrics; the second branch (product-catalog) recognized as a separate fault.
  • Error rate about 50% (about 10% at baseline), computed for span_kind="SPAN_KIND_SERVER" and k8s_namespace_name="otel".
  • Slowest operation established for the shop (namespace otel), without the flagd stream, with a trace ID from Slow traces.
  • The trace ID opens a trace with a charge span carrying app.loyalty.level = gold.
  • Three queries (PromQL, LogQL, TraceQL) copied from Explore, two of them saved on the dashboard Incydent payment <login>, deleted in step 6.
  • You can point to two views that are empty or misleading because of configuration or filters: Related logs after the span_kind filter, and Patterns (empty for otel, stdout fragments without the filter).

The results are discussed with the instructor.

⭐Stretch: Diff flame graph for adHighCpu

A diff flame graph compares the CPU profile from two time windows, e.g. before and after deploying a new version. The lab has one version of each service, so you compare the state before and after a fault scenario was switched. Scenarios are switched for the whole group, which is why this part is optional.

  1. Drilldown → Profiles → All services → otel.ad, CPU profile type. Range Last 3 hours.
  2. Look for a change in CPU use level. In the lab baseline adHighCpu is on all the time, so you see a jump or a drop only if the scenario was switched (there may be an annotation tagged flagd; Annotations).
  3. Diff flame graph: baseline = window before the change, comparison = window after.

Check that: in the window with adHighCpu on, the class oteldemo.problempattern.CPULoad dominates.

If there is no change in the available range, compare two adjacent 30-minute windows of otel.ad. The diff should be nearly neutral, even though CPULoad is visible in both windows — a reference for what “no change” looks like on a Diff flame graph.

results matching ""

    No results matching ""