💪Exercise💪 — Payment incident in Drilldown
A business report: “customers can’t pay, every other attempt fails”. You run the diagnosis in the Drilldown apps without writing queries, from the symptom in metrics to a specific trace. Several other faults run in the lab at the same time, as in production, so part of the work is filtering out the noise. You finish with three ready queries (PromQL, LogQL, TraceQL) for a dashboard or an alert rule. Variant with an AI agent: Payment incident with an AI agent. Do one of the variants.
Goal
- You go Metrics → Logs → Traces Drilldown for the
paymentFailurescenario. - You establish the source service, the error rate, the error message, the attribute that distinguishes errors and an example trace ID.
- You export the generated queries to Explore and save them as dashboard panels.
- You recognize where Drilldown shows an empty or misleading view because of configuration or filters, not missing data.
Environment: the workshop lab, your personal account (Admin role in the organization).
Prerequisites
- Your lab login (
user<N>) and Grafana password from the lab handout. - The
paymentFailurescenario in its50%variant — the error-rate check is in the prerequisites of exercise 02. The lab baseline is10%: with it the rate in step 1 comes out at ~10%. - The folder
Uczestnicy/<login>for saving the dashboard. - Set the time range to Last 30 minutes in every app. If the variant was switched recently, narrow the range to the time since the switch, otherwise the rate mixes both variants. Drilldown runs dozens of parallel queries per view, and the workshop lab serves the whole group at once — don’t widen the range without need.
- The lab also runs copies of the shop from the SRE agent lab (namespaces
userN) with the same service names and their own, deliberately injected faults. The shop in the report is namespaceotel. The namespace label has a different name in every signal:k8s_namespace_namein span metrics,namespacein Loki,resource.k8s.namespace.namein Tempo.
Prepare an incident note and fill it in as you go:
| Field | Value |
|---|---|
| Services with errors (metrics) | |
| Source service (traces) | |
| Second fault visible in the same chain | |
Error rate of payment server spans |
|
| Error message and log level | |
| Attribute with the biggest difference in Comparison | |
| Slowest operation (service, operation, p90) and its trace ID | |
| Trace ID with an error | |
| PromQL query | |
| LogQL query | |
| TraceQL query |
💪Exercise💪 — steps
Step 1 — Metrics Drilldown: scale and reach
- Drilldown → Metrics → datasource Prometheus.
- Search for
traces_spanmetrics_calls→ ontraces_spanmetrics_calls_totalclick Select. - Add the label filters
k8s_namespace_name=otelandstatus_code=STATUS_CODE_ERROR. - Breakdown tab → By label =
service_name.
Check that: several services have errors, not just payment. Write them in the note. Metrics show the reach of the failure but not where it started — an error propagates up the call chain, and other faults run in the lab too.
- Remove both filters. Add the filter
service_name=payment. - Related logs tab.
Check that: the tab shows payment logs from Loki (the counter next to the tab name is above zero).
- Add the filter
span_kind=SPAN_KIND_SERVERand go back to Related logs.
Check that: the counter dropped to 0. Related logs searches for logs only when every filter label exists in Loki. service_name exists, span_kind does not. The view is empty because of the filter, not because there are no logs. k8s_namespace_name zeroes it the same way, because in Loki the namespace is the namespace label.
- Add the filter
k8s_namespace_name=otel. Breakdown → By label =status_code. Read the rates of theSTATUS_CODE_ERRORandSTATUS_CODE_UNSETseries and compute the error rate. - Menu (⋮) of the
traces_spanmetrics_calls_totalgraph at the top → Explore (opens in a new tab). Explore getssum(rate(traces_spanmetrics_calls_total{…}[$__rate_interval]))with your filters. Replacesum(withsum by (status_code) (, run it and copy the PromQL into the note.
Check that: the error rate of payment server spans is about 50%.
💡 Without the
span_kindfilter the rate comes out clearly lower. On failure,paymentends the internalchargespan without an error status and only then throws the exception, so only the gRPC server span gets statusERROR. Internal spans dilute the denominator.
Step 2 — Logs Drilldown: the error message
- Drilldown → Logs → datasource Loki → find
paymentin the service list → Show logs. - Labels tab → on
namespaceclick Select → onotelInclude. Then Select onstream. The service has two streams:otel(OTLP, one JSON line per record) andstdout(the same record from the container console, split into lines). Onotelclick Include. - Labels tab →
detected_level→ Select. Note the level at which the service logs payment errors. - Fields tab → on
attributes_err_messageclick Select.
Check that: the field has one value — the payment error message — detected from the line’s JSON without writing a parser. Select on attributes_loyalty_level shows a bronze / silver / gold / platinum distribution, but only from logs of successful transactions: the error log does not carry the loyalty level, the charge span does (step 3).
- Patterns tab.
Check that: for stream="otel" the Patterns tab is empty. Remove the stream filter and return to Patterns: there are patterns, but they are fragments of the stdout stream (traceId:'<_>',, message:'Payment request failed…'). The pattern ingester is enabled in the lab, but it tokenizes a JSON line only from log, message, msg and a few similar fields. OTLP content sits in body, so without pattern_ingester_tokenizable_json_fields_append: body in Loki there are no patterns for otel.
- Restore the filter
stream=otel. Logs tab → in the Line filter field (Filter logs by string) typeInvalid tokenand press Enter. - Expand a line and find the full error message (the
bodyfield). Write it in the note. - Menu (⋮) of the Logs panel → Explore. Copy the LogQL into the note.
Check that: the generated LogQL has a selector with service_name="payment", namespace="otel" and stream="otel", the filter |~ "(?i)Invalid token", and the | json | logfmt parsers that Logs Drilldown appends for the Fields tab.
Step 3 — Traces Drilldown: source and evidence
- Drilldown → Traces → datasource Tempo.
- Next to the Root spans / All spans switch, expand the menu (arrow) and choose Server spans. The
paymentserver span (Charge) is never the root of a trace. With the default Root spans,paymentshows only its own background spans (tcp.connect,dns.lookup), without payment errors. All spans is the heaviest query variant. - Errors rate signal → Breakdown tab → attribute
resource.service.name. You see the same services as in the metrics. - In the filter bar add
resource.k8s.namespace.name=otel, then the filterresource.service.name=checkout(Include oncheckoutor the filter bar) → Root cause errors tab.
Check that: the structure for checkout (PlaceOrder) has two error branches below it: PaymentService/Charge → payment and ProductCatalogService/GetProduct → product-catalog. For the “can’t pay” report the source is payment. The second branch is an independent fault from the lab baseline (productCatalogFailure). Write both in the note. Without the checkout filter, Root cause errors shows separate structures for every entry service and the payment chain gets lost in the noise.
- Change the service filter to
resource.service.name=payment. - Exceptions tab. Compare the exception message with the one from the logs.
- Comparison tab. Note the attribute with the biggest difference between error spans (selection) and the rest (baseline).
- Errored traces tab → open any trace. Write down the trace ID.
- In the trace view expand the
paymentserver span with error status, then its child spancharge.
Check that: the charge span has the attribute app.loyalty.level = gold and has no error status. The same gold is in the exception message. In a real incident this is the lead to a business rule that rejects a specific customer group.
- On the
paymentserver span use the Logs for this trace link.
Check that: the right pane shows payment logs with the same Invalid token message.
- Return to Traces Drilldown, Breakdown tab → menu (⋮) of the only panel → Explore. Explore gets a TraceQL metrics query: a
{ … }selector and| rate() by (…). Copy only the selector into the note, without the part after|.
Check that: the selector contains resource.service.name="payment", status=error and kind=server, plus your namespace filter (order may differ). With a service filter, Drilldown breaks the result down by the next attribute, e.g. resource.service.namespace, and adds it to the selector.
Step 4 — Traces Drilldown: the slowest operation
- Remove the service filter (keep the namespace filter). Stay on Server spans and switch the signal to Duration.
- Breakdown tab → attribute
resource.service.name. Services are sorted by average p90. The highest histogram buckets (10 min and more) areflagd(operationEventStream) — a long gRPC stream, not a slow request. Ifflagdis high on the list, exclude it (Exclude). - Root cause latency tab — note the service and operation that add the most time.
- Slow traces tab → open the slowest trace. In the trace view, in the Filters section, click Critical path (the docs call this switch Show critical path only in Span filters: Timeline).
Check that: the note has the service, operation and p90 of the slowest operation (in the lab baseline a few seconds, e.g. ad GetAds, checkout PlaceOrder or cart EmptyCart; values depend on lab load) and a trace ID from Slow traces whose critical path goes through that operation. Without the namespace filter the ranking mixes in the SRE agent lab’s shop copies, and without excluding flagd the “slowest operation” is often a stream lasting 10 minutes or more — a typical false lead in Duration.
Step 5 — queries on a dashboard
- In Explore paste the PromQL from step 1 (datasource Prometheus) → Add → Add to dashboard → New dashboard → Open dashboard.
- Save: title
Incydent payment <login>, folderUczestnicy/<login>→ Save. - Return to Explore, paste the LogQL from step 2 (datasource Loki) → Add → Add to dashboard → Existing dashboard →
Incydent payment <login>→ Open dashboard → Save.
Check that: the dashboard Incydent payment <login> has two panels: the error metric and logs with the Invalid token message.
Step 6 — cleanup
export GRAFANA_URL=https://grafana.workshop2.indexoutofrange.com
export LOGIN=<login>
printf 'Password: ' && read -rs P && GRAFANA_AUTH="$LOGIN:$P" && echo
for u in $(curl -sS -u "$GRAFANA_AUTH" "$GRAFANA_URL/api/search?type=dash-db&folderUIDs=uczestnicy-$LOGIN&query=incydent" | jq -j '.[] | .uid, " "'); do
curl -sS -u "$GRAFANA_AUTH" -X DELETE "$GRAFANA_URL/api/dashboards/uid/$u" | jq -r .message
done
Check that: the loop prints Dashboard Incydent payment <login> deleted.
Success criteria
- Every field of the incident note is filled in.
- The source service is
payment, established in Root cause errors with thecheckoutfilter, not from the list of services with errors in metrics; the second branch (product-catalog) recognized as a separate fault. - Error rate about 50% (about 10% at baseline), computed for
span_kind="SPAN_KIND_SERVER"andk8s_namespace_name="otel". - Slowest operation established for the shop (namespace
otel), without theflagdstream, with a trace ID from Slow traces. - The trace ID opens a trace with a
chargespan carryingapp.loyalty.level = gold. - Three queries (PromQL, LogQL, TraceQL) copied from Explore, two of them saved on the dashboard
Incydent payment <login>, deleted in step 6. - You can point to two views that are empty or misleading because of configuration or filters: Related logs after the
span_kindfilter, and Patterns (empty forotel,stdoutfragments without the filter).
The results are discussed with the instructor.
⭐Stretch: Diff flame graph for adHighCpu
A diff flame graph compares the CPU profile from two time windows, e.g. before and after deploying a new version. The lab has one version of each service, so you compare the state before and after a fault scenario was switched. Scenarios are switched for the whole group, which is why this part is optional.
- Drilldown → Profiles → All services →
otel.ad, CPU profile type. Range Last 3 hours. - Look for a change in CPU use level. In the lab baseline
adHighCpuis on all the time, so you see a jump or a drop only if the scenario was switched (there may be an annotation taggedflagd; Annotations). - Diff flame graph: baseline = window before the change, comparison = window after.
Check that: in the window with adHighCpu on, the class oteldemo.problempattern.CPULoad dominates.
If there is no change in the available range, compare two adjacent 30-minute windows of otel.ad. The diff should be nearly neutral, even though CPULoad is visible in both windows — a reference for what “no change” looks like on a Diff flame graph.
Related lessons
- Drilldown apps
- 💪Exercise💪 — Split view, inspector and saving an exploration
- Flame graphs — reading the Diff flame graph in the Stretch
- Grafana — alert rules from the PromQL query in this exercise