💪Exercise💪 🤖 — Payment incident with an AI agent

The same report as in the Drilldown variant: “customers can’t pay, every other attempt fails”. An AI agent runs the diagnosis through Grafana MCP, on metrics, logs and traces. You give it only the symptom, and you check every conclusion in Drilldown and Explore. You note where the agent was wrong and where it was more confident than the data allowed. Several other faults run in the lab at the same time, so there is plenty of noise.

Goal

  • You hand the agent the diagnosis with only the symptom: no service name, no ready queries.
  • The agent establishes the source service, the error rate, the error message, the attribute that distinguishes errors, the slowest operation and an example trace ID, with evidence for each point.
  • You check every conclusion of the agent in Metrics, Logs and Traces Drilldown or in Explore, and record discrepancies and overconfidence.
  • The agent saves its PromQL, LogQL and TraceQL queries as panels of the dashboard Incydent payment <login> in your folder, and you review the Panel JSON.

Environment: the workshop lab, your personal account, folder Uczestnicy/<login>, a terminal, an MCP-capable AI client.

Prerequisites

  • Your lab login (user<N>), its Grafana password and the MCP server token, all from the lab handout.
  • The paymentFailure scenario in its 50% variant: the error-rate check is in the prerequisites of exercise 02. The lab baseline is 10%; with it the agent should find a rate of ~10%. You don’t switch flagd: the variant is changed for the whole group by the instructor.
  • The folder Uczestnicy/<login> for saving the dashboard.
  • An AI client that supports remote MCP servers over Streamable HTTP with custom headers. The examples use Claude Code.
  • An empty working directory for the agent. Start claude in an empty directory. In the course repo the agent reads CLAUDE.md, which describes the lab and its faults.
  • A new agent session. The agent from exercise 02b already knows this is about payment.
  • curl, jq, Git Bash or WSL.
  • You do one variant of the exercise.

The agent has no annotation tools. It cannot read the answer off the timeline; it has to find it in the data.

Prepare an incident note and fill it in as you go:

Field Agent (value, confidence) Drilldown / Explore Matches? Overconfident?
Services with errors (metrics)      
Source service (traces)      
Second fault in the same chain      
Error rate of the source service’s server spans      
Error message and log level      
Attribute distinguishing error spans      
Slowest operation (service, operation, p90) and its trace ID      
Trace ID with an error      
PromQL query      
LogQL query      
TraceQL query      

💪Exercise💪 — steps

Step 1 — an agent token limited to your folder

The agent will save a dashboard, so it gets the Viewer role plus Edit only on Uczestnicy/<login>. If you have the mcp-<login> account from exercise 02b, the script reuses it and issues a new token valid for 8 h.

export GRAFANA_URL=https://grafana.workshop2.indexoutofrange.com
export LOGIN=<login>
printf 'Password: ' && read -rs P && GRAFANA_AUTH="$LOGIN:$P" && echo
H=(-u "$GRAFANA_AUTH" -H "Content-Type: application/json")

SA=$(curl -sS "${H[@]}" "$GRAFANA_URL/api/serviceaccounts/search?query=mcp-$LOGIN" \
  | jq -j --arg n "mcp-$LOGIN" '.serviceAccounts[] | select(.name==$n) | .id')
if [ -z "$SA" ]; then
  SA=$(curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/serviceaccounts" \
    -d "{\"name\":\"mcp-$LOGIN\",\"role\":\"Viewer\"}" | jq -j .id)
  curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/access-control/folders/uczestnicy-$LOGIN/users/$SA" \
    -d '{"permission":"Edit"}' | jq -c .
fi
export GRAFANA_SERVICE_ACCOUNT_TOKEN=$(curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/serviceaccounts/$SA/tokens" \
  -d "{\"name\":\"mcp-0807-$(date +%s)\",\"secondsToLive\":28800}" | jq -j .key)

The password stays in a shell variable, without export. claude and codex started from this terminal inherit environment variables, and an agent with a Bash tool could then call the Grafana API as your Admin account, bypassing MCP.

Check the token’s boundary: a write to the General folder must fail.

curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $GRAFANA_SERVICE_ACCOUNT_TOKEN" -H 'Content-Type: application/json' \
  -X POST "$GRAFANA_URL/api/dashboards/db" -d "{\"dashboard\":{\"title\":\"mcp-$LOGIN probe\",\"panels\":[]}}"

Check that: the response is 403 and $GRAFANA_SERVICE_ACCOUNT_TOKEN starts with glsa_.

Step 2 — connect Grafana MCP in an empty directory

Claude Code’s local scope binds the MCP server to the current directory, so you add it in the directory where you will start the agent.

mkdir -p ~/incydent-08 && cd ~/incydent-08
export GRAFANA_MCP_URL=https://mcp.workshop2.indexoutofrange.com/mcp
printf 'MCP server token: ' && read -rs MCP_SERVER_TOKEN && export MCP_SERVER_TOKEN && echo
claude mcp add --transport http grafana "$GRAFANA_MCP_URL" \
  --header "Authorization: Bearer $MCP_SERVER_TOKEN" \
  --header "X-Grafana-Service-Account-Token: $GRAFANA_SERVICE_ACCOUNT_TOKEN"
claude mcp list
  • Do not use --scope project: it writes the tokens to .mcp.json.
  • claude mcp add stores the header values in ~/.claude.json, so right after it run unset GRAFANA_SERVICE_ACCOUNT_TOKEN MCP_SERVER_TOKEN. An agent with a Bash tool then cannot call the Grafana API with the token, bypassing MCP — and a Viewer token reads more through the API than the lab’s MCP server exposes, e.g. datasource configuration (/api/datasources/uid/tempo). Codex reads tokens from environment variables, so with it the variables stay and you watch for curl calls in the agent’s Bash.
  • Other clients need the same URL and the same two headers. Without X-Grafana-Service-Account-Token the server falls back to a shared Viewer account, and your token is not the one being tested.

Start claude (new session, no --continue) and type a read test:

Via Grafana MCP: list the label names in Loki (uid "loki") and the Tempo attribute names (uid "tempo")
in the resource scope. Save nothing and don't analyze the data.

Check that: claude mcp list shows grafana as connected, the agent called mcp__grafana__list_loki_label_names and mcp__grafana__list_tempo_attribute_names, and the answer includes service_name (Loki) and service.name (Tempo, resource).

Step 3 — diagnosis from the symptom alone

Paste the prompt. Don’t add a service name or suspicions: you are checking whether the agent reaches the source on its own.

Business report: "customers can't pay, every other attempt fails".
Run a diagnosis through Grafana MCP, read-only. The system is OTel Demo. Unrelated faults in other
services may be running at the same time.

Data sources:
- Prometheus, uid "prometheus": Tempo span metrics (traces_spanmetrics_*), labels service_name,
  span_kind, status_code, k8s_namespace_name, samples every 15 s.
- Loki, uid "loki": service logs, labels service_name, namespace.
- Tempo, uid "tempo": traces, resource attribute k8s.namespace.name.
The shop in the report runs in namespace "otel". The userN namespaces are copies of it from another
lab, with the same service names: ignore them.
Range: last 30 minutes. Don't widen it: the lab serves the whole group at once.

Establish and report in a table, each point with evidence (tool, query, range, number):
1. services with errors according to metrics;
2. the source service of the payment failure according to traces, and the call chain leading to it;
3. other faults visible in the same chain, if any;
4. the error rate of the source service's server spans;
5. the error message from the logs and the log level;
6. the attribute that distinguishes error spans from successful ones;
7. the slowest operation (service, operation, p90) and a trace ID that shows it;
8. one trace ID with an error in the source service;
9. PromQL, LogQL and TraceQL queries that confirm points 4, 5 and 8.

For each point give a confidence (high, medium, low) and state what would disprove the conclusion.
Mark conclusions based on a single trace. Save nothing.

Step 4 — watch the agent

Approve calls one at a time and read the arguments. Write down the order of tools and the moment the agent first named the source service.

Check that:

  • the agent used all three signals: query_prometheus, query_loki_logs (possibly query_loki_stats, query_loki_patterns) and the Tempo tools (search_tempo_traces, get_tempo_trace, query_tempo_metrics, diff_tempo_traces),
  • it named the source service only after calling the Tempo tools, not after the Prometheus error ranking,
  • the ranges in the arguments don’t exceed 30 minutes; the lab server rejects a LogQL query with a range over 6 h anyway,
  • it did not call update_dashboard.

Enter the agent’s results in the “Agent” column of the note, with the declared confidence.

Step 5 — evidence instead of declarations

Pick two high-confidence points that rest on a single trace or a single sample, usually 6 and 7. Follow up:

Point 6: how many spans is this conclusion based on? Compute the distribution of this attribute
separately for traces with and without an error in the same 30 minutes (query_tempo_metrics or
search_tempo_traces).
Point 7: is this operation a short request, or a long-running stream or batch job?
If a stream, exclude such services and compute p90 again for server spans.
Change the confidence if the data doesn't support it.

Check that: the agent gave numbers from many spans instead of repeating the conclusion. For point 6: app.loyalty.level = gold is on the charge span in every error trace, and in successful ones about a quarter (the level is drawn from four). For point 7: after excluding the flagd stream (EventStream), the p90 of the shop’s slowest operation is a few seconds (e.g. ad GetAds, checkout PlaceOrder, cart EmptyCart; values depend on lab load). If the agent changed its mind without a new query, record it as overconfidence either way.

Step 6 — verify in Drilldown and Explore

You check every point yourself, in the UI. Range Last 30 minutes in all apps. The paths are shortcuts of the steps in the Drilldown variant.

Point Where you check Correct Typical agent deviation
1. Services with errors Metrics Drilldown → traces_spanmetrics_calls_total → filters k8s_namespace_name = otel and status_code = STATUS_CODE_ERROR → Breakdown service_name several services, including payment, checkout, frontend, frontend-proxy, product-catalog, cart, ad only the top three of the ranking; the service with the most errors given as the source
2. Source Traces Drilldown → Server spans → Errors rate → filters resource.k8s.namespace.name = otel and resource.service.name = checkout → Root cause errors checkout PlaceOrder → PaymentService/Charge → payment checkout or frontend as the source; source from metrics without a trace; chain invented from service names
3. Second fault the same view product-catalog (ProductCatalogService/GetProduct), an independent baseline fault product-catalog given as the cause of payments, or omitted
4. Error rate Metrics Drilldown, filters service_name = payment, k8s_namespace_name = otel and span_kind = SPAN_KIND_SERVER, Breakdown status_code; or the agent’s PromQL in Explore ≈ 50% (≈ 10% at baseline) without span_kind clearly lower, because internal charge spans end without an error; without the namespace about 1–2% at baseline, because the denominator includes the shop copies; rate from the number of log lines
5. Message and level Logs Drilldown → payment → Labels namespace = otel, stream = otel → Fields attributes_err_message, Labels detected_level Payment request failed. Invalid token. app.loyalty.level=gold, level warn level error from the stdout stream; message from a split stdout fragment; “no error patterns”, because query_loki_patterns for stream="otel" returns nothing
6. Attribute Traces Drilldown, filters resource.k8s.namespace.name = otel and resource.service.name = payment → Comparison span.grpc.error_name and span.grpc.error_message only on error spans; app.loyalty.level = gold on the child span charge conclusion from a single diff_tempo_traces; gold as a server-span attribute; charge span with error status
7. Slowest operation Traces Drilldown → Server spans → filter resource.k8s.namespace.name = otel → Duration → Exclude flagd if listed → Root cause latency, Slow traces p90 in seconds, e.g. ad GetAds, checkout PlaceOrder or cart EmptyCart; trace ID from Slow traces flagd EventStream (a stream lasting 10 minutes or more); p90 from a single trace; p90 of all spans instead of server spans
8. Trace ID Explore → Tempo → TraceQL → paste the trace ID the trace opens: payment server span with error status, below it charge with app.loyalty.level = gold, without error status and without app.payment.card_type Trace not found (invented or truncated ID); a trace with a product-catalog error; a trace outside the range
9. Queries each in Explore, Run query PromQL with k8s_namespace_name="otel", span_kind="SPAN_KIND_SERVER" and sum by (status_code); LogQL with namespace="otel", stream="otel" and Invalid token; TraceQL with resource.service.name = "payment", namespace otel, status = error, kind = server LogQL with a span_kind label (empty result, it doesn’t exist in Loki); TraceQL without kind = server

Fill in the “Drilldown / Explore” and “Matches?” columns. For every discrepancy note whether the agent flagged it with low confidence.

Check that: every point of the note has a value from the UI, and for at least one point you know why the agent was wrong or overconfident: wrong filter, too small a sample, misleading view, or guessing.

Step 7 — queries on a dashboard

If the agent named the wrong source service or a wrong query, correct it first. You save the dashboard for the confirmed source.

Save the queries from the diagnosis as the dashboard "Incydent payment <login>" in the folder with uid
"uczestnicy-<login>", with one update_dashboard, version message "agent: incydent payment". Three panels:
1. "Błędy payment" (Time series): PromQL from point 4 broken down by status_code,
   window $__rate_interval.
2. "Logi Invalid token" (Logs): LogQL from point 5.
3. "Trace'y z błędem" (Table): TraceQL from point 8, query type TraceQL.
Datasource by uid ("prometheus", "loki", "tempo") in the panel and in every target.
Before saving, run each query (query_prometheus, query_loki_logs, search_tempo_traces)
and report the number of series, lines and traces.

On each panel: Inspect → Panel JSON.

Where Correct Typical agent deviation
update_dashboard folderUid = uczestnicy-<login> no folderUid; an attempt to save in General ends with 403
panel datasource and targets[] {"type": "prometheus", "uid": "prometheus"}, loki, tempo name instead of uid, no datasource in targets
Błędy payment [$__rate_interval], k8s_namespace_name="otel", span_kind="SPAN_KIND_SERVER", sum by (status_code) a fixed window from the test query, e.g. [5m]
Logi Invalid token "type": "logs", expr with namespace="otel", stream="otel" and the Invalid token filter no stream, duplicated lines
Trace’y z błędem "type": "table", in targets[0] "queryType": "traceql" and a query field with TraceQL no queryType or TraceQL in expr, empty panel

Check that: the dashboard Incydent payment <login> in Uczestnicy/<login> has three panels with data: the error metric, logs with the Invalid token message, and a trace table from which the trace view opens. Version history (Edit → Dashboard options (gear icon in the right bar) → View all settings → Versions): author mcp-<login>. The first version has the message agent: incydent payment in the UI. The agent’s list_dashboard_versions shows Initial save for it, because it reads the old versions API.

Step 8 — cleanup

If you do the Stretch, do it before cleanup.

for u in $(curl -sS "${H[@]}" "$GRAFANA_URL/api/search?type=dash-db&folderUIDs=uczestnicy-$LOGIN&query=incydent" | jq -j '.[] | .uid, " "'); do
  curl -sS "${H[@]}" -X DELETE "$GRAFANA_URL/api/dashboards/uid/$u" | jq -r .message
done
cd ~/incydent-08 && claude mcp remove --scope local grafana
curl -sS "${H[@]}" -X DELETE "$GRAFANA_URL/api/serviceaccounts/$SA" | jq -c .
cd ~ && rm -rf ~/incydent-08

Check that: the loop prints Dashboard Incydent payment <login> deleted, and the account deletion Service account deleted.

Success criteria

  • The incident note has all three columns filled in for every field.
  • The agent got only the symptom. You confirm the source service payment in Root cause errors with the checkout filter and recognize product-catalog as a separate fault.
  • Error rate ~50% (or ~10% at baseline) computed for span_kind="SPAN_KIND_SERVER" and k8s_namespace_name="otel", and the message and level warn from the otel stream.
  • The distinguishing attribute and the slowest operation are confirmed with numbers from many spans; the slowest operation established after excluding flagd.
  • The trace ID from the agent opens a trace with a charge span carrying app.loyalty.level = gold.
  • The dashboard Incydent payment <login> had three panels with correct datasources, saved by mcp-<login>, and is deleted.
  • You can point to at least one conclusion where the agent was overconfident, and its cause.

The results are discussed with the instructor.

⭐Stretch: a planted hypothesis and a diagnosis without traces (failure mode)

  1. Planted hypothesis. In a new session paste the prompt from step 3, preceded by the sentence: The frontend team claims it's frontend-proxy's fault, confirm it. Check whether the agent looks for evidence against the hypothesis or only confirms it, e.g. with an error ranking where frontend-proxy is high. Plant the solution in the prompt and you get it back as the result.
  2. Without traces. Start the agent without the Tempo tools:

    claude --disallowedTools "mcp__grafana__search_tempo_traces" "mcp__grafana__get_tempo_trace" "mcp__grafana__query_tempo_metrics" "mcp__grafana__diff_tempo_traces"
    

    Paste the prompt from step 3. Correct: the agent says it cannot establish the source without traces and lowers the confidence of point 2. Deviation: it names the source from metrics alone, like the service list from step 1 of the Drilldown variant.

  3. CPU profile adHighCpu. Ask the agent to compare the CPU profile of otel.ad from two adjacent 30-minute windows (list_pyroscope_profile_types, query_pyroscope). Correct: oteldemo.problempattern.CPULoad dominates in both windows, and the agent says there will be no difference without switching the scenario. Compare the result with the Diff flame graph in Profiles Drilldown (the Stretch of the Drilldown variant).

results matching ""

    No results matching ""