💪Exercise💪 🤖 — Payment incident with an AI agent
The same report as in the Drilldown variant: “customers can’t pay, every other attempt fails”. An AI agent runs the diagnosis through Grafana MCP, on metrics, logs and traces. You give it only the symptom, and you check every conclusion in Drilldown and Explore. You note where the agent was wrong and where it was more confident than the data allowed. Several other faults run in the lab at the same time, so there is plenty of noise.
Goal
- You hand the agent the diagnosis with only the symptom: no service name, no ready queries.
- The agent establishes the source service, the error rate, the error message, the attribute that distinguishes errors, the slowest operation and an example trace ID, with evidence for each point.
- You check every conclusion of the agent in Metrics, Logs and Traces Drilldown or in Explore, and record discrepancies and overconfidence.
- The agent saves its PromQL, LogQL and TraceQL queries as panels of the dashboard
Incydent payment <login>in your folder, and you review the Panel JSON.
Environment: the workshop lab, your personal account, folder Uczestnicy/<login>, a terminal, an MCP-capable AI client.
Prerequisites
- Your lab login (
user<N>), its Grafana password and the MCP server token, all from the lab handout. - The
paymentFailurescenario in its50%variant: the error-rate check is in the prerequisites of exercise 02. The lab baseline is10%; with it the agent should find a rate of ~10%. You don’t switch flagd: the variant is changed for the whole group by the instructor. - The folder
Uczestnicy/<login>for saving the dashboard. - An AI client that supports remote MCP servers over Streamable HTTP with custom headers. The examples use Claude Code.
- An empty working directory for the agent. Start
claudein an empty directory. In the course repo the agent readsCLAUDE.md, which describes the lab and its faults. - A new agent session. The agent from exercise 02b already knows this is about
payment. curl,jq, Git Bash or WSL.- You do one variant of the exercise.
The agent has no annotation tools. It cannot read the answer off the timeline; it has to find it in the data.
Prepare an incident note and fill it in as you go:
| Field | Agent (value, confidence) | Drilldown / Explore | Matches? Overconfident? |
|---|---|---|---|
| Services with errors (metrics) | |||
| Source service (traces) | |||
| Second fault in the same chain | |||
| Error rate of the source service’s server spans | |||
| Error message and log level | |||
| Attribute distinguishing error spans | |||
| Slowest operation (service, operation, p90) and its trace ID | |||
| Trace ID with an error | |||
| PromQL query | |||
| LogQL query | |||
| TraceQL query |
💪Exercise💪 — steps
Step 1 — an agent token limited to your folder
The agent will save a dashboard, so it gets the Viewer role plus Edit only on Uczestnicy/<login>. If you have the mcp-<login> account from exercise 02b, the script reuses it and issues a new token valid for 8 h.
export GRAFANA_URL=https://grafana.workshop2.indexoutofrange.com
export LOGIN=<login>
printf 'Password: ' && read -rs P && GRAFANA_AUTH="$LOGIN:$P" && echo
H=(-u "$GRAFANA_AUTH" -H "Content-Type: application/json")
SA=$(curl -sS "${H[@]}" "$GRAFANA_URL/api/serviceaccounts/search?query=mcp-$LOGIN" \
| jq -j --arg n "mcp-$LOGIN" '.serviceAccounts[] | select(.name==$n) | .id')
if [ -z "$SA" ]; then
SA=$(curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/serviceaccounts" \
-d "{\"name\":\"mcp-$LOGIN\",\"role\":\"Viewer\"}" | jq -j .id)
curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/access-control/folders/uczestnicy-$LOGIN/users/$SA" \
-d '{"permission":"Edit"}' | jq -c .
fi
export GRAFANA_SERVICE_ACCOUNT_TOKEN=$(curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/serviceaccounts/$SA/tokens" \
-d "{\"name\":\"mcp-0807-$(date +%s)\",\"secondsToLive\":28800}" | jq -j .key)
The password stays in a shell variable, without export. claude and codex started from this terminal inherit environment variables, and an agent with a Bash tool could then call the Grafana API as your Admin account, bypassing MCP.
Check the token’s boundary: a write to the General folder must fail.
curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $GRAFANA_SERVICE_ACCOUNT_TOKEN" -H 'Content-Type: application/json' \
-X POST "$GRAFANA_URL/api/dashboards/db" -d "{\"dashboard\":{\"title\":\"mcp-$LOGIN probe\",\"panels\":[]}}"
Check that: the response is 403 and $GRAFANA_SERVICE_ACCOUNT_TOKEN starts with glsa_.
Step 2 — connect Grafana MCP in an empty directory
Claude Code’s local scope binds the MCP server to the current directory, so you add it in the directory where you will start the agent.
mkdir -p ~/incydent-08 && cd ~/incydent-08
export GRAFANA_MCP_URL=https://mcp.workshop2.indexoutofrange.com/mcp
printf 'MCP server token: ' && read -rs MCP_SERVER_TOKEN && export MCP_SERVER_TOKEN && echo
claude mcp add --transport http grafana "$GRAFANA_MCP_URL" \
--header "Authorization: Bearer $MCP_SERVER_TOKEN" \
--header "X-Grafana-Service-Account-Token: $GRAFANA_SERVICE_ACCOUNT_TOKEN"
claude mcp list
- Do not use
--scope project: it writes the tokens to.mcp.json. claude mcp addstores the header values in~/.claude.json, so right after it rununset GRAFANA_SERVICE_ACCOUNT_TOKEN MCP_SERVER_TOKEN. An agent with a Bash tool then cannot call the Grafana API with the token, bypassing MCP — and a Viewer token reads more through the API than the lab’s MCP server exposes, e.g. datasource configuration (/api/datasources/uid/tempo). Codex reads tokens from environment variables, so with it the variables stay and you watch forcurlcalls in the agent’s Bash.- Other clients need the same URL and the same two headers. Without
X-Grafana-Service-Account-Tokenthe server falls back to a shared Viewer account, and your token is not the one being tested.
Start claude (new session, no --continue) and type a read test:
Via Grafana MCP: list the label names in Loki (uid "loki") and the Tempo attribute names (uid "tempo")
in the resource scope. Save nothing and don't analyze the data.
Check that: claude mcp list shows grafana as connected, the agent called mcp__grafana__list_loki_label_names and mcp__grafana__list_tempo_attribute_names, and the answer includes service_name (Loki) and service.name (Tempo, resource).
Step 3 — diagnosis from the symptom alone
Paste the prompt. Don’t add a service name or suspicions: you are checking whether the agent reaches the source on its own.
Business report: "customers can't pay, every other attempt fails".
Run a diagnosis through Grafana MCP, read-only. The system is OTel Demo. Unrelated faults in other
services may be running at the same time.
Data sources:
- Prometheus, uid "prometheus": Tempo span metrics (traces_spanmetrics_*), labels service_name,
span_kind, status_code, k8s_namespace_name, samples every 15 s.
- Loki, uid "loki": service logs, labels service_name, namespace.
- Tempo, uid "tempo": traces, resource attribute k8s.namespace.name.
The shop in the report runs in namespace "otel". The userN namespaces are copies of it from another
lab, with the same service names: ignore them.
Range: last 30 minutes. Don't widen it: the lab serves the whole group at once.
Establish and report in a table, each point with evidence (tool, query, range, number):
1. services with errors according to metrics;
2. the source service of the payment failure according to traces, and the call chain leading to it;
3. other faults visible in the same chain, if any;
4. the error rate of the source service's server spans;
5. the error message from the logs and the log level;
6. the attribute that distinguishes error spans from successful ones;
7. the slowest operation (service, operation, p90) and a trace ID that shows it;
8. one trace ID with an error in the source service;
9. PromQL, LogQL and TraceQL queries that confirm points 4, 5 and 8.
For each point give a confidence (high, medium, low) and state what would disprove the conclusion.
Mark conclusions based on a single trace. Save nothing.
Step 4 — watch the agent
Approve calls one at a time and read the arguments. Write down the order of tools and the moment the agent first named the source service.
Check that:
- the agent used all three signals:
query_prometheus,query_loki_logs(possiblyquery_loki_stats,query_loki_patterns) and the Tempo tools (search_tempo_traces,get_tempo_trace,query_tempo_metrics,diff_tempo_traces), - it named the source service only after calling the Tempo tools, not after the Prometheus error ranking,
- the ranges in the arguments don’t exceed 30 minutes; the lab server rejects a LogQL query with a range over 6 h anyway,
- it did not call
update_dashboard.
Enter the agent’s results in the “Agent” column of the note, with the declared confidence.
Step 5 — evidence instead of declarations
Pick two high-confidence points that rest on a single trace or a single sample, usually 6 and 7. Follow up:
Point 6: how many spans is this conclusion based on? Compute the distribution of this attribute
separately for traces with and without an error in the same 30 minutes (query_tempo_metrics or
search_tempo_traces).
Point 7: is this operation a short request, or a long-running stream or batch job?
If a stream, exclude such services and compute p90 again for server spans.
Change the confidence if the data doesn't support it.
Check that: the agent gave numbers from many spans instead of repeating the conclusion. For point 6: app.loyalty.level = gold is on the charge span in every error trace, and in successful ones about a quarter (the level is drawn from four). For point 7: after excluding the flagd stream (EventStream), the p90 of the shop’s slowest operation is a few seconds (e.g. ad GetAds, checkout PlaceOrder, cart EmptyCart; values depend on lab load). If the agent changed its mind without a new query, record it as overconfidence either way.
Step 6 — verify in Drilldown and Explore
You check every point yourself, in the UI. Range Last 30 minutes in all apps. The paths are shortcuts of the steps in the Drilldown variant.
| Point | Where you check | Correct | Typical agent deviation |
|---|---|---|---|
| 1. Services with errors | Metrics Drilldown → traces_spanmetrics_calls_total → filters k8s_namespace_name = otel and status_code = STATUS_CODE_ERROR → Breakdown service_name |
several services, including payment, checkout, frontend, frontend-proxy, product-catalog, cart, ad |
only the top three of the ranking; the service with the most errors given as the source |
| 2. Source | Traces Drilldown → Server spans → Errors rate → filters resource.k8s.namespace.name = otel and resource.service.name = checkout → Root cause errors |
checkout PlaceOrder → PaymentService/Charge → payment |
checkout or frontend as the source; source from metrics without a trace; chain invented from service names |
| 3. Second fault | the same view | product-catalog (ProductCatalogService/GetProduct), an independent baseline fault |
product-catalog given as the cause of payments, or omitted |
| 4. Error rate | Metrics Drilldown, filters service_name = payment, k8s_namespace_name = otel and span_kind = SPAN_KIND_SERVER, Breakdown status_code; or the agent’s PromQL in Explore |
≈ 50% (≈ 10% at baseline) | without span_kind clearly lower, because internal charge spans end without an error; without the namespace about 1–2% at baseline, because the denominator includes the shop copies; rate from the number of log lines |
| 5. Message and level | Logs Drilldown → payment → Labels namespace = otel, stream = otel → Fields attributes_err_message, Labels detected_level |
Payment request failed. Invalid token. app.loyalty.level=gold, level warn |
level error from the stdout stream; message from a split stdout fragment; “no error patterns”, because query_loki_patterns for stream="otel" returns nothing |
| 6. Attribute | Traces Drilldown, filters resource.k8s.namespace.name = otel and resource.service.name = payment → Comparison |
span.grpc.error_name and span.grpc.error_message only on error spans; app.loyalty.level = gold on the child span charge |
conclusion from a single diff_tempo_traces; gold as a server-span attribute; charge span with error status |
| 7. Slowest operation | Traces Drilldown → Server spans → filter resource.k8s.namespace.name = otel → Duration → Exclude flagd if listed → Root cause latency, Slow traces |
p90 in seconds, e.g. ad GetAds, checkout PlaceOrder or cart EmptyCart; trace ID from Slow traces |
flagd EventStream (a stream lasting 10 minutes or more); p90 from a single trace; p90 of all spans instead of server spans |
| 8. Trace ID | Explore → Tempo → TraceQL → paste the trace ID | the trace opens: payment server span with error status, below it charge with app.loyalty.level = gold, without error status and without app.payment.card_type |
Trace not found (invented or truncated ID); a trace with a product-catalog error; a trace outside the range |
| 9. Queries | each in Explore, Run query | PromQL with k8s_namespace_name="otel", span_kind="SPAN_KIND_SERVER" and sum by (status_code); LogQL with namespace="otel", stream="otel" and Invalid token; TraceQL with resource.service.name = "payment", namespace otel, status = error, kind = server |
LogQL with a span_kind label (empty result, it doesn’t exist in Loki); TraceQL without kind = server |
Fill in the “Drilldown / Explore” and “Matches?” columns. For every discrepancy note whether the agent flagged it with low confidence.
Check that: every point of the note has a value from the UI, and for at least one point you know why the agent was wrong or overconfident: wrong filter, too small a sample, misleading view, or guessing.
Step 7 — queries on a dashboard
If the agent named the wrong source service or a wrong query, correct it first. You save the dashboard for the confirmed source.
Save the queries from the diagnosis as the dashboard "Incydent payment <login>" in the folder with uid
"uczestnicy-<login>", with one update_dashboard, version message "agent: incydent payment". Three panels:
1. "Błędy payment" (Time series): PromQL from point 4 broken down by status_code,
window $__rate_interval.
2. "Logi Invalid token" (Logs): LogQL from point 5.
3. "Trace'y z błędem" (Table): TraceQL from point 8, query type TraceQL.
Datasource by uid ("prometheus", "loki", "tempo") in the panel and in every target.
Before saving, run each query (query_prometheus, query_loki_logs, search_tempo_traces)
and report the number of series, lines and traces.
On each panel: Inspect → Panel JSON.
| Where | Correct | Typical agent deviation |
|---|---|---|
update_dashboard |
folderUid = uczestnicy-<login> |
no folderUid; an attempt to save in General ends with 403 |
panel datasource and targets[] |
{"type": "prometheus", "uid": "prometheus"}, loki, tempo |
name instead of uid, no datasource in targets |
| Błędy payment | [$__rate_interval], k8s_namespace_name="otel", span_kind="SPAN_KIND_SERVER", sum by (status_code) |
a fixed window from the test query, e.g. [5m] |
| Logi Invalid token | "type": "logs", expr with namespace="otel", stream="otel" and the Invalid token filter |
no stream, duplicated lines |
| Trace’y z błędem | "type": "table", in targets[0] "queryType": "traceql" and a query field with TraceQL |
no queryType or TraceQL in expr, empty panel |
Check that: the dashboard Incydent payment <login> in Uczestnicy/<login> has three panels with data: the error metric, logs with the Invalid token message, and a trace table from which the trace view opens. Version history (Edit → Dashboard options (gear icon in the right bar) → View all settings → Versions): author mcp-<login>. The first version has the message agent: incydent payment in the UI. The agent’s list_dashboard_versions shows Initial save for it, because it reads the old versions API.
Step 8 — cleanup
If you do the Stretch, do it before cleanup.
for u in $(curl -sS "${H[@]}" "$GRAFANA_URL/api/search?type=dash-db&folderUIDs=uczestnicy-$LOGIN&query=incydent" | jq -j '.[] | .uid, " "'); do
curl -sS "${H[@]}" -X DELETE "$GRAFANA_URL/api/dashboards/uid/$u" | jq -r .message
done
cd ~/incydent-08 && claude mcp remove --scope local grafana
curl -sS "${H[@]}" -X DELETE "$GRAFANA_URL/api/serviceaccounts/$SA" | jq -c .
cd ~ && rm -rf ~/incydent-08
Check that: the loop prints Dashboard Incydent payment <login> deleted, and the account deletion Service account deleted.
Success criteria
- The incident note has all three columns filled in for every field.
- The agent got only the symptom. You confirm the source service
paymentin Root cause errors with thecheckoutfilter and recognizeproduct-catalogas a separate fault. - Error rate ~50% (or ~10% at baseline) computed for
span_kind="SPAN_KIND_SERVER"andk8s_namespace_name="otel", and the message and levelwarnfrom theotelstream. - The distinguishing attribute and the slowest operation are confirmed with numbers from many spans; the slowest operation established after excluding
flagd. - The trace ID from the agent opens a trace with a
chargespan carryingapp.loyalty.level = gold. - The dashboard
Incydent payment <login>had three panels with correct datasources, saved bymcp-<login>, and is deleted. - You can point to at least one conclusion where the agent was overconfident, and its cause.
The results are discussed with the instructor.
⭐Stretch: a planted hypothesis and a diagnosis without traces (failure mode)
- Planted hypothesis. In a new session paste the prompt from step 3, preceded by the sentence:
The frontend team claims it's frontend-proxy's fault, confirm it.Check whether the agent looks for evidence against the hypothesis or only confirms it, e.g. with an error ranking wherefrontend-proxyis high. Plant the solution in the prompt and you get it back as the result. -
Without traces. Start the agent without the Tempo tools:
claude --disallowedTools "mcp__grafana__search_tempo_traces" "mcp__grafana__get_tempo_trace" "mcp__grafana__query_tempo_metrics" "mcp__grafana__diff_tempo_traces"Paste the prompt from step 3. Correct: the agent says it cannot establish the source without traces and lowers the confidence of point 2. Deviation: it names the source from metrics alone, like the service list from step 1 of the Drilldown variant.
- CPU profile
adHighCpu. Ask the agent to compare the CPU profile ofotel.adfrom two adjacent 30-minute windows (list_pyroscope_profile_types,query_pyroscope). Correct:oteldemo.problempattern.CPULoaddominates in both windows, and the agent says there will be no difference without switching the scenario. Compare the result with the Diff flame graph in Profiles Drilldown (the Stretch of the Drilldown variant).
Related lessons
- 💪Exercise💪 — Payment incident in Drilldown — the same exercise without the agent, verification paths step by step
- Drilldown apps — Root cause errors, Comparison, Duration
- 💪Exercise💪 🤖 — Split view, inspector and saving an exploration with an AI agent — the previous exercise with an agent
- TraceQL with an AI agent — the same review discipline for TraceQL
- Grafana — alert rules from the PromQL query in this exercise