đȘExerciseđȘ đ€ â Split view, inspector and saving an exploration with an AI agent
The same problem as in the manual variant: the same PromQL gives different values in a panel and in Explore. An AI agent explores the
paymentmetric and logs through Grafana MCP and finally saves the LogQL as a panel. The agent is a third place where the same query is computed, with its own step and window. You measure in Query inspector how they differ.
Goal
- The agent explores the
paymentmetric and logs (query_prometheus,query_loki_logs) and returns Explore links (generate_deeplink). - You assemble a split view with synced time by hand and share it with a short link.
- You measure the panelâs and Exploreâs
intervalMs,maxDataPointsandStepin Query inspector and compare them with the agentâsstepSecondsandrate()window. - The agent aligns the panelâs interval with a patch and adds a Logs panel with LogQL to the dashboard (
update_dashboard).
Environment: workshop lab, your own Grafana account, your folder Uczestnicy/<login>, a terminal, an MCP-capable AI client.
Prerequisites
- Your lab login (
user<N>), its Grafana password and the MCP server token, all from the lab handout; your folderUczestnicy/<login>. - The
paymentFailurescenario at the50%variant. How to check the ratio: prerequisites of the manual variant. You donât switch flagd: the instructor changes the variant for the whole group. Steps 5â8 work without the scenario too. - An AI client that supports remote MCP servers over Streamable HTTP with custom headers. The examples use Claude Code.
- A clone of the materials repo:
git clone https://github.com/ProtopiaTech-Labs/grafana-admin.git materialy. Run the commands in thematerialydirectory and start Claude Code there too, because the agent readspliki/08-explore-vs-panel.jsonfrom it. Donât start the agent in the course repo: there it readsCLAUDE.md, which describes the lab and its faults. curl,jq, Git Bash or WSL.- Do one variant of the exercise.
Who does what:
| Step | Who | Why |
|---|---|---|
| PromQL and LogQL queries, Explore links | agent | query_prometheus, query_loki_logs, generate_deeplink |
| split view, time sync, split view short link | you, in the UI | generate_deeplink builds a link to one Explore pane; split view state and time sync are set in the UI |
intervalMs, maxDataPoints, Step |
you, in Query inspector | they depend on the graph width in the browser; the labâs server doesnât render panels and the agent canât see them |
| Min step in Explore | you, in the UI | an Explore query option, not an object in Grafana |
| dashboard import, panel Min interval, Logs panel | agent | update_dashboard (full JSON or operations) |
đȘExerciseđȘ â steps
Step 1 â an agent token limited to your folder
The agent saves a dashboard, so it gets the Viewer role plus Edit only on Uczestnicy/<login>. If the service account mcp-<login> already exists from another exercise, the script reuses it and issues a new token valid for 8 h.
export GRAFANA_URL=https://grafana.workshop2.indexoutofrange.com
export LOGIN=<login>
printf 'Password: ' && read -rs P && GRAFANA_AUTH="$LOGIN:$P" && echo
H=(-u "$GRAFANA_AUTH" -H "Content-Type: application/json")
SA=$(curl -sS "${H[@]}" "$GRAFANA_URL/api/serviceaccounts/search?query=mcp-$LOGIN" \
| jq -j --arg n "mcp-$LOGIN" '.serviceAccounts[] | select(.name==$n) | .id')
if [ -z "$SA" ]; then
SA=$(curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/serviceaccounts" \
-d "{\"name\":\"mcp-$LOGIN\",\"role\":\"Viewer\"}" | jq -j .id)
curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/access-control/folders/uczestnicy-$LOGIN/users/$SA" \
-d '{"permission":"Edit"}' | jq -c .
fi
export GRAFANA_SERVICE_ACCOUNT_TOKEN=$(curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/serviceaccounts/$SA/tokens" \
-d "{\"name\":\"mcp-0802-$(date +%s)\",\"secondsToLive\":28800}" | jq -j .key)
The password stays in a shell variable, without export. claude and codex started from this terminal inherit environment variables, and an agent with a Bash tool could then call the Grafana API as your Admin account, bypassing MCP.
Check the tokenâs boundary: a write to the General folder must fail.
curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $GRAFANA_SERVICE_ACCOUNT_TOKEN" -H 'Content-Type: application/json' \
-X POST "$GRAFANA_URL/api/dashboards/db" -d "{\"dashboard\":{\"title\":\"mcp-$LOGIN probe\",\"panels\":[]}}"
Check that: the response is 403 and $GRAFANA_SERVICE_ACCOUNT_TOKEN starts with glsa_.
Step 2 â connecting Grafana MCP
Run the commands in the materialy directory where you will start claude later: the local scope binds the MCP server to the current directory. If the grafana server is already added with an old token, first run claude mcp remove --scope local grafana.
export GRAFANA_MCP_URL=https://mcp.workshop2.indexoutofrange.com/mcp
printf 'MCP server token: ' && read -rs MCP_SERVER_TOKEN && export MCP_SERVER_TOKEN && echo
claude mcp add --transport http grafana "$GRAFANA_MCP_URL" \
--header "Authorization: Bearer $MCP_SERVER_TOKEN" \
--header "X-Grafana-Service-Account-Token: $GRAFANA_SERVICE_ACCOUNT_TOKEN"
claude mcp list
- The default
localscope keeps the tokens in~/.claude.json, outside the repo. Donât use--scope project: it writes the tokens to.mcp.json. claude mcp addstores the header values in~/.claude.json, so right after it rununset GRAFANA_SERVICE_ACCOUNT_TOKEN MCP_SERVER_TOKEN. An agent with a Bash tool then canât call the Grafana API with the token bypassing MCP â and a Viewer token reads more through the API than the labâs MCP server exposes, e.g. datasource configuration (/api/datasources/uid/tempo). Codex reads tokens from environment variables, so with it the variables stay and you watch thecurlcalls in the agentâs Bash.- Other clients need the same URL and the same two headers. Without
X-Grafana-Service-Account-Tokenthe server falls back to a shared Viewer account, and your token is not the one being used.
Start claude and type a read test:
Via Grafana MCP: in the Loki datasource with uid "loki" list the label names and the values of the stream label.
In the Prometheus datasource with uid "prometheus" list the metrics starting with traces_spanmetrics.
Don't save anything.
Check that: claude mcp list shows grafana as connected, the agent called mcp__grafana__list_loki_label_* and mcp__grafana__list_prometheus_metric_names, and the answer contains the labels service_name and stream with values otel and stdout, and the metric traces_spanmetrics_calls_total.
Step 3 â the agent explores the payment metric and logs
The prompt says what the agent must find but gives no PromQL or LogQL:
Via Grafana MCP, read-only. The "payment" service of the OTel Demo, last 1 h. The shop runs
in namespace "otel"; the userN namespaces are copies from another lab, skip them.
1. Prometheus, uid "prometheus" (Tempo span metrics, samples every 15 s, labels service_name,
span_kind, status_code, k8s_namespace_name): requests per second to payment broken down
by status_code, and the error ratio. Count only server spans. Report the PromQL, stepSeconds,
the rate() window, the number of series and the ratio.
2. Loki, uid "loki": payment log lines about failed payments. If the service has several
streams with the same records, pick one and justify it. Report the LogQL, the number of lines
returned and three example lines.
3. generate_deeplink: an Explore link for each of the two queries (with the query in the queries parameter), with the same time range.
State exactly what each link restores: datasource, range, query.
Don't save anything.
Approve tool calls one at a time and read the arguments. For query_prometheus note stepSeconds, startTime, endTime and the rate() window â you compare them in step 6.
Check that: the agent called query_prometheus, query_loki_logs and generate_deeplink twice, and the answer gives queries, numbers and two links. Compare with the table:
| Where | Correct | Typical agent deviation |
|---|---|---|
| PromQL: matchers | service_name="payment", k8s_namespace_name="otel", span_kind="SPAN_KIND_SERVER" in numerator and denominator |
no span_kind, ratio too low: internal charge spans end without an error and dilute the denominator; no k8s_namespace_name: the ratio mixes 18 copies of the shop |
| PromQL: breakdown | sum by (status_code), two series: STATUS_CODE_UNSET and STATUS_CODE_ERROR |
no aggregation (series per pod and operation) or STATUS_CODE_OK in the answer, which doesnât exist in the data |
| ratio | 0.35â0.65 at the 50% variant, ~0.1 at baseline (over the whole hour; the shop makes ~3 payments a minute, so a single 5m window is noisy) |
one instant sample presented as the hourly average; ratio computed from the number of log lines |
stepSeconds and rate() window |
stated explicitly in the answer | âdefaultâ, no value |
| LogQL: selector | service_name="payment", namespace="otel", stream="otel" |
no stream: duplicated records and stack: 'Error: âŠ' + fragments from stdout |
| number of lines | the agent says query_loki_logs returns 10 lines by default and at most 100 (limit), and reports resultsTruncated / totalLinesScanned from the response metadata |
âthere were 10 errors in the hourâ or â100 errorsâ |
| links | the link opens Explore with the agentâs query (generate_deeplink with queries; with shorten: true as /goto/âŠ) |
link without queries: Explore opens with an empty editor; link range different from the query range |
Note the differences. If the agent dropped span_kind, the namespace or stream, correct it in one sentence and have it rerun the query.
Step 4 â split view and short link (by hand)
- Open the agentâs Prometheus link. If the editor is empty, the agent didnât pass
queries: paste the agentâs PromQL. Run query. - Split â in the right pane datasource Loki, the agentâs LogQL.
- The chain icon next to the time picker (Synced times) is highlighted: Split copies the range and syncs time right away. Donât click it, or you unsync the panes. Drag-select a few minutes of the left graph where the
STATUS_CODE_ERRORseries is above zero. - Share in the top right (in split view just the icon) copies a short link (Copy shortened URL). Open the link in a new tab.
- Close the right pane (Close). In the left one change the datasource from Prometheus to Loki.
Check that:
- the right pane narrows to the selected window and shows
paymentlogs about failed payments, - the short link has the form
/goto/<uid>and restores both split panes with the same range, while the agentâs links restore only one pane, - after the datasource change the Loki editor has the selector
{service_name="payment"}:span_kindandk8s_namespace_namedrop out because they donât exist in Loki.
Step 5 â the agent imports the ânarrow vs wide panelâ dashboard
Read the file pliki/08-explore-vs-panel.json and save it through update_dashboard in the folder
with uid "uczestnicy-<login>". Don't add uid or id, don't change queries, titles or gridPos.
Version message: "agent: import 08". Report the UID assigned by Grafana.
Check that: update_dashboard has folderUid = uczestnicy-<login>, and the dashboard 08 â Panel wÄ
ski vs szeroki opens in Uczestnicy/<login>. Panel WÄ
ski (narrow) is 6 columns wide, Szeroki (wide) 24, both with the same query with [$__interval]. If the agent added a uid, check that it didnât overwrite someone elseâs dashboard (Edit â Dashboard options (gear icon in the right bar) â View all settings â Versions has one version).
Step 6 â measuring: panel, Explore and agent
- On panel WÄ
ski: Inspect â Query â Refresh. Note
queries[0].maxDataPoints,queries[0].intervalMs(request) andStepfromframes[0].schema.meta.executedQueryString(response). - Panel menu of WÄ ski â Explore (no split view). Query inspector â Query â Refresh. Note the same three values.
- Ask the agent:
Get the query of panel "WÄ
ski" from the dashboard you just saved (get_dashboard_panel_queries),
and run it through query_prometheus for the last 24 h. Report stepSeconds, the window you substituted
for $__interval, and the series maximum. Don't save anything.
| Where | maxDataPoints |
intervalMs |
Step / stepSeconds |
rate() window |
Maximum |
|---|---|---|---|---|---|
| Panel WÄ ski | Â | Â | Â | Â | Â |
| Explore from the panel menu | Â | Â | Â | Â | Â |
agentâs query_prometheus |
â | â | Â | Â | Â |
Check that: the panel has a larger step than Explore (typically panel 300000 and 5m0s, Explore 60000 and 1m0s; the values depend on screen width). The Explore editor shows [5m]: when opening from the panel menu Grafana substitutes $__interval with the panelâs value, so the window is the same and only the step differs. The agent has a third set of values that it chose itself. Correct: the agent says $__interval is a Grafana macro and states which window it used in its place. Typical deviation: it sends $__interval literally and gets a parse error, or compares its maximum with the panel without knowing the panelâs step.
Step 7 â aligning the interval
- Ask the agent for a patch:
Fix panel "WÄ
ski" on this dashboard through update_dashboard with uid and a list of operations:
panel Min interval 5m, in the query the window [5m] instead of [$__interval]. Don't change panel "Szeroki".
Before saving, check the paths with get_dashboard_property. Version message: "agent: min interval 5m".
Then run this query through query_prometheus for 24 h with a 300 s step and report the maximum.
- In Explore from step 6: Options under the query â Min step =
5m(the[5m]window is already there) â Run query. - Repeat the step 6 measurement in the panel and in Explore.
Check that:
- the panel and Explore both have
Step=5m0s, and the three maxima (panel, Explore, agent) are close. In ExploreintervalMsmay stay60000: Min step is applied by the datasource backend.Stepdecides the result. - the Panel JSON of panel WÄ
ski has
"interval": "5m"at panel level, andtargets[0].exprhas[5m]. Typical deviation:intervalwritten intotargets[0], or panel Szeroki changed too.
Step 8 â the agent saves LogQL as a Logs panel
Add to this dashboard a panel of type Logs titled "payment logs" with the LogQL query from step 3
(datasource uid "loki"). Use update_dashboard with uid and operations, appending the panel at the end
of the panels array, below the existing panels. Before saving, run the query through query_loki_logs
and report the number of lines. Version message: "agent: logs panel".
Check that:
- the dashboard has a third panel of type Logs with
paymentlog lines, and the Panel JSON has"type": "logs",datasource{"type": "loki", "uid": "loki"}in the panel and intargets[0], and anexprwithstream="otel", - version history (Edit â Dashboard options â View all settings â Versions): author
mcp-<login>, every version with the message from the prompts. The agentâslist_dashboard_versionsshowsInitial savefor the first version because it reads the old versions API; the Versions list in the UI shows the right message. The diff of the last version covers only the new panel.
Typical deviation: gridPos overlapping an existing panel, "type": "logs-panel", or a panel without datasource in targets.
Step 9 â cleanup
If you do the Stretch, do it before cleanup. Delete the dashboard:
for u in $(curl -sS "${H[@]}" "$GRAFANA_URL/api/search?type=dash-db&folderUIDs=uczestnicy-$LOGIN&query=szeroki" | jq -j '.[] | .uid, " "'); do
curl -sS "${H[@]}" -X DELETE "$GRAFANA_URL/api/dashboards/uid/$u" | jq -r .message
done
If you also work with the agent in exercise 07b, keep the service account and the MCP server (the token expires after 8 h). If not:
claude mcp remove --scope local grafana
curl -sS "${H[@]}" -X DELETE "$GRAFANA_URL/api/serviceaccounts/$SA" | jq -c .
Check that: the loop prints Dashboard 08 â Panel wÄ
ski vs szeroki deleted, and the account deletion Service account deleted.
Success criteria
- The short link
/goto/<uid>restores the metric + logs split view with synced time; the agentâs links open one Explore pane. - The step 6 table has three rows: the panel with a larger step than Explore (typically
300000/5m0svs60000/1m0s) and the same window (the panelâs substituted$__interval, typically[5m]), the agent with its ownstepSecondsand window. - After step 7 the panel and Explore both have
Step=5m0s, and the agentâs maximum at a 300 s step is close. - The dashboard
08 â Panel wÄ ski vs szerokiinUczestnicy/<login>has a Logs panel withstream="otel", and the versions saved by the agent have authormcp-<login>. - You have a list of differences between the agentâs queries and the step 3 table.
âStretch: the agent explains a discrepancy it canât see (failure mode)
- Explanation without measurement. In a new session, before step 7, ask:
Why does panel "WÄ ski" on the dashboard 08 â Panel wÄ ski vs szeroki show a different maximum than Explore? Report the panel's intervalMs and maxDataPoints.The agent has the panelâs queries but doesnât know the graph width in the browser. Correct: it says it canât measure these values and points to Query inspector. Deviation: it reports a specificintervalMsit couldnât have measured. Compare with the step 6 table. - A split view URL assembled by hand. Ask the agent for an Explore link with split view (Loki with
Invalid tokenon the left, the PromQL from step 3 on the right) built as thepanesparameter (schemaVersion=1, keysaandb), withoutgenerate_deeplink. Open the link. Check that the JSON is encoded correctly and that the datasources have uidlokiandprometheus. Compare with the Stretch of the manual variant.
Related lessons
- Split view, inspector and saving an exploration â the same exercise without an agent
- Explore â Query inspector and panel â Explore
- Payment incident with an AI agent â the next exercise with an agent
- TraceQL with an AI agent â the same token and MCP setup with a read-only agent