đŸ’ȘExerciseđŸ’Ș đŸ€– — Split view, inspector and saving an exploration with an AI agent

The same problem as in the manual variant: the same PromQL gives different values in a panel and in Explore. An AI agent explores the payment metric and logs through Grafana MCP and finally saves the LogQL as a panel. The agent is a third place where the same query is computed, with its own step and window. You measure in Query inspector how they differ.

Goal

  • The agent explores the payment metric and logs (query_prometheus, query_loki_logs) and returns Explore links (generate_deeplink).
  • You assemble a split view with synced time by hand and share it with a short link.
  • You measure the panel’s and Explore’s intervalMs, maxDataPoints and Step in Query inspector and compare them with the agent’s stepSeconds and rate() window.
  • The agent aligns the panel’s interval with a patch and adds a Logs panel with LogQL to the dashboard (update_dashboard).

Environment: workshop lab, your own Grafana account, your folder Uczestnicy/<login>, a terminal, an MCP-capable AI client.

Prerequisites

  • Your lab login (user<N>), its Grafana password and the MCP server token, all from the lab handout; your folder Uczestnicy/<login>.
  • The paymentFailure scenario at the 50% variant. How to check the ratio: prerequisites of the manual variant. You don’t switch flagd: the instructor changes the variant for the whole group. Steps 5–8 work without the scenario too.
  • An AI client that supports remote MCP servers over Streamable HTTP with custom headers. The examples use Claude Code.
  • A clone of the materials repo: git clone https://github.com/ProtopiaTech-Labs/grafana-admin.git materialy. Run the commands in the materialy directory and start Claude Code there too, because the agent reads pliki/08-explore-vs-panel.json from it. Don’t start the agent in the course repo: there it reads CLAUDE.md, which describes the lab and its faults.
  • curl, jq, Git Bash or WSL.
  • Do one variant of the exercise.

Who does what:

Step Who Why
PromQL and LogQL queries, Explore links agent query_prometheus, query_loki_logs, generate_deeplink
split view, time sync, split view short link you, in the UI generate_deeplink builds a link to one Explore pane; split view state and time sync are set in the UI
intervalMs, maxDataPoints, Step you, in Query inspector they depend on the graph width in the browser; the lab’s server doesn’t render panels and the agent can’t see them
Min step in Explore you, in the UI an Explore query option, not an object in Grafana
dashboard import, panel Min interval, Logs panel agent update_dashboard (full JSON or operations)

đŸ’ȘExerciseđŸ’Ș — steps

Step 1 — an agent token limited to your folder

The agent saves a dashboard, so it gets the Viewer role plus Edit only on Uczestnicy/<login>. If the service account mcp-<login> already exists from another exercise, the script reuses it and issues a new token valid for 8 h.

export GRAFANA_URL=https://grafana.workshop2.indexoutofrange.com
export LOGIN=<login>
printf 'Password: ' && read -rs P && GRAFANA_AUTH="$LOGIN:$P" && echo
H=(-u "$GRAFANA_AUTH" -H "Content-Type: application/json")

SA=$(curl -sS "${H[@]}" "$GRAFANA_URL/api/serviceaccounts/search?query=mcp-$LOGIN" \
  | jq -j --arg n "mcp-$LOGIN" '.serviceAccounts[] | select(.name==$n) | .id')
if [ -z "$SA" ]; then
  SA=$(curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/serviceaccounts" \
    -d "{\"name\":\"mcp-$LOGIN\",\"role\":\"Viewer\"}" | jq -j .id)
  curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/access-control/folders/uczestnicy-$LOGIN/users/$SA" \
    -d '{"permission":"Edit"}' | jq -c .
fi
export GRAFANA_SERVICE_ACCOUNT_TOKEN=$(curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/serviceaccounts/$SA/tokens" \
  -d "{\"name\":\"mcp-0802-$(date +%s)\",\"secondsToLive\":28800}" | jq -j .key)

The password stays in a shell variable, without export. claude and codex started from this terminal inherit environment variables, and an agent with a Bash tool could then call the Grafana API as your Admin account, bypassing MCP.

Check the token’s boundary: a write to the General folder must fail.

curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $GRAFANA_SERVICE_ACCOUNT_TOKEN" -H 'Content-Type: application/json' \
  -X POST "$GRAFANA_URL/api/dashboards/db" -d "{\"dashboard\":{\"title\":\"mcp-$LOGIN probe\",\"panels\":[]}}"

Check that: the response is 403 and $GRAFANA_SERVICE_ACCOUNT_TOKEN starts with glsa_.

Step 2 — connecting Grafana MCP

Run the commands in the materialy directory where you will start claude later: the local scope binds the MCP server to the current directory. If the grafana server is already added with an old token, first run claude mcp remove --scope local grafana.

export GRAFANA_MCP_URL=https://mcp.workshop2.indexoutofrange.com/mcp
printf 'MCP server token: ' && read -rs MCP_SERVER_TOKEN && export MCP_SERVER_TOKEN && echo
claude mcp add --transport http grafana "$GRAFANA_MCP_URL" \
  --header "Authorization: Bearer $MCP_SERVER_TOKEN" \
  --header "X-Grafana-Service-Account-Token: $GRAFANA_SERVICE_ACCOUNT_TOKEN"
claude mcp list
  • The default local scope keeps the tokens in ~/.claude.json, outside the repo. Don’t use --scope project: it writes the tokens to .mcp.json.
  • claude mcp add stores the header values in ~/.claude.json, so right after it run unset GRAFANA_SERVICE_ACCOUNT_TOKEN MCP_SERVER_TOKEN. An agent with a Bash tool then can’t call the Grafana API with the token bypassing MCP — and a Viewer token reads more through the API than the lab’s MCP server exposes, e.g. datasource configuration (/api/datasources/uid/tempo). Codex reads tokens from environment variables, so with it the variables stay and you watch the curl calls in the agent’s Bash.
  • Other clients need the same URL and the same two headers. Without X-Grafana-Service-Account-Token the server falls back to a shared Viewer account, and your token is not the one being used.

Start claude and type a read test:

Via Grafana MCP: in the Loki datasource with uid "loki" list the label names and the values of the stream label.
In the Prometheus datasource with uid "prometheus" list the metrics starting with traces_spanmetrics.
Don't save anything.

Check that: claude mcp list shows grafana as connected, the agent called mcp__grafana__list_loki_label_* and mcp__grafana__list_prometheus_metric_names, and the answer contains the labels service_name and stream with values otel and stdout, and the metric traces_spanmetrics_calls_total.

Step 3 — the agent explores the payment metric and logs

The prompt says what the agent must find but gives no PromQL or LogQL:

Via Grafana MCP, read-only. The "payment" service of the OTel Demo, last 1 h. The shop runs
in namespace "otel"; the userN namespaces are copies from another lab, skip them.

1. Prometheus, uid "prometheus" (Tempo span metrics, samples every 15 s, labels service_name,
   span_kind, status_code, k8s_namespace_name): requests per second to payment broken down
   by status_code, and the error ratio. Count only server spans. Report the PromQL, stepSeconds,
   the rate() window, the number of series and the ratio.
2. Loki, uid "loki": payment log lines about failed payments. If the service has several
   streams with the same records, pick one and justify it. Report the LogQL, the number of lines
   returned and three example lines.
3. generate_deeplink: an Explore link for each of the two queries (with the query in the queries parameter), with the same time range.
   State exactly what each link restores: datasource, range, query.

Don't save anything.

Approve tool calls one at a time and read the arguments. For query_prometheus note stepSeconds, startTime, endTime and the rate() window — you compare them in step 6.

Check that: the agent called query_prometheus, query_loki_logs and generate_deeplink twice, and the answer gives queries, numbers and two links. Compare with the table:

Where Correct Typical agent deviation
PromQL: matchers service_name="payment", k8s_namespace_name="otel", span_kind="SPAN_KIND_SERVER" in numerator and denominator no span_kind, ratio too low: internal charge spans end without an error and dilute the denominator; no k8s_namespace_name: the ratio mixes 18 copies of the shop
PromQL: breakdown sum by (status_code), two series: STATUS_CODE_UNSET and STATUS_CODE_ERROR no aggregation (series per pod and operation) or STATUS_CODE_OK in the answer, which doesn’t exist in the data
ratio 0.35–0.65 at the 50% variant, ~0.1 at baseline (over the whole hour; the shop makes ~3 payments a minute, so a single 5m window is noisy) one instant sample presented as the hourly average; ratio computed from the number of log lines
stepSeconds and rate() window stated explicitly in the answer “default”, no value
LogQL: selector service_name="payment", namespace="otel", stream="otel" no stream: duplicated records and stack: 'Error: 
' + fragments from stdout
number of lines the agent says query_loki_logs returns 10 lines by default and at most 100 (limit), and reports resultsTruncated / totalLinesScanned from the response metadata “there were 10 errors in the hour” or “100 errors”
links the link opens Explore with the agent’s query (generate_deeplink with queries; with shorten: true as /goto/
) link without queries: Explore opens with an empty editor; link range different from the query range

Note the differences. If the agent dropped span_kind, the namespace or stream, correct it in one sentence and have it rerun the query.

  1. Open the agent’s Prometheus link. If the editor is empty, the agent didn’t pass queries: paste the agent’s PromQL. Run query.
  2. Split → in the right pane datasource Loki, the agent’s LogQL.
  3. The chain icon next to the time picker (Synced times) is highlighted: Split copies the range and syncs time right away. Don’t click it, or you unsync the panes. Drag-select a few minutes of the left graph where the STATUS_CODE_ERROR series is above zero.
  4. Share in the top right (in split view just the icon) copies a short link (Copy shortened URL). Open the link in a new tab.
  5. Close the right pane (Close). In the left one change the datasource from Prometheus to Loki.

Check that:

  • the right pane narrows to the selected window and shows payment logs about failed payments,
  • the short link has the form /goto/<uid> and restores both split panes with the same range, while the agent’s links restore only one pane,
  • after the datasource change the Loki editor has the selector {service_name="payment"}: span_kind and k8s_namespace_name drop out because they don’t exist in Loki.

Step 5 — the agent imports the “narrow vs wide panel” dashboard

Read the file pliki/08-explore-vs-panel.json and save it through update_dashboard in the folder
with uid "uczestnicy-<login>". Don't add uid or id, don't change queries, titles or gridPos.
Version message: "agent: import 08". Report the UID assigned by Grafana.

Check that: update_dashboard has folderUid = uczestnicy-<login>, and the dashboard 08 — Panel wąski vs szeroki opens in Uczestnicy/<login>. Panel Wąski (narrow) is 6 columns wide, Szeroki (wide) 24, both with the same query with [$__interval]. If the agent added a uid, check that it didn’t overwrite someone else’s dashboard (Edit → Dashboard options (gear icon in the right bar) → View all settings → Versions has one version).

Step 6 — measuring: panel, Explore and agent

  1. On panel Wąski: Inspect → Query → Refresh. Note queries[0].maxDataPoints, queries[0].intervalMs (request) and Step from frames[0].schema.meta.executedQueryString (response).
  2. Panel menu of Wąski → Explore (no split view). Query inspector → Query → Refresh. Note the same three values.
  3. Ask the agent:
Get the query of panel "Wąski" from the dashboard you just saved (get_dashboard_panel_queries),
and run it through query_prometheus for the last 24 h. Report stepSeconds, the window you substituted
for $__interval, and the series maximum. Don't save anything.
Where maxDataPoints intervalMs Step / stepSeconds rate() window Maximum
Panel Wąski          
Explore from the panel menu          
agent’s query_prometheus — —      

Check that: the panel has a larger step than Explore (typically panel 300000 and 5m0s, Explore 60000 and 1m0s; the values depend on screen width). The Explore editor shows [5m]: when opening from the panel menu Grafana substitutes $__interval with the panel’s value, so the window is the same and only the step differs. The agent has a third set of values that it chose itself. Correct: the agent says $__interval is a Grafana macro and states which window it used in its place. Typical deviation: it sends $__interval literally and gets a parse error, or compares its maximum with the panel without knowing the panel’s step.

Step 7 — aligning the interval

  1. Ask the agent for a patch:
Fix panel "Wąski" on this dashboard through update_dashboard with uid and a list of operations:
panel Min interval 5m, in the query the window [5m] instead of [$__interval]. Don't change panel "Szeroki".
Before saving, check the paths with get_dashboard_property. Version message: "agent: min interval 5m".
Then run this query through query_prometheus for 24 h with a 300 s step and report the maximum.
  1. In Explore from step 6: Options under the query → Min step = 5m (the [5m] window is already there) → Run query.
  2. Repeat the step 6 measurement in the panel and in Explore.

Check that:

  • the panel and Explore both have Step = 5m0s, and the three maxima (panel, Explore, agent) are close. In Explore intervalMs may stay 60000: Min step is applied by the datasource backend. Step decides the result.
  • the Panel JSON of panel Wąski has "interval": "5m" at panel level, and targets[0].expr has [5m]. Typical deviation: interval written into targets[0], or panel Szeroki changed too.

Step 8 — the agent saves LogQL as a Logs panel

Add to this dashboard a panel of type Logs titled "payment logs" with the LogQL query from step 3
(datasource uid "loki"). Use update_dashboard with uid and operations, appending the panel at the end
of the panels array, below the existing panels. Before saving, run the query through query_loki_logs
and report the number of lines. Version message: "agent: logs panel".

Check that:

  • the dashboard has a third panel of type Logs with payment log lines, and the Panel JSON has "type": "logs", datasource {"type": "loki", "uid": "loki"} in the panel and in targets[0], and an expr with stream="otel",
  • version history (Edit → Dashboard options → View all settings → Versions): author mcp-<login>, every version with the message from the prompts. The agent’s list_dashboard_versions shows Initial save for the first version because it reads the old versions API; the Versions list in the UI shows the right message. The diff of the last version covers only the new panel.

Typical deviation: gridPos overlapping an existing panel, "type": "logs-panel", or a panel without datasource in targets.

Step 9 — cleanup

If you do the Stretch, do it before cleanup. Delete the dashboard:

for u in $(curl -sS "${H[@]}" "$GRAFANA_URL/api/search?type=dash-db&folderUIDs=uczestnicy-$LOGIN&query=szeroki" | jq -j '.[] | .uid, " "'); do
  curl -sS "${H[@]}" -X DELETE "$GRAFANA_URL/api/dashboards/uid/$u" | jq -r .message
done

If you also work with the agent in exercise 07b, keep the service account and the MCP server (the token expires after 8 h). If not:

claude mcp remove --scope local grafana
curl -sS "${H[@]}" -X DELETE "$GRAFANA_URL/api/serviceaccounts/$SA" | jq -c .

Check that: the loop prints Dashboard 08 — Panel wąski vs szeroki deleted, and the account deletion Service account deleted.

Success criteria

  • The short link /goto/<uid> restores the metric + logs split view with synced time; the agent’s links open one Explore pane.
  • The step 6 table has three rows: the panel with a larger step than Explore (typically 300000 / 5m0s vs 60000 / 1m0s) and the same window (the panel’s substituted $__interval, typically [5m]), the agent with its own stepSeconds and window.
  • After step 7 the panel and Explore both have Step = 5m0s, and the agent’s maximum at a 300 s step is close.
  • The dashboard 08 — Panel wąski vs szeroki in Uczestnicy/<login> has a Logs panel with stream="otel", and the versions saved by the agent have author mcp-<login>.
  • You have a list of differences between the agent’s queries and the step 3 table.

⭐Stretch: the agent explains a discrepancy it can’t see (failure mode)

  1. Explanation without measurement. In a new session, before step 7, ask: Why does panel "Wąski" on the dashboard 08 — Panel wąski vs szeroki show a different maximum than Explore? Report the panel's intervalMs and maxDataPoints. The agent has the panel’s queries but doesn’t know the graph width in the browser. Correct: it says it can’t measure these values and points to Query inspector. Deviation: it reports a specific intervalMs it couldn’t have measured. Compare with the step 6 table.
  2. A split view URL assembled by hand. Ask the agent for an Explore link with split view (Loki with Invalid token on the left, the PromQL from step 3 on the right) built as the panes parameter (schemaVersion=1, keys a and b), without generate_deeplink. Open the link. Check that the JSON is encoded correctly and that the datasources have uid loki and prometheus. Compare with the Stretch of the manual variant.

results matching ""

    No results matching ""