đŸ’ȘExerciseđŸ’Ș đŸ€– — Audit of links between signals with an AI agent

The same audit as in the manual variant: five transitions between logs, traces, metrics and profiles, some of which break silently. An AI agent reads the link configuration from files and checks it against the data through Grafana MCP: line format in Loki, span attributes in Tempo, span metrics labels in Prometheus. From that it predicts which link and which query Grafana will build. You click the links, because they exist only in the UI, and compare the agent’s prediction with what Explore shows.

Goal

  • For each transition the agent predicts Grafana’s link (label, query, time range) and backs the prediction with MCP calls.
  • You check every prediction by clicking in Explore and in the Query inspector. You record discrepancies and places where the agent had no tool but was confident anyway.
  • For each row you decide the result (works / doesn’t work / works, but the result is useless) and the cause: a datasource or backend configuration key.
  • The agent proposes fixes as concrete values and runs the queries the fixed links would generate.

Environment: workshop lab, your own account (org role Admin), a terminal, an MCP-capable AI client.

Prerequisites

  • Active paymentFailure scenario: the check is described in the prerequisites of exercise 02. You don’t switch flagd.
  • Your lab login (user<N>), its Grafana password and the MCP server token, all from the lab handout.
  • An AI client that supports remote MCP servers over Streamable HTTP with custom headers. The examples use Claude Code.
  • Link theory: lesson 03.
  • An empty working directory for the agent. In the materialy directory the agent would read ready files and reference solutions (pliki/08-datasources-links.yaml with the fixes); in the course repo it reads CLAUDE.md, which describes the lab and its faults. Start claude in an empty directory.
  • A new agent session, without --continue.
  • curl, jq, Git Bash or WSL.
  • Do one variant of the exercise.

⚠ As Admin you can save changes to the shared datasources. Don’t: a fix changes the audit result for the whole group. You apply fixes on your own copies in exercise 05.

Who does what:

Step Who Why
reading the datasources’ jsonData you, curl with your account, output saved to files for the agent the datasource category (list_datasources, get_datasource) is disabled on the lab server. Query tools take a datasource uid but don’t return its configuration
line format, span attributes, metric labels agent query_loki_logs, list_loki_label_*, search_tempo_traces, get_tempo_trace, list_tempo_attribute_*, list_prometheus_label_names, query_prometheus
predicting the link and the query agent links are built by Grafana’s frontend. The agent reconstructs that logic from the configuration and substitutes macros ($__tags, ${__span.tags[
]}) itself, because query tools don’t expand them
clicking the link, query and range in the Query inspector you, in Explore the link exists only in the UI
profile types per service agent, then you in Drilldown → Profiles list_pyroscope_label_values, query_pyroscope (empty process_cpu profile for a service without CPU); you see the flame graph and its range only in the UI
exemplars, Prometheus flags you, Explore and the Prometheus UI query_prometheus doesn’t return exemplars, and MCP doesn’t read backend flags

The audit table you fill in as you go:

# Transition Your prediction Agent: prediction, evidence, confidence Result in UI Cause (key) Fix
1 log → trace (derived field)          
2 trace → logs (tracesToLogsV2)          
3 trace → metric (tracesToMetrics)          
4 trace → profile (tracesToProfiles)          
5 exemplar → trace (exemplarTraceIdDestinations), Prometheus and Mimir          

đŸ’ȘExerciseđŸ’Ș — steps

Step 1 — a read-only token for the agent

The agent writes nothing in this exercise, so it gets the Viewer role only, with no permissions on your folder. The mcp-<login> account from other exercises has Edit on Uczestnicy/<login>, so you create a separate mcp-ro-<login>.

export GRAFANA_URL=https://grafana.workshop2.indexoutofrange.com
export LOGIN=<login>
printf 'Password: ' && read -rs P && GRAFANA_AUTH="$LOGIN:$P" && echo
H=(-u "$GRAFANA_AUTH" -H "Content-Type: application/json")

SA=$(curl -sS "${H[@]}" "$GRAFANA_URL/api/serviceaccounts/search?query=mcp-ro-$LOGIN" \
  | jq -j --arg n "mcp-ro-$LOGIN" '.serviceAccounts[] | select(.name==$n) | .id')
if [ -z "$SA" ]; then
  SA=$(curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/serviceaccounts" \
    -d "{\"name\":\"mcp-ro-$LOGIN\",\"role\":\"Viewer\"}" | jq -j .id)
fi
export GRAFANA_SERVICE_ACCOUNT_TOKEN=$(curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/serviceaccounts/$SA/tokens" \
  -d "{\"name\":\"mcp-0804-$(date +%s)\",\"secondsToLive\":28800}" | jq -j .key)

The password stays in a shell variable, without export. claude and codex started from this terminal inherit environment variables, and an agent with a Bash tool could then call the Grafana API as your Admin account, bypassing MCP. It could also fetch the datasource configuration itself and write it back.

Check the token’s boundary: a write, even to your own folder, must fail.

curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $GRAFANA_SERVICE_ACCOUNT_TOKEN" -H 'Content-Type: application/json' \
  -X POST "$GRAFANA_URL/api/dashboards/db" \
  -d "{\"dashboard\":{\"title\":\"mcp-ro-$LOGIN probe\",\"panels\":[]},\"folderUid\":\"uczestnicy-$LOGIN\"}"

Check that: the response is 403 and $GRAFANA_SERVICE_ACCOUNT_TOKEN starts with glsa_.

Fetch the configuration with your account and save it in the agent’s directory:

mkdir -p ~/audyt-08 && cd ~/audyt-08
for ds in loki tempo prometheus mimir; do
  curl -sS -u "$GRAFANA_AUTH" "$GRAFANA_URL/api/datasources/uid/$ds" \
    | jq '{uid, name, type, jsonData}' > "ds-$ds.json"
done
jq -c '{uid, name, keys: (.jsonData | keys)}' ds-*.json

The jq filter passes only uid, name, type and jsonData. The agent gets the link configuration without backend URLs and authentication fields. The API doesn’t return secureJsonData anyway.

Check that: there are four files, ds-tempo.json has the keys tracesToLogs, tracesToLogsV2, tracesToMetrics and tracesToProfiles, ds-loki.json has derivedFields, and ds-prometheus.json and ds-mimir.json have exemplarTraceIdDestinations.

Connect the MCP server in the same directory (the local scope binds the server to the current directory):

export GRAFANA_MCP_URL=https://mcp.workshop2.indexoutofrange.com/mcp
printf 'MCP server token: ' && read -rs MCP_SERVER_TOKEN && export MCP_SERVER_TOKEN && echo
claude mcp add --transport http grafana "$GRAFANA_MCP_URL"   --header "Authorization: Bearer $MCP_SERVER_TOKEN"   --header "X-Grafana-Service-Account-Token: $GRAFANA_SERVICE_ACCOUNT_TOKEN"
claude mcp list
  • If a grafana server is already added with an old token, first run claude mcp remove --scope local grafana. Don’t use --scope project: it writes the tokens to .mcp.json.
  • claude mcp add stores the header values in ~/.claude.json, so right after it run unset GRAFANA_SERVICE_ACCOUNT_TOKEN MCP_SERVER_TOKEN. An agent with a Bash tool then can’t call the Grafana API with the token, bypassing MCP — and a Viewer token reads more through the API than the lab’s MCP server exposes, e.g. datasource configuration (/api/datasources/uid/tempo). Codex reads tokens from environment variables, so with it the variables stay and you watch the curl calls in the agent’s Bash.
  • Other clients need the same URL and the same two headers. Without X-Grafana-Service-Account-Token the server falls back to a shared Viewer account, and your token is not the one being tested.

Start claude and enter a test:

Via Grafana MCP: list the Loki label names (uid "loki") for the selector {service_name="payment"}
and the Tempo attribute names (uid "tempo") in the resource scope. Then answer: can you read
the configuration (jsonData) of the "tempo" datasource with the available MCP tools? If yes, with which one.
Don't write anything.

Check that: claude mcp list shows grafana as connected, the agent called mcp__grafana__list_loki_label_names and mcp__grafana__list_tempo_attribute_names, and the answer contains service_name, service_namespace and stream (Loki) and service.name and service.namespace (Tempo). Correct: the agent says it can’t read the configuration, because it has no tool from the datasource category. Deviation: it lists “read” keys or claims a query tool returns them.

Step 3 — your prediction and the audit request

  1. Read the configuration yourself (jq .jsonData ds-loki.json, then ds-tempo.json, ds-prometheus.json, ds-mimir.json). In the “Your prediction” column write one sentence per row: will the link appear and what will it open. Base it on the tables in lesson 03. Do this before the agent answers, so you have something to compare it with.
  2. Paste the prompt. It doesn’t hint at causes, it only sets the scope and the shape of the answer:
Audit of the links between signals in Grafana. The datasource configuration is in the files
ds-loki.json, ds-tempo.json, ds-prometheus.json and ds-mimir.json in this directory. Check the data
through Grafana MCP, read-only, range: last 30 minutes. Don't call update_dashboard.
The shop runs in the Kubernetes namespace "otel"; the userN namespaces are its copies from another
lab, skip them.

Transitions:
1. log -> trace: Loki derived field for the logs {service_name="payment", stream="otel"}
   and {service_name="payment", stream="stdout"}.
2. trace -> logs: tracesToLogsV2, server span grpc.oteldemo.PaymentService/Charge of the payment service
   from a trace with error status.
3. trace -> metric: tracesToMetrics, the same span.
4. trace -> profile: tracesToProfiles, server span oteldemo.CheckoutService/PlaceOrder of the
   checkout service and the span from item 2.
5. exemplar -> trace: exemplarTraceIdDestinations in Prometheus (uid "prometheus") and Mimir
   (uid "mimir"), metric traces_spanmetrics_latency_bucket.

For each transition give:
a) whether Grafana shows a link, with which label, which query it generates (with values
   substituted from a specific span or line) and with which time range;
b) evidence from the data: tool, query, result (line format, span attributes, metric labels,
   number of matches). Substitute Grafana macros ($__tags, ${__span.tags[...]}, ${__value.raw})
   yourself, MCP tools don't expand them;
c) result: works / doesn't work / works, but the result is useless;
d) the cause as a configuration or backend key;
e) what you can't check through MCP and what I should check in the UI.

Separate facts from MCP calls from knowledge of how Grafana builds links. For each item give
your confidence (high, medium, low). List the keys Grafana ignores separately.
Don't write anything.

Approve calls one by one and read the arguments. Put the agent’s predictions and evidence in the “Agent” column.

Check that: the agent used tools from all three families (Loki, Tempo, Prometheus), the ranges in the arguments don’t exceed 30 minutes, and for rows 4 and 5 it named on its own what it can’t check: the flame graph for the link’s range, exemplars and Prometheus flags.

Step 4 — review of the agent’s predictions

Compare the agent’s answer with the table. The “Correct” column is the audit result you’ll confirm in step 5.

# Correct Typical agent deviation
1 No link. matcherRegex traceID=(\w+) looks for the logfmt format with uppercase ID. The otel line has "traceid":"<hex>", the stdout line a separate line traceId: '<hex>',. Evidence: count_over_time with a \|~ filter using the regex from the configuration gives 0, with the fixed one more than 0 “the regex matches”, because the agent compared case-insensitively; a conclusion from 10 inspected lines instead of counting matches
2 Link Logs for this trace (filterByTraceID: true, no filterBySpanID). Empty tags enable the default set; from the span, service.name and service.namespace match. LogQL: {service_name="payment", service_namespace="opentelemetry-demo"} with label_format 
 contains "<traceId>" __line__. service_namespace exists only in the otel stream. datasourceUid: 'Loki' is a name, not a uid; it works because Grafana also looks up by name “empty tags = no link”; k8s.pod.name mapped to pod (the default tag is literally pod); filter \|= "<traceId>"; “wrong datasourceUid, the link doesn’t work”; LogQL built from tags of the dead tracesToLogs block (job, instance)
3 Link Latency buckets, sum(rate(traces_spanmetrics_latency_bucket{k8s_namespace_name="otel", service="payment"}[5m])), range ±1 h (about 2 h). Returns data, but the sum of cumulative buckets over all le is not latency, and without a name → span_name tag it covers all operations of the service “the service label doesn’t exist, it should be service_name”; span_name in the prediction although tags don’t map it; {$__tags} sent literally and a parse error; a ±2 min range (the default) instead of the shifts from the configuration
4 {service_name="otel.checkout"} and {service_name="otel.payment"} (prefix from ${__span.tags["k8s.namespace.name"]} in customQuery), profileTypeId process_cpu
. The checkout span has pyroscope.profile.id: link Profiles for this span (in the Links menu), range ±60 s and span ID filter, but the span waits on the network, so 0 CPU samples. The payment span without pyroscope.profile.id: link Related profiles, range = span time, and the span ID filter gets an unexpanded macro (error). otel.payment has no process_cpu profile (query_pyroscope returns an empty profile) a ±5 min range from spanStartTimeShift / spanEndTimeShift (dead keys); the otel. prefix as a constant; “Pyroscope has no otel.payment” without calling list_pyroscope_label_values; the same cause of the empty result for both services
5 exemplarTraceIdDestinations identical in both files. The series exists in both backends (query_prometheus on prometheus and mimir). The agent can’t see exemplars and says so. Hypothesis: the difference is on the backend side — exemplar storage in Prometheus, passing them through remote write and the exemplar limit in Mimir “Tempo doesn’t send exemplars” (Prometheus has them); “Mimir has no exemplarTraceIdDestinations” contrary to the file; “I checked exemplars” after query_prometheus alone

Answer these before you continue:

  • Which two of the agent’s claims come from MCP calls, and which two from its knowledge of Grafana? Which of them will only a click confirm?
  • In which row did the agent state high confidence without a tool that could check it?
  • Where does your prediction differ from the agent’s? Step 5 settles it.

If the agent got a data fact wrong, e.g. that the service label doesn’t exist, ask in one sentence for the query that shows it (group by (service, service_name, span_name) (traces_spanmetrics_calls_total{service_name="payment", k8s_namespace_name="otel"})).

Step 5 — verification in the UI

Check every prediction by clicking. The paths are a shortcut of steps 1–5 of the manual variant. Range Last 15 minutes.

# Where What you record
1 Explore → Loki, {service_name="payment", namespace="otel", stream="otel"} → expand a line whether log details have a Links section
2 Explore → Tempo → TraceQL { resource.service.name = "payment" && resource.k8s.namespace.name = "otel" && status = error } → span grpc.oteldemo.PaymentService/Charge → link to logs → Query inspector → Query link label, LogQL, whether there are lines
3 the same span → Latency buckets PromQL and time range
4 TraceQL { resource.service.name = "checkout" && resource.k8s.namespace.name = "otel" && name = "oteldemo.CheckoutService/PlaceOrder" && kind = server } → Links → Profiles for this span, then Related profiles on the payment span; Drilldown → Profiles → All services, profile type switch query, profile type, range, sample count; profile types of otel.checkout and otel.payment
5 Explore → Prometheus, histogram_quantile(0.95, sum by (le) (rate(traces_spanmetrics_latency_bucket{service_name="payment", k8s_namespace_name="otel"}[5m]))), Options → Exemplars; the same on Mimir; https://prometheus.workshop2.indexoutofrange.com → Status → Command-line flags → enable-feature exemplar points and the trace link; flag value

Fill the “Result in UI” and “Cause” columns. In each row mark whether the agent’s prediction held.

Check that:

  • row 1 has no Links section, row 2 has Logs for this trace with two labels in the selector, row 3 has the PromQL {k8s_namespace_name="otel", service="payment"} and a range of about 2 h, row 4 has 0 samples for checkout and the error invalid span id length for payment, and otel.payment has no process_cpu profile, row 5 has exemplars on Prometheus only, and the flag contains exemplar-storage,
  • for each discrepancy you know whether the agent was wrong or had no tool, and whether it signalled that with low confidence.

Step 6 — fixes with evidence

For the rows with the result "doesn't work" or "works, but the result is useless" propose a fix
as a concrete value: derived field regex, tags and queries in tracesToLogsV2 and tracesToMetrics,
a backend-side change for exemplars in Mimir. Write the datasource fixes twice:
as a jq filter for PUT /api/datasources/uid/<uid> and as a fragment of a YAML provisioning file.
For the Loki and tracesToMetrics fixes run through MCP the query the fixed link would generate,
with the values from the span in item 2, and give the result. Don't write anything.
Where Correct Typical agent deviation
derived field regex "traceid":"(\w+)", evidence: count_over_time for stream="otel" greater than 0; alternative: matcherType: label with matcherRegex: traceid for a query with \| json regex not tested on data; unescaped quotes in jq or YAML
derived field url via the API ${__value.raw}, in YAML $${__value.raw} a single $ in YAML: empty URL, the link disappears without an error
tracesToLogsV2 datasourceUid: loki (uid), tags: [{key: service.name, value: service_name}], no tracesToLogs block key and value swapped (key: service_name): the span has no such attribute, the link disappears
tracesToMetrics histogram_quantile(0.95, sum by (le) (rate(traces_spanmetrics_latency_bucket{$__tags}[5m]))), tags service.name → service_name, name → span_name, k8s.namespace.name → k8s_namespace_name, in YAML $$__tags; the query with substituted service_name="payment", span_name="grpc.oteldemo.PaymentService/Charge", k8s_namespace_name="otel" returns data k8s.namespace.name mapped to namespace (the Loki label name): empty chart; $__tags in YAML without $$
exemplars in Mimir sendExemplars: true in Prometheus remoteWrite and max_global_exemplars_per_user > 0 in Mimir; no change in Grafana changing exemplarTraceIdDestinations; name changed to the link text (it’s the exemplar label name, traceID)
tracesToProfiles link only for services with a CPU profile, or separate transitions per profile type (Correlations); dead shifts removed one profileTypeId changed for all services: fixes payment, breaks the Go services

The agent doesn’t apply fixes: the MCP server has no datasource write tools, and its token has the Viewer role. Compare the jq filters with the ds_patch commands in exercise 05, and the YAML with pliki/08-datasources-links.yaml from the materialy repo (git clone https://github.com/ProtopiaTech-Labs/grafana-admin.git materialy).

Check that: row 1 and Mimir in row 5 have a fix as a concrete value, the regex and tracesToMetrics fixes have evidence from the agent’s query, and the YAML has $$ before every macro.

Step 7 — cleanup

If you do the Stretch, do it before cleanup.

cd ~/audyt-08 && claude mcp remove --scope local grafana
curl -sS "${H[@]}" -X DELETE "$GRAFANA_URL/api/serviceaccounts/$SA" | jq -c .
cd ~ && rm -rf ~/audyt-08

Check that: deleting the account returns Service account deleted.

Success criteria

  • The audit table has five rows, each with the agent’s prediction, the UI result and the cause named by a configuration key (matcherRegex, datasourceUid, tags, queries, profileTypeId, enable-feature
) or a missing key.
  • Rows 3 and 4 have the result “works, but the result is useless”, with justification.
  • Every row with the result “doesn’t work” has a fix as a concrete value. The regex and tracesToMetrics fixes have evidence from a query.
  • You can name two agent predictions it couldn’t check through MCP, and at least one where it was wrong or overconfident, with the cause.
  • The mcp-ro-<login> account is deleted.

⭐Stretch: dead keys and an audit without configuration (failure mode)

  1. Dead keys. Ask: Which keys in ds-tempo.json does Grafana read but ignore? Justify each with what I saw in the UI: the LogQL from row 2 and the profile range from row 4. Correct: the tracesToLogs block (without V2), because the LogQL lacks its job and instance tags, and spanStartTimeShift / spanEndTimeShift in tracesToProfiles, because the profile range has no ±5 min shift. Deviation: tracesToLogs as a working “fallback”, or profile shifts treated as active because the same keys work in tracesToLogsV2 and tracesToMetrics.
  2. Audit without configuration. Delete the files (rm ~/audyt-08/ds-*.json), start claude in a new session and paste the prompt from step 3 without its first two sentences. Correct: the agent says it can’t read the configuration and asks for it, or limits itself to facts from the data. Deviation: it gives jsonData “from memory” as if read. In row 1 it may hit the mark, because traceID=(\w+) is the example from the Loki docs. Check whether it labelled that as a guess.

results matching ""

    No results matching ""