đȘExerciseđȘ đ€ â Audit of links between signals with an AI agent
The same audit as in the manual variant: five transitions between logs, traces, metrics and profiles, some of which break silently. An AI agent reads the link configuration from files and checks it against the data through Grafana MCP: line format in Loki, span attributes in Tempo, span metrics labels in Prometheus. From that it predicts which link and which query Grafana will build. You click the links, because they exist only in the UI, and compare the agentâs prediction with what Explore shows.
Goal
- For each transition the agent predicts Grafanaâs link (label, query, time range) and backs the prediction with MCP calls.
- You check every prediction by clicking in Explore and in the Query inspector. You record discrepancies and places where the agent had no tool but was confident anyway.
- For each row you decide the result (works / doesnât work / works, but the result is useless) and the cause: a datasource or backend configuration key.
- The agent proposes fixes as concrete values and runs the queries the fixed links would generate.
Environment: workshop lab, your own account (org role Admin), a terminal, an MCP-capable AI client.
Prerequisites
- Active
paymentFailurescenario: the check is described in the prerequisites of exercise 02. You donât switch flagd. - Your lab login (
user<N>), its Grafana password and the MCP server token, all from the lab handout. - An AI client that supports remote MCP servers over Streamable HTTP with custom headers. The examples use Claude Code.
- Link theory: lesson 03.
- An empty working directory for the agent. In the
materialydirectory the agent would read ready files and reference solutions (pliki/08-datasources-links.yamlwith the fixes); in the course repo it readsCLAUDE.md, which describes the lab and its faults. Startclaudein an empty directory. - A new agent session, without
--continue. curl,jq, Git Bash or WSL.- Do one variant of the exercise.
â ïž As Admin you can save changes to the shared datasources. Donât: a fix changes the audit result for the whole group. You apply fixes on your own copies in exercise 05.
Who does what:
| Step | Who | Why |
|---|---|---|
reading the datasourcesâ jsonData |
you, curl with your account, output saved to files for the agent |
the datasource category (list_datasources, get_datasource) is disabled on the lab server. Query tools take a datasource uid but donât return its configuration |
| line format, span attributes, metric labels | agent | query_loki_logs, list_loki_label_*, search_tempo_traces, get_tempo_trace, list_tempo_attribute_*, list_prometheus_label_names, query_prometheus |
| predicting the link and the query | agent | links are built by Grafanaâs frontend. The agent reconstructs that logic from the configuration and substitutes macros ($__tags, ${__span.tags[âŠ]}) itself, because query tools donât expand them |
| clicking the link, query and range in the Query inspector | you, in Explore | the link exists only in the UI |
| profile types per service | agent, then you in Drilldown â Profiles | list_pyroscope_label_values, query_pyroscope (empty process_cpu profile for a service without CPU); you see the flame graph and its range only in the UI |
| exemplars, Prometheus flags | you, Explore and the Prometheus UI | query_prometheus doesnât return exemplars, and MCP doesnât read backend flags |
The audit table you fill in as you go:
| # | Transition | Your prediction | Agent: prediction, evidence, confidence | Result in UI | Cause (key) | Fix |
|---|---|---|---|---|---|---|
| 1 | log â trace (derived field) | Â | Â | Â | Â | Â |
| 2 | trace â logs (tracesToLogsV2) |
 |  |  |  |  |
| 3 | trace â metric (tracesToMetrics) |
 |  |  |  |  |
| 4 | trace â profile (tracesToProfiles) |
 |  |  |  |  |
| 5 | exemplar â trace (exemplarTraceIdDestinations), Prometheus and Mimir |
 |  |  |  |  |
đȘExerciseđȘ â steps
Step 1 â a read-only token for the agent
The agent writes nothing in this exercise, so it gets the Viewer role only, with no permissions on your folder. The mcp-<login> account from other exercises has Edit on Uczestnicy/<login>, so you create a separate mcp-ro-<login>.
export GRAFANA_URL=https://grafana.workshop2.indexoutofrange.com
export LOGIN=<login>
printf 'Password: ' && read -rs P && GRAFANA_AUTH="$LOGIN:$P" && echo
H=(-u "$GRAFANA_AUTH" -H "Content-Type: application/json")
SA=$(curl -sS "${H[@]}" "$GRAFANA_URL/api/serviceaccounts/search?query=mcp-ro-$LOGIN" \
| jq -j --arg n "mcp-ro-$LOGIN" '.serviceAccounts[] | select(.name==$n) | .id')
if [ -z "$SA" ]; then
SA=$(curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/serviceaccounts" \
-d "{\"name\":\"mcp-ro-$LOGIN\",\"role\":\"Viewer\"}" | jq -j .id)
fi
export GRAFANA_SERVICE_ACCOUNT_TOKEN=$(curl -sS "${H[@]}" -X POST "$GRAFANA_URL/api/serviceaccounts/$SA/tokens" \
-d "{\"name\":\"mcp-0804-$(date +%s)\",\"secondsToLive\":28800}" | jq -j .key)
The password stays in a shell variable, without export. claude and codex started from this terminal inherit environment variables, and an agent with a Bash tool could then call the Grafana API as your Admin account, bypassing MCP. It could also fetch the datasource configuration itself and write it back.
Check the tokenâs boundary: a write, even to your own folder, must fail.
curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $GRAFANA_SERVICE_ACCOUNT_TOKEN" -H 'Content-Type: application/json' \
-X POST "$GRAFANA_URL/api/dashboards/db" \
-d "{\"dashboard\":{\"title\":\"mcp-ro-$LOGIN probe\",\"panels\":[]},\"folderUid\":\"uczestnicy-$LOGIN\"}"
Check that: the response is 403 and $GRAFANA_SERVICE_ACCOUNT_TOKEN starts with glsa_.
Step 2 â link configuration for the agent and MCP connection
Fetch the configuration with your account and save it in the agentâs directory:
mkdir -p ~/audyt-08 && cd ~/audyt-08
for ds in loki tempo prometheus mimir; do
curl -sS -u "$GRAFANA_AUTH" "$GRAFANA_URL/api/datasources/uid/$ds" \
| jq '{uid, name, type, jsonData}' > "ds-$ds.json"
done
jq -c '{uid, name, keys: (.jsonData | keys)}' ds-*.json
The jq filter passes only uid, name, type and jsonData. The agent gets the link configuration without backend URLs and authentication fields. The API doesnât return secureJsonData anyway.
Check that: there are four files, ds-tempo.json has the keys tracesToLogs, tracesToLogsV2, tracesToMetrics and tracesToProfiles, ds-loki.json has derivedFields, and ds-prometheus.json and ds-mimir.json have exemplarTraceIdDestinations.
Connect the MCP server in the same directory (the local scope binds the server to the current directory):
export GRAFANA_MCP_URL=https://mcp.workshop2.indexoutofrange.com/mcp
printf 'MCP server token: ' && read -rs MCP_SERVER_TOKEN && export MCP_SERVER_TOKEN && echo
claude mcp add --transport http grafana "$GRAFANA_MCP_URL" --header "Authorization: Bearer $MCP_SERVER_TOKEN" --header "X-Grafana-Service-Account-Token: $GRAFANA_SERVICE_ACCOUNT_TOKEN"
claude mcp list
- If a
grafanaserver is already added with an old token, first runclaude mcp remove --scope local grafana. Donât use--scope project: it writes the tokens to.mcp.json. claude mcp addstores the header values in~/.claude.json, so right after it rununset GRAFANA_SERVICE_ACCOUNT_TOKEN MCP_SERVER_TOKEN. An agent with a Bash tool then canât call the Grafana API with the token, bypassing MCP â and a Viewer token reads more through the API than the labâs MCP server exposes, e.g. datasource configuration (/api/datasources/uid/tempo). Codex reads tokens from environment variables, so with it the variables stay and you watch thecurlcalls in the agentâs Bash.- Other clients need the same URL and the same two headers. Without
X-Grafana-Service-Account-Tokenthe server falls back to a shared Viewer account, and your token is not the one being tested.
Start claude and enter a test:
Via Grafana MCP: list the Loki label names (uid "loki") for the selector {service_name="payment"}
and the Tempo attribute names (uid "tempo") in the resource scope. Then answer: can you read
the configuration (jsonData) of the "tempo" datasource with the available MCP tools? If yes, with which one.
Don't write anything.
Check that: claude mcp list shows grafana as connected, the agent called mcp__grafana__list_loki_label_names and mcp__grafana__list_tempo_attribute_names, and the answer contains service_name, service_namespace and stream (Loki) and service.name and service.namespace (Tempo). Correct: the agent says it canât read the configuration, because it has no tool from the datasource category. Deviation: it lists âreadâ keys or claims a query tool returns them.
Step 3 â your prediction and the audit request
- Read the configuration yourself (
jq .jsonData ds-loki.json, thends-tempo.json,ds-prometheus.json,ds-mimir.json). In the âYour predictionâ column write one sentence per row: will the link appear and what will it open. Base it on the tables in lesson 03. Do this before the agent answers, so you have something to compare it with. - Paste the prompt. It doesnât hint at causes, it only sets the scope and the shape of the answer:
Audit of the links between signals in Grafana. The datasource configuration is in the files
ds-loki.json, ds-tempo.json, ds-prometheus.json and ds-mimir.json in this directory. Check the data
through Grafana MCP, read-only, range: last 30 minutes. Don't call update_dashboard.
The shop runs in the Kubernetes namespace "otel"; the userN namespaces are its copies from another
lab, skip them.
Transitions:
1. log -> trace: Loki derived field for the logs {service_name="payment", stream="otel"}
and {service_name="payment", stream="stdout"}.
2. trace -> logs: tracesToLogsV2, server span grpc.oteldemo.PaymentService/Charge of the payment service
from a trace with error status.
3. trace -> metric: tracesToMetrics, the same span.
4. trace -> profile: tracesToProfiles, server span oteldemo.CheckoutService/PlaceOrder of the
checkout service and the span from item 2.
5. exemplar -> trace: exemplarTraceIdDestinations in Prometheus (uid "prometheus") and Mimir
(uid "mimir"), metric traces_spanmetrics_latency_bucket.
For each transition give:
a) whether Grafana shows a link, with which label, which query it generates (with values
substituted from a specific span or line) and with which time range;
b) evidence from the data: tool, query, result (line format, span attributes, metric labels,
number of matches). Substitute Grafana macros ($__tags, ${__span.tags[...]}, ${__value.raw})
yourself, MCP tools don't expand them;
c) result: works / doesn't work / works, but the result is useless;
d) the cause as a configuration or backend key;
e) what you can't check through MCP and what I should check in the UI.
Separate facts from MCP calls from knowledge of how Grafana builds links. For each item give
your confidence (high, medium, low). List the keys Grafana ignores separately.
Don't write anything.
Approve calls one by one and read the arguments. Put the agentâs predictions and evidence in the âAgentâ column.
Check that: the agent used tools from all three families (Loki, Tempo, Prometheus), the ranges in the arguments donât exceed 30 minutes, and for rows 4 and 5 it named on its own what it canât check: the flame graph for the linkâs range, exemplars and Prometheus flags.
Step 4 â review of the agentâs predictions
Compare the agentâs answer with the table. The âCorrectâ column is the audit result youâll confirm in step 5.
| # | Correct | Typical agent deviation |
|---|---|---|
| 1 | No link. matcherRegex traceID=(\w+) looks for the logfmt format with uppercase ID. The otel line has "traceid":"<hex>", the stdout line a separate line traceId: '<hex>',. Evidence: count_over_time with a \|~ filter using the regex from the configuration gives 0, with the fixed one more than 0 |
âthe regex matchesâ, because the agent compared case-insensitively; a conclusion from 10 inspected lines instead of counting matches |
| 2 | Link Logs for this trace (filterByTraceID: true, no filterBySpanID). Empty tags enable the default set; from the span, service.name and service.namespace match. LogQL: {service_name="payment", service_namespace="opentelemetry-demo"} with label_format ⊠contains "<traceId>" __line__. service_namespace exists only in the otel stream. datasourceUid: 'Loki' is a name, not a uid; it works because Grafana also looks up by name |
âempty tags = no linkâ; k8s.pod.name mapped to pod (the default tag is literally pod); filter \|= "<traceId>"; âwrong datasourceUid, the link doesnât workâ; LogQL built from tags of the dead tracesToLogs block (job, instance) |
| 3 | Link Latency buckets, sum(rate(traces_spanmetrics_latency_bucket{k8s_namespace_name="otel", service="payment"}[5m])), range ±1 h (about 2 h). Returns data, but the sum of cumulative buckets over all le is not latency, and without a name â span_name tag it covers all operations of the service |
âthe service label doesnât exist, it should be service_nameâ; span_name in the prediction although tags donât map it; {$__tags} sent literally and a parse error; a ±2 min range (the default) instead of the shifts from the configuration |
| 4 | {service_name="otel.checkout"} and {service_name="otel.payment"} (prefix from ${__span.tags["k8s.namespace.name"]} in customQuery), profileTypeId process_cpuâŠ. The checkout span has pyroscope.profile.id: link Profiles for this span (in the Links menu), range ±60 s and span ID filter, but the span waits on the network, so 0 CPU samples. The payment span without pyroscope.profile.id: link Related profiles, range = span time, and the span ID filter gets an unexpanded macro (error). otel.payment has no process_cpu profile (query_pyroscope returns an empty profile) |
a ±5 min range from spanStartTimeShift / spanEndTimeShift (dead keys); the otel. prefix as a constant; âPyroscope has no otel.paymentâ without calling list_pyroscope_label_values; the same cause of the empty result for both services |
| 5 | exemplarTraceIdDestinations identical in both files. The series exists in both backends (query_prometheus on prometheus and mimir). The agent canât see exemplars and says so. Hypothesis: the difference is on the backend side â exemplar storage in Prometheus, passing them through remote write and the exemplar limit in Mimir |
âTempo doesnât send exemplarsâ (Prometheus has them); âMimir has no exemplarTraceIdDestinationsâ contrary to the file; âI checked exemplarsâ after query_prometheus alone |
Answer these before you continue:
- Which two of the agentâs claims come from MCP calls, and which two from its knowledge of Grafana? Which of them will only a click confirm?
- In which row did the agent state high confidence without a tool that could check it?
- Where does your prediction differ from the agentâs? Step 5 settles it.
If the agent got a data fact wrong, e.g. that the service label doesnât exist, ask in one sentence for the query that shows it (group by (service, service_name, span_name) (traces_spanmetrics_calls_total{service_name="payment", k8s_namespace_name="otel"})).
Step 5 â verification in the UI
Check every prediction by clicking. The paths are a shortcut of steps 1â5 of the manual variant. Range Last 15 minutes.
| # | Where | What you record |
|---|---|---|
| 1 | Explore â Loki, {service_name="payment", namespace="otel", stream="otel"} â expand a line |
whether log details have a Links section |
| 2 | Explore â Tempo â TraceQL { resource.service.name = "payment" && resource.k8s.namespace.name = "otel" && status = error } â span grpc.oteldemo.PaymentService/Charge â link to logs â Query inspector â Query |
link label, LogQL, whether there are lines |
| 3 | the same span â Latency buckets | PromQL and time range |
| 4 | TraceQL { resource.service.name = "checkout" && resource.k8s.namespace.name = "otel" && name = "oteldemo.CheckoutService/PlaceOrder" && kind = server } â Links â Profiles for this span, then Related profiles on the payment span; Drilldown â Profiles â All services, profile type switch |
query, profile type, range, sample count; profile types of otel.checkout and otel.payment |
| 5 | Explore â Prometheus, histogram_quantile(0.95, sum by (le) (rate(traces_spanmetrics_latency_bucket{service_name="payment", k8s_namespace_name="otel"}[5m]))), Options â Exemplars; the same on Mimir; https://prometheus.workshop2.indexoutofrange.com â Status â Command-line flags â enable-feature |
exemplar points and the trace link; flag value |
Fill the âResult in UIâ and âCauseâ columns. In each row mark whether the agentâs prediction held.
Check that:
- row 1 has no Links section, row 2 has Logs for this trace with two labels in the selector, row 3 has the PromQL
{k8s_namespace_name="otel", service="payment"}and a range of about 2 h, row 4 has0 samplesforcheckoutand the errorinvalid span id lengthforpayment, andotel.paymenthas noprocess_cpuprofile, row 5 has exemplars on Prometheus only, and the flag containsexemplar-storage, - for each discrepancy you know whether the agent was wrong or had no tool, and whether it signalled that with low confidence.
Step 6 â fixes with evidence
For the rows with the result "doesn't work" or "works, but the result is useless" propose a fix
as a concrete value: derived field regex, tags and queries in tracesToLogsV2 and tracesToMetrics,
a backend-side change for exemplars in Mimir. Write the datasource fixes twice:
as a jq filter for PUT /api/datasources/uid/<uid> and as a fragment of a YAML provisioning file.
For the Loki and tracesToMetrics fixes run through MCP the query the fixed link would generate,
with the values from the span in item 2, and give the result. Don't write anything.
| Where | Correct | Typical agent deviation |
|---|---|---|
| derived field regex | "traceid":"(\w+)", evidence: count_over_time for stream="otel" greater than 0; alternative: matcherType: label with matcherRegex: traceid for a query with \| json |
regex not tested on data; unescaped quotes in jq or YAML |
derived field url |
via the API ${__value.raw}, in YAML $${__value.raw} |
a single $ in YAML: empty URL, the link disappears without an error |
tracesToLogsV2 |
datasourceUid: loki (uid), tags: [{key: service.name, value: service_name}], no tracesToLogs block |
key and value swapped (key: service_name): the span has no such attribute, the link disappears |
tracesToMetrics |
histogram_quantile(0.95, sum by (le) (rate(traces_spanmetrics_latency_bucket{$__tags}[5m]))), tags service.name â service_name, name â span_name, k8s.namespace.name â k8s_namespace_name, in YAML $$__tags; the query with substituted service_name="payment", span_name="grpc.oteldemo.PaymentService/Charge", k8s_namespace_name="otel" returns data |
k8s.namespace.name mapped to namespace (the Loki label name): empty chart; $__tags in YAML without $$ |
| exemplars in Mimir | sendExemplars: true in Prometheus remoteWrite and max_global_exemplars_per_user > 0 in Mimir; no change in Grafana |
changing exemplarTraceIdDestinations; name changed to the link text (itâs the exemplar label name, traceID) |
tracesToProfiles |
link only for services with a CPU profile, or separate transitions per profile type (Correlations); dead shifts removed | one profileTypeId changed for all services: fixes payment, breaks the Go services |
The agent doesnât apply fixes: the MCP server has no datasource write tools, and its token has the Viewer role. Compare the jq filters with the ds_patch commands in exercise 05, and the YAML with pliki/08-datasources-links.yaml from the materialy repo (git clone https://github.com/ProtopiaTech-Labs/grafana-admin.git materialy).
Check that: row 1 and Mimir in row 5 have a fix as a concrete value, the regex and tracesToMetrics fixes have evidence from the agentâs query, and the YAML has $$ before every macro.
Step 7 â cleanup
If you do the Stretch, do it before cleanup.
cd ~/audyt-08 && claude mcp remove --scope local grafana
curl -sS "${H[@]}" -X DELETE "$GRAFANA_URL/api/serviceaccounts/$SA" | jq -c .
cd ~ && rm -rf ~/audyt-08
Check that: deleting the account returns Service account deleted.
Success criteria
- The audit table has five rows, each with the agentâs prediction, the UI result and the cause named by a configuration key (
matcherRegex,datasourceUid,tags,queries,profileTypeId,enable-featureâŠ) or a missing key. - Rows 3 and 4 have the result âworks, but the result is uselessâ, with justification.
- Every row with the result âdoesnât workâ has a fix as a concrete value. The regex and
tracesToMetricsfixes have evidence from a query. - You can name two agent predictions it couldnât check through MCP, and at least one where it was wrong or overconfident, with the cause.
- The
mcp-ro-<login>account is deleted.
âStretch: dead keys and an audit without configuration (failure mode)
- Dead keys. Ask:
Which keys in ds-tempo.json does Grafana read but ignore? Justify each with what I saw in the UI: the LogQL from row 2 and the profile range from row 4.Correct: thetracesToLogsblock (withoutV2), because the LogQL lacks itsjobandinstancetags, andspanStartTimeShift/spanEndTimeShiftintracesToProfiles, because the profile range has no ±5 min shift. Deviation:tracesToLogsas a working âfallbackâ, or profile shifts treated as active because the same keys work intracesToLogsV2andtracesToMetrics. - Audit without configuration. Delete the files (
rm ~/audyt-08/ds-*.json), startclaudein a new session and paste the prompt from step 3 without its first two sentences. Correct: the agent says it canât read the configuration and asks for it, or limits itself to facts from the data. Deviation: it givesjsonDataâfrom memoryâ as if read. In row 1 it may hit the mark, becausetraceID=(\w+)is the example from the Loki docs. Check whether it labelled that as a guess.
Related lessons
- Audit of links between signals â the same exercise without the agent, verification paths step by step
- Logs, traces and datasource links â default tags, link time ranges, silent failures
- Links between signals in datasource configuration â where you apply the audit fixes
- Payment incident with an AI agent â the next exercise with an agent
- TraceQL with an AI agent â the same Grafana MCP setup and review discipline for traces
- Correlations (Grafana docs) â transitions defined outside the datasource