💪Exercise💪 — Links between signals in datasource configuration
You apply the audit fixes on your own copies of the Loki, Tempo and Prometheus datasources, which point at the same backends as the shared ones. You configure them through the API, the way automation does, and finally compare the result with a provisioning file. The shared datasources stay untouched.
Goal
- Configure a derived field (log → trace),
tracesToLogsV2(trace → logs),tracesToMetrics(trace → metric) andexemplarTraceIdDestinations(metric → trace). - Check each transition in Explore and on a dashboard panel.
- Trigger two silent failures: a link that disappears and a link that opens an empty result.
- Check backend requirements and Drilldown app versions through the API.
Environment: workshop lab, your own account (org role Admin).
Prerequisites
- The audit from exercise 04 or its AI agent variant — you know what’s broken and why.
- Your lab login (
user<N>) and password from the lab handout. curl,jq, Git Bash or WSL.- A clone of the materials repo:
git clone https://github.com/ProtopiaTech-Labs/grafana-admin.git materialy. The filepliki/08-datasources-links.yamlis in thematerialydirectory.
Once per terminal session:
export GRAFANA_URL=https://grafana.workshop2.indexoutofrange.com
export LOGIN=<login>
printf 'Password: ' && read -rs P && GRAFANA_AUTH="$LOGIN:$P" && echo
J='Content-Type: application/json'
# Read datasource → jq change → write. $1 = uid, $2 = jq filter ($l = login).
ds_patch() {
curl -sS -u "$GRAFANA_AUTH" "$GRAFANA_URL/api/datasources/uid/$1" \
| jq --arg l "$LOGIN" "$2" \
| curl -sS -u "$GRAFANA_AUTH" -H "$J" -X PUT "$GRAFANA_URL/api/datasources/uid/$1" -d @- \
| jq -r .message
}
Your datasources: Loki <login> (uid loki-<login>), Tempo <login> (uid tempo-<login>), Prometheus <login> (uid prometheus-<login>). The whole organization sees datasources, so the name and uid carry your login. Links between them point only at your copies.
💡 Link labels in the commands below (
Trace w Tempo,p95 tej operacji) are Polish on purpose: they matchpliki/08-datasources-links.yaml, which you compare against in step 8.
💪Exercise💪 — steps
Step 1 — datasource copies without links
for ds in \
"loki|Loki|http://loki-gateway.monitoring.svc.cluster.local" \
"tempo|Tempo|http://tempo-query-frontend.monitoring.svc.cluster.local:3200" \
"prometheus|Prometheus|http://prometheus-and-grafana-kub-prometheus.monitoring.svc.cluster.local:9090"; do
IFS='|' read -r type name url <<<"$ds"
jq -n --arg t "$type" --arg n "$name $LOGIN" --arg u "$url" --arg id "$type-$LOGIN" \
'{name: $n, uid: $id, type: $t, access: "proxy", url: $u, jsonData: {}}' \
| curl -sS -u "$GRAFANA_AUTH" -H "$J" -X POST "$GRAFANA_URL/api/datasources" -d @- | jq -r .message
curl -sS -u "$GRAFANA_AUTH" "$GRAFANA_URL/api/datasources/uid/$type-$LOGIN/health" | jq -r .status
done
Check that: each pair of lines is Datasource added and OK. On a repeat run, instead of Datasource added you get an error about an existing name: the copies are already there, carry on.
Step 2 — derived field: log → trace
-
Starting state: Explore →
Loki <login>, range Last 15 minutes:{service_name="payment", namespace="otel", stream="otel"}Expand a line: log details have no Links section.
-
Add a derived field with the fixed regex (Derived fields):
ds_patch "loki-$LOGIN" '.jsonData.derivedFields = [{ name: "TraceID", matcherType: "regex", matcherRegex: "\"traceid\":\"(\\w+)\"", datasourceUid: "tempo-\($l)", url: "${__value.raw}", urlDisplayLabel: "Trace w Tempo"}]' -
Run the query again and expand a line.
Check that: ds_patch prints Datasource updated, log details have a Trace w Tempo link in the Links section, and clicking it opens the trace in Tempo <login> in the right split-view pane.
💡 Through the API you write
${__value.raw}. In a provisioning file the same value is$${__value.raw}: Grafana substitutes environment variables in provisioning files, and$$is the escape (Use environment variables). A single$in YAML gives an empty URL and the link disappears without an error.
Step 3 — trace → logs and trace → metric
-
Add both blocks to your Tempo (Configure trace to logs, Configure trace to metrics):
ds_patch "tempo-$LOGIN" '.jsonData.tracesToLogsV2 = { datasourceUid: "loki-\($l)", spanStartTimeShift: "-1m", spanEndTimeShift: "1m", tags: [{key: "service.name", value: "service_name"}], filterByTraceID: true} | .jsonData.tracesToMetrics = { datasourceUid: "prometheus-\($l)", spanStartTimeShift: "-2m", spanEndTimeShift: "2m", tags: [{key: "service.name", value: "service_name"}, {key: "name", value: "span_name"}, {key: "k8s.namespace.name", value: "k8s_namespace_name"}], queries: [{name: "p95 tej operacji", query: "histogram_quantile(0.95, sum by (le) (rate(traces_spanmetrics_latency_bucket{$__tags}[5m])))"}]}'The
k8s.namespace.nametag narrows the metric to the span’s namespace. Without it the p95 mixes the shop with its copies from the SRE agent lab (userN), which have the same service and operation names. -
Explore →
Tempo <login>→ Search → TraceQL:{ resource.service.name = "payment" && resource.k8s.namespace.name = "otel" && status = error && kind = server } - Open a trace and expand the
grpc.oteldemo.PaymentService/Chargespan of thepaymentservice. - Click Logs for this trace. In the right pane open Query inspector → Query and read the LogQL.
- Go back to the span and click p95 tej operacji. Read the PromQL and the time range.
Check that:
- the LogQL starts with
{service_name="payment"}(only your tag, withoutservice_namespacefrom the default set) and filters by trace ID throughlabel_format … contains "<trace ID>" __line__; the result is the lines of this trace, - the PromQL contains
k8s_namespace_name="otel", service_name="payment", span_name="grpc.oteldemo.PaymentService/Charge"— values from the span, not constants.nameis a Grafana pseudo-tag with the span’s operation name, alongsidespanId,traceIdandduration, - the metric time range is the span time ±2 min, and the chart has data.
Step 4 — two silent failures
-
A link that disappears. Replace the tag key in
tracesToLogsV2with the Loki label name — a common mistake when copying configuration:ds_patch "tempo-$LOGIN" '.jsonData.tracesToLogsV2.tags = [{key: "service_name", value: "service_name"}]'Refresh the trace from step 3 and expand the same span.
-
A link that opens an empty result. Restore the logs tag, and in metrics map the namespace to the Loki label name:
ds_patch "tempo-$LOGIN" '.jsonData.tracesToLogsV2.tags = [{key: "service.name", value: "service_name"}] | .jsonData.tracesToMetrics.tags |= map(if .key == "k8s.namespace.name" then .value = "namespace" else . end)'Refresh the trace and click p95 tej operacji.
-
Undo the second change:
ds_patch "tempo-$LOGIN" '.jsonData.tracesToMetrics.tags |= map(if .key == "k8s.namespace.name" then .value = "k8s_namespace_name" else . end)'
Check that:
- after change 1 the Logs for this trace link disappeared without any message: the span has no
service_nameattribute, so$__tagsis empty, and Grafana doesn’t create a Loki link without a selector, - after change 2 the metrics link is there, but the PromQL has
namespace="otel"and returns an empty result without an error: the attribute exists on the span, but span metrics carry this dimension ask8s_namespace_name, notnamespace(that’s the Loki name). Grafana always creates the metrics link, even when no tag matches.
Step 5 — exemplars: metric → trace
-
Check exemplar storage in Prometheus (read through the Grafana proxy):
curl -sS -u "$GRAFANA_AUTH" "$GRAFANA_URL/api/datasources/proxy/uid/prometheus-$LOGIN/api/v1/status/flags" \ | jq -r '.data."enable-feature"' -
Explore →
Prometheus <login>, range Last 15 minutes:histogram_quantile(0.95, sum by (le) (rate(traces_spanmetrics_latency_bucket{service_name="payment", k8s_namespace_name="otel"}[5m])))Options → Exemplars on → Run query. Hover an exemplar point: the tooltip has the value and labels, including
traceID, but no trace link. -
Add the exemplar destination (Exemplars):
ds_patch "prometheus-$LOGIN" '.jsonData.exemplarTraceIdDestinations = [{ name: "traceID", datasourceUid: "tempo-\($l)", urlDisplayLabel: "Trace w Tempo"}]' -
Run the query again, hover an exemplar and click Trace w Tempo.
Check that: the flag contains exemplar-storage, and clicking the exemplar opens the trace in Tempo <login>. name must be the exemplar label name (Tempo writes traceID), not the link name.
Step 6 — derived field on a dashboard
- Go back to the query from step 2 (
Loki <login>). - Add → Add to dashboard → New dashboard → Open dashboard → Save: title
08 — logi payment <login>, folderUczestnicy/<login>→ Save. - On the Logs panel expand any line.
Check that: the Trace w Tempo link is visible on the dashboard panel too. A derived field belongs to the datasource, so it works wherever there are logs. Correlations work only in Explore.
Step 7 — backend requirements and Drilldown versions
-
The Loki pattern ingester (a requirement of the Patterns tab in Logs Drilldown), configuration read through the Grafana proxy:
curl -sS -u "$GRAFANA_AUTH" "$GRAFANA_URL/api/datasources/proxy/uid/loki-$LOGIN/config" \ | grep -E -A1 '^pattern_ingester:|pattern_ingester_tokenizable_json_fields_(default|append):' -
Drilldown app versions:
for p in grafana-metricsdrilldown-app grafana-lokiexplore-app grafana-exploretraces-app grafana-pyroscope-app; do printf '%-32s ' "$p" curl -sS -u "$GRAFANA_AUTH" "$GRAFANA_URL/api/plugins/$p/settings" | jq -r '.info.version // "brak"' done
Check that: pattern_ingester has enabled: true, the list of JSON fields to tokenize doesn’t contain body (hence no patterns for the otel stream in exercise 07), and the command prints four versions (brak = not installed). The API doesn’t show which one is pinned: in the lab only grafana-pyroscope-app (the plugins: list → preinstall_sync in the lab configuration, lesson 06). Grafana downloads the other three from the catalog at startup. The lab’s grafana.ini has preinstall_auto_update = false, but the plugin directory sits on a non-persistent volume (emptyDir), so every new Grafana pod installs their latest versions.
Step 8 — configuration as a file and cleanup
-
Fetch the
jsonDataof your three datasources:for t in loki tempo prometheus; do curl -sS -u "$GRAFANA_AUTH" "$GRAFANA_URL/api/datasources/uid/$t-$LOGIN" | jq '{name, uid, jsonData}' done - Compare the result with
pliki/08-datasources-links.yaml. It’s the same configuration as a provisioning file for the shared datasources (uidloki,tempo,prometheus) — the audit fix deployed as code (Provision data sources). -
Delete the dashboard from step 6 and your datasource copies:
for u in $(curl -sS -u "$GRAFANA_AUTH" "$GRAFANA_URL/api/search?type=dash-db&folderUIDs=uczestnicy-$LOGIN&query=logi%20payment" | jq -j '.[] | .uid, " "'); do curl -sS -u "$GRAFANA_AUTH" -X DELETE "$GRAFANA_URL/api/dashboards/uid/$u" | jq -r .message done for t in loki tempo prometheus; do curl -sS -u "$GRAFANA_AUTH" -X DELETE "$GRAFANA_URL/api/datasources/uid/$t-$LOGIN" | jq -r .message done
Check that: the keys from steps 2–5 (derivedFields[0], tracesToLogsV2, tracesToMetrics, exemplarTraceIdDestinations) match the file, apart from the uid and $$. The file also has keys the copies didn’t need (timeout, maxLines, nodeGraph, serviceMap, httpMethod, a fixed tracesToProfiles, the Label derived field from the Stretch). Cleanup prints Dashboard 08 — logi payment <login> deleted and three times Data source deleted.
Success criteria
- Log → trace: the Trace w Tempo link in Explore and on the dashboard’s Logs panel opens the trace.
- Trace → logs: LogQL with
{service_name="payment"}and a trace ID filter returns the lines of this trace. - Trace → metric: the PromQL contains
service_name,span_nameandk8s_namespace_nametaken from the span and returns data. - Metric → trace: an exemplar point opens the trace.
- You can tell apart the two silent failures from step 4: a disappearing link (tag missing on the span) and an empty result (label missing in the backend).
- The datasource copies and the
08 — logi payment <login>dashboard are deleted.
⭐Stretch: a Label-type derived field next to the regex
Before step 8 add a second derived field that matches the name of a label or parsed field instead of the line content:
ds_patch "loki-$LOGIN" '.jsonData.derivedFields += [{
name: "TraceID (label)", matcherType: "label", matcherRegex: "traceid",
datasourceUid: "tempo-\($l)", url: "${__value.raw}", urlDisplayLabel: "Trace w Tempo (label)"}]'
Compare log details for two queries:
{service_name="payment", namespace="otel", stream="otel"}
{service_name="payment", namespace="otel", stream="otel"} | json
Check that: the Trace w Tempo (label) link appears only with | json (only then is traceid a field), while the regex link appears in both cases.
Related lessons
- Logs, traces and datasource links
- Audit of links between signals
- Drilldown apps
- Integrations & TraceQL — Tempo integrations with logs, metrics and profiles
- Exemplars from the SDK — exemplars end to end
- Correlations (Grafana docs) — the same stack, transitions defined outside the datasource