💪Exercise💪 — Split view, inspector and saving an exploration
The same PromQL gives different values in a panel and in Explore, and a user reports it as a Grafana bug. In this exercise you measure the cause in Query inspector and set up the panel so the result doesn’t depend on screen width. AI agent variant: Split view, inspector and saving an exploration with an AI agent. Do one of the variants.
Goal
- Put the
paymentservice’s metric and logs side by side in split view with synced time and share the view with a short link. - Measure the panel’s and Explore’s
intervalMsandmaxDataPointsin Query inspector. - Align the panel’s and Explore’s interval with explicit query options.
- Save a LogQL query from Explore as a dashboard panel.
Environment: workshop lab, your own Grafana account (organisation Admin role).
Prerequisites
- Your lab login (
user<N>) and password from the lab handout, and your folderUczestnicy/<login>. -
The
paymentFailurescenario at the50%variant. In Explore (Prometheus, uidprometheus) compute the error ratio ofpaymentserver spans:sum(rate(traces_spanmetrics_calls_total{service_name="payment", k8s_namespace_name="otel", span_kind="SPAN_KIND_SERVER", status_code="STATUS_CODE_ERROR"}[15m])) / sum(rate(traces_spanmetrics_calls_total{service_name="payment", k8s_namespace_name="otel", span_kind="SPAN_KIND_SERVER"}[15m]))A result of
0.35–0.65means the50%variant. About0.1(usually0.03–0.2) is the lab baseline (10%variant);0or no result means the scenario is off. The mere presence of aSTATUS_CODE_ERRORseries proves nothing, because it also exists at10%. The window is 15 minutes because the shop makes only ~3 payments a minute: in a5mwindow the baseline ratio jumps from0to over0.3.The
k8s_namespace_name="otel"filter narrows the result to the shop. Copies of the shop from the SRE-agent lab (namespacesuserN) write to the same span metrics, logs and traces with the same service names. Without the filter the ratio mixes the shop with copies where the scenario is off, so at baseline it comes out too low and can look like the scenario is off.The instructor switches the
50%variant for the whole group. After the switch wait ~15 min before measuring the ratio:rate()with a15mwindow mixes the old and new variants. Steps 5–7 work without the scenario too. - A clone of the materials repo:
git clone https://github.com/ProtopiaTech-Labs/grafana-admin.git materialy. The filepliki/08-explore-vs-panel.jsonis in thematerialydirectory.
💪Exercise💪 — steps
Step 1 — the metric in Explore
- Explore → datasource Prometheus → Code mode.
-
Range Last 1 hour, query:
sum by (status_code) (rate(traces_spanmetrics_calls_total{service_name="payment", k8s_namespace_name="otel", span_kind="SPAN_KIND_SERVER"}[$__rate_interval])) - Run query.
Check that: the graph has two series, STATUS_CODE_UNSET and STATUS_CODE_ERROR. At the 50% variant they have similar height. At baseline ERROR sits at zero most of the time, with single spikes: the $__rate_interval window is ~1 min here, and a minute holds ~3 payments.
Step 2 — logs alongside in split view
- Split → change the right pane’s datasource to Loki.
-
Query (Code mode):
{service_name="payment", namespace="otel", stream="otel"} |~ `(?i)fail|error` - Check the chain icon next to the time picker (Synced times). Split copies the range and syncs both panes’ time right away, so the icon is highlighted. Don’t click it: clicking unsyncs the panes. The documentation (Split and compare) describes linking time with a click, but in the lab the panes are linked as soon as you Split.
- Drag-select a few minutes of the left graph where the
STATUS_CODE_ERRORseries is above zero. A window with no errors givesNo logs foundon the right.
Check that: the right pane narrows to the same window and shows payment log lines about failed payments (Payment request failed. Invalid token. …).
💡 The
paymentservice has two streams in Loki with the sameservice_name.stream="otel"is the OTLP logs: Alloy turns them into a JSON line inotelcol.exporter.loki(body,traceid,severity,attributes).stream="stdout"is the same records printed by the container console, cut into single lines, withouttraceidon one line. Withoutstream="otel"you get both streams and fragments likestack: 'Error: …' +.
Step 3 — short link
- Click Share in the top right (in split view just the icon). The button copies a short link (Copy shortened URL). The arrow next to it expands the variants: Copy URL, Copy absolute shortened URL, Copy absolute URL (range replaced with fixed timestamps).
- Open the link in a new tab (or a private window after logging in).
Check that: the link has the form /goto/<uid> and restores both split panes with the same range.
Step 4 — metric → logs
- Close the right pane (Close).
- In the left pane change the datasource from Prometheus to Loki.
Check that: the Loki editor has the selector {service_name="payment"}. Grafana carries over the matchers from the PromQL selector and keeps only labels that exist in Loki (Switch from metrics to logs): span_kind and k8s_namespace_name drop out (the namespace is the namespace label in Loki), and status_code was never a matcher (it’s only in by).
Step 5 — import the “narrow vs wide panel” dashboard
- Dashboards → New → Import dashboard → paste the contents of
pliki/08-explore-vs-panel.json. - Folder:
Uczestnicy/<login>. The file has nouid, so Grafana assigns a new one and won’t overwrite someone else’s dashboard. But every further import creates another copy: on a repeat, delete the previous one first (step 9). - Import.
The dashboard (08 — Panel wąski vs szeroki) has two panels with the same query, range 24 h:
sum(rate(traces_spanmetrics_calls_total{service_name="payment", k8s_namespace_name="otel", status_code="STATUS_CODE_ERROR"}[$__interval]))
Panel Wąski (narrow) is 6 grid columns wide, panel Szeroki (wide) 24.
Step 6 — measuring in Query inspector
You need three values from the inspector:
queries[0].maxDataPointsandqueries[0].intervalMs— in the request,Step— in the response, inframes[0].schema.meta.executedQueryString(e.g.Expr: … Step: 5m0s). This is the step Prometheus actually evaluated the query with.
- On panel Wąski: panel menu → Inspect → Query → Refresh. Note the three values.
- Panel menu of Wąski → Explore. Don’t use split view: the measurement must cover one full-width pane.
- In Explore: Query inspector → Query → Refresh. Note the three values and the
rate()window in the query editor. - Compare the graph maximum in the panel and in Explore.
| Where | maxDataPoints |
intervalMs |
Step |
rate() window |
Graph maximum |
|---|---|---|---|---|---|
| Panel Wąski | $__interval |
||||
| Explore from the panel menu |
Check that:
maxDataPointsmatches the graph width in pixels, so the values depend on the screen. Panel Wąski hasintervalMs=300000andStep=5m0s(on a screen wider than ~2000 px,120000and2m0s). Explore has a smaller step:60000and1m0son a 1600–1920 px screen,120000and2m0sat ~1300 px.- The Explore editor shows
[5m](with a panel step of2m0s,[2m]), not[$__interval]. When opening from the panel menu, Grafana substitutes variables, including$__interval, with the panel’s values. Therate()window is therefore the same, and the maxima differ because of the step: Explore computes a point every minute and hits short peaks that the panel with a 5 min step skips.
Step 7 — aligning the interval
- Edit panel Wąski → Query options → Min interval =
5m. Replace[$__interval]with[5m]. Save → Save. - In Explore from step 6 expand Options under the query → Min step =
5m(Min step). The[5m]window is already there. Run query. - Repeat the step 6 measurement in both places.
Check that: the panel and Explore both have Step = 5m0s and the same maximum. The panel sends intervalMs = 300000. In Explore intervalMs stays unchanged (e.g. 60000): Min step is a query option applied by the datasource backend, not the frontend. Step decides the result.
💡 Explore opened from the panel menu gets the queries with variables substituted, the time range and the datasource. Min interval (Min interval), Max data points, the panel’s transformations and expressions stay on the dashboard.
$__rate_intervalalone won’t align results, because Grafana derives it from$__interval. The organisation-wide floor is Scrape interval (timeInterval) in the datasource configuration.
Step 8 — saving a LogQL query as a panel
- In Explore set datasource Loki and the query from step 2.
- Add → Add to dashboard → Existing dashboard → pick the imported dashboard → Open dashboard.
- Save → Save. Explore doesn’t save the dashboard for you.
Check that: the dashboard has a third panel of type Logs (title New Panel) with the LogQL query from step 2.
Step 9 — cleanup
Delete the imported dashboard, including copies from any repeated imports:
export GRAFANA_URL=https://grafana.workshop2.indexoutofrange.com
export LOGIN=<login>
printf 'Password: ' && read -rs P && GRAFANA_AUTH="$LOGIN:$P" && echo
for u in $(curl -sS -u "$GRAFANA_AUTH" "$GRAFANA_URL/api/search?type=dash-db&folderUIDs=uczestnicy-$LOGIN&query=szeroki" | jq -j '.[] | .uid, " "'); do
curl -sS -u "$GRAFANA_AUTH" -X DELETE "$GRAFANA_URL/api/dashboards/uid/$u" | jq -r .message
done
Check that: the loop prints Dashboard 08 — Panel wąski vs szeroki deleted for every copy. You can’t delete the short link from step 3: the API only creates short links, and unused ones expire only once [short_links] expire_time is set (default -1, no expiry; the lab keeps the default).
Success criteria
- The short link
/goto/<uid>restores the metric + logs split view with synced time. - The step 6 table: the panel has a larger
intervalMsandStepthan Explore (typically300000/5m0svs60000/1m0s), and in Explore therate()window is the panel’s substituted$__interval. - After step 7 the panel and Explore both have
Step=5m0s. - The imported dashboard in
Uczestnicy/<login>has a Logs panel with the LogQL query and is saved.
⭐Stretch: an Explore URL built by hand
Build an Explore link without clicking, the way a ticketing system or runbook would. The panes parameter is encoded JSON with the pane definitions; schemaVersion must be 1 (Generate Explore URLs from external tools). In Git Bash / WSL:
PANES=$(jq -cn '{a: {datasource: "loki",
queries: [{refId: "A", expr: "{service_name=\"payment\", namespace=\"otel\", stream=\"otel\"} |= `Invalid token`",
datasource: {type: "loki", uid: "loki"}}],
range: {from: "now-30m", to: "now"}}}')
echo "https://grafana.workshop2.indexoutofrange.com/explore?schemaVersion=1&orgId=1&panes=$(jq -rn --arg p "$PANES" '$p|@uri')"
Open the generated URL. Then add a second key (b) with a Prometheus query (datasource: "prometheus", expr from step 1) and check that the link opens split view.
Check that: the link opens Explore with the query ready, and after adding b — two panes. Compare this URL’s length with the short link from step 3. Chat tools truncate the full URL, which is why sharing uses Copy shortened URL.