💪Exercise💪 — Split view, inspector and saving an exploration

The same PromQL gives different values in a panel and in Explore, and a user reports it as a Grafana bug. In this exercise you measure the cause in Query inspector and set up the panel so the result doesn’t depend on screen width. AI agent variant: Split view, inspector and saving an exploration with an AI agent. Do one of the variants.

Goal

  • Put the payment service’s metric and logs side by side in split view with synced time and share the view with a short link.
  • Measure the panel’s and Explore’s intervalMs and maxDataPoints in Query inspector.
  • Align the panel’s and Explore’s interval with explicit query options.
  • Save a LogQL query from Explore as a dashboard panel.

Environment: workshop lab, your own Grafana account (organisation Admin role).

Prerequisites

  • Your lab login (user<N>) and password from the lab handout, and your folder Uczestnicy/<login>.
  • The paymentFailure scenario at the 50% variant. In Explore (Prometheus, uid prometheus) compute the error ratio of payment server spans:

    sum(rate(traces_spanmetrics_calls_total{service_name="payment", k8s_namespace_name="otel", span_kind="SPAN_KIND_SERVER", status_code="STATUS_CODE_ERROR"}[15m]))
    /
    sum(rate(traces_spanmetrics_calls_total{service_name="payment", k8s_namespace_name="otel", span_kind="SPAN_KIND_SERVER"}[15m]))
    

    A result of 0.35–0.65 means the 50% variant. About 0.1 (usually 0.03–0.2) is the lab baseline (10% variant); 0 or no result means the scenario is off. The mere presence of a STATUS_CODE_ERROR series proves nothing, because it also exists at 10%. The window is 15 minutes because the shop makes only ~3 payments a minute: in a 5m window the baseline ratio jumps from 0 to over 0.3.

    The k8s_namespace_name="otel" filter narrows the result to the shop. Copies of the shop from the SRE-agent lab (namespaces userN) write to the same span metrics, logs and traces with the same service names. Without the filter the ratio mixes the shop with copies where the scenario is off, so at baseline it comes out too low and can look like the scenario is off.

    The instructor switches the 50% variant for the whole group. After the switch wait ~15 min before measuring the ratio: rate() with a 15m window mixes the old and new variants. Steps 5–7 work without the scenario too.

  • A clone of the materials repo: git clone https://github.com/ProtopiaTech-Labs/grafana-admin.git materialy. The file pliki/08-explore-vs-panel.json is in the materialy directory.

💪Exercise💪 — steps

Step 1 — the metric in Explore

  1. Explore → datasource Prometheus → Code mode.
  2. Range Last 1 hour, query:

    sum by (status_code) (rate(traces_spanmetrics_calls_total{service_name="payment", k8s_namespace_name="otel", span_kind="SPAN_KIND_SERVER"}[$__rate_interval]))
    
  3. Run query.

Check that: the graph has two series, STATUS_CODE_UNSET and STATUS_CODE_ERROR. At the 50% variant they have similar height. At baseline ERROR sits at zero most of the time, with single spikes: the $__rate_interval window is ~1 min here, and a minute holds ~3 payments.

Step 2 — logs alongside in split view

  1. Split → change the right pane’s datasource to Loki.
  2. Query (Code mode):

    {service_name="payment", namespace="otel", stream="otel"} |~ `(?i)fail|error`
    
  3. Check the chain icon next to the time picker (Synced times). Split copies the range and syncs both panes’ time right away, so the icon is highlighted. Don’t click it: clicking unsyncs the panes. The documentation (Split and compare) describes linking time with a click, but in the lab the panes are linked as soon as you Split.
  4. Drag-select a few minutes of the left graph where the STATUS_CODE_ERROR series is above zero. A window with no errors gives No logs found on the right.

Check that: the right pane narrows to the same window and shows payment log lines about failed payments (Payment request failed. Invalid token. …).

💡 The payment service has two streams in Loki with the same service_name. stream="otel" is the OTLP logs: Alloy turns them into a JSON line in otelcol.exporter.loki (body, traceid, severity, attributes). stream="stdout" is the same records printed by the container console, cut into single lines, without traceid on one line. Without stream="otel" you get both streams and fragments like stack: 'Error: …' +.

  1. Click Share in the top right (in split view just the icon). The button copies a short link (Copy shortened URL). The arrow next to it expands the variants: Copy URL, Copy absolute shortened URL, Copy absolute URL (range replaced with fixed timestamps).
  2. Open the link in a new tab (or a private window after logging in).

Check that: the link has the form /goto/<uid> and restores both split panes with the same range.

Step 4 — metric → logs

  1. Close the right pane (Close).
  2. In the left pane change the datasource from Prometheus to Loki.

Check that: the Loki editor has the selector {service_name="payment"}. Grafana carries over the matchers from the PromQL selector and keeps only labels that exist in Loki (Switch from metrics to logs): span_kind and k8s_namespace_name drop out (the namespace is the namespace label in Loki), and status_code was never a matcher (it’s only in by).

Step 5 — import the “narrow vs wide panel” dashboard

  1. Dashboards → New → Import dashboard → paste the contents of pliki/08-explore-vs-panel.json.
  2. Folder: Uczestnicy/<login>. The file has no uid, so Grafana assigns a new one and won’t overwrite someone else’s dashboard. But every further import creates another copy: on a repeat, delete the previous one first (step 9).
  3. Import.

The dashboard (08 — Panel wąski vs szeroki) has two panels with the same query, range 24 h:

sum(rate(traces_spanmetrics_calls_total{service_name="payment", k8s_namespace_name="otel", status_code="STATUS_CODE_ERROR"}[$__interval]))

Panel Wąski (narrow) is 6 grid columns wide, panel Szeroki (wide) 24.

Step 6 — measuring in Query inspector

You need three values from the inspector:

  • queries[0].maxDataPoints and queries[0].intervalMs — in the request,
  • Step — in the response, in frames[0].schema.meta.executedQueryString (e.g. Expr: … Step: 5m0s). This is the step Prometheus actually evaluated the query with.
  1. On panel Wąski: panel menu → Inspect → Query → Refresh. Note the three values.
  2. Panel menu of Wąski → Explore. Don’t use split view: the measurement must cover one full-width pane.
  3. In Explore: Query inspector → Query → Refresh. Note the three values and the rate() window in the query editor.
  4. Compare the graph maximum in the panel and in Explore.
Where maxDataPoints intervalMs Step rate() window Graph maximum
Panel Wąski       $__interval  
Explore from the panel menu          

Check that:

  • maxDataPoints matches the graph width in pixels, so the values depend on the screen. Panel Wąski has intervalMs = 300000 and Step = 5m0s (on a screen wider than ~2000 px, 120000 and 2m0s). Explore has a smaller step: 60000 and 1m0s on a 1600–1920 px screen, 120000 and 2m0s at ~1300 px.
  • The Explore editor shows [5m] (with a panel step of 2m0s, [2m]), not [$__interval]. When opening from the panel menu, Grafana substitutes variables, including $__interval, with the panel’s values. The rate() window is therefore the same, and the maxima differ because of the step: Explore computes a point every minute and hits short peaks that the panel with a 5 min step skips.

Step 7 — aligning the interval

  1. Edit panel Wąski → Query options → Min interval = 5m. Replace [$__interval] with [5m]. Save → Save.
  2. In Explore from step 6 expand Options under the query → Min step = 5m (Min step). The [5m] window is already there. Run query.
  3. Repeat the step 6 measurement in both places.

Check that: the panel and Explore both have Step = 5m0s and the same maximum. The panel sends intervalMs = 300000. In Explore intervalMs stays unchanged (e.g. 60000): Min step is a query option applied by the datasource backend, not the frontend. Step decides the result.

💡 Explore opened from the panel menu gets the queries with variables substituted, the time range and the datasource. Min interval (Min interval), Max data points, the panel’s transformations and expressions stay on the dashboard. $__rate_interval alone won’t align results, because Grafana derives it from $__interval. The organisation-wide floor is Scrape interval (timeInterval) in the datasource configuration.

Step 8 — saving a LogQL query as a panel

  1. In Explore set datasource Loki and the query from step 2.
  2. Add → Add to dashboard → Existing dashboard → pick the imported dashboard → Open dashboard.
  3. Save → Save. Explore doesn’t save the dashboard for you.

Check that: the dashboard has a third panel of type Logs (title New Panel) with the LogQL query from step 2.

Step 9 — cleanup

Delete the imported dashboard, including copies from any repeated imports:

export GRAFANA_URL=https://grafana.workshop2.indexoutofrange.com
export LOGIN=<login>
printf 'Password: ' && read -rs P && GRAFANA_AUTH="$LOGIN:$P" && echo
for u in $(curl -sS -u "$GRAFANA_AUTH" "$GRAFANA_URL/api/search?type=dash-db&folderUIDs=uczestnicy-$LOGIN&query=szeroki" | jq -j '.[] | .uid, " "'); do
  curl -sS -u "$GRAFANA_AUTH" -X DELETE "$GRAFANA_URL/api/dashboards/uid/$u" | jq -r .message
done

Check that: the loop prints Dashboard 08 — Panel wąski vs szeroki deleted for every copy. You can’t delete the short link from step 3: the API only creates short links, and unused ones expire only once [short_links] expire_time is set (default -1, no expiry; the lab keeps the default).

Success criteria

  • The short link /goto/<uid> restores the metric + logs split view with synced time.
  • The step 6 table: the panel has a larger intervalMs and Step than Explore (typically 300000 / 5m0s vs 60000 / 1m0s), and in Explore the rate() window is the panel’s substituted $__interval.
  • After step 7 the panel and Explore both have Step = 5m0s.
  • The imported dashboard in Uczestnicy/<login> has a Logs panel with the LogQL query and is saved.

⭐Stretch: an Explore URL built by hand

Build an Explore link without clicking, the way a ticketing system or runbook would. The panes parameter is encoded JSON with the pane definitions; schemaVersion must be 1 (Generate Explore URLs from external tools). In Git Bash / WSL:

PANES=$(jq -cn '{a: {datasource: "loki",
  queries: [{refId: "A", expr: "{service_name=\"payment\", namespace=\"otel\", stream=\"otel\"} |= `Invalid token`",
             datasource: {type: "loki", uid: "loki"}}],
  range: {from: "now-30m", to: "now"}}}')
echo "https://grafana.workshop2.indexoutofrange.com/explore?schemaVersion=1&orgId=1&panes=$(jq -rn --arg p "$PANES" '$p|@uri')"

Open the generated URL. Then add a second key (b) with a Prometheus query (datasource: "prometheus", expr from step 1) and check that the link opens split view.

Check that: the link opens Explore with the query ready, and after adding b — two panes. Compare this URL’s length with the short link from step 3. Chat tools truncate the full URL, which is why sharing uses Copy shortened URL.

results matching ""

    No results matching ""