Grafana Loki
Loki is the log aggregation backend. Unlike traditional log systems (Elasticsearch, Splunk), Loki indexes only labels, not the full log content — making it significantly cheaper to operate at scale.
Role in the Stack
| Function | Details |
|---|---|
| Log storage | Stores logs in Azure Blob Storage with label-based indexing |
| LogQL engine | Query language combining label filtering with content filtering |
| Trace correlation | TraceID derived field enables click-through to Tempo |
| Pattern detection | Pattern ingester for Logs Drilldown auto-analysis |
| Structured metadata | Supports OpenTelemetry attributes as structured metadata |
Deployment — Simple Scalable Mode
| Component | Replicas | CPU | RAM | Storage | Purpose |
|---|---|---|---|---|---|
| Write | 2 | 800m | 2Gi | 10Gi PV | Distributor + Ingester (accepts and buffers logs) |
| Read | 3 | 2000m | 4Gi | 10Gi PV | Query Frontend + Querier (processes LogQL queries) |
| Backend | 1 | 800m | 2Gi | 10Gi PV | Compactor + Index Gateway + Ruler |
| Gateway | 1 | 200m | 256Mi | — | Nginx reverse proxy, routing |
What Feeds Into Loki
| Source | Log Type | Path |
|---|---|---|
| Alloy (file tail) | Container stdout/stderr | Alloy tails /var/log/pods/ → CRI parse → Loki |
| Alloy (OTLP) | Structured application logs | App OTel SDK → Alloy OTLP receiver → Loki |
Both streams carry Kubernetes labels (namespace, pod, container) and OpenTelemetry attributes.
Storage
- Backend: Azure Blob Storage
- Containers:
loki-chunks(log data),loki-ruler(ruler state) - Schema: v13 with 24h index period
- Caching: Chunks cache (1Gi), Results cache (512Mi)
Limits
| Limit | Value |
|---|---|
| Ingestion rate | 100 MB/s |
| Ingestion burst | 200 MB |
| Max query parallelism | 32 |
| Query split interval | 15 minutes |
| Reject old samples | Disabled (accepts logs up to 7 days old) |
| Structured metadata | Allowed |
Features Enabled
Pattern Ingester — Enabled specifically for the Logs Drilldown feature. Automatically detects patterns in log lines and groups them, allowing users to identify noise and anomalies without writing queries.
Structured Metadata — OpenTelemetry attributes (like traceID, spanID, service attributes) are stored as structured metadata, queryable via LogQL without indexing as labels.
Integration with Other Components
Logs → Traces — Loki is configured with a TraceID derived field that detects trace IDs in log lines. Clicking a trace ID in Grafana opens the corresponding trace in Tempo.
Traces → Logs — The Tempo datasource links traces back to Loki logs with:
- Filter by
TraceID - Tag mapping:
service.name→servicelabel in Loki - Time shift: ±1 hour around the trace timestamp
Logs Drilldown — The pattern ingester feeds the Patterns tab in Logs Drilldown, enabling automatic grouping and noise filtering.
Grafana Datasource
- Type:
loki - URL:
http://loki-gateway.monitoring.svc.cluster.local - Derived fields: TraceID → link to Tempo
- Use for: Log queries (LogQL), trace-correlated log exploration, pattern detection