Grafana Loki
Loki is the log aggregation backend. Unlike traditional log systems (Elasticsearch, Splunk), Loki indexes only labels, not the full log content — making it significantly cheaper to operate at scale.
Role in the Stack
| Function | Details |
|---|---|
| Log storage | Stores logs in Azure Blob Storage with label-based indexing |
| LogQL engine | Query language combining label filtering with content filtering |
| Trace correlation | TraceID derived field enables click-through to Tempo |
| Pattern detection | Pattern ingester for Logs Drilldown auto-analysis |
| Structured metadata | Supports OpenTelemetry attributes as structured metadata |
Versions
| Chart | grafana-community/loki 13.6.2 |
| App | Loki 3.7.1 |
Deployment — Distributed (Microservices) Mode
| Component | Replicas | CPU (req) | RAM (req) | Storage | Purpose |
|---|---|---|---|---|---|
| Distributor | 2 | 200m | 512Mi | — | Validates incoming streams, applies rate limits, fans out to ingesters |
| Ingester | 3 | 500m | 1Gi | 10Gi PV | Buffers logs in chunks, flushes to Azure blob (zoneAwareReplication: false) |
| Querier | 2 | 500m | 1Gi | — | Executes LogQL against ingesters + storage, deduplicates |
| Query Frontend | 1 | 200m | 256Mi | — | Splits / queues / caches queries, parallelises shardable ones |
| Query Scheduler | 1 | 100m | 256Mi | — | Queues queries between frontend and queriers |
| Compactor | 1 | 200m | 512Mi | 10Gi PV | Compacts TSDB indexes, applies retention |
| Index Gateway | 1 | 200m | 512Mi | 10Gi PV | Serves TSDB index reads to queriers |
| Ruler | 1 | 200m | 512Mi | 10Gi PV | Recording / alerting rule evaluation |
| Gateway | 1 | 200m | 256Mi | — | Nginx reverse proxy — sole entry point for clients |
What Feeds Into Loki
| Source | Log Type | Path |
|---|---|---|
| Alloy (file tail) | Container stdout/stderr | Alloy tails /var/log/pods/ → CRI parse → Loki |
| Alloy (OTLP) | Structured application logs | App OTel SDK → Alloy OTLP receiver → Loki |
Both streams carry Kubernetes labels (namespace, pod, container) and OpenTelemetry attributes.
Storage
- Backend: Azure Blob Storage
- Containers:
loki-chunks(log data),loki-ruler(ruler state) - Schema: v13 with 24h index period
- Caching: Chunks cache (1Gi), Results cache (512Mi)
Limits
| Limit | Value |
|---|---|
| Ingestion rate | 100 MB/s |
| Ingestion burst | 200 MB |
| Max query parallelism | 32 |
| Query split interval | 15 minutes |
| Reject old samples | Disabled (accepts logs up to 7 days old) |
| Structured metadata | Allowed |
Features Enabled
Pattern Ingester — Enabled specifically for the Logs Drilldown feature. Automatically detects patterns in log lines and groups them, allowing users to identify noise and anomalies without writing queries.
Structured Metadata — OpenTelemetry attributes (like traceID, spanID, service attributes) are stored as structured metadata, queryable via LogQL without indexing as labels.
Integration with Other Components
Logs → Traces — Loki is configured with a TraceID derived field that detects trace IDs in log lines. Clicking a trace ID in Grafana opens the corresponding trace in Tempo.
Traces → Logs — The Tempo datasource links traces back to Loki logs with:
- Filter by
TraceID - Tag mapping:
service.name→servicelabel in Loki - Time shift: ±1 hour around the trace timestamp
Logs Drilldown — The pattern ingester feeds the Patterns tab in Logs Drilldown, enabling automatic grouping and noise filtering.
Grafana Datasource
- Type:
loki - URL:
http://loki-gateway.monitoring.svc.cluster.local - Derived fields: TraceID → link to Tempo
- Use for: Log queries (LogQL), trace-correlated log exploration, pattern detection