Prometheus
Prometheus serves as the short-term metrics hub in our stack. It does not scrape targets directly — Alloy handles all scraping and sends metrics to Prometheus via remote write.
Role in the Stack
| Function | Details |
|---|---|
| Short-term storage | Holds recent metrics (hours) for fast queries |
| PromQL engine | Serves as a query engine for Grafana dashboards and Drilldown |
| Remote write receiver | Accepts metrics from Alloy and Tempo metrics generator |
| Remote write sender | Forwards all metrics to Mimir for long-term retention |
| Exemplar storage | Links metric data points to trace IDs for cross-signal navigation |
Configuration Highlights
- Remote write receiver: Enabled — accepts metrics pushed by Alloy
- Native histograms: Enabled — supports the new histogram format for more efficient percentile queries
- ServiceMonitor scraping: Disabled — Alloy handles all scraping
- Remote write to Mimir:
http://mimir-nginx.monitoring.svc.cluster.local:80/api/v1/push
What Feeds Into Prometheus
| Source | Signal | Path |
|---|---|---|
| Alloy | Application metrics (OTLP) | Alloy OTLP receiver → remote write to Prometheus |
| Alloy | Kubernetes metrics (kube-state-metrics, node-exporter, kubelet) | Alloy scraper → remote write to Prometheus |
| Alloy | Exporter metrics (PostgreSQL, Redis) | Alloy scraper → remote write to Prometheus |
| Tempo | Span metrics (RED: rate, errors, duration) | Tempo metrics generator → remote write to Prometheus |
What Prometheus Feeds
| Destination | Purpose |
|---|---|
| Mimir | Long-term storage via remote write |
| Grafana | Short-term queries, dashboards, exemplars |
| Alertmanager | Alert rule evaluation and routing |
Integration with Other Components
Exemplars — When an application emits a metric with a trace ID (via OTel SDK), Prometheus stores it as an exemplar. In Grafana, clicking an exemplar point on a metric graph jumps directly to the corresponding trace in Tempo.
Service Map — Grafana’s Tempo datasource uses Prometheus as the source for the Service Map (node graph) visualization, querying span metrics to render service-to-service dependencies.
Alertmanager — Prometheus evaluates alert rules and sends notifications to Alertmanager, which handles deduplication, grouping, silencing, and routing to notification channels.
Grafana Datasource
- Type:
prometheus - URL:
http://prometheus-and-grafana-kub-prometheus.monitoring.svc.cluster.local:9090 - Exemplars: Enabled — links to Tempo by TraceID
- Access: NodePort 30090, Ingress at
prometheus.<domain>