Grafana Pyroscope
Pyroscope is the continuous profiling backend. It answers the question traditional observability can’t: “the service is slow — but which function is slow?”
Role in the Stack
| Function | Details |
|---|---|
| Profile storage | Stores CPU, memory, lock, and other profiles in Azure Blob Storage |
| Flame graph rendering | Serves flame graph data to Grafana |
| Diff analysis | Compares profiles across time windows to detect regressions |
| Multi-language | Supports Go, Java, Python, .NET, Node.js, Rust, Ruby, C++ (via eBPF) |
Deployment
- Mode: Single binary (monolithic) — suitable for the workshop scale
- Replicas: 1
- Resources: 100m–1 CPU, 512Mi–2Gi RAM
- Storage: Azure Blob Storage (
pyroscope-datacontainer)
What Feeds Into Pyroscope
| Source | Collection Method | Languages | Profile Types |
|---|---|---|---|
| Alloy eBPF | Kernel-level stack sampling at 97 Hz | All (any process on the node) | CPU only |
| Alloy SDK scrape | Pull-based scraping via pprof/JFR endpoints | Go, Java, .NET, Python, Node.js, Rust | CPU, memory, goroutines, locks, exceptions |
eBPF vs SDK Profiling
| Aspect | eBPF (via Alloy) | SDK (via Alloy scrape) |
|---|---|---|
| Code changes | None | Annotation: profiles.grafana.com/cpu_scrape: "true" |
| Profile types | CPU only | CPU, heap, goroutines, mutex, block, exceptions |
| Overhead | < 1% | 1–5% |
| Stack quality | Kernel + user-space | User-space with language-specific detail |
| Coverage | Every process on the node | Only annotated pods |
In our setup, both run simultaneously — eBPF provides baseline CPU profiling for everything, SDK provides richer profiles for instrumented services.
Profile Types Available Per Service
| Service | Language | CPU | Memory | Goroutines | Locks | Exceptions |
|---|---|---|---|---|---|---|
ad |
Java | JFR | JFR | — | — | — |
cart |
.NET | CORECLR | CORECLR | — | CORECLR | CORECLR |
checkout |
Go | pprof | pprof | pprof | pprof | — |
product-catalog |
Go | pprof | pprof | pprof | pprof | — |
recommendation |
Python | SDK | SDK | — | — | — |
frontend |
Node.js | SDK | — | — | — | — |
payment |
Node.js | SDK | — | — | — | — |
shipping |
Rust | SDK | — | — | — | — |
email |
Ruby | eBPF | — | — | — | — |
| Others | Various | eBPF | — | — | — | — |
Integration with Other Components
Traces → Profiles
Grafana’s Tempo datasource maps service_name from trace spans to Pyroscope profiles. When viewing a trace span in Grafana, you can click through to see the CPU/memory profile of that service at the time the span was recorded.
Profiles Drilldown
Pyroscope is the data source for the Profiles Drilldown app in Grafana. It provides:
- All Services overview (CPU usage per service)
- Per-service profile type selection
- Flame graph visualization
- AI-powered flame graph explanation (via Grafana LLM plugin)
- Diff flame graphs for regression detection (red = slower, green = faster)
Alloy Integration
Alloy’s pyroscope.ebpf component runs as a privileged DaemonSet and pushes profiles to Pyroscope. The pyroscope.scrape component discovers SDK-instrumented pods and pulls profiles via standard endpoints.
Grafana Datasource
- Type:
grafana-pyroscope-datasource - URL:
http://pyroscope.monitoring.svc.cluster.local:4040 - Plugin required:
grafana-pyroscope-app(installed) - Use for: Flame graphs, CPU analysis, memory leak detection, lock contention, regression comparison