Overview
Grafana Pyroscope
Source: https://grafana.com/oss/pyroscope/
Grafana Pyroscope is an open-source continuous profiling platform that enables you to understand resource usage (CPU, memory, etc.) at the code level in production environments with minimal overhead.
Key Features
- Continuous profiling — always-on profiling in production, not just during debugging sessions
- Multiple profile types — CPU, heap/memory, goroutines, mutex contention, block/I/O, off-CPU
- Multi-language support — Go, Java, Python, .NET, Node.js, Rust, C++ (via eBPF)
- Flame graph visualization — intuitive visualization of where time and resources are spent
- Diff flame graphs — compare profiles between deployments to detect regressions
- Grafana integration — native datasource with drill-down from metrics and traces to profiles
Data Flow
┌─────────────────────────────────────────────────────┐
│ Data Sources │
│ │
│ ┌──────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ eBPF │ │ Pyroscope SDK│ │ JFR / CORECLR│ │
│ │ (all │ │ (Go, Python, │ │ (Java, .NET) │ │
│ │ languages)│ │ Node.js) │ │ │ │
│ └─────┬────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
│ └───────────────┼─────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────────┐ │
│ │ Grafana Alloy │ │
│ │ pyroscope.ebpf → eBPF-based CPU profiling │ │
│ │ pyroscope.scrape → SDK-based profile scraping │ │
│ │ pyroscope.write → forward to Pyroscope │ │
│ └────────────────────┬───────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────────┐ │
│ │ Pyroscope Server (:4040) │ │
│ │ Storage: Azure Blob / S3 / local disk │ │
│ └────────────────────────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────────┐ │
│ │ Grafana (:3000) │ │
│ │ Flame graphs, diff views, trace correlation │ │
│ └────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘
Internal Architecture
Pyroscope has a microservices-based architecture. All components are compiled into a single binary, and the -target parameter controls which component(s) the process runs as. This allows the same binary to operate as a monolith or as individual microservices.
Components
| Component | Role | Stateful |
|---|---|---|
| Distributor | Receives incoming profiles from clients and routes them to ingesters | No |
| Ingester | Writes profiles to local disk, periodically flushes blocks to long-term storage | Yes |
| Compactor | Merges blocks from multiple ingesters, removes duplicate samples, reduces storage | No |
| Query-frontend | Receives queries, accelerates execution (splitting, caching), dispatches to query-scheduler | No |
| Query-scheduler | Maintains a per-tenant query queue, ensures fair scheduling | No |
| Querier | Pulls queries from scheduler, fetches data from ingesters (recent) and store-gateways (historical) | No |
| Store-gateway | Provides access to blocks in long-term object storage | No |
The Write Path
Profiles from clients
│
▼
┌───────────────┐
│ Distributor │
└───────┬───────┘
│ routes by tenant + series
▼
┌─────────────┼─────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Ingester │ │ Ingester │ │ Ingester │
│ (replica)│ │ (replica)│ │ (replica)│
└─────┬────┘ └─────┬────┘ └─────┬────┘
│ │ │
│ flush blocks to storage │
▼ ▼ ▼
┌────────────────────────────────────────┐
│ Long-Term Object Storage │
│ (S3 / Azure Blob / GCS / local) │
└────────────────────┬───────────────────┘
│
▼
┌───────────────┐
│ Compactor │
│ merge blocks, │
│ deduplicate │
└───────────────┘
- Distributor receives push requests and routes each profile series to ingesters
- Each series is replicated to 3 ingesters by default
- Ingesters append profiles to a per-tenant database on local disk
- In-memory profiles are periodically flushed to disk as blocks
- Blocks are uploaded to long-term object storage
- Compactor merges blocks from multiple ingesters into single blocks and removes duplicate samples
The Read Path
Query from Grafana
│
▼
┌──────────────────┐
│ Query-frontend │
│ (split, cache) │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Query-scheduler │
│ (per-tenant │
│ fair queuing) │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Querier │
└───┬──────────┬───┘
│ │
recent │ │ historical
data │ │ data
▼ ▼
┌──────────┐ ┌───────────────┐
│ Ingesters│ │ Store-gateway │
│ (memory) │ │ (object store) │
└──────────┘ └───────────────┘
- Query-frontend receives the query, splits it by time range, checks the cache
- Query-scheduler queues the sub-queries with fair per-tenant scheduling
- Querier picks up work and fetches data from:
- Ingesters — for recent, in-memory data
- Store-gateways — for historical data in object storage
- Results are merged and returned to Grafana
Deployment Modes
| Mode | -target |
Description | Use Case |
|---|---|---|---|
| Monolithic | all (default) |
All components in a single process | Development, small workloads, quick start |
| Microservices | per component (e.g. ingester) |
Each component runs as a separate process | Production — independent scaling, isolated failure domains |
💡 In this workshop we deploy Pyroscope in monolithic mode (
-target=all) as a single replica, which is sufficient for a training environment.
Long-Term Storage
Pyroscope stores each tenant’s profiles in on-disk blocks containing an index, metadata, and Parquet tables. Blocks are uploaded to object storage for durability.
| Backend | Use Case |
|---|---|
| Amazon S3 | Production (AWS) |
| Azure Blob Storage | Production (Azure — used in our setup) |
| Google Cloud Storage | Production (GCP) |
| OpenStack Swift | Production (OpenStack) |
| Local filesystem | Development, single-node only |
Collection Methods
| Method | Languages | Overhead | Code Changes | Description |
|---|---|---|---|---|
| eBPF | All | < 1% | None | Kernel-level sampling via Grafana Alloy |
| Pyroscope SDK | Go, Python, Java, .NET, Node.js | 1-5% | Minimal | In-process profiler with richer data |
| JFR | Java, Kotlin | 1-3% | None (agent) | Java Flight Recorder integration |
| CORECLR Profiler | .NET | 1-3% | None (agent) | .NET CLR profiling |
| Pyroscope scrape | Go (pprof) | < 1% | Annotation only | Pull-based via pod annotations |
Profile Types
| Type | What It Measures | When to Use |
|---|---|---|
| CPU | Time spent executing code | High CPU usage, slow endpoints |
| Heap (Alloc) | Currently allocated memory | Memory leaks, high RAM |
| Goroutine / Thread | Active threads/goroutines | Goroutine leaks, deadlocks |
| Mutex / Lock | Time waiting for locks | Lock contention |
| Block / I/O | Time blocked on I/O | Slow network/disk ops |
| Off-CPU | Time when thread is not on CPU | I/O waits, scheduling |
When to Use Pyroscope
- ✅ Performance optimization — find hot spots in production code
- ✅ Memory leak diagnosis — heap profile shows what holds memory
- ✅ Regression analysis — diff flame graphs before and after deployment
- ✅ Cloud cost reduction — identify inefficient code → smaller instances
- ✅ Latency debugging — trace shows slow span, profile shows why
- ❌ Does not replace traces or metrics — it’s a complementary signal