Overview

Grafana Pyroscope

Source: https://grafana.com/oss/pyroscope/

Grafana Pyroscope is an open-source continuous profiling platform that enables you to understand resource usage (CPU, memory, etc.) at the code level in production environments with minimal overhead.

Key Features

• Continuous profiling — always-on profiling in production, not just during debugging sessions
• Multiple profile types — CPU, heap/memory, goroutines, mutex contention, block/I/O, off-CPU
• Multi-language support — Go, Java, Python, .NET, Node.js, Rust, C++ (via eBPF)
• Flame graph visualization — intuitive visualization of where time and resources are spent
• Diff flame graphs — compare profiles between deployments to detect regressions
• Grafana integration — native datasource with drill-down from metrics and traces to profiles

Data Flow

┌─────────────────────────────────────────────────────┐
│                   Data Sources                       │
│                                                      │
│  ┌──────────┐  ┌──────────────┐  ┌──────────────┐  │
│  │ eBPF     │  │ Pyroscope SDK│  │ JFR / CORECLR│  │
│  │ (all     │  │ (Go, Python, │  │ (Java, .NET) │  │
│  │ languages)│  │  Node.js)    │  │              │  │
│  └─────┬────┘  └──────┬───────┘  └──────┬───────┘  │
│        │               │                 │          │
│        └───────────────┼─────────────────┘          │
│                        ▼                             │
│  ┌────────────────────────────────────────────────┐ │
│  │              Grafana Alloy                      │ │
│  │  pyroscope.ebpf → eBPF-based CPU profiling     │ │
│  │  pyroscope.scrape → SDK-based profile scraping │ │
│  │  pyroscope.write → forward to Pyroscope        │ │
│  └────────────────────┬───────────────────────────┘ │
│                        ▼                             │
│  ┌────────────────────────────────────────────────┐ │
│  │           Pyroscope Server (:4040)              │ │
│  │  Storage: Azure Blob / S3 / local disk         │ │
│  └────────────────────────────────────────────────┘ │
│                        ▼                             │
│  ┌────────────────────────────────────────────────┐ │
│  │              Grafana (:3000)                    │ │
│  │  Flame graphs, diff views, trace correlation   │ │
│  └────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘

Internal Architecture

Source: https://grafana.com/docs/pyroscope/latest/reference-pyroscope-architecture/about-grafana-pyroscope-architecture/

Pyroscope has a microservices-based architecture. All components are compiled into a single binary, and the -target parameter controls which component(s) the process runs as. This allows the same binary to operate as a monolith or as individual microservices.

Components

The Write Path

                    Profiles from clients
                            │
                            ▼
                    ┌───────────────┐
                    │  Distributor  │
                    └───────┬───────┘
                            │  routes by tenant + series
                            ▼
              ┌─────────────┼─────────────┐
              ▼             ▼             ▼
        ┌──────────┐  ┌──────────┐  ┌──────────┐
        │ Ingester │  │ Ingester │  │ Ingester │
        │ (replica)│  │ (replica)│  │ (replica)│
        └─────┬────┘  └─────┬────┘  └─────┬────┘
              │              │              │
              │   flush blocks to storage   │
              ▼              ▼              ▼
        ┌────────────────────────────────────────┐
        │         Long-Term Object Storage       │
        │    (S3 / Azure Blob / GCS / local)     │
        └────────────────────┬───────────────────┘
                             │
                             ▼
                     ┌───────────────┐
                     │   Compactor   │
                     │ merge blocks, │
                     │ deduplicate   │
                     └───────────────┘

1. Distributor receives push requests and routes each profile series to ingesters
2. Each series is replicated to 3 ingesters by default
3. Ingesters append profiles to a per-tenant database on local disk
4. In-memory profiles are periodically flushed to disk as blocks
5. Blocks are uploaded to long-term object storage
6. Compactor merges blocks from multiple ingesters into single blocks and removes duplicate samples

The Read Path

                    Query from Grafana
                            │
                            ▼
                  ┌──────────────────┐
                  │ Query-frontend   │
                  │ (split, cache)   │
                  └────────┬─────────┘
                           │
                           ▼
                  ┌──────────────────┐
                  │ Query-scheduler  │
                  │ (per-tenant      │
                  │  fair queuing)   │
                  └────────┬─────────┘
                           │
                           ▼
                  ┌──────────────────┐
                  │    Querier       │
                  └───┬──────────┬───┘
                      │          │
            recent    │          │  historical
            data      │          │  data
                      ▼          ▼
              ┌──────────┐ ┌───────────────┐
              │ Ingesters│ │ Store-gateway  │
              │ (memory) │ │ (object store) │
              └──────────┘ └───────────────┘

1. Query-frontend receives the query, splits it by time range, checks the cache
2. Query-scheduler queues the sub-queries with fair per-tenant scheduling
3. Querier picks up work and fetches data from:
   • Ingesters — for recent, in-memory data
   • Store-gateways — for historical data in object storage
4. Results are merged and returned to Grafana

Deployment Modes

💡 In this workshop we deploy Pyroscope in monolithic mode (-target=all) as a single replica, which is sufficient for a training environment.

Long-Term Storage

Pyroscope stores each tenant’s profiles in on-disk blocks containing an index, metadata, and Parquet tables. Blocks are uploaded to object storage for durability.

Collection Methods

Profile Types

When to Use Pyroscope

• ✅ Performance optimization — find hot spots in production code
• ✅ Memory leak diagnosis — heap profile shows what holds memory
• ✅ Regression analysis — diff flame graphs before and after deployment
• ✅ Cloud cost reduction — identify inefficient code → smaller instances
• ✅ Latency debugging — trace shows slow span, profile shows why
• ❌ Does not replace traces or metrics — it’s a complementary signal

Pyroscope vs Pixie

Both tools use eBPF for profiling but serve fundamentally different purposes. Pyroscope is a dedicated continuous profiling platform with long-term storage. Pixie is a real-time debugging tool that captures full network traffic and keeps data in-memory.

Feature Comparison

When to Use Which

Use Pyroscope when:

• You need to understand why code is slow — which functions consume CPU, allocate memory, or contend on locks
• You want to compare profiles between deployments (diff flame graphs)
• You need long-term profiling data for trend analysis
• You want to drill down from a trace span to see the exact code profile for that operation
• You’re profiling Java, Python, .NET, or Node.js services (SDK support)

Use Pixie when:

• You need to see full HTTP/gRPC/SQL request bodies without instrumentation
• You’re debugging network issues — TCP drops, DNS failures, retransmits
• You want instant visibility into a running cluster without deploying any agents to applications
• You’re investigating database query performance with full SQL capture
• Data must stay in the cluster for security/compliance reasons

Use both together:

• Pixie shows you which requests are slow (full request tracing with latency) → Pyroscope shows you why they’re slow (code-level CPU/memory profile)
• Pixie captures network-level symptoms, Pyroscope reveals application-level root causes