Client Instrumentation

Instrumenting Applications with Pyroscope

Source: https://grafana.com/docs/pyroscope/latest/configure-client/

There are three ways to send profiling data to Pyroscope. Each approach offers different trade-offs between setup effort, data richness, and language support.

Instrumentation Methods

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    Three Ways to Send Profiles                        β”‚
β”‚                                                                       β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚  1. Grafana      β”‚  β”‚  2. SDK Direct   β”‚  β”‚  3. SDK via Alloy   β”‚  β”‚
β”‚  β”‚     Alloy        β”‚  β”‚     Push         β”‚  β”‚     (hybrid)        β”‚  β”‚
β”‚  β”‚                  β”‚  β”‚                  β”‚  β”‚                     β”‚  β”‚
β”‚  β”‚  No code changes β”‚  β”‚  SDK in app code β”‚  β”‚  SDK in app code    β”‚  β”‚
β”‚  β”‚  eBPF / pull     β”‚  β”‚  pushes directly β”‚  β”‚  pushes to local    β”‚  β”‚
β”‚  β”‚  mode profiling  β”‚  β”‚  to Pyroscope    β”‚  β”‚  Alloy, which       β”‚  β”‚
β”‚  β”‚                  β”‚  β”‚                  β”‚  β”‚  forwards to        β”‚  β”‚
β”‚  β”‚                  β”‚  β”‚                  β”‚  β”‚  Pyroscope          β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚           β”‚                     β”‚                        β”‚            β”‚
β”‚           β–Ό                     β–Ό                        β–Ό            β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚                     Pyroscope Server                            β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Method 1 β€” Auto-instrumentation with Grafana Alloy

Grafana Alloy

Grafana Alloy collects profiles without any code changes. It supports:

  • eBPF profiling β€” CPU profiles from natively compiled languages (C/C++, Go, Rust, Zig) and high-level languages (Java, .NET, Python, Ruby, PHP, Node.js, Perl)
  • Pull mode β€” scrapes pprof endpoints from Go and Java applications

Best for: quick setup, polyglot environments, when you can’t modify application code.

Method 2 β€” Direct SDK Instrumentation

Install a language-specific Pyroscope SDK in your application. The SDK automatically pushes profiles periodically to Pyroscope server.

Best for: maximum control over profiling configuration, richer profile types (mutex, block, allocations), dynamic labels.

Method 3 β€” SDK via Alloy (Hybrid)

Applications instrumented with SDKs send profiles to Alloy’s pyroscope.receive_http component, which forwards them to Pyroscope. Benefits:

  • Lower latency β€” local Alloy instance vs direct internet connection
  • Centralized metadata β€” Alloy adds infrastructure labels (node, namespace, pod)
  • Buffering & retry β€” Alloy handles transient failures

Best for: production environments where you want SDK-level data with infrastructure-level enrichment.

Choosing the Right Method

Factor Alloy (eBPF / pull) SDK Direct SDK via Alloy
Code changes None SDK integration SDK integration
Setup effort Low (DaemonSet) Medium (per-app) Medium-High
Profile types CPU only (eBPF) CPU, heap, mutex, block, goroutines CPU, heap, mutex, block, goroutines
Dynamic labels No Yes Yes
Metadata enrichment Yes (K8s labels) Manual Yes (Alloy adds infra labels)
Failure handling Built-in App-side Alloy handles retries

Supported Language SDKs

Go Java Python .NET Node.js Ruby Rust eBPF

Language Package Profile Types Labels
Go github.com/grafana/pyroscope-go CPU, heap, goroutines, mutex, block pyroscope.TagWrapper()
Java io.pyroscope:agent CPU (itimer/cpu/wall), alloc, lock Pyroscope.LabelsWrapper
Python pyroscope-io CPU pyroscope.tag_wrapper()
.NET Pyroscope (NuGet) CPU, wall, alloc, lock, exceptions, heap Pyroscope.LabelsWrapper.Do()
Node.js @pyroscope/nodejs CPU, wall, heap Pyroscope.wrapWithLabels()
Ruby pyroscope-beta CPU Tags via config
Rust pyroscope + pyroscope-pprofrs CPU Tags via config
eBPF Grafana Alloy pyroscope.ebpf CPU Kubernetes labels

SDK Examples by Language

Go

import "github.com/grafana/pyroscope-go"

func main() {
  runtime.SetMutexProfileFraction(5)
  runtime.SetBlockProfileRate(5)

  pyroscope.Start(pyroscope.Config{
    ApplicationName: "my-go-app",
    ServerAddress:   "http://pyroscope-server:4040",
    Tags:            map[string]string{"hostname": os.Getenv("HOSTNAME")},
    ProfileTypes: []pyroscope.ProfileType{
      pyroscope.ProfileCPU,
      pyroscope.ProfileAllocObjects,
      pyroscope.ProfileAllocSpace,
      pyroscope.ProfileInuseObjects,
      pyroscope.ProfileInuseSpace,
      pyroscope.ProfileGoroutines,
      pyroscope.ProfileMutexCount,
      pyroscope.ProfileMutexDuration,
      pyroscope.ProfileBlockCount,
      pyroscope.ProfileBlockDuration,
    },
  })
}

Dynamic labels β€” tag specific code paths:

pyroscope.TagWrapper(context.Background(),
  pyroscope.Labels("controller", "slow_controller"),
  func(c context.Context) {
    slowCode()
  })

Java

Option A β€” from code (e.g. Spring Boot @PostConstruct):

PyroscopeAgent.start(
  new Config.Builder()
    .setApplicationName("my-java-app")
    .setProfilingEvent(EventType.ITIMER)
    .setFormat(Format.JFR)
    .setServerAddress("http://pyroscope-server:4040")
    .build()
);

Option B β€” as javaagent (no code changes):

export PYROSCOPE_APPLICATION_NAME=my.java.app
export PYROSCOPE_SERVER_ADDRESS=http://pyroscope-server:4040

java -javaagent:pyroscope.jar -jar app.jar

Key Java configuration:

Variable Default Description
PYROSCOPE_PROFILING_INTERVAL 10ms CPU sampling interval
PYROSCOPE_FORMAT collapsed Use jfr for multiple profile types
PYROSCOPE_PROFILER_EVENT itimer CPU event: itimer, cpu, wall
PYROSCOPE_PROFILER_ALLOC disabled Allocation threshold (e.g. 512k)
PYROSCOPE_PROFILER_LOCK disabled Lock threshold (e.g. 10ms)
PYROSCOPE_UPLOAD_INTERVAL 10s How often to send data

Dynamic labels in Java:

Pyroscope.LabelsWrapper.run(
  new LabelsSet("controller", "slow_controller"),
  () -> { slowCode(); }
);

Python

import pyroscope

pyroscope.configure(
  application_name = "my-python-app",
  server_address   = "http://pyroscope-server:4040",
  sample_rate      = 100,
  oncpu            = True,
  gil_only         = True,
  tags             = {
    "region": os.getenv("REGION"),
  }
)
Parameter Default Description
sample_rate 100 Samples per second
oncpu True Report only CPU time (vs wall time)
gil_only True Only profile GIL-holding threads

Dynamic labels in Python:

with pyroscope.tag_wrapper({"controller": "slow_controller"}):
    slow_code()

.NET

.NET profiling uses a native CLR profiler β€” no code changes needed, only environment variables:

PYROSCOPE_APPLICATION_NAME=my-dotnet-app
PYROSCOPE_SERVER_ADDRESS=http://pyroscope-server:4040
PYROSCOPE_PROFILING_ENABLED=1
CORECLR_ENABLE_PROFILING=1
CORECLR_PROFILER={BD1A650D-AC5D-4896-B64F-D6FA25D6B26A}
CORECLR_PROFILER_PATH=/dotnet/Pyroscope.Profiler.Native.so
LD_PRELOAD=/dotnet/Pyroscope.Linux.ApiWrapper.x64.so

Available profile types (all toggled via environment variables):

Variable Default Description
PYROSCOPE_PROFILING_CPU_ENABLED true CPU profiling
PYROSCOPE_PROFILING_WALLTIME_ENABLED false Wall time profiling
PYROSCOPE_PROFILING_ALLOCATION_ENABLED false Memory allocation profiling
PYROSCOPE_PROFILING_LOCK_ENABLED false Lock contention profiling
PYROSCOPE_PROFILING_EXCEPTION_ENABLED false Exception profiling
PYROSCOPE_PROFILING_HEAP_ENABLED false Live heap profiling (.NET 7+)

Dynamic labels in .NET:

var labels = Pyroscope.LabelSet.Empty.BuildUpon()
    .Add("controller", "slow_controller")
    .Build();
Pyroscope.LabelsWrapper.Do(labels, () =>
{
  SlowCode();
});

Node.js

const Pyroscope = require('@pyroscope/nodejs');

Pyroscope.init({
  serverAddress: 'http://pyroscope-server:4040',
  appName: 'my-node-app',
  tags: {
    region: 'eu-west-1',
  },
});

Pyroscope.start();
Parameter Default Description
flushIntervalMs 60000 How often to send profiles (ms)
heapSamplingIntervalBytes 524288 Bytes between heap samples
wall.CollectCpuTime false Enable CPU time profiling

Dynamic labels in Node.js:

Pyroscope.wrapWithLabels({ controller: 'slow_controller' }, () =>
  slowCode()
);

Tag and Label Rules

Tags (labels) allow filtering profiles in Grafana. All Pyroscope SDKs follow the same rules:

  • Valid characters: ASCII letters, digits, underscores ([a-zA-Z_][a-zA-Z0-9_])
  • Periods (.) are not valid in tag names
  • Two types of labels:
    • Static β€” set at initialization, apply to all profiles (e.g. region, hostname)
    • Dynamic β€” set per code block using language-specific wrappers (e.g. controller, endpoint)

πŸ’‘ Tip: Use dynamic labels to tag specific endpoints or controllers. This lets you filter flame graphs to see exactly which code paths are expensive for a given route.

results matching ""

    No results matching ""