eBPF for Observability

eBPF in the Observability Stack

eBPF enables a fundamentally different approach to collecting telemetry: instead of instrumenting applications, you instrument the kernel itself. This gives you visibility into every process on a node without touching a single line of application code.

What eBPF Can Observe

Network Traffic

By hooking into syscalls like send(), recv(), connect(), and accept(), eBPF programs can:

• Capture full HTTP request/response bodies (method, path, headers, status)
• Parse database protocols (SQL queries, Redis commands, gRPC calls)
• Track TCP metrics — retransmits, drops, RTT, connection states
• Monitor DNS queries and resolution times
• Map service-to-service communication without service mesh

CPU Profiling

Using perf events, eBPF can sample CPU stack traces at regular intervals (typically ~100 Hz):

• Captures stack traces from all languages — compiled and interpreted
• Produces flame graphs showing where CPU time is spent
• Overhead is typically < 1% of CPU
• No application restart or recompilation needed

System-Level Metrics

Through tracepoints and kprobes:

• File I/O — read/write latency per file, throughput
• Disk I/O — block device latency, queue depth
• Memory — page faults, allocation patterns, OOM events
• Scheduling — run queue latency, context switches, off-CPU time
• Security — process execution, file access, network connections

eBPF-Based Observability Tools

Kubernetes Requirements

Running eBPF-based tools in Kubernetes is not a simple “deploy and go” — it has specific infrastructure and security requirements:

Node-Level Requirements

Deployment Model

eBPF tools run as DaemonSets — one pod per node, because eBPF programs attach to the kernel of the node they run on. A single pod cannot observe other nodes.

Security & Privileges

This is the most common blocker. eBPF programs need elevated privileges to load code into the kernel:

# Typical DaemonSet security context for eBPF tools
securityContext:
  privileged: true        # or use fine-grained capabilities:
  # capabilities:
  #   add: ["SYS_ADMIN", "SYS_RESOURCE", "SYS_PTRACE", "PERFMON", "BPF"]
  runAsUser: 0

⚠️ PodSecurityStandards: The Restricted profile blocks eBPF entirely. You need at least Privileged for eBPF DaemonSets. If your cluster enforces PSS, you’ll need a namespace-level exception for the monitoring namespace.

Volume Mounts

eBPF tools typically need access to host paths:

Managed Kubernetes Considerations

Common Pitfalls

• Kernel too old — eBPF features are added incrementally; a 4.x kernel may lack ring buffers, BTF, or bpf_link
• Missing BTF — without /sys/kernel/btf/vmlinux, CO-RE tools won’t start; some cloud images strip BTF to save space
• PSP/PSS/SCC blocking — security policies reject privileged pods; must whitelist the monitoring namespace
• ARM nodes — eBPF support on ARM64 is generally good since 5.x, but some tools may not ship ARM binaries
• Windows nodes — eBPF on Windows is experimental and limited; mixed clusters need nodeSelectors to schedule only on Linux nodes

eBPF vs Traditional Instrumentation

When to Use eBPF-Based Observability

• ✅ Polyglot environments — many languages, no single SDK covers all
• ✅ Legacy services — can’t modify source code
• ✅ Network debugging — TCP issues, DNS failures, inter-service traffic
• ✅ Quick wins — instant visibility without development effort
• ✅ Security monitoring — process execution, file access auditing

When SDK Instrumentation Is Better

• ✅ Business-level spans — e.g., “process order”, “validate payment”
• ✅ Custom metrics — application-specific KPIs
• ✅ Cross-cluster traces — distributed trace context propagation
• ✅ Long-term storage — export to backends with configurable retention
• ✅ Fine-grained control — custom attributes, sampling strategies

💡 Best practice: Use eBPF for baseline visibility (network, CPU, system) and complement with OpenTelemetry SDK instrumentation for application-specific observability. The two approaches are complementary, not competing.

eBPF in This Workshop

In this workshop, you will encounter eBPF in two contexts:

1. Pixie — uses eBPF to auto-trace protocols, profile CPU, and monitor network traffic across the cluster
2. Pyroscope (via Grafana Alloy) — uses eBPF for continuous CPU profiling with pyroscope.ebpf component running as a privileged DaemonSet