OpenTelemetry Profiling: eBPF, the Data Model, OTTL

TL;DR: Profiling is OpenTelemetry’s fourth signal, next to metrics, logs and traces. It answers a question the others can’t - which function is actually eating the CPU - by sampling the call stack at a fixed rate. It has its own index-based data model (about 90% smaller than storing raw strings), a two-collector eBPF architecture in Kubernetes, and a real sustainability angle: wasted CPU at cluster scale is wasted energy. It’s also still marked unstable, and OTTL can barely touch it yet. This is what it is, how it’s wired, and where the sharp edges are.

I wrote separately about shaping telemetry with OTTL in the Collector - enriching, redacting, cutting cardinality across the three mature signals. This post is about the one that’s still arriving. If metrics, logs and traces tell you whether, what and where, profiling tells you how the code behaves while it runs.

What each signal is actually for

The four signals aren’t redundant; each answers a question the others can’t:

Signal	Answers	Example question
Metrics	Is the system healthy / performant?	Is latency within bounds?
Logs	What specifically happened?	What was the error for request X?
Traces	Where is the time and where are the errors?	Which service is the bottleneck?
Profiles	How does the code behave at runtime?	Which function is burning 40% of the CPU?

A trace narrows a slow request to a service. A profile tells you it’s a regex compiled inside a hot loop. That’s the gap it fills. Technically it’s stack sampling - a sampler captures the current call stack at a regular interval (say 100 Hz), producing a timestamped series of samples.

The angle that makes it more than a debugging tool

Profiling used to be something you switched on during an incident and switched off after, because it was expensive. Continuous profiling dropped that cost enough to leave it on in production permanently - and that unlocks an argument beyond debugging: sustainability. In a Kubernetes cluster, cumulative CPU waste is a real carbon cost. Profiling makes that waste measurable, which makes it fixable. That’s a reason to run it even when nothing is on fire - not “which function is slow during this incident” but “which function is quietly costing us CPU, and money, and emissions, every hour of every day.”

The data model, and why it’s index-based

OpenTelemetry deliberately did not reuse pprof (Go’s format) - it’s too Go-centric and carries no trace context. Instead it designed its own model, built around index tables to keep the size down:

ProfileData
├── resource attributes   (service.name, pod, node - as in every signal)
└── Profile
    ├── start / end timestamp
    ├── samples[]
    │   ├── sample_type (CPU | memory)
    │   ├── attributes
    │   ├── measured values
    │   └── trace context (trace_id, span_id)
    ├── mapping table   (memory addresses)
    ├── location table  (stack levels; index 0 = top of stack)
    ├── function table  (file ID, function ID, line number)
    └── string table    (human-readable names for every index)

Why the indirection? A sample doesn’t store the string "handleRequest" a thousand times - it stores an index into the string table. 100 samples with full strings run about 500 KB; the same samples with indices run about 50 KB. That’s roughly a 90% reduction, which is the difference between “continuous profiling in production” being viable and not.

Note the trace_id and span_id on each sample: in principle you can navigate from a trace to the profile that captured it and back. In practice, context propagation is still being refined by the OTel Profiling SIG - one of several “coming soon” items.

Where the profiles come from

Per-language, the tooling already exists - Go has pprof, Java has JFR and async-profiler, Python has py-spy and cProfile, Node has its CPU profiler, Rust has its own options. But the interesting path is eBPF, which samples stacks at the kernel level with no changes to application code - language-agnostic, whole-node coverage.

The Kubernetes architecture: two collectors

This is the part that trips people up. Profiling needs two collectors, not one:

graph LR
    subgraph node["each node"]
      A["eBPF profiler<br/>(DaemonSet)"]
    end
    A -->|OTLP| B["gateway collector<br/>(Deployment/StatefulSet)<br/>service.profile.support=true"]
    B --> C["backend<br/>(Pyroscope / Grafana / Dynatrace)"]

The eBPF collector runs as a DaemonSet, one per node. It’s deliberately minimal: the only receiver is profiling, the only exporter is OTLP, and the processors are just enough to attach metadata - resource detection, k8s attributes, resource, batch, memory limiter. It has one knob worth knowing, samples_per_second.

The gateway collector is a normal Deployment or StatefulSet, but it must have the feature gate service.profile.support=true set - without it the collector won’t even start the profiles pipeline. It receives profiles over OTLP, runs whatever transform/filter/export you configure, and ships to a backend that speaks OTLP for profiles (Pyroscope, Grafana, Dynatrace).

service:
  feature_gates:
    service.profile.support: true

The honest part: OTTL barely touches profiles yet

If you read the OTTL post, you know the whole point of the Collector is shaping data in flight. For profiles, as of mid-2026, you mostly can’t:

Operation	Available?
Access resource attributes	✅
Access profile-level attributes	✅
Filter and basic attribute transforms	✅
Iterate over samples	❌
Read function names from the function table	❌
Aggregate sample values	❌
Distinguish a CPU profile from a memory one	❌

In other words: you can filter and enrich at the resource/profile level, but the transformations that would actually be useful on profile data - anything that reaches into the samples - aren’t implemented in the OTTL profile context yet. The video author’s workaround is telling: an experimental profile-to-metrics connector that implements the logic directly in Go against the profile data structure, precisely because OTTL can’t express it. When your escape hatch is “drop to Go,” the language support isn’t there.

So should you turn it on?

A measured yes, with eyes open:

It’s still unstable. The API can change. Treat it as something you pilot, not something you build a compliance process around.
Continuous profiling’s overhead is claimed low, not proven here. The sustainability pitch assumes the cost of always-on profiling is small; the source asserts this without overhead numbers. Measure it in your own cluster before you commit to always-on.
The two-collector setup is real operational surface. An eBPF DaemonSet on every node plus a feature-gated gateway is more to run and monitor than adding a processor to an existing pipeline.
Backends are still catching up. You need one that speaks OTLP for profiles; the list is growing but shorter than for the other signals.

The trajectory is clearly upward - the SIG is actively working on richer OTTL for profiles, better trace↔profile correlation, better compression, and more backends. The fourth signal is coming. It’s just worth being precise about the fact that, today, it’s the one signal you can collect far more easily than you can shape.