Shaping Telemetry Before It Costs You: OTTL in the OpenTelemetry Collector

TL;DR: The OpenTelemetry Collector transforms telemetry - metrics, logs, traces, profiles - with no changes to application code. The language for that is OTTL, and it’s how you enrich attributes, redact PII, and control cardinality before data ever reaches a backend (and a bill). The catch is that OTTL fails silently: wrong context, wrong type, missing where, wrong statement order - none of them error, they just quietly do nothing or quietly cost CPU. This post covers the three jobs OTTL is actually for, the cache[] trick that makes log parsing click, the pipeline order that isn’t arbitrary, and the traps that make it feel broken.

---

Most observability advice is about getting more data out. The problem you actually hit in production is the opposite: you have too much of it, some of it is a compliance liability, and a chunk of your bill is one high-cardinality label somebody added without thinking. The place to fix all three is the same place, and it isn’t your application code.

Why shape telemetry at all

The OpenTelemetry Collector sits between your instrumented applications and your backends (Grafana, Jaeger, Prometheus, Dynatrace - anything that speaks OTLP). Its whole value proposition is that the filtering, enrichment, PII redaction and cardinality control all happen in the Collector, not in your services. You don’t redeploy twelve microservices to drop a noisy attribute; you change one Collector config.

A Collector pipeline is three parts:

Receiver → [Processor → Processor → ...] → Exporter

and each signal - traces, metrics, logs, profiles - gets its own pipeline. The receivers take data in (OTLP, filelog, and many more), the exporters send it on, and the interesting work happens in the processors in the middle. Two of those processors - transform and filter - are driven by OTTL.

OTTL in one screen

OTTL - the OpenTelemetry Transformation Language - is an expression language built into the Collector for manipulating telemetry in flight. A statement looks like this:

editor(target, value) where condition

There are two kinds of function. Editors change data - set, delete_key, delete_matching_keys, replace_pattern, merge_maps, limit, keep_keys. Converters are pure functions that compute a value - ConvertCase, SHA256, ParseJSON, ParseKeyValue, IsMatch, Concat, Substring, and the type casts Int / Double / String.

The one structural thing to internalise is context. OTTL operates on a hierarchy:

resource → scope → span | datapoint | log | profile

The resource context sees resource attributes (service.name, k8s.pod.name). The span context sees span fields (name, duration, attributes, events). The log context sees a log’s body, severity and attributes. And here’s the first trap: accessing a span field from the log context silently does nothing - no error, no effect. More on that below, because it’s the thing that will waste your afternoon.

The three jobs OTTL is actually for

1. Enrichment

Fill in what instrumentation left out:

- set(resource.attributes["service.name"], resource.attributes["k8s.deployment.name"])
    where resource.attributes["service.name"] == nil
- set(resource.attributes["deployment.environment"], "production")
    where IsMatch(resource.attributes["k8s.namespace.name"], "^prod")
- set(attributes["http.method"], ConvertCase(attributes["http.method"], "upper"))
- set(attributes["slow_request"], true) where duration > 5_000_000_000

2. PII redaction

The compliance job - and the one you want running before data leaves your boundary:

- replace_pattern(body, "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}", "REDACTED_EMAIL")
- replace_pattern(body, "\\d{4}[- ]?\\d{4}[- ]?\\d{4}[- ]?\\d{4}", "REDACTED_CC")
- set(attributes["user.id"], SHA256(attributes["user.id"]))
- delete_matching_keys(attributes, "(?i)(authorization|cookie|password|token|secret)")

SHA256 on a user ID is the neat one: the same input always hashes to the same output, so you keep the ability to correlate a session without ever exposing who the user is.

3. Cardinality control

The one that quietly owns your bill. Every unique label combination is a time series; an unbounded ID in a metric label is how you end up with millions of them:

# /api/user/123456 → /api/user/{id}
- replace_pattern(attributes["http.target"], "/[0-9]+", "/{id}")
# 200, 201, 204 → 2xx
- set(attributes["http.status_bucket"],
    Concat([Substring(String(attributes["http.status_code"]), 0, 1), "xx"], ""))

The cache[] trick

This is the pattern that makes OTTL click for logs. cache[] is a temporary map that lives for the duration of the transform block - the scratch space for multi-step work, most importantly turning a JSON blob in a log body into real attributes:

- set(cache["decoded"], String(body))
- replace_pattern(cache["decoded"], "[email regex]", "REDACTED")
- merge_maps(attributes, ParseJSON(cache["decoded"]), "upsert")
- set(attributes["severity_text"], attributes["level"])
- delete_key(cache, "decoded")

Step by step: copy the body into cache as a string, redact PII in the cache copy, parse the cache as JSON and merge it into attributes, promote a field, then clean the cache up. Redact-then-parse, not parse-then-redact - so the sensitive data is gone before it becomes structured and queryable.

Pipeline order is not arbitrary

The order of processors is a correctness property, not a style choice:

enrich → clean → parse → filter → cardinality

Step	Why it goes here
Enrich first	filter, clean and cardinality steps may depend on enriched attributes; without enrichment a filter can’t match deployment.environment
Clean	less data to carry through everything downstream
Parse	turns a JSON blob into attributes so filters can operate on parsed fields
Filter	drop whole records (health checks, debug logs) before the expensive work
Cardinality	run only on the data that survived - smaller CPU cost

Get this backwards - filter after cardinality, say - and you spend CPU normalising URLs on records you’re about to throw away.

graph LR
    A["Receiver<br/>(OTLP / filelog)"] --> B["enrich<br/>fill missing attrs"]
    B --> C["clean<br/>redact PII"]
    C --> D["parse<br/>JSON body → attrs"]
    D --> E["filter<br/>drop noise"]
    E --> F["cardinality<br/>normalise labels"]
    F --> G["Exporter<br/>(OTLP → backend)"]

The silent failure modes

This is the section that turns OTTL from “looks like docs” into “won’t cost me an afternoon.” Almost every OTTL mistake fails quietly:

• Type mismatch is skipped silently. An int where a string is expected doesn’t error - the statement is just ignored. Different instrumentation libraries send the same field as different types, so always cast with Int(), Double(), String() when the type is uncertain.
• Wrong context = quiet nothing. span.name in the log context will never do anything, and never tell you why.
• No where = CPU on every record. A regex with no condition runs against every single record, even ones whose body contains nothing to match. where isn’t just correctness, it’s cost.
• Statement order matters. An attribute you deleted in step 1 is unavailable in step 2. Statements run sequentially, not as a set.

And choose your error mode deliberately:

Mode	Behaviour	When
ignore	log the error, keep processing	production
silent	keep going, no logging	production (low noise)
propagate	drop the whole record, return the error	dev/staging - catch everything

Testing before you deploy

You do not iterate OTTL in production. The workflow:

1. ottl.run - a browser playground running a real Collector compiled to WASM. Prototype against sample data with before/after visible.
2. otelcol validate --config pipeline.yaml - catches syntax errors (missing quotes, wrong function names, bad paths) before deploy.
3. The debug exporter - add it to the pipeline and the Collector prints transformed data to stdout, so you see the real post-transform records.
4. Then deploy.

And monitor the Collector itself - it emits metrics for records accepted, refused, dropped and exported per pipeline. Watch otelcol_processor_dropped_* especially: a sudden jump usually means an OTTL filter or a pipeline is doing something you didn’t intend.

In Kubernetes

The usual shape is a DaemonSet collector on each node (tails logs with filelog, does light preprocessing, forwards OTLP) feeding a gateway Deployment or StatefulSet that runs the full OTTL pipeline and exports to the backend. The k8sattributes processor is what enriches records with pod, namespace, deployment and container metadata - which is exactly the enrichment your later filters lean on. If you run the OpenTelemetry Operator, the Collector becomes a CRD and the operator manages the DaemonSet/Deployment lifecycle for you.

And a fourth signal on the way

Everything above is about the three mature signals - metrics, logs and traces. OpenTelemetry has a fourth: profiling, which answers how the code behaves at runtime - which function eats 40% of the CPU. It flows through the same Collector, but it has its own data model, its own two-collector eBPF architecture in Kubernetes, and - importantly for this post - only partial OTTL support so far. Because shaping profiles is a different enough problem, I’ve given it its own writeup: OpenTelemetry profiling, the fourth signal.

The point

Instrumentation decides what data exists. The Collector decides what data survives - and OTTL is the language of that decision. Enrich so your filters have something to match, redact before anything sensitive becomes queryable, cut cardinality last on the records that made it through, and remember that OTTL will let you write something that does nothing at all without a single complaint. Prototype in ottl.run, validate before deploy, and watch the dropped-records metric like it’s a smoke alarm.