Five Container-Image Corrections DevOps Keeps Getting Wrong

TL;DR: Five container-build habits worth re-examining. Alpine vs slim is a libc decision, not a byte count. Layer order is your cache strategy - and the anti-pattern isn’t COPY . ., it’s shipping junk into the build context. Multi-stage + distroless beats a full OS and is friendlier than scratch. “One process per container is a vibe, not a law” - true below Kubernetes scale, but only if you know what observability you’re trading away. Pin digests, not tags - and automate the bumps, or you trade “tag moved” for “image rots.” Bonus: why pasting a build error into an LLM gives you a correct answer to the wrong question.

---

There’s an old story about five monkeys in a cage with a ladder and a banana at the top. Every time one climbs, all five get sprayed with cold water. Soon none of them climb. Swap out a monkey and the newcomer, reaching for the ladder, gets beaten by the other four before the water ever comes. Keep swapping until no original monkey remains, and the group still attacks anyone who tries - none of them knowing why. They just know: we don’t climb the ladder here.

Container building is full of ladders nobody climbs anymore. “Use Alpine, it’s smaller.” “Never write COPY . ..” “One process per container.” Some of these were true once, some were never quite true, and most get repeated because that’s how we do it here - which Grace Hopper called the most dangerous phrase in the language. The container is the smallest unit of modern infrastructure, so a bad habit doesn’t cost you once; it costs you across thousands of builds a day, hitting build time, stability, and your cloud bill at the same moment.

Here are five corrections - each with the demo numbers that justify it, and each with the caveat the confident version leaves out.

Correction 1 - Alpine vs slim is a libc argument, not a size argument

The pitch for Alpine is always the image size. That’s the wrong axis to optimise on, and it hides the real difference: Alpine uses musl libc; Debian, Ubuntu and Fedora use glibc - and the overwhelming majority of precompiled upstream releases are built against glibc.

This matters even if you never write a line of C. Python, Node and Go all bottom out in C-level system calls (open, read, write), so native extensions and prebuilt binaries are bound to a specific libc. On Alpine the consequence is one of three: the package has no matching binary distribution, it compiles from source on every build, or it silently runs slower.

One package demonstrates all three. duckdb - not an exotic C binding, but the database half the data-tooling world now reaches for by reflex - on Python/Alpine dies like this:

$ docker run --rm python:3.14-alpine pip install --only-binary=:all: duckdb
ERROR: Could not find a version that satisfies the requirement duckdb (from versions: none)
ERROR: No matching distribution found for duckdb

from versions: none is the tell. There is no version problem: DuckDB 1.5.5 publishes ten Linux wheels - manylinux, x86_64 and aarch64, CPython 3.10 through 3.14. Not one musllinux. To Alpine, the package simply does not exist. Change one word in that command, alpine to slim, and it finishes in about five seconds with a 21 MB wheel and no compiler in sight.

The --only-binary=:all: flag is doing deliberate work there: it forbids the source fallback, which is what makes the failure honest. Drop it and Alpine does something worse than fail - it quietly pulls DuckDB’s 18 MB source tarball and starts compiling C++. Add the toolchain the internet will tell you to add, and that path does eventually finish:

$ apk add build-base cmake ninja && pip install duckdb
Successfully built duckdb        # 800 seconds, on ten cores

Thirteen minutes and twenty seconds, against five seconds for the wheel. Roughly 160× - and it succeeds, which is precisely what makes it dangerous. The price of avoiding it is 128 MB of base image: python:3.14-alpine is 77 MB, python:3.14-slim is 205 MB. For a build that runs hundreds of times a day, that trade is not close.

The caveat the confident version skips: “Alpine has no binary wheels” was fully true before PEP 656 and the musllinux wheel tag. Today a good part of the ecosystem does publish musl wheels - numpy, pandas, grpcio, lxml, cryptography, and as of 2.9.12 psycopg2-binary, which for years played exactly the role DuckDB plays above and would now quietly disprove the point. Coverage is still thinner than manylinux and the gaps are unpredictable, so the practical rule holds - but treat it as ecosystem coverage, not a law of physics, and expect this paragraph to age. Check per-dependency rather than assuming globally; one pip install --only-binary=:all: against an Alpine image settles it in seconds.

Correction 2 - Layer order is your cache strategy

Docker reuses a layer as long as nothing earlier changed. That single sentence contains both the mechanism and the mistake.

If you copy source code before installing dependencies, editing one character of code invalidates the dependency-install layer and everything below it. Every build is a cold build, even though your dependencies haven’t moved in weeks. Think of it as an onion: the deeper the change, the more layers you have to peel and rebuild. So things that change rarely belong deep (early in the Dockerfile), and things that change constantly belong on the outside.

WORKDIR /app

COPY package.json package-lock.json ./   # changes rarely
RUN npm ci                                # this layer should stay cached
COPY . .                                  # changes on every commit
CMD ["npm", "start"]

A code change now invalidates only the last two layers. npm ci drops to zero time because it hits the cache. (npm ci over npm install, by the way, is its own recommendation - a deterministic install from the lockfile, correct both in CI and in image builds.)

Here’s the reframing that actually matters, though. People see COPY . ., reflexively call it an anti-pattern, and replace it with a mile-long list of individual COPY lines to hand-pick files. That hurts readability, breaks every time someone adds a file, and doesn’t fix the real problem - the build context is transferred in full regardless.

The anti-pattern isn’t COPY . .. The anti-pattern is shipping junk into the build context.

The fix works exactly like .gitignore:

node_modules
.git
*.log
dist
*.tmp

In the demo, 50 MB of synthetic bloat (generated with dd) vanishes from the transfer the moment .dockerignore exists - the build reports a near-empty context. COPY . . stays put, and the Dockerfile stays readable. Many teams either don’t know the file exists or treat it as some obscure feature nobody uses. It’s the cheapest win in the entire Dockerfile.

Correction 3 - Multi-stage, and choosing the final base

If your build produces a self-contained artifact, shipping a whole operating system alongside it is pure waste. The rule fits in one line: builder stage is the bloat zone, final stage is runtime only.

The first stage - conventionally named builder - gets the full project context, the toolchain, and the dev dependencies. The final stage copies only the artifact out of it. The Go progression from the demo is dramatic:

Approach	Size
Go on Alpine (full runtime)	272 MB
Multi-stage, copy the binary only	11 MB
FROM scratch + binary	2.3 MB

FROM golang:1.26 AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /app ./cmd/server

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=builder /app /app
ENTRYPOINT ["/app"]

But scratch is double-edged, and this is where I’d slow you down. distroless is to scratch what slim is to Alpine: a touch bigger, noticeably less painful. It still ships no shell and no package manager (so the attack surface stays small), but it carries the runtime basics scratch doesn’t - CA certificates, timezone data, a real /etc/passwd - and a :nonroot tag that runs as UID 65532. Take the tag: the plain one still defaults to root, so the “safer by default” reputation is only half-earned. The difference shows up on incident day: in an image with no CA certs, TLS connections fail in ways that are hard to diagnose, and with no shell you can’t exec in to look.

The caveat the demo skips: that 2.3 MB number is Go-specific. Go and Rust compile to standalone binaries, so the road to scratch is short. Python and Node have to carry their runtime with the app, so distroless or scratch is much harder there - and the source doesn’t show how. If you take one thing from this section: compiled language → always multi-stage; interpreted language → multi-stage still helps, but slim is usually your floor.

Correction 4 - “One process per container is a vibe, not a law”

This is the contrarian one, and it deserves its asterisk stated loudly.

The dogma is well-founded. One process per container gives you a single lifecycle the orchestrator can reason about, granular scaling (scale the bottleneck, not the pair), clean stdout/stderr, and clean SIGTERM delivery on shutdown. Good reasons, all of them.

The counter-argument: you have a small Python server that, for whatever reason, needs to sit behind nginx, and you are not at Kubernetes scale. The dogma says two containers. The pragmatic answer is that below that scale, the cost of that complexity can exceed the benefit - “the one-process dogma costs more complexity than it saves sometimes.” You run both under supervisord, a process-control system that acts as a watchdog: define your programs in supervisord.conf, set auto-start and auto-restart, copy your code plus the nginx and supervisor configs into the image. It works without fireworks - nginx on port 80, proxying the app on 8000.

The argument is operational, not technical. And it’s fair. But the source sells it as a clean win, and it isn’t - here’s what you’re trading:

• The orchestrator goes blind to failures. If supervisord quietly restarts a crashed backend, Kubernetes or Docker never sees it: the health check passes, the restart doesn’t appear in metrics, RESTARTS stays at zero. You lose the signal exactly when you need it most.
• PID 1 and zombie processes. The PID 1 process is responsible for reaping orphaned children. supervisord handles this; a naive CMD ["./start.sh"] backgrounding two processes with & does not.
• Signal propagation. SIGTERM has to reach both processes for a graceful shutdown. That has to be configured deliberately, or a deploy ends by killing live connections.
• Scaling stops being granular. Scale the backend and you scale nginx too - usually harmless, occasionally costly.
• In Kubernetes the problem is already solved. A sidecar in the same Pod gives you shared networking and lifecycle without breaking the dogma - which is why the advice self-limits to “when you’re not at K8s scale.”

So: below Kubernetes, two tightly-coupled processes under supervisord is a legitimate trade, not a sin - as long as you export supervisor’s state so those restarts don’t become invisible. In Kubernetes, use a sidecar. And never run two processes by hand with a & in a start script; use supervisor, or tini/dumb-init, or separate containers.

Correction 5 - Digests over tags

A tag like node:26-slim usually gives you what you expect - but tags can be moved, by accident or on purpose, and not just latest or nightly. Versioned tags move too.

docker buildx imagetools inspect node:26-slim

returns an immutable SHA digest. Written into the Dockerfile, it pins the version permanently instead of relying on the goodwill of whoever publishes the image:

FROM node:26-slim@sha256:abc123...

The risk is distinctly higher for internal images than for official ones - internal registries see far more accidental tag reuse.

The caveat the source sells past: a pinned digest is immutable, which is its virtue and its flaw. A pinned image never receives a CVE patch until someone bumps the hash. Pinning without automation (Renovate, Dependabot) simply trades the risk “the tag moved under me” for the risk “my image is quietly rotting.” Pin and automate the bumps, or you’ve made things worse.

A sidebar: when the AI gives a correct answer to the wrong question

There’s a thread running under all five of these that’s worth pulling out on its own, because it’s the most useful thing here for anyone leaning on a coding agent.

The source puts it bluntly - trusting an LLM to build or refine a container from scratch is “bad, very bad” - and the mechanism is more instructive than a plain “it hallucinates,” because the model’s advice is formally correct:

1. Your Alpine build fails because a dependency has no musl wheel.
2. You paste No matching distribution found for duckdb into the model.
3. It replies apk add build-base cmake ninja - and it works. The build goes green.
4. The error is gone from view, and a thirteen-minute compile has replaced a five-second download in every pipeline that runs this image.

That isn’t a hallucination. It’s a correct answer to a badly-framed question. The model was handed an error message and it removed the error message. It was never given the context that the right decision is to change the base image, because the question was “fix this error,” not “is this base image right?”

The general pattern, worth keeping well beyond Docker: an agent optimises the signal you show it. A green build is an easy signal to optimise; build time and cost-at-scale are not, because they aren’t in the context. The risk isn’t the agent that’s wrong - it’s the agent that passes exactly the test you set it.

Enforcing this: droast

Most of the above is mechanical enough to lint. droast is a Rust Dockerfile linter that “roasts” your file - roughly 85 checks across security, reproducibility, performance and maintainability, printed as info/warn/error lines: npm install where npm ci belongs, a COPY . . with no .dockerignore, an unpinned base, and so on. It reads a droast.toml, emits GitHub annotations, JSON or SARIF for CI, and has a --no-roast mode for thinner skins. Put it in the pipeline and most of these corrections enforce themselves.

The decision rules

graph TD
    A["Choosing a base image"] --> B{Compiled to a<br/>standalone binary?}
    B -->|"Go / Rust"| C{Needs TLS,<br/>timezones, /etc?}
    C -->|no| D["FROM scratch<br/>2-3 MB"]
    C -->|yes| E["distroless<br/>small + safe defaults"]
    B -->|"Python / Node"| F{Verified musl<br/>wheels exist?}
    F -->|no| G["slim (glibc)<br/>the safe default"]
    F -->|yes| H["Alpine, measured<br/>per-dependency"]

Rule	Why
Default to slim, not Alpine	musl breaks precompiled binaries; only choose Alpine after verifying wheels
If you’re on Alpine, don’t paper over it with build-base	needing the toolchain is a signal you picked the wrong base
Compiled language → always multi-stage	for Go/Rust, skipping it wastes an order of magnitude
Final base: distroless	scratch only for fully static binaries with no TLS/tz needs
.dockerignore before any other optimisation	cheapest win; keep COPY . . readable
Pin digests and automate the bumps	one without the other is worse than a tag
Measure before you believe	every number here came off one machine and one dependency profile; yours will differ

Applied here, honestly

The only fair way to end a list like this is to run it against my own image. This site ships as a container - an Astro build in a node:22-bookworm-slim builder, dist/ copied into nginx - and the audit came back uneven.

Correction	Status on this site
slim over Alpine	already there. sharp and esbuild ship glibc binaries; musl would mean compiling both on every build
layer order + .dockerignore	already there. Lockfile and npm ci sit above COPY . ., and node_modules never enters the context - the host copy holds macOS binaries that would poison a linux build
multi-stage	already there. Node, npm and the toolchain stay in the builder; the runtime carries dist/ and two config files
distroless final base	no - and it stays no. ngx_otel_module loads dynamically against the libraries the official nginx image is linked with, so distroless means building nginx myself. The trade taken instead: non-root user, no application toolchain in the image
one process per container	already there. nginx alone; the Prometheus exporter is a sidecar, which is the Kubernetes-shaped answer from Correction 4
digest pins	this was the actual gap. Both FROM lines now carry a @sha256:, and Renovate landed in the same change - because a pin without automation is worse than the tag it replaced

One real gap, one deliberate refusal, four already right. And the refusal is the part I’d defend hardest: “always use distroless” is a fine rule right up until the dynamically-loaded telemetry module you depend on needs a libc that isn’t there. Inherited practice with the reason still attached isn’t dogma - it’s an opinion you can argue with. That’s the whole difference.

The through-line is the monkeys. None of these are hard once you ask why the habit exists - the trap is inheriting the flinch without the reason. Slim over Alpine, order your layers, copy only the artifact, know what supervisord costs you, pin what you can automate. And when the agent hands you a fix that makes the error disappear, ask whether it answered your question or just silenced it.