Argo Events + Argo Workflows as CI: Webhook to Pipeline Without a Vendor

TL;DR: The first post in this series drew the map: push to Codeberg, production some minutes later, no CI vendor anywhere. This one zooms into the first half of that sentence. Argo Events turns a webhook into a Kubernetes event; Argo Workflows turns that event into a running pipeline - in my cluster, on my compute, defined in the same repo as everything else. The interesting part isn’t either tool. It’s that one generic pipeline definition serves every project I own, and that a deadman cron exists because the failure mode of self-hosted CI isn’t a red build - it’s silence.

The claim from part one, cashed out

Part one asserted that a git push is the only way anything reaches my cluster. For that to be true, something inside the cluster has to be listening. In a CI vendor’s world, that listener is their SaaS: GitHub receives your push and their runners react. Self-hosting the reaction means owning three problems the vendor was quietly solving: getting the webhook into the cluster safely, translating it into work, and noticing when nothing happens.

That’s the anatomy of this post: entry, translation, execution - and the deadman that watches all three.

Entry: a webhook is just an HTTP request

The path from Codeberg to the pipeline is deliberately boring:

an HTTPRoute exposes one hostname and one path per EventSource - all the tenants share /tenants/codeberg, the control plane has its own, and nothing else is reachable
a NetworkPolicy wraps the EventSource pods so the only traffic that reaches them comes from the gateway, and the only place they may talk to is the event bus
the webhook secret arrives via ExternalSecret - it lives in the secret store, not in a manifest

# HTTPRoute: one hostname, one path, one backend - the tenants' EventSource
kind: HTTPRoute
metadata:
  name: argo-events-codeberg-tenants
  namespace: argo-events
  annotations:
    external-dns.alpha.kubernetes.io/hostname: events.mistwolf.tech
spec:
  parentRefs:
    - kind: Gateway
      name: dev-gateway
      namespace: kgateway-system
  rules:
    - matches: [{ path: { value: /tenants/codeberg } }]
      backendRefs: [{ name: codeberg-tenants-eventsource-svc, port: 12000 }]
---
# NetworkPolicy: ingress from the gateway namespace, egress to the event bus, nothing else
kind: NetworkPolicy
metadata:
  name: eventsource-ingress-kgateway-only
  namespace: argo-events
spec:
  podSelector:
    matchLabels:
      controller: eventsource-controller
  policyTypes: [Ingress, Egress]

The EventSource is Argo Events’ half of the handshake: it speaks HTTP, validates the signature, and republishes the payload onto an internal event bus. From that moment the push is no longer a web thing - it’s a Kubernetes-native event that anything in the cluster can subscribe to.

Translation: one Sensor per tenant, and why that’s the design

A Sensor subscribes to the bus and decides what an event means. Mine are per tenant - codeberg-wostal-eu watches this blog’s repo, its siblings watch the other projects. Each sensor does exactly one thing: filter the event (right repo? branch push or tag?) and submit a Workflow from a template with a handful of parameters - repo URL, tenant name, revision.

# sensor-codeberg-wostal-eu.yaml (abridged): the "push to main" dependency
- name: push
  eventSourceName: codeberg-tenants
  eventName: codeberg-event
  filters:
    data:
      - path: body.repository.full_name
        value: ["tomasz-wostal-eu/wostal-eu"]
      - path: body.ref
        value: ["refs/heads/main"]
    # Skip semantic-release's own "chore(release): X [skip ci]" commits
    # so a release commit doesn't re-trigger the main pipeline.
    exprs:
      - expr: "!(msg =~ '[[]skip ci[]]')"
        fields:
          - name: msg
            path: body.head_commit.message

Two filters in the real file earn their lines the way most config does - by remembering an incident. The [skip ci] expression exists because semantic-release pushes its own release commit, and without the filter every release triggered one more, pointless, pipeline run. And the tag dependency carries a guard rejecting an all-zeros after SHA, because deleting a tag also arrives as a push event - and used to spawn a tag workflow that reliably died at git-clone. (Even the escaping is a war story: the [[] character-class trick is there because backslash escapes break the expr evaluator - found on argo-events 1.9.10, still true on the 1.9.11 this cluster runs today.) A CI vendor would have absorbed those edge cases for me; owning the listener means owning its folklore.

This is the point where the series’ recurring theme shows up again. The sensor is written intention: someone who knew what a push to main should mean wrote it down once, and now the event can’t mean anything else. There is no UI where a helpful colleague - or a helpful agent - can add “just one extra step” to a specific build. Changing what a push means requires a commit to the control-plane repo, which means review, which means the intention stays legible.

Execution: the generic pipeline

Here’s the part I’d defend as the actual engineering in this setup. There is not a pipeline per project. There are three WorkflowTemplates - tenant-main-pipeline, tenant-tag-pipeline, tenant-pr-pipeline - and every project flows through the same three, parameterized by repo and tenant name.

They compose from a shared tenant-steps template that reads like a vocabulary of things I believe every pipeline owes me: git-clone, helm-lint, semantic-release, docker-build-push, helm-package-push, and codeberg-commit-status to report the verdict back to the forge. A tenant’s main-branch push runs clone, lint, then semantic-release - which either decides no release is warranted and exits, or cuts a tag. The tag fires the tag pipeline, which builds and pushes the image and chart. Where the artifacts go and what happens after (Kargo, Rollouts, the k6 gate) was part one’s territory; deep dives on those are parts three and four.

The whole main-branch pipeline is three DAG tasks, and the tasks are references rather than definitions:

# extras/hetzner/hel1/argo-workflows/workflowtemplate-tenant-main.yaml
kind: WorkflowTemplate
metadata:
  name: tenant-main-pipeline
spec:
  arguments:
    parameters:
      - { name: repo, value: "" }
      - { name: chart-name, value: "" }
      - { name: sha, value: "" }
      - { name: branch, value: "refs/heads/main" }
  templates:
    - name: main
      dag:
        tasks:
          - name: clone
            templateRef: { name: tenant-steps, template: git-clone }
          - name: lint
            templateRef: { name: tenant-steps, template: helm-lint }
            # depends: clone; consumes tasks.clone.outputs.artifacts.source
          - name: release
            templateRef: { name: tenant-steps, template: semantic-release }

Four parameters and three templateRefs. Every project I own runs that exact object. What differs between a blog and a Go service is a handful of values the sensor fills in - the repo, the image name, the chart name, and since the blog moved to my personal namespace, which OCI registry the chart goes to - never the shape of the pipeline. Here is one of them mid-flight, from the drill on 27 August:

argo get -n argo-workflows wostal-eu-main-mb4bv

# Name:       wostal-eu-main-mb4bv
# Status:     Succeeded
# Duration:   2 minutes 22 seconds
# Progress:   3/3
#
# STEP                     TEMPLATE                       DURATION  MESSAGE
#  v wostal-eu-main-mb4bv  main
#  |-v clone               tenant-steps/git-clone         20s
#  |-v lint                tenant-steps/helm-lint         14s
#  \-v release(0)          tenant-steps/semantic-release  1m

Twenty seconds of clone, fourteen of lint, a minute of semantic-release deciding whether the world deserves a new version.

# the interesting log line - what semantic-release actually decided.
# Part one explains why this is a curl and not `argo logs`: podGC removes the
# pod the moment the workflow ends, so on a finished run the live-log readers
# come back empty and the archived artifact is where the log actually lives.
curl -s "$ARGO/artifact-files/argo-workflows/workflows/wostal-eu-main-mb4bv/\
wostal-eu-main-mb4bv-release/outputs/main-logs" | grep -i "next release version"

[6:41:10 PM] [semantic-release] › ℹ  The next release version is 1.23.1

Two properties fall out of the generic design, one obvious, one less so:

The obvious one: a fix lands everywhere at once. When I improved the lint step, every project got it on its next push. Nobody “migrates pipelines” because there’s exactly one to migrate.

The less obvious one: projects can’t drift into snowflakes. A project that needs something special has to either argue the vocabulary should grow - a review conversation in the control-plane repo - or it doesn’t get it. That friction is the feature. Every CI vendor’s marketplace optimizes for the opposite: making the special case frictionless, one YAML paste at a time, until no two pipelines in the org agree on what “build” means.

The vocabulary does grow, just slowly and in public, and it is worth being exact about what grew this month, because the two additions are not the same kind of thing. confluence-publish is a real step: it sits in drift-warden’s own main pipeline after semantic-release and renders the runbooks listed in pages.yaml into a read-only Confluence space on every merge. Three containers, because a first attempt to do checkout, Mermaid rendering and upload in one image failed on the toolchains. jira-sync is not a step at all. It is a cluster-scoped template that the drift-warden sensor submits as its own workflow on three events: a pull request opened moves the issue to In Progress, a merge moves it to Done, a push to main only leaves a comment. Nothing in the pull request text ever reaches the step’s source: the template fetches titles and messages from the Codeberg API itself and parses them with jq, after one version interpolated a commit message into a shell script and learned why that is a bad idea. Any tenant sensor could submit the same template; today none does. So the honest count is: one step added to the control plane’s own pipeline, one bridge added beside every pipeline, and the shared tenant vocabulary unchanged since August.

The deadman, because silence is the real failure mode

A CI vendor’s outage page tells you when your builds aren’t running. Self-hosted, nobody tells you. And the nasty property of event-driven CI is that a pipeline that never starts looks identical to a quiet afternoon. Broken webhook delivery, a wedged event bus, an expired secret - none of them produce a red X anywhere. The system returns a success signal because no signal was ever the thing being checked.

So there’s a CronWorkflow called ci-deadman. Twice an hour, for every repository with a main-branch pipeline, it asks Codeberg for the newest commit on main that is not a [skip ci] release commit, waits out a grace period in case the webhook is still in flight, and then checks one thing: does that commit carry any commit status? Every main pipeline sets one, and statuses never expire - unlike Workflow objects, whose TTL is shorter than the lookback window, which is how an earlier version of this check produced a false alarm. A commit on main with zero statuses means a merge happened and no pipeline ever saw it. That is the whole test, and it has fired for real - three consecutive failures one August afternoon while I was rotating CI tokens, which is exactly the class of breakage that produces no red X anywhere else. The gate framing from part one applies verbatim: the value of the deadman is not how often it fires, but that nothing happening now has a place to be caught.

The pattern generalised faster than I expected. A second cron, mirror-deadman, watches something narrower: drift-warden pushes to a private GitHub mirror for disaster recovery, and a mirror that quietly stops syncing is a backup you discover is missing on the day you need it. Every six hours it asks the Codeberg API how long ago the mirror last synced.

Fourteen hours after I added it, it went red for the first time, and the way it failed is the better lesson:

FAILURES:
- cannot read push mirrors for mistwolftech/drift-warden (token needs repository admin read)

What broke was not the mirror. It was the deadman’s ability to ask: the token it uses had lost the scope that lets it read push-mirror status, collateral from a credential cleanup the day before. Three runs failed over twelve hours before the scope went back on.

Note what a weaker implementation does here. An unreadable answer is not a failed answer, so treat it as no news, stay green, and the operator sees an unbroken run of ticks while the backup goes unwatched. This is the whole design rule in one line: a check that cannot run must fail, not pass. An observer that goes blind and reports success is worse than no observer, because by then you trust it.

What this costs, honestly

The vendor was charging money; this charges attention, and pretending otherwise would undercut the series. The bill as I’ve actually paid it:

You are the runner team now. Workflow controller upgrades, executor changes, resource requests for build pods - mine, not GitHub’s. In practice this folds into the same Renovate-driven update flow as everything else in the control plane, but it’s real work when a major version moves.
No marketplace. Every step exists because I wrote it or ported it. For helm-lint that’s trivial. For anything exotic, the vendor’s ecosystem advantage is real and I simply don’t have it.
Debugging is kubectl, not a web console. I consider kubectl logs on a pipeline pod a feature - it’s the same debugging motion as every other workload I run. A team that lives in CI web UIs will experience it as a regression.
The compute bill, which is smaller than the attention bill. Unlike the controllers in part one, pipeline pods only exist while they run. What they ask for while they do:

Step	CPU request	Memory request
`git-clone`	100m	128Mi
`helm-lint`	100m	128Mi
`semantic-release`	200m	512Mi
`docker-build-push`	500m	512Mi
`helm-package-push`	200m	256Mi
`codeberg-commit-status`	50m	64Mi

A main-branch push peaks at 200m and 512Mi for about a minute; a tag pipeline that actually builds an image peaks at 500m and 512Mi. The two standing costs are the crons: ci-deadman runs twice an hour at 50m / 64Mi, and the Renovate CronWorkflow is the genuinely expensive one at 200m / 512Mi requested and a full core / 2Gi allowed. Renovate scanning every repo costs more than every build I run.

What it buys back: the pipeline is subject to the same regime as everything else - versioned, reviewed, reverted with git revert, promoted nowhere because it is the control plane. When part one said the entire deployment story lives in one repo, this layer is why that sentence includes CI.

The test you can run on your own setup

Close the laptop and answer from memory: if your webhook deliveries silently stopped this morning, what would tell you - and when? If the honest answer is “the next person who wonders why their merge didn’t deploy,” you have the same deadman gap I had, whether your CI is self-hosted or not. The cron is twenty lines. The property it buys - silence fails loudly - is the difference between a system you trust and a system that hasn’t scared you yet.

Next in the series: Kargo - promotion as a first-class object, and why “what’s allowed to reach production” deserves the same written-down treatment as “what a push means.”