Stale git index.lock: A Silent Failure Postmortem

TL;DR: My blog went quiet for eighteen days and no part of my system noticed. The cause was a zero-byte .git/index.lock, left behind by a crashed process, which made every git write in my local clone fail. The automation around it kept reporting success, because none of it ever asked whether the work had left the laptop. The interesting part is not the lock. It is that I had built careful alerting for my cluster and none at all for the pipeline that turns my work into published words.

The symptom

I only found out because I asked.

Nothing was broken in any way a monitor could see. The site was up, the cluster was green, the deployment pipeline I had spent a year building was idle and healthy. The most recent article on wostal.eu was dated 29 August. It was 16 September.

The weekly automations had run. Both of them, every Monday, on schedule, reporting success. The content plan said an article should have shipped on the 14th. The calendar event had come and gone with a ten minute reminder. And the blog had not moved for two and a half weeks.

The autopsy

The first thing I tried was a commit, which is when the system finally spoke:

fatal: Unable to create '/…/wostal-eu/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again.

That message is a lie, or at least an outdated guess. There was no other git process. There had not been one for days:

$ ls -la .git/index.lock
-rw-------  1 tomasz  staff  0  Sep  7 08:10 .git/index.lock

$ ps aux | grep -c "[g]it"
0

Zero bytes, dated 7 September at 08:10. Git takes that lock before it writes the index and removes it afterwards; if the process dies in between, the file stays, and every subsequent write refuses to start. It is a correct and conservative design. It is also completely silent to everyone except the process unlucky enough to hit it.

Nine days of consequences follow from that one empty file:

no commits, so drafts stayed exactly where they were
no git fetch, so my local main sat on a release from 21 August while production had already shipped two newer posts. Every local answer to “what is published?” was wrong, and confidently so
no push, so the finished article never reached the remote at all

Then I checked my notes repository, and found the same disease with a different date:

$ ls -la .git/HEAD.lock
-rw-------  1 tomasz  staff  0  Aug 26 19:24 .git/HEAD.lock

26 August, 19:24. Which is, to the minute, the timestamp of the last commit in that repository. It had been frozen for three weeks and the daily log that was supposed to record every operation had simply stopped gaining entries, which is precisely the signal I would have used to notice.

Why nothing screamed

Three things had to line up, and all three are design decisions rather than accidents.

The error had exactly one witness. Git wrote it to the stderr of a process nobody was watching. It did not touch a log aggregator, did not increment a counter, did not exist in any form five seconds later. My cluster pages me when a rule evaluator goes unhealthy, because I built a check that runs outside the metrics stack and asks it a question. Nothing asked my laptop anything.

The automation succeeded at its actual job. The Monday task that prepares drafts did prepare them. It read the plan, wrote the text, and tried to commit. Whether the commit landed was, as far as the task was concerned, somebody else’s problem, and the “somebody else” did not exist. A job that reports success for work that never persisted is worse than a job that fails, because it actively consumes the attention you would otherwise spend checking.

Nobody compared intent against reality. This is the real one. The whole system was full of statements about what should happen: a publishing manifest with a date, a calendar event, a plan in my notes, a frontmatter date field. Four independent declarations of intent, and not a single process whose job was to compare any of them to the world.

That is the same failure mode I have already been bitten by twice in the cluster. A push webhook that silently fails to arrive, so a merge never triggers a pipeline. A metrics database down for five hours while every alert that could have told me sat downstream of it. Both times the fix was the same shape, and both times I somehow filed it as a cluster concern rather than a general one.

The second failure, which was not a lock

While fixing the first problem I found a second one that had nothing to do with git.

The article scheduled for that week referenced seven screenshots. The directory that should have held them contained a README explaining the naming convention and nothing else. The b-roll session booked for the previous Saturday had not happened, so the images had never been captured, and the article had been sitting there for two weeks quietly guaranteeing seven broken images on publication day.

This one is worth separating out, because it is a different class. The lock was a machine failing silently. This was a human commitment failing silently, and the system had no more visibility into the second than the first. A check for it costs nothing:

$ ls public/images/blog/gitops-control-plane/
README.txt

Seven references, one file, and nobody asking.

What actually detects this

The fix is not more automation. I had plenty. The fix is assertions: statements that must be true, evaluated somewhere that can still speak when the thing it watches is broken.

I am adding a content-deadman CronWorkflow to the same Argo Workflows instance that already runs my pipeline deadman. Five assertions, once a day, silent on success, Telegram on failure:

#	Assertion
A1	The newest post on production is younger than twelve days
A2	Every manifest due within 72 hours points at a branch that exists on the remote, and the post on it has `draft: false`
A3	Every image referenced by those posts exists in the branch
A4	The manifest date and the frontmatter date describe the same day
A5	Something, anything, is queued for the next fourteen days

A2 is the one that earns its keep, and the reason is worth stating plainly: it detects a local problem using only remote state. The lock was on my laptop and no cluster workload will ever see it. But “the manifest promises Monday and the branch is not on the remote” is true and checkable from anywhere, and it would have fired on 14 September with three days to spare.

The shape of the check, in the interest of not being hand-wavy:

# A2a: does the branch exist ON ORIGIN? A local-only branch is exactly the
# September failure: manifest ready, work committed, none of it ever pushed.
if ! curl -sf -H "${AUTH}" "${API}/wostal-eu/branches/${branch}" >/dev/null; then
  fail "${name} publishes in ${delta_h}h but branch '${branch}' does NOT exist on origin"
fi

There is a satisfying and slightly embarrassing detail here. I ran A1 by hand against the live site while writing this, and it fires immediately: nineteen days. The first thing my new watchdog will do is tell me something I already know, and it will not go quiet until the backlog actually ships.

Where a check has to live

The part I got wrong for a year is placement, so it is worth being explicit. A check can only assert about state it can reach.

Runs in	Can see	Cannot see
The cluster	the remote repository, production, the publishing queue, the social queue	your working copy, your unpushed commits, your stale locks
Your laptop	locks, unpushed branches, missing local files	essentially everything else

Which means the honest answer is two small things rather than one clever thing. The cluster gets the five assertions above. The laptop gets an hourly launchd agent with two questions of its own: is there a lock file in .git older than thirty minutes, and does any branch have commits the remote has never seen. Neither needs to be smart. Both need to exist.

What I deliberately did not build

An agent. The temptation, once you have described the problem, is to put a language model in the middle of it and let it watch the pipeline, notice the drift and fix things. I have that capability wired into this workflow already, and I am keeping it out of this particular loop.

The reason is that every question above is an invariant. Does the branch exist. Is the file there. Do two dates match. test -f answers that class of question deterministically, for free, and cannot talk itself into a wrong answer. A model in that position adds API keys in my cluster, a new failure mode, and non-determinism in exchange for nothing, and the maintenance budget for this entire stack is three to five hours a month. Spending it on babysitting a watchdog would be a poor trade.

There is a layer where judgement genuinely helps, and it sits above the invariants: is this draft actually ready, does it repeat something I published in June, does it hold the conventions I claim to follow. That layer can be as clever as it likes, because nothing downstream of it publishes without me.

The boring part

The immediate fixes took about an hour and none of them were interesting, which is the point. Delete two lock files. Prune twenty-two worktree entries pointing at directories that no longer existed. Fix a scheduled task that had been aiming at a Hugo content path since the site moved to Astro months ago, silently working around the absence. Backfill the log entries for the two weeks that left no trace. Repair one calendar collision where two articles had been scheduled for the same day.

Every one of those was invisible until somebody looked, and “somebody looked” is not a control. That is the whole lesson, and I needed eighteen days of silence to take my own cluster’s advice.

I will report back on what the watchdog catches, including the false positives. Especially the false positives.