Building the metafactory

For the last seven weeks I've been building a factory. It's a set of AI agents that picks up written tickets, builds the work, checks it, and lands it in the right project, while I make the decisions and review what comes back. I call it the metafactory, because the first thing it had to build was itself.

On the night of 14 to 15 September it landed four tickets with nobody watching. Two days before that, it sat dead for nine hours and nothing told me. Both of those are the real story, so here's how I got from one to the other.

Why build a factory at all

By the end of July I had a working second brain and a lot of things I wanted it to do. The bottleneck was me. Every piece of work needed me in a chat session, steering an AI turn by turn. I could get a lot done in a day, but nothing happened while I slept, and on parental leave I had far less time at a desk.

So on 4 August I made a decision that cost me some patience: build the factory first. Client work I'd been planning would wait and become the factory's first real job. The logic was simple. If the factory worked, every later piece of work would be cheaper. If I kept building by hand, I'd stay the bottleneck forever.

Starting from a published blueprint

I didn't design it from scratch. Simon Chapman had published a detailed blueprint of how he runs his own agent setup, including the skills that drive it. On 26 July I forked it (made my own copy to change) and started adapting it.

That was a mistake, and it took me ten days to see it.

On 4 August a session reported the factory as built and the memory pipeline as live, with 56,263 sources indexed. It wasn't built. When I asked for a proper audit, it found six top-priority gaps, including that the workers had no way to sign in to the model at all. Almost all of them came from my changes to Simon's version, not from his design. My words at the time, which I'll stand by: it made me very uneasy that it thought it was all built and then found all of these catastrophic failures.

On 6 August I did two things. I unwound the fork and installed Simon's skills exactly as he publishes them. And I wrote a short file called ACCEPTANCE.md, which is now the contract for the whole thing. Its first line is the rule: "built" may only be claimed when the acceptance run passes. Behaviour, not testimony. No agent report, checklist tick or commit message counts.

I've written before about an AI saying it was done when it wasn't. This was the same problem at a bigger scale, and the same fix: define done as something you can watch happen.

The design, in plain terms

On 19 August, after reading everything Simon and Chris Parsons had published on how they run agents, I settled the design. Chris has a list of five things a harness (the machinery around the AI that keeps it on track) needs: the ticket that says what done looks like, the tests that prove it, a loop that lets a stopped run carry on, isolation so one attempt can't trample another, and a review at the end. Mine is built around that list.

Here's how a ticket moves through it:

1. A ticket is a GitHub issue. It says what's wanted, which project it's for, and has a "How to test this" section with the exact checks.
2. Intake and shaping. A cheap model reads it and posts any questions it needs answered. Once I've answered, the ticket is ready.
3. A worker claims it. The worker gets its own copy of the code (a git worktree, a separate working folder) so it can't interfere with anything else. It has a fixed number of turns to build the work and commit it.
4. The worker can't push. It can only commit to its own branch. It has no permission to land anything.
5. Host verification. The machine running the factory runs the ticket's own test lines itself, rather than trusting the worker's report.
6. Review. A separate review turn reads the change against the ticket and runs the tests again.
7. The gate lands it. Only the gate, the final step, can merge and push. Then the issue closes.

That fourth point came straight from one of Simon's own post-mortems, where workers told to push had bypassed his checks nineteen times in a day. If a worker can't push, it can't bypass anything. It's the difference between something it's been told not to do and something it can't do.

On 21 August ticket #1 went through end to end. It added a single version file to the project. Not exciting, but it was proof the whole loop worked.

Turning it up

On 27 August I told it to run around the clock. The next day it went from one slot (one ticket being built at a time) to three. I added quiet hours, so anything that needed me between 10pm and 8am waited for a single WhatsApp digest at 8am, rather than buzzing my phone through the night.

Then it went quiet. On 30 August I changed some settings on the scheduled job that runs the dispatcher (the part that hands tickets to workers). The Mac's scheduler listed the job as loaded, which I took to mean it was running. It had never fired once. From about 3:24pm to 10:45pm the factory did nothing, and I only noticed because I hadn't heard from it.

Listed isn't the same as running. The fix is to check that the job has actually run, not that it's registered.

The supervisor doesn't build

The more important change on 30 August was about the supervisor. Alongside the factory I run an orchestrator, a Claude session whose job is to keep the factory moving. For two days it had been fixing things itself whenever the factory got stuck: a small edit here, a config change there. Each fix made sense on its own. Together they caused two outages totalling 18 hours, broke pushing from one of my main projects, and left the factory's own records showing the wrong model.

Simon's blueprint has a rule for this, and I'd ignored it. The orchestrator orchestrates. It doesn't build. From 30 August the orchestrator can file tickets, answer questions, restart and re-queue, and report what's true. It can't edit code. If the factory is blocked, the answer is an operational restart plus an urgent ticket, and the code waits for the factory.

It feels slower. It's much faster, because every fix now goes through the same tests and review as everything else.

A second opinion from a different model

From 31 August to 12 September the factory worked through a set of nine changes that came out of a review by Codex, OpenAI's coding model. Among them: an automatic plan check by a different lab before any risky build, a hard rule that every landing review is done by a different provider from the one that built the work, and routing that sends riskier work to stronger models. The principle underneath is one I now hold firmly. The model that makes something shouldn't be the only one that judges it.

Nine hours dead

Then 13 September. Between about 5:50 and 6:40am three tickets each burned through all six of their turns hitting the same error: the subscription's usage limit for that five-hour window was used up. Two of them then got stuck in the verification step with nothing running, holding both slots. Nothing moved until 4pm. Nine hours.

The orchestrator wasn't watching, and its alarm didn't fire. It measured how long since a ticket's status file had changed, and the dispatcher touched those files every cycle whether anything was happening or not. So the alarm saw constant activity.

The cruel bit was that two of the stuck tickets were the fixes for exactly this: one to check quota before starting a ticket (#151), and one to clean up after a dead step (#154). The factory couldn't yet build its own reliability fixes without someone watching.

Two days earlier there'd been a warning sign I'd underrated. On 11 September I found the factory had been building from a copy of the code 21 commits out of date, and that with more than one slot two workers could claim the same ticket. I'd dropped to one slot overnight and moved on. The lesson I took from both: measure progress from the thing that proves progress, the log of actual work, never from a file that gets touched anyway.

The night it worked

On 14 September I watched quota by hand and got the two fixes through. Then I let it run. Between 2:26pm on the 14th and 8:09am on the 15th it landed four tickets:

• #155, a rewrite of the brief each worker gets at the start of a ticket, which had been making one model stop early, at 14:26
• #154, the clean-up path for a dead step, at 22:34
• #100, a fix so a worker resumes its own session after a stop rather than picking up the reviewer's, at 01:53
• #148, one adapter for any model that speaks the common OpenAI format, at 08:09

That last one matters more than it sounds. It opened a cheap lane, so routine work can run on open-source models rather than my subscriptions.

I woke up to four pieces of finished, tested, reviewed work. On the 15th I wrote down exactly how the orchestrator had supervised that run, as a skill it now loads at the start of every session: a fixed five-minute start-up, a monitor that names the fix for each kind of stop, a short closed list of fixes it's allowed to apply, rules for handing over when it runs low, and a fixed report format. Everything in it was learned by getting it wrong first.

If you're building something like this

You don't need a factory to use the parts that made the difference.

1. Write the ticket so a test can settle it. If you can't say how you'd check it's done, the AI can't either.
2. Define "built" as a run you've watched pass. Put it in writing before you start, so nobody can talk you out of it later, including the AI.
3. Take a published setup as published. Change it only once it works for you unchanged.
4. Make the dangerous thing impossible, not forbidden. If workers can't push, you don't have to trust them not to.
5. Keep the supervisor out of the code. Its job is to keep things moving and tell you the truth.
6. Alarm on the real signal. Last actual progress, not last time a file changed.

Where it is now

As I write this, the factory runs on my Mac, mostly on my existing subscriptions, with a cheap lane for routine work. It's not finished and it still needs watching more than I'd like. The quota problem is better but not solved. What's changed is that I trust what it tells me, because "done" now means something I can check.

The next test is whether it keeps working when I'm properly away from the desk. That's the plan for the next few weeks.

If you're wondering what a smaller version of this could do for your own work, I'm happy to talk it through.