Supermog: We Built a System That Builds Itself

bobbyg603
Supermog: We Built a System That Builds Itself

Right now, while you read this, a fleet of agents in our lab is ripping through GitHub issues. They will open pull requests, hand them to separate, independent agents for review, then prove the changes work by driving real browsers and real phones, posting the video to every PR. They will address the review feedback, keep each branch rebased as the work merges around it, and then merge. They did this last night while we slept. They will do it again tonight. They never get tired, never get bored on rebase number forty, never ask whose turn it is to review — and honestly, we’re a little in awe of them, and immensely grateful for them. Together, they are Supermog.

Here’s the part that still makes us grin: most of Supermog was built by Supermog. Somewhere in week one the loop closed. We stopped implementing features and started filing issues (“you should rebase your own branches,” “you should merge your own PRs”) and the system grew the ability while we watched. Filing an issue at midnight and waking up to the implemented, reviewed, video-verified PR never stopped feeling like magic. This project was, without competition, the most fun we’ve ever had building software: two weeks of pair-building the seed with Claude, then progressively getting out of the way as the thing took over its own development.

And the tireless little factory running it all? Two Raspberry Pi 5s we found in a drawer, left over from other projects.

The fleet at work: two Pironman 5 towers, two old Android phones on verification duty, and, if you look closely, the pixel mog hammering away on each OLED.

What Supermog Actually Is 🤖

Supermog is a v2. The original Mog turned an issue into a PR inside a Docker sandbox and did nothing else, and everything it lacked became this system’s spec.

Three pieces:

A GitHub App. Supermog installs like any other app, with narrowly scoped permissions (contents, issues, PRs, statuses, workflows; read-only where read-only will do). Work is triggered by webhooks: a trigger label on an issue, or an @supermogger mention on a PR with a directive: review, verify, eat (address feedback), merge, rebase, remog. Plain prose after a mention is treated as review feedback and acted on.

A control plane at the edge. A Cloudflare Worker plus one Durable Object holds the queue, job history, secrets, and the dashboard. It costs approximately nothing and never falls over when a Pi does. Cloudflare is just the first home: the core is runtime-neutral, and support for deploying the control plane to more infrastructure providers is in the works.

Agents on whatever hardware is around. Workers scale horizontally and don’t care what they run on: a worker is any box that can dial out to the control plane over a single WebSocket and pull jobs. Ours happen to be the two Pis; nothing about the design needs them to be. No inbound ports, no tunnels into the house. An issue moves through three lanes, one job at a time: Develop implements it, reviews the PR and addresses the feedback; QA runs the checks, brings the repo’s stack up and verifies the result; Release rebases onto whatever merged in the meantime. Logs and evidence are kept on the box’s own disk and streamed to the dashboard.

Security got real attention, because the failure mode is “a robot with commit access”: org and repo allowlists on every trigger path, a write-only secrets portal (GitHub Actions style: you can paste a value in, nothing ever displays it back), secrets stripped from child process environments, log masking, and containers that can’t reach the host. The agents only touch repos you’ve explicitly listed.

How It Grew: Every Bottleneck Became a Feature 🌱

Supermog wasn’t designed; it was iterated, and the repo’s history reads like a lab notebook. The pattern repeated a dozen times: hit a bottleneck, file an issue about it, and, increasingly, let Supermog build the fix itself.

  • Day 1: one runner, browser verification. The first version implemented an issue and then proved it: a Chromium instance drove the actual feature and posted screenshots and video to the PR. Not “tests pass” — here are the pixels.
  • Day 1: device verification. A Pi 5 doesn’t have the muscle to run an Android emulator, so we plugged in the real thing: old phones from the same drawer. A physical device gets the PR’s build over adb, an agent drives the real app, and the recording lands on the PR. A constraint that became a feature: verification on actual hardware.
  • Week 1: one unified verify step. Browser and device verification collapsed into a single general-purpose verification stage: one QA agent, whatever surfaces the repo declares. Someday we’d like it to drive a full desktop, not just a browser or a phone.
  • Week 2: a second runner, then a real control plane. One Pi was too slow. Adding the second forced the honest architecture: a server/agent split, one WebSocket per box, and the server rebuilt runtime-neutral so it could move to Cloudflare Workers.
  • Week 2: the SD card era ends. pi-1’s SD card died mid-review (mmc0: Card stuck being busy!) while the agent kept heartbeating from page cache. We moved both boxes into Pironman 5 cases: the NVMe SSD moved from a USB-C enclosure to real PCIe and became the OS disk. Faster clones, no more card roulette, and the three fans took the SoC down from ~55 °C (131 °F) to comfortable.
  • Week 2: the staleness war. With our bots continuously writing PRs, branches rot fast. Fixes landed in layers: rebase-on-command, then automatic branch updates whenever anything merges (GitHub’s server-side update when clean, an agent job when conflicted), then Depends-on: #N for stacked PRs — and finally for issues, a piece Grok implemented through Supermog’s own pipeline.
  • Week 2: it files its own bugs. When a run hits something wrong, the agent reports it through BugSplat’s user feedback tool, tagged with Supermog’s own trigger tag, so the bot picks its own report back up as work. The agent is both the user and the developer. The loop closed.
  • Week 2: CI moved home. GitHub Actions billing broke our pipeline three times: every check across every repo went red until we raised the spending limit, again. Then the obvious hit us: these boxes are on 24/7 anyway. The Pis became our self-hosted Actions runners: zero billed minutes, faster native arm64 builds, and the Actions queue got its own lane on the status page.
  • This week: it merges and ships itself. Two human steps fell in one week. Supermog publishes a supermog/verified commit status summarizing its round, arms GitHub auto-merge on its own PRs and un-arms if a later round reopens findings; branch protection does the rest. Then a push to main started rolling the fleet: the control plane drains a box, swaps its image, waits for the new build to report over the socket, and moves to the next. Its very first outing stopped itself on a bad image reference and left the box safely drained on the old build — exactly the failure behavior you’d want from a system that ships itself.
  • This week: three lanes, three queues. A box used to run one job at a time, so a slow verification held up everything behind it. The fix wasn’t a bigger queue, it was admitting an issue is only ever in one state: Develop, QA or Release, each with its own line and its own capacity on each box. A job is one lane’s work. When it finishes it puts the issue at the bottom of the next lane’s line and hands its slot straight back — no job ever sits on a slot it isn’t using, and a queued issue holds nothing at all. QA stays single-file per box, because it brings up the repo’s entire stack; the other two run several at a time, and the limits are set per machine, so a laptop with sixteen cores is not held to a Raspberry Pi’s numbers. Getting there meant deleting a lot of clever code that had been carefully arranging for work to wait.
  • This week: we swapped models mid-outage. On September 3, ChatGPT, Claude and Grok went down within minutes of each other — the correlated failure everyone assumes is rare. Our worker started hanging mid-job. Which model implements and which one reviews are settings, so we pointed both roles at a provider that was still answering and the queue kept moving. No redeploy, no code change, about two hours lost. The lesson we took: independent review is what matters, not which model performs it — and anything you depend on should be swappable while it is running.

Along the way the dashboard grew everything an operator actually needs: per-runner detail pages, queue reordering by drag, metrics and token tracking, a lessons store (per-repo things the bot learned the hard way, injected into future prompts), a usage guard that pauses claiming work near API rate limits, human-needed notifications, one-click pause/update, and, because why not, brandable theming.

The Activity view: nine runs in flight across Develop, QA and Release on two boxes, with each lane's own queue underneath.

Nine runs in flight across two boxes, thirty more waiting. Develop, QA and Release each hold their own line, so a slow verification blocks only QA.

A box's hardware page: power draw with per-run annotations, memory, load, and temperature.

Each box’s hardware page. Yes, that says 7.7 watts — the whole factory draws less than a lightbulb, and the Pironman fans keep it under 40 °C flat out.

The Lessons page: per-repo lessons the bot learned from review feedback, injected into future runs.

The lessons store: things reviewers caught once, injected into every future run so they never get caught twice.

The numbers so far: 2 Raspberry Pi 5s · 440+ issues and PRs in ~2 weeks · 16 pipeline stages from issue to merge · 1 WebSocket per box and zero inbound ports · $0 in CI minutes on the self-hosted runners.

The Fun Part 🔨

Each Pironman 5 has a tiny SSD1306 OLED. Ours run a pixel mog who hammers away whenever the box is actively mogging: locked 30-to-90-degree mallet swing, frame-wide BOOM dashes on impact, and a nap when the queue is idle. We didn’t drop a PNG on it. We drew him at 128×64, one bit per pixel, in a browser preview that is the panel buffer, locked the frame numbers in tests so the TypeScript and the Python on the Pi cannot drift, and let the Pi paint that same buffer onto the glass. A tap of the power button flips to host stats.

Taking the screen meant not starting Pironman’s own software (it would have stolen I2C 0x3C), and then the tower coolers went quiet. On Raspberry Pi OS that fan isn’t Pironman’s to drive: the kernel pwm-fan sits off below 50 °C, and Pironman only reads the tach. Their own FAQ is one line in config.txt. We dropped the first trip to 30 °C; the guy kept slamming and the SoC stopped cooking. Completely unnecessary, and everyone’s favorite feature. We also filmed the case build — the timelapse, and the story of editing it entirely inside Claude Code, got a short post of its own.

Autocatalytic, Not Recursive 🔁

Martin Casado (a16z) draws a distinction we’ve adopted: recursive self-improvement is a system copying and upgrading itself wholesale; autocatalytic is using the thing as a tool to make the thing faster — the way engineers have used software to build software since the first compiler. He prefers the second term for what’s actually happening in AI right now, and it describes Supermog exactly. Supermog doesn’t rewrite its own weights. It writes, reviews, verifies, and merges the code of the system that runs it, while doing the same for our other projects. Somewhere north of half of Supermog’s recent features shipped through Supermog.

It’s also an experiment in how far you can go with little to no human review. The honest answer so far: further than we expected, with guardrails. Independent review by a second model catches real bugs; verification-with-evidence catches the “compiles but doesn’t work” class; a hard cap on how many times an issue may go back from QA to Develop keeps the bot from thrashing; and when a round genuinely can’t vouch for a branch, it says so, posts why, and pages a human instead of merging.

What’s Next 🚀

Runners on our macOS and Windows machines, which have enough muscle to run Android emulators, so they won’t need a phone plugged in — and, with per-machine limits, to run a lot more work in parallel than a Pi can. Full-desktop verification. Real merge-order scheduling. And more dogfooding through Automate It, because the point was never the Pis — it’s what a small team ships when the pipeline works this hard.

Should We Make the Source Available? 🤔

We built Supermog for ourselves, and honestly, we’re curious whether it would be useful beyond our walls. Would your team run something like this? Would you want to license the source, study how it works, or have us stand a fleet up for you? We genuinely don’t know what the demand looks like, which is why we’re asking. If this sounds like something you’d use, tell us: ping @workingdevshero on X or email bobby@workingdevshero.com. If enough hands go up, we’ll figure out the right way to share it.

And if you just want the throughput without owning the machinery — that’s literally what we do. Say hi.