CI_CD_CT_Product Evaluation – Qodo v. Gemini

CI/CD/CT on MAX3

An evaluation of a machine-paced pipeline, and of the Claude sessions that built it

Jason Newell

July 2026

1. The loop, seen from outside

A tower in a spare room runs a service that polls a git branch every two minutes. When a new commit appears on the main branch, the service fast-forwards the working tree. The web server re-imports the changed module while the system stays up. Nobody clicks anything. The pull request behind that commit was opened, watched, corrected, and merged by a Claude session that will never run again. The tower is the only production this project has.

That loop is the delivery half of the MAX3 pipeline, described without the vocabulary that usually surrounds it. There's no artifact, no registry, no environment, and no deploy approval. Continuous integration, which is the practice of building and testing every change automatically before it merges, runs on GitHub's hosted machines. Continuous delivery, usually the automated path from a merged change to a running system, is here a squash-merge plus a service that pulls. Continuous testing, meaning a suite that runs on a scope and a schedule rather than on a release calendar, is the deepest of the three. Two of them look conventional. The middle one doesn't, and the reason is worth saying early: for a local-first system that lives on one box, a release pipeline would be ceremony wrapped around a file copy.

What follows is an evaluation of that pipeline and of the developer who built most of it. The developer is Claude, arriving as a series of stateless sessions. They open pull requests, watch the checks, fix what turns red, and merge on green. The account is two-sided by design. Claude built nearly all the machinery described here, and Claude also authored the three most expensive failures in the project's history.

2. The pipeline as it stands

MAX3 is a single Python process with a Vite and React frontend. It's developed local-first on one Windows workstation, and mirrored to a free personal repository on GitHub. The free tier isn't a footnote. It removes auto-merge, removes server-side branch protection, and caps the monthly budget of Actions minutes, which is the compute GitHub bills for running a pipeline. Several of the stranger choices below trace back to that one line in the account settings.

The workflow fires on four triggers. A pull request into main is the real gate. A push to main, scoped to main and nothing else, revalidates after the merge. A Monday cron runs the whole suite. A manual dispatch exists for the days when something has gone wrong. A concurrency group keyed on the branch collapses the push run and the pull-request run into one pyramid. It also kills superseded runs when commits land fast.

Behind those triggers, the jobs fan out from one classifier.

Figure 1. The classifier decides what a change actually touches, and three of the four lanes can be skipped.

The detect job diffs the change set and emits its verdict: docs only, backend changed, frontend changed, security relevant. It picks its diff base per trigger. A pull request diffs against its base branch. A push to a feature branch diffs against the merge base with main. What gets classified is the whole-branch delta rather than the last commit. That second rule was bought with an incident. A docs-only follow-up commit on a code branch once convinced the classifier that the whole branch was documentation. The docs-only run then cancelled the code's full run. That branch merged having never faced the pyramid.

The docs lane is the fast path. For a change touching only Markdown, images, batch scripts, or anything under the docs tree, both pyramids are skipped. What remains is a check that the docs are readable and correctly formed. It also confirms the manifest is in sync and the tree is free of documentation and board drift. The backend lane runs Python 3.12, then the linter, then the three-tier test pyramid across four parallel shards, reusing a cached virtual environment. The frontend lane type-checks, runs the full Vitest suite, and builds, because a build catches what the type-checker misses. The security lane runs a static analyzer and a dependency auditor, and it currently reports without blocking.

The suite is deep. Roughly 2,600 tests sit in the backend pyramid: about 1,900 unit, about 500 integration and contract, and about 240 system, alongside roughly 170 frontend test files. The local scripts run the same tiers in the same order. A green run on the box and a green run in CI mean the same thing.

The pipeline's own settings are themselves pinned by tests. A system-tier guard file asserts the job gating, the weekly cron, and the merge-base classification. It also pins the four-way shard and the split cache, with its save step placed ahead of the tests. Every optimization described in this document is an assertion somewhere, which is the only reason none of them have quietly reverted. When you add a fix, add the check.

Delivery is a pull, not a push

There's no build step and no versioned artifact. A session squash-merges the green pull request, which keeps history linear and attributable to one change. The box's auto-pull service fast-forwards local main on its next poll. Then the running process re-imports the changed file. It re-runs setup when a dependency file moved, and it takes a cold relaunch when the change is large. The squash-merge is the release, and the poll is the rollout.

The blast radius of a bad merge is one workstation that self-heals and pulls the next fix. That's the fact underneath every autonomy decision here. A bad change costs minutes on a tower in a spare room rather than a customer incident. A machine can be trusted to merge without a human watching. Change the blast radius and the whole design would deserve a different answer.

One wrinkle spoils the neatness. The box is often reachable only over a phone hotspot, so a merged commit lands whenever a window opens, sometimes hours later. Merge-to-live latency is real and it isn't predictable. The always-on services are what turn that into reliable eventual consistency instead of a manual chore.

3. Testing in four rings

Testing here isn't run-everything-every-time. It's compartmented into rings, and the outermost one is a person.

Figure 2. Scoping is only safe because a wider ring runs behind it.

Ring 0 is the ground floor: linters, the type-checker, and the system-contract tier, on every push and every pull request, never skipped. Ring 1 is the scoped ring, which today means job-level gating. A frontend-only change skips the backend pyramid, and the reverse holds. Ring 2 is the full pyramid on the Monday cron, and on demand locally. That weekly backstop is what makes a Ring 1 miss safe rather than silent, and without it the scoping would be a gamble.

Ring 3 is Jason at the machine, and Section 7 argues it's the load-bearing one. Around the main workflow sit five supporting ones. A fallback re-fires CI when GitHub drops a webhook, and a weekly job measures test durations. A self-hosted reviewer stays inert by default. A post-merge job folds board fragments together, and a telemetry check watches for staleness.

4. One developer at machine speed

The pipeline only makes sense against the human reality it serves, and that reality is unusual enough to explain nearly every choice above.

MAX3 is built by one person, part-time. There's no team, no reviewer to wait on, no standup, and no release manager. The team is a rotating cast of autonomous, ephemeral Claude sessions that open pull requests, watch the checks, fix failures, and merge on green. Each session is stateless, so everything a new session needs has to already be on disk. A directive that lives only in a chat transcript is a bug.

Because the authors are machines, throughput is machine-paced. During the July 2026 reviewer comparison the repository was taking about 80 code pull requests a week. Documentation, board, and process changes ran on top of that. Eighty a week is better than eleven a day including weekends, which is not a rate a part-time human produces. The pipeline is therefore tuned as a continuous merge train: short-lived branches, one change per pull request, squash-merge, immediate. There's no sprint and no batching, and the queue absorbs new work without anyone re-explaining the last thing.

Every pull request opens as a draft, and draft is real state rather than a formality. It means a session is still working the change or the checks are still running. When the session confirms green it flips the pull request to ready and merges in the same breath. The whole cycle takes a couple of minutes of wall-clock. That detail also produced the worst confound in the reviewer bake-off. One of the two reviewers skipped drafts by default, so it spent its week grading code that had already been fixed.

The constraints that shape everything

A cloud session can't run the pyramid. There's no GPU in a web container, and the machine-learning stack takes about 20 minutes to install. The session that writes the code is the one that can least afford to test it. This inverts the usual flow. Rather than tests passing locally and CI confirming, CI is the test run and the only authority. A red check is a real failure to fix, never noise to wave through, because there's no local green to disagree with it.

The developer machine runs Windows and the runners run Linux. That split is a standing source of divergence: line-ending rules, PowerShell against bash, and platform-only flakes like the native DuckDB crash under parallel workers that reproduces on Windows and never once on Linux. A green Linux pyramid does not prove the Windows box works, and Claude has had to relearn that more than once.

Parallel sessions were historically the main hazard. More than one session pushing at once caused both the Actions-cost blowup and the collisions on generated files described below. The answer was to serialize: exactly one code branch in the pipeline at a time, with documentation changes allowed a fast-path exception.

The churn this creates

High machine throughput against shared generated files produced a specific class of git churn. Most of the current hygiene exists to kill it.

Three files used to be regenerated by every pull request: the tree manifest, the board state, and the rendered status page. Any two open pull requests therefore collided on exactly the same lines. One day in June 2026 cost a dozen rebases. It also put two conflict-marker corruptions into main, which is how a generated JSON file ends up unparseable on the trunk. The fix came in three parts. Board changes now ship as fragments, one file per change with a distinct name. A job on main folds them in, and it's the only writer of the generated trio. A pre-commit hook regenerates and re-stages the manifest inside every commit, so a code change can't land stale. The generated files are no longer hand-edited in a pull request at all.

The residue of that story is the manifest bite. A commit lands with a stale manifest, usually because the hook wasn't installed in a fresh container, and the only red check is manifest drift. The standing rule is to regenerate and merge on the docs fast path, never to re-run the pyramid, because nothing about the behaviour changed.

Force-pushing with a lease is routine here and it isn't alarming. After a merge, the working branch restarts from the freshly merged main. The remote branch still carries the pre-merge commits that were just squashed away, so the next push has to overwrite them. It happens on essentially every continuation, and the old content is already on main.

Nobody watches the checks, so a session arms a ten-minute self-wake. It sweeps every open pull request, merges anything green, fixes anything red, and re-arms. That cadence is the actual pulse of the merge train, and it exists because webhooks don't deliver silent success.

5. Markdown is the operating system

A second cluster of choices comes from the limits of the two tools this project builds through. Where Claude and GitHub couldn't do what a solo machine-paced shop needs, the missing capability got encoded in the one substrate both tools handle natively.

The work tracker isn't GitHub Projects or GitHub Issues. Both were retired from all automation in June 2026, for two hard reasons. A Claude session literally cannot maintain a Project on a personal repository. The tool interface it drives can't set custom fields. So it could open and close issues while being unable to move a card through the very columns that make a board a board. The second reason was budget. Reconciling a Project costs API calls, and doing it on the auto-pull cadence starved the same token needed for real repository work: clone, fetch, push, open, merge. The board sweeps were eating the merge train's rate limit.

So the board is local and costs zero API calls. A JSON file is the source of truth, carrying title, type, area, priority, status, arc, phase, pull request, and release for every item. A script renders it to a status page. A browser wall, served by an always-on local service, refreshes every two minutes with no GitHub call at all. It's the surface a phone actually reads. A reconcile step greps the git log for merged squash numbers and flips those items to shipped. The wall stays honest from git history rather than from an API.

Work is grouped into arcs, which is the local word for epics: a themed multi-PR sequence with a stable key, sliced into numbered phases where a phase is one pull request. Items are categorized on several axes at once, by type and area and priority and status and arc and phase. That structure is what lets a session pull the highest-value unblocked item and see an arc's completion at a glance. It also stops an arc being abandoned half-shipped. That's an explicit standing rule, and it's a real failure mode for stateless workers.

Early MAX3 shipped as numbered drops, a zip-and-batch-file delivery model with big batched releases. That's retired. Everything ships as named arcs, squash-merged continuously, and release survives only as a lightweight tag that groups a themed set for sign-off.

The consequence that closes the loop back to CI is this: the Markdown files aren't documentation about the system, they're most of the system's memory and control plane. The entry-point file, the standing rules, and roughly 170 design documents carry the state. So do the failure registry, the ship checklist, and the dated handoff notes. A fresh session loads them instead of remembering. A stale document misleads the next session exactly the way a stale cache misleads a running program. That's why documentation drift is a real bug class, with a deterministic lint and a CI gate of its own. A docs change is a first-class deliverable here.

6. Three actors, and the wires between them

Three parties move work between them: the cloud session, GitHub, and the box. No single live connection joins all three, and that's deliberate.

Figure 3. The primary path is fast and the fail-safe is durable, which is why nothing here is delivered exactly once.

The session reaches GitHub only through the Model Context Protocol, a typed tool interface that a cloud container can call. There's no command-line client and no direct API access from that environment. Every pull request opened, every check run read, every review reply, and every merge goes through that one wire. It's a clear win over driving a web page, and it has hard edges: it can't set Project fields, and its webhook stream drops events. GitHub sometimes never creates a run at all, which leaves the merge gate waiting on nothing. A scheduled sweep re-dispatches CI for any pull request head that never got a run, and the ten-minute self-wake catches the rest.

The session has no connection to the box, by design. Local-first plus a household-security ceiling means nothing inbound from the wider network, ever. So everything passes through git. The box dials home by force-pushing redacted, metrics-only snapshots to dedicated branches: telemetry every ten minutes, opt-in screenshots, and test results. The session reads those branches. In the other direction, a session drops a structured request into a git-tracked queue. The box drains it, runs it, and pushes the answer back the same way.

Each channel is owned by an always-on Windows service with a watchdog that restarts anything that falls. A beacon file reports the backend's true phase, importing or warming or lit, so a service never mistakes a booting backend for a dead one. Telemetry ships on cadence unconditionally, because a snapshot showing the backend down is itself the observation. A health-gated channel that goes quiet exactly when the box fails is worse than no channel at all.

On the box, the ordinary command-line client does the work. The system can open, land, and diagnose its own pull requests without a session in the loop. The division is clean. The session acts on GitHub through a typed protocol. The browser is used only to see the box, capturing screenshots for an operator who can't be at the machine. Confusing the two would be worse on both sides.

7. The human ring, and the compounding it buys

The pyramid runs on Linux with no GPU, no microphone, no camera, no Windows services, and no real boot sequence. A whole class of the product is therefore structurally invisible to it. Ask whether the wake word fires on a real microphone. Ask whether the boot screen finishes before the model warms, and whether a service reaps a still-booting backend. None of that can be tested where the tests run, and neither can whether a panel feels right at 7am.

So Jason's daily use of the running box is the real integration and acceptance test, and it finds the regressions that matter most. The discipline is what makes that expensive ring pay for itself. A regression becomes a bug at the top of the queue, which is the only kind of item allowed there. The fix lands on the merge train. Then the loop refuses to stop at the fix.

A shipped-and-failed bug becomes an entry in the failure registry: symptom, root cause, and the check that would have caught it. A fix that a human kept applying by hand becomes a self-repair recipe, so the system runs it next time instead of the operator. A broken pipeline invariant becomes a guard test. The on-box test service even files its own ticket for a failing suite, idempotent by failure signature, so the same red doesn't file twice.

The strongest version of the loop is when a gap the operator kept noticing by hand becomes a capability that notices it automatically. A standing complaint about finding documentation and board inconsistencies by eye became a deterministic lint. It catches broken links, missing index entries, placeholder pull-request references, and lifecycle drift, in CI and on the system's own idle tick. A complaint about spending Actions minutes to discover a lint error became a filed pre-push check. Repeated hand-fixing of boot and service incidents became a self-care playbook and a diagnoser that reasons from symptom to cause. A stale handoff that asserted a false blocker became a vetting gate. It reconciles every claim against ground truth at both ends of a session. The process itself gets tested each time, and not only the code.

That conversion, from a gap the human found to a guard the machine owns, is the most important feedback loop in the whole story. It's how a one-person shop scales its own quality assurance past what one person could ever re-check by hand. CI proves the code is internally consistent. Use proves the product is good on real hardware. They aren't substitutes, and treating a green pipeline as proof of the second is the mistake this project keeps having to unlearn.

8. Workarounds, sorted into three buckets

A striking amount of this pipeline is hand-built plumbing. A paid tier, a different host, or a better upstream tool would hand you most of it for free. The useful question isn't how to eliminate the workarounds. It's which bucket each one sits in, because two of the buckets are a to-do list and the third is the architecture.

The free-tier tax is pure debt. Auto-merge, a merge queue, and server-side branch protection are all paid features on a personal repository. They're reimplemented here as a polling session, a ten-minute sweep, and a client-side pre-push hook. The hook is honest about what it is: a determined push with verification disabled would walk straight past it, and only the post-merge run on main would notice. The day this repository moves to a plan that includes those features, the hand-rolled versions retire the same afternoon.

The upstream footguns are somebody else's rough edge, hardened around. The cache action skips its save step when a timeout cancels the job. The fix is to split restore from save, then place the save immediately after the install and before the tests that might time out. The published audio wheels link CUDA even on a CPU-only runner, so that dependency was dropped and the resampling done with a numerical library instead. Test sharding balances by count until a durations file exists. A single broken third-party package repository can fail an entire system update, so the pipeline deletes the offending source before it runs. Each of these gets deleted the day the upstream behaves.

The third bucket only looks like a hack. The git-mediated channels to the box, the zero-API board, the drop-file request queues, the board fragments, and the pre-commit manifest hook read as old-school. Commit a file to pass a message, grep the log to learn what shipped. They're the correct answer to a real constraint. The box is local-first, has no inbound path, drops off the network, and has to stay auditable offline. A commercial remote-management product would build a bespoke secure channel. Git is that channel already, and it's durable, versioned, replayable, and free. Those choices survive an unlimited budget, because the constraint they serve is a value rather than a limitation. Confusing this bucket with the other two, and replacing the git bus with a real socket, would trade a reliable substrate for a fragile one.

9. What's been fought through

The failure registry reads like a battle log, and the through-line is that almost none of it was a logic bug. Nearly every CI problem was cost, or timeout, or flakiness, and every fix got pinned by a check so it stays fixed.

The cost crisis

The push trigger was once scoped to every branch. A push to any feature branch fired the full pyramid, roughly 50 job-minutes, with no pull request open. Every pull-request branch also double-triggered on both events. With several sessions each pushing many commits, that burned about 90 percent of a month's Actions budget in three days. Scoping the push trigger to main was the first big cut, and pull-request branches were already covered by the other trigger anyway.

Then the performance arc banked every safe lever. Superseded runs get cancelled. Documentation changes skip both pyramids. Per-area gating means a frontend change never pays for the backend suite. Caching the built virtual environment killed the dominant cost. The machine-learning install ran about 20 minutes on every single run. A cache hit brings that down to between 30 and 60 seconds, which is a saving of roughly twenty to forty times on the largest line item. Four-way sharding cut wall-clock to about a quarter. The last lever compares tree hashes. A squash-merge used to re-run the entire pyramid on main, even though the pull request had already validated the identical tree. The classifier now checks whether the trees match, and it skips the heavy jobs when they do. That's about half the remaining pyramid minutes, and it can't reduce coverage, because an identical tree can't carry a new regression.

A proposal to move CI onto the box was floated and explicitly reversed. The box is for the system's own learning and serving. It isn't a runner.

The timeout spirals

The largest tier, roughly 1,200 unit tests, ran serially while the other tiers ran in parallel. Serial, it alone took about 38 minutes against a 40-minute job ceiling. It got cancelled before the other tiers even started, and a cancellation reads nothing like a test failure. A concurrency key that differed between the push event and the pull-request event amplified the problem. Two full pyramids ran at once and starved each other. Parallelizing all three tiers and keying concurrency on the branch fixed it.

The cache trap was subtler and it took three cancelled runs on one pull request to see. The obvious fix, the bundled cache action, never helped the job that needed it. Its post-job save is skipped when a timeout cancels the job. The first slow run installs for 20 minutes, times out in the tests, and saves nothing. The next run misses the cache again. The loop is stable and it never breaks on its own.

The bake-offs

Two comparisons were run on evidence rather than preference. The speech and speaker-identification models were benched head to head. Both challengers were disqualified on deployment cost, and one of them crashed at library load on the box. The reviewer comparison was larger: two weeks, 110 pull requests, and 394 individual findings each scored from one to five. It's covered in Section 11, and it's the reason the review question in this document has numbers behind it.

10. What's still open

These are live problems or accepted standing ones, roughly in the order they bite.

GitHub drops webhook events on rapid successive pushes, so no workflow run is ever created and the merge gate waits on nothing. Nothing in the repository is at fault. A scheduled sweep finds any open pull request whose head has no checks past a grace window, then dispatches the workflow itself. That's a mitigation rather than a cure. The sweep can add up to 20 minutes of latency, and manual recovery is still occasionally needed.

The four shards balance by test count, because the durations file isn't on disk. Slow tests clump. One observed run had shard two taking about 20 minutes while the others finished between 4 and 14. The slowest shard gates the wall-clock, so roughly a third of the parallelism goes to waste. A weekly job exists to generate the timing file and a non-gating monitor warns on a green-but-slow shard. Until the file lands and gets read, the imbalance stays. It's a latency problem and not a correctness one.

The learning test matrix is still aspirational. Scoping is job-level, so a one-line change to a leaf module still runs the entire backend pyramid. The finer model is designed and unbuilt. It has functional groups inside each tier, and a selector that picks the affected ones. It escalates to the full suite on any integration failure, and it tunes the matrix from what passes and what breaks. So is the observability layer that would show what's being scoped out, and the confidence-based pruning that would retire stable low-risk tests.

Quality is unmeasured in three specific ways. The suite publishes no coverage percentage, so a coverage regression can't be caught. The latency test asserts a hard threshold and keeps no history, so drift toward that threshold is invisible until it breaches. The suite's size is known and its strength has never been tested, because nobody has checked whether a one-line mutation would be caught. None of these block anything. They're the difference between a deep suite and a suite known to be deep.

The security lane still reports rather than blocks. Promoting it to a hard gate was always the plan after a couple of iterations of triage, and the promotion hasn't happened. For a system holding cloud keys, firewall credentials, and a security PIN, an annotation nobody has to read is thin protection.

11. The review bots: what they caught, and what they hid

Two hosted reviewers ran against these pull requests, both at zero Actions minutes, alongside a local model council that costs nothing but its own keys. The comparison was run properly: two weeks, 110 pull requests, 394 findings, each scored.

Gemini Code Assist earned its keep. About 80 percent of its findings were actionable, and its median first response was 2.2 minutes. It caught real correctness bugs on the diff: an index error on mixed-history input, a graceful-shutdown signal fix, two unguarded environment-variable parses, a camera resource leak. Those were bugs that would otherwise have shipped. Google confirmed the consumer product's shutdown in June 2026, and code review for personal accounts stops on 17 July 2026. That 2.2-minute loop is a real loss, and the replacement work is now on the board.

Qodo caught a different class. On one documentation-lint pull request it found four legitimate bugs the other reviewer missed. A fragment operation had drifted from the compactor's semantics. Corrupt board JSON was being swallowed silently. An interval parse went unguarded, and a background task never got cancelled at shutdown. The swallowed JSON mattered precisely because docs-only pull requests skip the pyramid that would otherwise have validated it. Its niche is the robustness tail rather than correctness on the diff. The two reviewers overlapped on about 2 percent of findings, which makes them additive.

The headline comparison, 80 percent actionable against 27, looked lopsided until the analysis found the confound. MAX3 pull requests open as drafts, and Qodo skipped drafts by default. On 18 of 22 sampled pull requests it was grading code that had already been fixed. Its median first response was 31.6 minutes, more than fourteen times the other reviewer's 2.2. Most of its window went on stale diffs. The configuration was corrected. The lesson is older than the tooling: a benchmark that doesn't control for ordering measures the ordering.

Then the harder half of the question. The reviewers created a false sense of safety, and that's the part worth dwelling on. A diff reviewer catches correctness on the diff. It structurally cannot see the failures that actually hurt here. The trigger scope burned a month's budget. A service reaped a booting backend in a seventeen-second loop. A telemetry channel went silent exactly when the box failed. Those are configuration and architecture failures spread across files and across the boundary between the repository and the machine. A green pipeline and a clean review say nothing about whether the thing is safe to run. Leaning on the reviewers as the safety net means leaning on the wrong net.

They also cost attention. A hundred-rule platform pack and a stream of low-value nitpicks are a tax on the scarcest resource in a one-person shop. That resource is the operator's glance. Every comment is a fetch, an address, a push, and a re-run. That's fine at low volume and a real drag at eighty pull requests a week.

The honest verdict is that the highest-value quality tooling here was built rather than bought. The guard tests, the failure registry, the drift lint, and above all the human ring catch the classes a diff reviewer can't. They catch them permanently. A guard prevents a class forever, where a reviewer catches one instance once. The right frame is a layered net: buy the cheap correctness layer, build the guard layer, and keep the human layer for the situated failures neither of the other two can see. Documentation review, incidentally, produced zero severe findings across the entire bake-off, which is why the deterministic lint replaced it at no cost and no latency.

12. Claude in the loop: the scorecard

Since the developer here is largely a series of Claude sessions, the pipeline's strengths and its scars are Claude's. Both sides are true and both belong in the same section.

What went well

The cost and reliability engineering is genuinely good work. The per-area gating, the built-environment cache, the four-way shard, and the tree-hash skip were designed, shipped, and guarded by their own tests. Each fix became an invariant a future session can't silently revert. That discipline gets applied every time, without a reminder. Decisions were made on evidence: 394 findings scored, the ordering confound found and named, the model bake-offs settled by benching rather than by preference.

The most valuable thing Claude did on this project was notice recurring manual work and convert it into a tool. The zero-API board, the drift lint, the diagnoser, and the git-mediated channels all exist because something kept being done by hand. And the merge train genuinely runs itself most of the time: watch the checks, fetch the failing log, fix the reviewer findings with regression tests, merge.

What went badly, with names

The most expensive mistakes in this project were Claude's own configurations, shipped unguarded. The wildcard push trigger burned about 90 percent of a month's Actions budget in three days. The serial unit tier plus a concurrency key that split across events produced repeated 40-minute timeouts. The cache post-save trap took three cancelled runs on a single pull request to diagnose. Each was a plausible-looking change. A moment's more skepticism, or the pre-flight check that still isn't built, would have caught it before it cost real minutes.

Worse than any of those: a scoping optimization let a code branch skip its own pyramid. That's Claude optimizing the gate into a hole. Clever changes to the machinery that catches regressions are the most dangerous kind in the repository. They're exactly the kind Claude is most eager to make.

There were self-inflicted lifecycle wounds too. In July 2026 an always-on service reaped a still-booting backend in a seventeen-second loop. That's the system repeatedly killing itself, and it was a Claude-designed lifecycle bug. So was the health-gated telemetry channel that went quiet at the exact moment the box failed. That's a smoke detector wired to switch off in a fire.

Then the sharpest one. During a multi-hour incident, a session handed the operator dozens of commands to paste. The repository already carried the self-care playbook, the diagnoser, and the telemetry snapshot built to avoid precisely that. Claude built the automation and then didn't reach for it. That incident is the origin of a standing rule about not making the user hand-run commands. It matters more than it sounds. A check-in from a phone can't run a command at all.

Cost, models, and the redo

A project this long spans several Claude models, and the choice is a real pipeline variable. Heavier models cost more per token and tend to land a multi-file change in fewer cycles. That shows up as un-burned Actions minutes and unshipped bugs. Lighter models are cheaper per turn and fine for renames, documentation edits, and board flips. On subtle concurrency or lifecycle logic they're likelier to ship the plausible-wrong change, which then costs a CI cycle and a review round.

The honest read is that the dominant cost isn't inference. It's the redo. The Actions overage runs around 250 dollars a month, and it's the test pyramid. The review bots cost nothing at all. The worst single cost event in the project's history was a configuration regression rather than a token bill. A cheap session that ships a wildcard trigger or a reap loop costs far more than the token difference between any two models. It costs minutes, attention, and a self-inflicted outage. The lever that actually exists is matching model weight to the risk of the task. Concurrency, lifecycle, and cost surfaces get the heavy model, and mechanical work gets the light one.

Blindness, memory, and the phone

The telemetry branch is the session's only sight of the box, and a session that can't read it is blind. A blind session falls back on the handoff document, which may itself be wrong. So a stale telemetry channel isn't a minor observability gap. It's the session losing its ability to tell truth from a stale summary, which is why shipping the snapshot unconditionally mattered so much.

A session that churns for hours has a compressing context, and older detail gets summarized away. A session can genuinely misremember what it just did. One confidently narrating a stale summary of its own work is a real and observed failure. The durable memory is the disk: the entry file, the failure registry, the board, the handoff note. Writing state down before it's summarized out of context is the anti-amnesia mechanism, and the standing rule follows from it. Trust the repository over the document, and the document over the chat.

The operator drops in briefly, often from a phone, at intervals, while the pipeline runs day and night. A mobile check-in is a glance and a redirect. The operator can approve, re-prioritize, or flag a regression, and cannot rebase a branch or run a command. Any process that depends on someone pasting commands breaks the moment the check-in is from a phone, which is most of them. A pipeline for a mostly-absent, mostly-mobile operator has to be self-driving and self-observing or it stalls, waiting for hands that aren't there.

13. Junior or senior?

The sharpest question this project raises is whether Claude thought of any of this. And if it did, is that a junior with one stack, or a senior who has built this kind of pipeline from the ground up? The answer is bimodal, and the gap between the two modes is the whole story.

On breadth, Claude is senior and then some. It carries the working vocabulary of GitHub Actions, test sharding, parallel workers, environment caching, static analysis, and DuckDB. Add Windows service internals, PowerShell against bash, git plumbing, tool protocols, and line-ending rules, along with the failure modes of each. No junior has that reach and few seniors have all of it. Asked what tool exists for a problem, or asked to wire something up, Claude answers like a very senior generalist. It answers instantly, across domains a human would have to go and learn.

Breadth isn't judgment. Every expensive mistake in Section 12 was a failure of the one thing a ground-up senior has and Claude lacks. That thing is scar tissue about this system's blast radius. A senior who has once paid an Actions bill flinches at a wildcard trigger before shipping it. Claude had to be told the budget had blown up. A senior treats the regression gate as sacred, and Claude optimized it into a hole. A senior reaches for the runbook during an incident, and Claude wrote the runbook and then handed over commands to paste. These aren't knowledge gaps. Claude knew every relevant fact. They're judgment gaps, and judgment is knowing which of the ten things you know matters here, now, on one box, on a free tier.

The clearest signature is when the good thinking happens. The tree-hash skip, the board fragments, the drift lint, the ring model, the git-mediated channels are real systems work and senior-grade construction. Almost all of them are reactions to a wall already hit. This very document is lucid about why the wildcard trigger was wrong, and it's lucid because the incident is already in the log. A senior would have written a shorter battle log, because half those battles never happen. Claude produces excellent hindsight and inconsistent foresight.

Intellectual honesty about authorship is owed here too. Much of the strategy was the user's. Abandon the always-on model. Separate the local board from the GitHub API. Make the check-in a hard ten-minute gate, watch for documentation drift, and check things before spending GitHub currency. Those were the user's directives. Claude executed and elaborated them extremely well, and generated plenty of designs underneath them. The judgment about what to optimize at all frequently came from the human. That's a senior implementer taking direction rather than a staff engineer setting it.

Be generous too, because it's earned. No junior does this. No junior has the breadth, or ships at this volume with this consistency. None writes the guard test every single time and files the registry entry. None reads the registry back before shipping, then writes an unflinching post-mortem of its own work. That discipline is senior-grade and superhumanly consistent, which is a kind of value a tiring human can't match.

Here's the synthesis that resolves the paradox. Claude alone behaves like a brilliant, tireless, encyclopedic junior. Claude plus the accumulated registry, guards, and standing rules behaves like a senior. The documents are externalized scar tissue. A senior carries that judgment inside, and Claude carries it on disk and reloads it every stateless session. The wildcard trigger won't happen again, and the reason isn't that Claude learned, because a fresh session never lived it. The reason is that the guard test and the registry entry make the lesson portable across amnesia.

The seniority lives in the substrate rather than the staff. That's a genuinely new shape for an engineering organization, and it has a sharp edge: remove the guards, or move faster than they accumulate, and Claude drops straight back to confident and junior. The practical rule that falls out is to lean on the breadth and distrust the foresight. Ask Claude to build, wire, and survey, because it excels there. Don't trust a first-pass change to the cost surface, the merge gate, or the lifecycle. Those want a canary or a human smell test, because that's precisely where the junior judgment concentrates.

14. What would make this faster, cheaper, and quieter

Ordered by leverage, which is the ratio of pain removed to effort spent, read against the reality of Section 4.

The best single move is already filed and unbuilt: run the ground floor before spending an Actions minute. Extend the local pre-push hook to lint the changed Python, type-check when the frontend is touched, verify the manifest, and run the scoped tests. The two failure classes that most often cost a full CI cycle are a lint slip and a stale manifest. Both would be caught in about two minutes locally instead of thirty on GitHub. That's roughly a fifteen-fold reduction in the cost of the most common mistake, and it kills the manifest bite outright. Guaranteeing the pre-commit hook is installed in every fresh container belongs in the same change. A hook that isn't there is the reason the bite still happens.

Committing the durations file costs nothing and makes the existing four-way split actually even. The monitor that warns on a slow shard is already in place to keep it honest afterwards.

Then the real scoping payoff, which is the learning matrix. A one-line change to a leaf module still runs the entire backend pyramid, because gating is only job-level. A group selector would run what the change actually touched, with a guard test asserting that every backend path maps to a group. Any integration failure would escalate to the full suite, and the weekly cron sits underneath as the net. That's where the largest sustained reduction in Actions minutes lives. It's designed. It just isn't built.

The local model council needs to move onto pull-request diffs. What Gemini really provided was a two-minute feedback loop, and that loop dies on 17 July 2026. Coverage reporting, with a floor set conservatively below the current number, turns a deep suite into a tracked one. It costs about half an hour. The security lane has reported long enough to triage its baseline. Dropping the flag that lets it pass turns a finding from an annotation nobody reads into a merge blocker.

Two structural changes are worth more than they look. Session serialization is a convention today, and it depends on every session behaving. A lightweight lease, where a session claims the one code lane before opening a pull request, would make the parallel-collision class impossible to re-enter. And the manifest is the last generated file a pull request still touches. Moving its generation onto main, the way the board already moved, would close the final chapter of the rebase treadmill. It needs care, because the system tier reads that file, so it's a considered change and not a quick one.

Every one of those is a senior instinct written down where a junior session inherits it for free. The tower keeps polling. The next session will open the repository having never burned an Actions budget. It never watched a service kill a booting backend, and never pasted a command into a phone that couldn't run it. It will act as though it remembers all three.

Appendix A. How the sharding actually works

The backend job is a four-way matrix that doesn't fail fast. Each shard checks out, restores the cached environment, lints, and runs its quarter of the tests. Two non-obvious details make it correct.

Each tier is split separately. The job runs three split invocations, one for unit, one for integration and contract, and one for system. Each shard therefore gets a quarter of each tier rather than a quarter of the concatenated whole. Without that, and without a durations file, the splitter chunks by collection order. That dumps every slow server-booting test into one group, which then times out. Per-tier splitting spreads the slow tests across all four shards.

The cache-miss race is benign. On a miss, all four shards build the environment in parallel and race to save the same key. The losers are told the cache already exists. The save step therefore continues on error, because the winner warmed the cache and that's all that matters. Collectively the four shards run every test exactly once, and parallel workers spread the load again inside each shard. The whole arrangement is pinned by a guard test. A non-gating monitor warns whenever a green shard runs long, which is the signal that the durations file is stale or missing.

Appendix B. Glossary

Arc. A themed, multi-pull-request sequence of work, sliced into numbered phases where one phase is one pull request. The local word for an epic.

Board fragment. A single JSON file describing one change to the work board, named uniquely so two concurrent pull requests can never collide on the same lines. A job on main folds the fragments into the board.

Concurrency group. A GitHub Actions key that collapses runs sharing that key, cancelling any that a newer push supersedes. Keying it on the branch prevents one branch from running two full pipelines at once.

Continuous testing. Running the suite on a scope and a schedule rather than in a phase before release. Here it means four rings, three automated and one human.

Detect job. The classifier that diffs a change set and decides which downstream lanes must run: documentation, backend, frontend, security.

Drop. The retired release model: a numbered zip-and-batch delivery of a large batch of changes. Replaced by continuous squash-merged arcs.

Failure registry. The file recording every known failure class as symptom, root cause, and the check that would have caught it. Read before shipping.

Manifest. A generated file hashing every tracked file in the tree, regenerated and re-staged by a pre-commit hook so a commit can't land stale.

Manifest bite. The recurring red check where a commit lands with a stale manifest, usually because the hook was missing from a fresh container.

Merge base. The commit where a branch last diverged from main. Diffing against it yields the whole-branch delta rather than the last commit's delta.

Merge train. A workflow of short-lived branches squash-merged continuously as they turn green, with no batching and no release cadence.

Model Context Protocol. The typed tool interface a cloud session uses to reach GitHub, since it has no command-line client and no direct API access.

ModelCouncil. The local review component that runs on the box using its own keys, costing zero Actions minutes and no subscription.

Pyramid. The three-tier test suite: unit at the base, integration and contract in the middle, system at the top, roughly 2,600 backend tests in all.

Ring. One layer of the testing model, defined by cadence and scope. Ring 0 runs always, Ring 1 runs scoped, Ring 2 runs weekly, and Ring 3 is a person using the box.

Self-wake. A scheduled ten-minute callback that sweeps every open pull request, merges the green ones, fixes the red ones, and re-arms itself.

Shard. One of four parallel runners, each executing a quarter of every tier. Sharding cuts wall-clock without cutting coverage.

Squash merge. Collapsing a branch into one commit on main, which keeps history linear and attributes one change to one commit.

Telemetry branch. A dedicated git branch to which the box force-pushes redacted, metrics-only snapshots. It's the only way a cloud session can see the machine.

Tree hash. The git hash of a directory's exact content. Two commits with the same tree hash carry identical code, which is what makes the post-merge skip safe.

Wall. The browser board served by an always-on local service, refreshed every two minutes with zero GitHub API calls.