How MAX3 keeps itself honest

An 8 to 10 minute tour of the documentation hierarchy, test pyramid, build pipeline, security model, and quality engineering practices behind a local-first AI assistant.

Why this article exists

MAX3 is a local-first voice assistant that runs entirely on a single Windows machine. It listens for speech, looks at a webcam, queries a local knowledge graph, and (eventually) talks back in a voice that sounds like the HAL9000 from 2001. None of that ships in a single release. It ships in milestones, each adding capability while leaving everything that worked before still working. That second part is the hard one.

Most projects accumulate broken-window debt as they grow: a refactor that quietly breaks an obscure code path, a dependency upgrade that changes a default, a feature flag that flips three weeks late. The way MAX3 fights that is unglamorous: every piece of meaningful behavior has a test that would have caught the last bug, every commit goes through a gate that refuses to pass on a single failing test, and every milestone carries the regressions catalogue of every prior milestone forward. This article is a tour of how that machinery is built and how it grows.

The documentation hierarchy

MAX3 has eleven Markdown files at the project root. They are not interchangeable. They are a hierarchy with a deliberate read-order, and an AI assistant or new contributor that reads them out of order will misunderstand the project.

Figure 1 — five layers of project documentation, top-down read order

Constitution: PRINCIPLES.md

PRINCIPLES.md is twenty-three numbered rules. They are immutable: they describe what MAX3 is, not what it currently does. Local-first by default means the user's machine never makes a network call without an explicit, audited reason. Reading the tape is faster than interviewing five components means correlation IDs and an event log are first-class debugging substrate, not an afterthought. Stub markers tell the truth means a feature that isn't done yet announces itself in the UI rather than silently degrading. Every other document in the project derives from these rules.

Operating manual: CLAUDE.md

CLAUDE.md is what an AI assistant — or a human contributor — needs to know on day one. It compresses the principles into actionable conventions: file naming, the manager/worker/orchestrator split, where new code goes, the canonical technology stack with no deviations, and a list of do-not-do gotchas the project learned the hard way. When CLAUDE.md says don't pass LogRecord built-in keys via extra=, that's because Python's logging module raises KeyError if you do, and the project crashed every resource-tracker tick for an afternoon discovering this.

Wiring diagrams: ARCHITECTURE, PHASE_1_PLAN, README

ARCHITECTURE.md is the wiring diagram in compact form: process topology, the seven internal layers, the voice-turn data flow with latency budgets, the event-bus contract with Pydantic types. PHASE_1_PLAN.md is the milestone schedule (M1 through M8) with definition-of-done for each. README.md is the install-and-run instructions for a new clone. These three files together let someone who has never seen the project become productive within a few hours.

Working procedures: TESTING, WORKFLOW, LOGGING

These are the operational documents — read while working, not before. TESTING.md describes the test pyramid, where each new test goes, and the regressions catalogue. WORKFLOW.md describes the test-and-commit gate, the 100% rule, and how self-modifying-code requests are gated rather than autonomous. LOGGING.md describes the structured log line shape, levels, anti-patterns, and diagnostic recipes for common failures.

Audit trails: LICENSES, integration guides

LICENSES.md catalogues every dependency with its license — every milestone re-audits the set, with M8 doing a complete review. Integration guides like POLISH_PASS_INTEGRATION.md document what code lives where and how it wires together when components ship across multiple commits.

The point of this hierarchy is not bureaucratic. It's that someone reading PRINCIPLES.md first will understand why TESTING.md is shaped the way it is. Someone reading TESTING.md first will think the test pyramid is over-engineered.

The test pyramid — three layers, two codebases, one rule

MAX3's test suite is structured by tier, not by codebase. Unit means the same thing in Python and TypeScript: pure logic, no I/O, milliseconds of runtime. Integration means component plus real I/O. System means full stack. A single command (test.bat unit, for example) runs every unit test across both backend and frontend, giving confidence at one level across the whole system.

Figure 2 — the test pyramid; counts are deliverable-only and grow each milestone

Unit tier — the foundation

The largest tier. A unit test exercises a single function, component, or store in isolation. Mocked dependencies are fine. Real I/O is forbidden. Examples: a Pydantic Event model serializing correctly, a Zustand store handling a panel switch, a CSS-token map not colliding with a Tailwind utility name. Backend uses pytest with @pytest.mark.unit; frontend uses Vitest under src/components/__tests__/, src/state/__tests__/, src/design/__tests__/.

Integration tier — components with real I/O

Real I/O against the local component-under-test, mocked everything else. A SQLite write that commits to disk is integration. A psutil call that reads system state is integration. A WebSocket handshake between two in-process components is integration. A few seconds of runtime is acceptable; minutes is not. Backend tests live in backend/tests/integration/; frontend tests live under src/views/__tests__/ where they exercise a view plus its store.

System tier — full-stack acceptance

The milestone acceptance suites. A system test boots the app and verifies an end-to-end behavior: clicking the STRATEGY panel navigates to the game catalogue; the M1 boot event reaches the WebSocket; the keyboard navigation suppresses while focus is in a text input. Backend uses FastAPI's TestClient; frontend tests live under src/__tests__/ and walk the full Layout component.

The rule that makes the pyramid honest

Every fixed bug yields a regression test at the lowest tier that catches it. Non-negotiable. The TESTING.md regressions catalogue lists every fix from M1 onward — when something breaks, you don't just patch it; you add a test that fails before the patch and passes after. This is how the project keeps the pyramid honest as a regression chain.

The current deliverable carries 923 tests at the time of writing (489 backend + 434 frontend). Earlier milestones carried fewer; later milestones will carry more. A milestone that ships without test additions is, by convention, a smell.

What runs before any zip ships

The build pipeline is the gate that keeps MAX3 honest at the package level. Before any zip is produced for a user to install, every check below must pass. If any one fails, no zip is produced. The pipeline is deterministic, fast, and checked into the repo.

Figure 3 — the seven gates that run before any deliverable is built

1. Clean caches

Before anything else: remove __pycache__, .pytest_cache, node_modules/.vite, dist/, and any prior build artifact. Stale caches are the most common source of works-on-my-machine failures.

2. Ruff lint and format

Ruff runs against the backend with the project's pinned version (currently 0.7.4) and the project's pyproject.toml configuration: line-length 100, target Python 3.12, and a curated rule selection (E, F, W, I, N, UP, B, C4, SIM, RUF). A missing import or unused variable is a real bug; the linter surfaces it before any test runs.

3. Backend pytest pyramid

All backend tiers in order: unit, integration, system. Backend tests carry @pytest.mark.unit / @pytest.mark.integration / @pytest.mark.system markers so each tier can be filtered. The pipeline runs the full set; partial runs are for local TDD.

4. Frontend vitest pyramid

Vitest runs against jsdom-emulated DOM with @testing-library/react. The same tier structure (unit / integration / system) applies; each tier has its own npm script (test:unit, test:integration, test:system) so frontend developers can iterate locally without running the whole pyramid.

4b. Strict TypeScript check

tsc --noEmit with the project's strictest tsconfig. This is the gate that catches type drift between the backend's Pydantic event schemas and the frontend's TypeScript event types. It mirrors what run.bat does on the user's machine, so the package never ships with a type error the user would discover at install time.

5. Cross-tree contract checks

Eighteen invariants the deliverable must hold for the user's existing tree to keep working: tokens.ts exports specific COLORS keys (canvas, node, bus, elevated, connection, headerAmber, gridLine, lensCore, lensEdge); EYE_PALETTES carry a label with an em-dash because VoiceSettings.tsx splits on it; log.ts has getLogger and setLogLevel with the right signatures; run.bat installs from the project root, not backend/. Each invariant has a test that scans the deliverable file and asserts the contract.

6. Manifest and zip

A SHA-256 manifest is generated for every file in the deliverable (currently ~220 entries — grows each milestone), the zip is built, and a final sanity check verifies no cache files leaked into the archive. Only after all six gates pass is the zip placed in the outputs directory.

How the suite stays current as features land

The pyramid is not built once and frozen. It grows in lockstep with code. The discipline is small and explicit, and applies to every change that lands.

• New feature → new tests in the matching tier and codebase. A new React component goes in src/components/ with a test in src/components/__tests__/. A new pytest fixture for a new backend manager goes in backend/tests/unit/ or integration/.
• Bug fix → regression test at the lowest tier that catches it. The bug is added to the regressions catalogue in TESTING.md. The catalogue is the project's institutional memory; pruning it is forbidden.
• Refactor → existing tests must still pass. If they do not, either the refactor changed observable behavior (in which case the refactor is actually a behavior change and needs new tests) or the test was over-specified and the refactor is fine.
• Type change → strict tsc must still pass; both backend and frontend type schemas are updated together when an event shape changes.
• Dependency upgrade → license re-audited in LICENSES.md; lockfiles regenerated; full pyramid run.

What Claude does as new functionality lands

Claude — the AI assistant building MAX3 across these conversations — runs a specific sequence on every change. Before declaring work complete:

• Sync the deliverable into a clean cloud test environment that mirrors the user's Windows install.
• Run the full pyramid (currently 923 tests; 489 backend + 434 frontend). Inspect failures, fix, re-run. The 100% rule applies: zero failures before claiming done.
• Run strict tsc against the same source the user's run.bat will check. Type errors block the zip even if all tests pass.
• Run the contract-check scaffolding that verifies the deliverable is shape-compatible with the user's existing tree files (the ones not in the zip).
• Regenerate the SHA-256 manifest for every file in the zip; the manifest is the install-time integrity check.
• Only then build the zip and present it.

When something the user reports doesn't match the cloud's view, Claude searches the past conversation transcripts (which persist across sessions) to reconstruct the missing context — what was changed, when, and why — before guessing at fixes.

Coverage today, and what's deliberately not tested

Figure 4 — approximate test counts by component tier

Coverage is uneven on purpose. Components with high churn (state stores, design tokens, layout) are heavily covered because they break frequently. Components that are stubs or deferred (Strategy games beyond Tic-Tac-Toe, the LLM router, the knowledge corpus ingestion workers) are covered only at the contract level — they don't yet do anything to test.

Pending test work — areas the project deliberately deferred — is tracked in TESTING.md as a section. Audio format policy is one example: the policy is documented but the tests that would enforce it haven't been written yet because the audio pipeline isn't lit. Performance regression tests are another: the latency budget exists in CLAUDE.md, but the tests that would fail a build for exceeding it require real Whisper inference and are deferred to phase 2.

Deprecated testing patterns

A handful of patterns were tried early and removed:

• Snapshot tests on rendered HTML — they failed for cosmetic reasons and provided no signal beyond "the markup didn't change." Forbidden in this project.
• Heavy E2E browser automation (Playwright, Cypress) — the system tier covers acceptance well enough at the React layer with jsdom, and full-browser tests are slow and flaky on Windows. Deferred indefinitely.
• Coverage percentage as a gate — the number is misleading because it rewards line coverage over behavior coverage. The 100% rule is on tests passing, not lines covered.

Security: localhost-only, redact-by-default

MAX3 is a local-first system. The default trust posture is restrictive. Crossing a trust boundary is always opt-in, audited, and reversible.

Figure 5 — four trust zones; each boundary is explicit and audited

Trust zones

Trust zone A is the local machine. By default, every MAX3 server listens on 127.0.0.1, not 0.0.0.0; LAN access requires an explicit configuration change. Trust zone B is the MAX3 process — a single Python interpreter where lower layers don't call upward, the event bus is the only safe coupling between distant layers, and the audit log records every cross-component event with a correlation ID. Trust zone C is the browser — same machine, separate process, talking to the backend over a localhost-only WebSocket. Trust zone D is the optional cloud LLM path; HTTPS, audited, opt-in.

Data handling

Two storage substrates: an event log (append-only SQLite, indefinite retention) that records summaries with correlation IDs, and a short-term transcript table (configurable 7 to 30 day retention) that holds verbatim user speech for the what-did-we-just-talk-about recall window.

Redaction is the default. Any field that may contain free-form user input is redacted-by-default in queries unless the user explicitly enables developer mode. The structured log formatter does not redact for you — the project's logging conventions explicitly forbid logging full audio buffers, raw transcripts, or anything that could echo a user-spoken phrase outside the short-term transcript table. An AST-scanning unit test (test_logging_safety.py) walks every backend file and fails the build if any logging call uses a forbidden field name.

Threat model in one paragraph

The threats MAX3 takes seriously are: a household member's data leaking to another household member without consent (multi-user privacy boundary leak); audit logs growing into a privacy liability if they record content rather than metadata; a self-improvement loop committing a confidently-wrong patch; a knowledge corpus surfacing harmful content. The threats MAX3 does not take seriously are: nation-state adversaries, supply-chain attacks on every transitive dependency, sophisticated browser exploits. The system runs on one machine, single-user-or-household, and the threat model is calibrated to that.

Quality engineering from the start

Several practices land on day one and grow rather than getting added later. Adding them later is more expensive than starting with them; the cost of a typo finder retrofit on a million-line codebase is much higher than the cost of typing properly from the first line.

• Lint as a hard stop. Ruff for Python, tsc for TypeScript, both run before any test. A missing import is a real bug; surface it fast.
• Structured logging with correlation IDs from day one. Every cross-component event carries a ULID that ties it to the user-facing chain that started it. This is the first place to look when something breaks.
• Append-only audit log from day one. Behavior changes the system makes to itself are versioned and rollbackable; reading the tape is the debugging substrate.
• Test pyramid with the lowest-tier rule from day one. Default to unit; reach upward only when the lower tier can't see the bug.
• Stubs that announce themselves. A feature that isn't done yet says so in the UI ([STUB: Wikidata ingest not yet active]) rather than silently returning empty results.
• Manifest-checked deliverables. SHA-256 checksums for every file in the zip; the install script verifies them and reports missing or modified files at extract time.

How the practices expand as the project grows

The practices don't change in nature; their scope grows. Lint started with eight ruff rules; it's now ten and will reach more as the codebase gets larger. Logging started as INFO-level only; it's now four levels (debug/info/warn/error) with WARN gating. The test pyramid started with one system test (M1 boot acceptance) and a dozen unit tests; it now has 923 tests across both codebases. The audit log started recording five event types; it's recording closer to fifty. The shape stays the same. The volume scales with the system.

Risks the project tracks explicitly

Some risks have specific, named tests guarding against them. Others are tracked in the design document as items to watch but cannot be fully tested:

Future test plans

Several test categories are deferred to specific milestones rather than being absent by accident:

• Voice loop end-to-end tests (post-M3): real Whisper inference, real TTS, latency budget assertions. Defer until the voice loop is lit; mocking these earlier produces tests that don't tell you the truth.
• Ingestion worker tests (M3 onward): each knowledge corpus (Wikipedia, Wikidata, MusicBrainz, OpenStreetMap) has its own ingestion worker with format-version pinning. Tests assert that schema drift in a source produces a graceful failure, not a corrupt graph.
• Multi-user privacy tests (M5 onward): cross-speaker audit assertions; the share gate is enforced rather than discretionary.
• Self-improvement gate tests (M7 onward): every patch the system suggests must pass the same gate a human commit does; the LLM-suggested patch path is gated, not autonomous, and the gate is itself tested.
• Performance regression tests (phase 2): the latency budget exists today as documentation; phase 2 adds tests that fail the build for exceeding it.

Components — where the tests live, mapped to what they cover

Definitions

Terms used throughout the article. Project-specific definitions take precedence over generic industry meanings.

The point of all this machinery is not to be impressive. It is to make the next change easy. A project where every commit shifts work that the next contributor has to redo is a project that loses momentum. A project where every commit leaves the system runnable, verifiable, and slightly better than it was before is a project that, over a year of part-time work, becomes something useful.