Four Models, One Repo, Zero Guarantees

My honest, results-not-typical field test of Grok, Claude, Copilot, and ChatGPT on GitHub. June 2026.

First, the fine print, because I have watched people read one stranger's post and bolt their whole pipeline to it. Everything here is my personal experience with these four tools, on my projects, with my standards files and my own gift for breaking things in fresh and creative ways. It is not a benchmark. It is not science. It is not a promise. Read it the way you read a 2 a.m. ad for a powder that melts belly fat while you sleep: specific, enthusiastic, and legally required to whisper results not typical. My results were these results. Yours may be better, worse, or weird in ways neither of us can predict.

So treat this like a testimonial from a guy who lost forty pounds and made six figures from his garage with one weird trick. Individual results may vary. Past performance does not guarantee future pull requests. I am not a doctor, a lawyer, or a paid spokesmodel for anyone named below. No repositories were permanently harmed in the making of these opinions, though one or two filed complaints. If you build your entire stack on this article and it tips over, that is mostly a you problem and a little bit a you-did-not-read-this-paragraph problem. With expectations now set to cautiously curious, here is what actually happened to me.

A year ago these four tools did clearly different jobs and the lines were easy to draw. In my use that stopped being true over the spring of 2026. They all converged on roughly the same shape: a terminal coding agent, some way to turn a GitHub issue into a pull request, a choice of underlying model, and a standards file the agent reads before it touches my code. The question I care about now is not who can edit a repo. They all can. It is which one I trust with what, where each one quietly breaks on me, and what it costs once I use it for real work instead of a demo.

Two things make any of this a moving target. Copilot moved to usage-based billing on June 1, and Claude's programmatic and CI usage splits into a separate credit pool on June 15. Grok's terminal agent is weeks-old beta. I am writing in June 2026, so treat the specifics as a snapshot and confirm anything load-bearing before you lean on it. See paragraph one.

Before any of this: the groundwork

A comparison like this can make it sound as though you pick a tool and start shipping. That is not how it has gone for me. Before any of these four earned its keep, I had to put in a lot of upfront work, and most of it had nothing to do with the model. Markdown files spelling out standards and requirements. A test suite with real gates. The coding scaffolding. The integration wiring. The conventions that tell an agent what “done” means here rather than in general. That groundwork is what makes any of these tools useful, and none of them will build it for me.

Every one of these tools has its own little gotchas, and no model is exempt. I find them fast, sometimes mid-task, and usually at the cost of an hour I wanted back. One wipes a file when I overload it. One drifts on a long roadmap. One floods me with vague errors until I throttle it. The pattern is always the same: I hit the quirk, I learn it, and from then on I am the one responsible for never hitting it blind again.

The only thing that has reliably worked is writing the lesson down where the tool will read it, in a revision-controlled md file. CLAUDE.md, AGENTS.md, a skills pack, whatever the tool calls it. I put the gotcha in the standards file, commit it, and the next session starts from what I already know instead of relearning it the hard way. The tools differ in plenty of ways below, but not in this. Whichever I choose, the groundwork is mine, and the md files are where it lives.

The matrix

Here is how the four shook out for me. I would read the bottom three rows first: standout strength, main weakness, and best-for carry most of the decision, and everything above them is my evidence.

One note on benchmarks, since they get quoted at me out of context. Grok Build reports about 70.8% on SWE-Bench Verified and Claude Opus 4.8 reports about 69.2% on SWE-Bench Pro, but those are different benchmarks at different difficulties, and Copilot and Codex run whatever model I point them at, so no single leaderboard number settled this for me. I judged them on fit, not on a decimal.

Grok (xAI)

GitHub integration

Grok gave me two distinct GitHub stories, and I keep them apart. The first is the connected-tool integration inside the Grok chat app: through the Connectors layer it reads my repo over the GitHub REST API, lists trees, opens files, searches code, and makes targeted writes and commits. This is the mature, dependable half, and it is read-heavy by design. The second is Grok Build, xAI's terminal coding agent, which launched in early beta in mid-May 2026 and advertises native repository, branch, and pull-request handling with a local-first model that keeps my code on my own machine. The early builds I touched had the GitHub piece flagged as uneven, so I still treat the in-chat connector as the safer bet.

Features and capabilities

In chat I got full-repo reading and search, single-file edits, branch creation, and genuinely strong documentation and planning output. Grok Build adds an interactive CLI with a plan mode that forces it to show proposed changes before touching anything, up to eight parallel sub-agents each working its own branch, an Arena mode that runs models side by side, and a Skills system that xAI states is compatible with Claude Code skills and CLAUDE.md files. It runs on grok-code-fast-1 and the newer grok-build-0.1, with a 256K-token context window.

Risks

The chat connector carries every limit in xAI's own notes: it is built for one file at a time, large edits depend on exact string matching, and it cannot see my CI. The worst version showed up when I overloaded it. Instead of a bad patch, it quietly replaced an entire file of code or documentation with a single one-line placeholder, and I did not see the extent of the damage until my gated test suite caught it. I never skip that gate after a Grok edit now. Large files also get truncated mid-edit, and reads, searches, and writes all count against the GitHub ceiling of 5,000 requests an hour on an authenticated token. Grok Build is the bigger unknown for me: it is days-old beta, entry runs about $300 a month at the top SuperGrok tier, and in my hands its CLI GitHub integration was the weakest part of the whole package.

Best uses

I reach for Grok to investigate an unfamiliar repo, draft integration guides and architecture docs, plan a large feature before anyone writes it, and make small, atomic edits I apply myself. If I am trying Grok Build, I treat it as an experiment, not a load-bearing part of my pipeline yet.

In practice, Grok was the sharper tool for me on small, specific fixes, where its focus showed and it did not wander. It was also stronger on architecture and on the detailed, step-level instructions a human coder or a tooling-research task actually needs, spelling out the how more concretely than the others did for me. When my deliverable was a clean plan a person would execute, or a tight change to one part of a system, that is where it earned its place.

Quick start

• In Grok, open Connectors and connect GitHub over OAuth with repo scope.
• Point it at a repo and have it read the target file in full before proposing any change.
• Keep edits to one file; for anything large, ask it to return the entire rewritten file rather than a patch.
• Apply locally on a feat/ or fix/ branch, commit in small increments, and always run your gated test suite before trusting the result.
• To try the agent: curl -fsSL https://x.ai/cli/install.sh | bash, sign in with a SuperGrok or X Premium Plus account, and start in plan mode for anything non-trivial.

Claude (Anthropic)

GitHub integration

Claude's GitHub work runs through Claude Code, which lives in the terminal, in VS Code and JetBrains, in the desktop and browser apps, and through an official GitHub Action (anthropics/claude-code-action@v1). Claude Code reads my codebase, edits files, runs commands, and opens pull requests. The Action runs the full Claude Code runtime inside my own GitHub runner, so my code stays on GitHub's infrastructure, and it responds to issue and PR comments, posts reviews, and turns a described task into a complete PR. It reads my CLAUDE.md and follows the standards I set there.

Features and capabilities

I got strong multi-step, multi-file refactoring with me supervising, deep debugging, instant PR creation from a plain-language description, and automated code review and CI-failure remediation in Actions. Opus 4.8 (released May 28, 2026) is the flagship I use for hard reasoning across large codebases; Sonnet 4.6 is my default for routine fixes and costs roughly 60% less.

Risks

The honest weak spot for me was cost and billing complexity, not capability. From June 15, 2026, programmatic use splits into a separate dollar-denominated Agent SDK credit pool: interactive Claude Code in my terminal and IDE keeps drawing on my subscription, but the GitHub Action, claude -p, and any third-party agent bill at full API rates from that separate pool, and when it runs dry my background jobs stop until it refreshes. I plan and budget for that before I wire Claude into CI. Claude can also be cautious, and Anthropic's February 2026 terms tightened how subscription OAuth tokens may be used outside official tools.

There is also a behavioral failure mode I have to name. On a large, multi-feature roadmap Claude got spacey on me, jumping between threads and leaving features partly built or their wiring unfinished, so a long task that looked complete sometimes was not. It is still the tool I trust most for large multi-product builds, but I hold it to an explicit roadmap, make it check off each feature as done, and verify the wiring myself rather than trusting that a big job came back whole.

Two operational notes from real use. Claude's GitHub access through the MCP integration kept getting overwhelmed under load and failing with nondescript messages that told me almost nothing, and my fix was a governor: throttle the calls and back off for a few minutes before retrying rather than hammering it. Its handling of GitHub Issues, and Projects in particular, was inconsistent and buggy for me. The sharpest example I confirmed: on personal-account repos, custom Project board fields like Status, Priority, Area, PR Arch, and PR simply cannot be set through the MCP from any session. The list_issue_fields call comes back empty for the project, every time. It works on org-owned repos and fails on personal ones, so the honest answer is yes for issue-level state, no for custom Project fields, and that is a GitHub MCP platform limit rather than a Claude oversight. For any personal-account repo this is such a pain, and I plan to manage that side by hand. The honest way I run Claude on GitHub housekeeping is to watch it the way I watch a teenager doing chores. The work gets done, but I check that it actually got done.

Best uses

Large multi-feature builds, complex reasoning, supervised refactors, deep debugging, and automated PR review where I want a model that respects a detailed standards file and stays in the loop with me. On the big builds I keep it to a written checklist so nothing ends up half-wired.

Quick start

• Install Claude Code (npm install -g @anthropic-ai/claude-code) and run claude inside your repo.
• Add a CLAUDE.md at the repo root with your conventions, test commands, and house style.
• For CI, add anthropics/claude-code-action@v1 to a workflow with your ANTHROPIC_API_KEY (Max subscribers can use an OAuth token instead).
• Mention @claude in an issue or PR comment to have it implement, review, or open a PR.
• Leave routine jobs on Sonnet 4.6; set model: claude-opus-4-8 only for the hard ones. Budget the Agent SDK credit for CI work.

Copilot (GitHub)

GitHub integration

This is the deepest native integration of the four, because Copilot is GitHub. It works inside VS Code and JetBrains, on GitHub.com, and in the mobile app, and chat lives in the IDE sidebar and on GitHub.com. Agent mode went generally available on both major IDE families in March 2026 and runs autonomous multi-step tasks: it decides which files to change, runs terminal commands like npm install or pytest, and iterates on errors on its own. The background coding agent turns an assigned issue into a pull request without me watching, and agentic code review gathers full project context and can hand its suggestions to the agent to generate fix PRs.

Features and capabilities

Inline completion that is still, for me, the best in class for daily flow, agent mode for medium-scale autonomous work, issue-to-PR automation, and model choice on paid tiers across GPT-5.4, Claude Opus 4.6, Gemini, and o3, switchable per task.

Risks

The real risk for me now is the bill. Copilot moved to usage-based GitHub AI Credits on June 1, 2026, metered by input, output, and cached tokens at each model's API rate. A quick chat question and a multi-hour autonomous agent run used to cost the same; they no longer do, and I found leaning on frontier models drains the allotment faster at the same sticker price. Enterprise also carries a hidden line item, since it requires GitHub Enterprise Cloud on top of the seat. Copilot is also, for me, the weakest of the four at high-level architecture and long-form planning; it completes my thought better than it reasons about a system from scratch.

Best uses

Everyday coding inside my editor, GitHub-native teams that live in PRs and Issues, and medium-scale changes where the project context the IDE already has does most of the work.

Quick start

• Enable Copilot on your plan and install the extension in VS Code or JetBrains.
• Use inline suggestions as you type; open the sidebar for chat and explanations.
• Switch to agent mode for a multi-step task and let it edit across files and run commands.
• Assign a GitHub issue to Copilot to get a background PR you can review.
• Turn on Copilot code review for the repo, and watch your AI Credit usage under the new metering.

ChatGPT (OpenAI)

GitHub integration

ChatGPT reaches GitHub through Codex, OpenAI's coding system, which spans a CLI, an IDE extension, a GitHub bot, and cloud delegation from inside ChatGPT. The cloud path is the one that defines it for me: I describe a task in plain language, Codex provisions a sandboxed cloud environment, clones my repo, does the work, runs my tests and linters, and hands back a diff or a draft pull request. The sandbox is network-isolated, ephemeral, and scoped to the repo. I connect a repo at chatgpt.com/codex and configure conventions and test commands in an AGENTS.md file.

Features and capabilities

The strongest hands-off autonomy of the four for me: I assign a well-scoped task and collect a finished, test-verified PR a few minutes later, including larger multi-file work, because the sandbox holds the whole repo. Multiple tasks run in parallel. The IDE extension adds inline completion, multi-file refactoring, and a side-panel agent that shares state with the CLI and the cloud, so I can start in the editor and finish in a sandbox. Codex Security, added in March 2026, is a dedicated agent for finding and fixing vulnerabilities. The current generation runs on GPT-5.5.

Risks

The async model is the strength and the catch in one. There is no real-time back and forth, so the output rode almost entirely on how clearly I wrote the task; a vague request came back as a vague PR. It rewards a precise spec and a real AGENTS.md and punishes hand-waving. For me, full Codex access effectively means ChatGPT Pro at $200 a month, and the cloud sandbox means my code runs on OpenAI's infrastructure rather than my own, which some teams I work with cannot accept.

Best uses

Batched, clearly specified work I want to hand off and forget: bug fixes with known reproduction steps, mechanical refactors, dependency bumps, and well-defined features where I can write a tight task description.

Quick start

• On ChatGPT Plus or higher, go to chatgpt.com/codex and connect your GitHub repo.
• Add an AGENTS.md with your test commands, lint setup, and conventions.
• Describe a task in the sidebar and choose Code for edits or Ask for explanations.
• Let Codex work in its sandbox, then review the diff and the test logs it returns.
• Approve and open the PR, or iterate with a refined task description. For local work, install the Codex CLI or IDE extension.

Choosing

I do not pick one. I assign by the kind of work. Copilot for the minute-to-minute coding in my editor and for teams that live on GitHub. Codex inside ChatGPT for batched, well-specified jobs I want to delegate and collect as a PR. Claude for large multi-feature builds and complex supervised refactoring, kept on a tight checklist so it does not drift and leave features half-wired. Grok for small, focused fixes, for architecture, and for the detailed coder-facing instructions and tooling research where it gave me the most usable level of detail, with Grok Build as an early-adopter option if I want a local-first CLI and can tolerate beta. I run two or three of these in parallel, pointed at different task classes. Your mix may differ, and say it with me, your results may vary.

This is a June 2026 snapshot from one engineer's actual desk. Model versions, prices, and the two billing changes here will keep moving, and the tools will sprout new quirks the week after I publish. Verify anything you are about to depend on. Results not typical. Void where prohibited. Consult your CI before starting any new pipeline.