Test the whole setup
A model name does not describe what your developers actually use. The agent, its instructions, skills, MCP tools, permissions and checks all change the result, so we version them and test them together.
Resources
How to benchmark a coding agent setup
Picking a coding agent setup is easier when you test it on your own work. These short notes explain how we think about it: what to test, how to keep the comparison fair, how to count cost, and what to do when data is missing.
In practice
What this looks like on your code
Every result links back to the setup, the tasks and the commit it came from. If something wasn't measured, you see that too, instead of a number that looks complete.
Keep the comparison fair
Use pull requests your team already merged. Start both setups from the same commit, run each task more than once, grade every attempt with the tests the PR merged, and show where data is missing.
Use the results to plan a rollout
Results show which kinds of work look ready for the new setup and which should stay on the current one. Your team decides, and your existing tools do the rollout. The tasks stay saved, so you can rerun them for the next change.
Keep a clear data boundary
Keep the setup you approved separate from what actually ran. Runtime records cover which models, tools and permissions were used, without requiring prompt or response text. Results come from tests, not from hidden reasoning traces.
Keep reading
Related pages
Core Thesis
Why coding agent setups are turning into shared infrastructure, and what that means for testing a change.
Read moreWhy public benchmarks fall short
Why a leaderboard score cannot choose your team's setup.
Read moreCost per accepted task
Why agent cost should be measured per task that passed, not per token.
Read moreApproved setup vs what actually ran
Why the setup you approved and the one that ran can differ.
Read moreSecurity
Review Fullbeam's data flow, repository access, data retention and what the product does not do.
Read moreGot a coding agent change coming up?
Read the core thesis first, then the notes on public benchmarks, cost and what actually ran. When you have a change in mind, we can benchmark it on pull requests your team already merged.
Your current setupThe change you want to tryPRs you already mergedAn answer for each kind of work