Fullbeam turns pull requests your team already merged into a private benchmark. Your current setup and the new one work the same tasks, several times each, and the tests from each pull request decide what passed. You see what improves, what regresses and what it costs, for each kind of work.
Every result links back to the setup, the tasks and the commit it came from. If something wasn't measured, you see that too, instead of a number that looks complete.
Two teams on the same model can get very different results. Fullbeam records everything that shapes the outcome: the agent and its version, the model, instructions, skills, MCP tools, permissions, routing, checks and runtime image. Change one piece and you have a new version to test, like backend-tools@17 to backend-tools@18.
One clean pass can be luck. Each task comes from a pull request your team merged and starts from the commit before the change. Both setups attempt it several times, and the tests from that pull request decide pass or fail.
An agent can switch models, load a plugin or hand work to a subagent after the task starts. Where a supported agent connection is installed, Fullbeam compares what ran with the setup you assigned. Without one, that part of the result shows as Unknown.
Pass rate, spread between attempts, cost per task that passed and missing data sit side by side. Reviewer changes, CI fixes and rework are reported separately, and only once they are linked to the benchmark.
Roll the new setup out where it holds up. Keep the current one wherever the new one fails or there isn't enough data yet.