The approved setup is the plan
It lists the model, agent, instructions, skills, tools, permissions, routing, checks and runtime your team approved for the benchmark and meant to give developers.
Resources / Runtime
The setup you approved vs what actually ran
During a session, an agent can switch to a fallback model, load a plugin, pick up a changed tool, widen a permission or hand a step to a subagent. So the setup you approved and the one that ran can differ.
In practice
What this looks like on your code
Every result links back to the setup, the tasks and the commit it came from. If something wasn't measured, you see that too, instead of a number that looks complete.
What ran is the record
It follows the models, tool versions, plugins, permissions and subagents actually used while the agent worked on the task. Fullbeam can record this where a supported agent connection is installed. Without one, it shows as Unknown.
Credit the setup that did the work
A cheap model starts a task, gets stuck, and an expensive fallback model finishes it. That pass belongs to the fallback, not the cheap model. In the same way, a new setup cannot pass with a tool or subagent that was never part of what you tested.
Missing data narrows the claim
If what ran is confirmed, the result counts for that setup. If the record is partial, differs from the plan or is missing, the claim gets narrower, or that kind of work stays in testing.
Got a coding agent change coming up?
Results belong to what actually ran. If a run used a model or tool that was never part of what you tested, the approved setup does not get credit for it.
Your current setupThe change you want to tryPRs you already mergedAn answer for each kind of work