Public leaderboards run someone else's tasks on someone else's setup. Fullbeam runs your current setup and the new one on tasks from your own merged pull requests, from the same starting commit, with the same budget and the same pass rules.
Every result links back to the setup, the tasks and the commit it came from. If something wasn't measured, you see that too, instead of a number that looks complete.
Pick merged pull requests: bug fixes, tricky regressions, migrations, incident follow-ups, features. No linked issue needed. Each task starts from the commit before the change, and the pull request's test changes are held back for grading. The inferred spec is checked against the solution before any agent sees it.
The same agent can pass a task once and fail it the next time. Run the tasks that matter several times. Fullbeam records how much pass rate and cost vary, failures only the new setup has, and runs that don't count, so one lucky attempt can't carry the result.
Fullbeam reports measured model and runtime cost per task that passed, counting every valid attempt, failures and retries included. Human engineering effort is not in that number. Steering, CI fixes, reviewer changes and rework are tracked separately and show as Unknown until they are linked to the benchmark.
Roll out where the new setup meets your bar. Keep it away from the kinds of work where it fails. Keep testing where there isn't enough data. Your team makes each call, and your own tools carry it out.
A new setup can win on backend maintenance and still lose on database migrations. The results keep those apart, so you can roll it out for one and not the other.