A leaderboard shows how a model did on public tasks with a published setup. Whether a change helps your team depends on your code, your agent setup and how your team reviews work. Those are different questions.
Every result links back to the setup, the tasks and the commit it came from. If something wasn't measured, you see that too, instead of a number that looks complete.
An agent that fixed a bug in a popular open source project has never seen your billing service or the fixtures its tests depend on. It does not know your service boundaries, your migration habits or the mistakes your reviewers keep catching. Those details decide whether a change gets merged.
Instructions, skills, MCP tools, permissions and checks can move the result as much as the model does. A leaderboard runs one published setup, which is rarely the one your team uses.
Engineers add context, correct wrong assumptions, fix failed CI and ask for follow-up changes. A one-shot score leaves all of that out.
Before you switch, you want to know how often a setup passes on your work and what each passing task costs, failed attempts and retries included. Most leaderboards do not report cost per passing task, and no public leaderboard runs on your code.
Use public benchmarks to build a shortlist. Then run the shortlist on pull requests your team already merged, graded by your own tests, before you change what your developers use.