Set up agents, models and provider keys

What it does

Before you compare, Fullbeam needs to know exactly what the current setup and the new setup are. A setup is the agent, the model, and the files that shape how the agent works, such as AGENTS.md, skills and config. Fullbeam reads setups from Git at an exact commit, so a result always points to the exact files that produced it.

You never install anything to run a comparison. Fullbeam runs every attempt in its own managed sandbox.

Supported agents

• Codex: supported in both comparison flows.
• Claude Code: supported in Compare your current and proposed setup only.
• OpenCode: not supported for comparisons today.

Fullbeam pins the agent version it runs. It does not use the version installed on your machines.

Option A: a pull request that changes the setup (Harness evaluations)

Use this for Codex. You describe the change as an ordinary pull request.

Before you start

• Fullbeam must have enabled Harness evaluations for your workspace. Until then, Start comparison stays disabled and lists what is missing.
• You need at least one validated task. See Build tasks from merged pull requests.

Steps

1. In a connected repository, create a branch that changes only harness files:
   • AGENTS.md (in any folder)
   • skills.md
   • files under .agents/skills/
   • .codex/config.toml (for example, the model)
   • .fullbeam/release.yaml
2. Open a pull request. Don't merge it.
3. Note the pull request number. You use it when you start the comparison.

What you'll see

When you resolve the pull request in Fullbeam, it shows the files that changed and a label for the kind of change:

• MODEL_ONLY: only the model or its reasoning settings differ.
• HARNESS_ONLY: instruction and skill files differ. Model settings don't.
• BUNDLE: both differ. The result can't be pinned on either change alone.
• UNCHANGED: both sides are the same, so there is nothing to compare.
• UNKNOWN: one side can't be resolved, for example because the model is unavailable. Fullbeam never substitutes a default.

A pull request that also changes other files, including anything under .github/, is rejected with the list of files. A change to reasoning effort alone is refused, because this flow can't enforce it.

Keys

In this preview flow, model calls go through Fullbeam's hosted runner account. You don't add a provider key for it.

Option B: two saved setups (Compare your current and proposed setup)

Use this for Codex or Claude Code when you want to compare instructions and skills.

Before you start

• Both setups must use the same agent, provider, model and network policy. Only instructions and skills can differ.
• Put skills in .agents/skills/<name>/SKILL.md for Codex or .claude/skills/<name>/SKILL.md for Claude Code.
• You need the workspace's primary owner to add a provider key.

Steps

1. Capture each setup from Git. Open the Find your setup in Git form at /home/<workspace>/harness. (This page is not in the sidebar. The Qualification page links to it when a setup is missing.) Choose the Repository, discover from a Branch or an Exact commit, give it a Release key and Version label, and select Discover setup. Do this once for the current setup and once for the new one.

2. Save each setup. Open Stack Registry. Under Prepare a setup for comparison, choose the captured setup, the Harness, Provider and Exact model, then select Prepare setup. This saves a Stack Release. It does not start any work.

3. Add your provider key. Open Qualification, expand Advanced release and connection controls, and find Provider credentials. Choose the Provider, enter a Profile name and the Provider API key, and set the limits:

   • Maximum concurrent attempts (1–4)
   • Per-attempt limit (micro-USD)
   • Evaluation limit (micro-USD)

   Select Store credential profile. The key is encrypted and never shown again.

Use an OpenAI key for Codex and an Anthropic key for Claude Code. An OpenRouter key runs an AGENTS.md setup on OpenCode.

You can also run on your own deployment. Choose the provider and enter where the key points:

• azure_openai: your Azure OpenAI resource, for example https://acme.openai.azure.com. Codex runs on it. The exact model is your deployment name.
• aws_bedrock: the AWS region of your Bedrock API key, for example eu-central-1. Claude Code runs on it. The exact model is a Bedrock model or inference profile ID.
• openai_compatible: a public HTTPS base URL that serves /chat/completions, for example https://llm.example.com/v1. OpenCode runs on it. Private addresses and ports other than 443 are refused.

The endpoint is stored with the key and can't be changed. To point elsewhere, add a new key. Fullbeam can't verify which model your deployment serves, so results record the model name it reports without verifying it.

Your key's spending limits are counted at a reference price of $5 per million input tokens and $25 per million output tokens, because Fullbeam can't know what your deployment charges. A model priced above that can cost you more than the limit, in proportion to its price; set the limits accordingly.

The Prepare a setup for comparison form offers only Fullbeam's own model catalog. To compare on your own deployment, have your agent start the comparison over MCP: prepare_comparison with model_access: "own_key", the key's provider_key_id and the exact model.

The setup comparison uses a per-attempt budget of $2, so pick a per-attempt limit of at least 2,000,000 micro-USD. The evaluation limit must cover the whole run for the credential to be offered.

Limits

• Fullbeam stores digests of your setup files, not the files themselves.
• Results from both flows are advisory in the preview.
• In Harness evaluations, the provider's runtime version, network isolation and reasoning effort are not verified.
• Never paste a provider key into a repository, issue, pull request, setup file or support message. If a key may be exposed, rotate it at the provider and store a new profile.

Agent collectors (separate from benchmarks)

Fullbeam also has collectors for Claude Code, Codex and OpenCode that record agent activity from your own machines, after each user authorizes their device. They are not needed to run a benchmark, and they don't affect comparison results. They are in limited release. Contact us if you want to try them.

Next: Run a comparison and read the results.