Troubleshooting and FAQ

Troubleshooting

I see a control, but the action is refused

Many actions need the workspace's primary owner, not just an owner. This includes adding provider keys, authorizing validation, working with suites and starting comparisons. Other owners can see these controls today, but Fullbeam refuses the action. Ask the primary owner, shown as Primary Owner on the Members page.

I can't install the GitHub App

Only a Fullbeam owner can start the install, and only a GitHub organization owner can approve it on GitHub. As a member, select Ask an owner to install it on Overview, then use Request owner approval or copy the install link and send it to your GitHub owner.

My repository or pull requests don't show up

1. Check on GitHub that the repository is selected in the Fullbeam app installation.
2. Wait for the background import to finish. Fullbeam imports the last 30 days of pull request history after installing.
3. In GitHub, check the app's recent webhook deliveries for errors.

The Import merged pull requests list shows only merged pull requests that Fullbeam has already imported, up to the 300 most recent.

A pull request is marked "Not usable"

The reason is shown next to it. The usual causes:

• The PR must change both tests and code. Its tests become the hidden grader, so a pull request with no test changes can't be graded.
• The PR contains binary changes.
• GitHub's diff for this PR could not be read.
• The PR is too large for one spec inference.

Pick a different pull request. These can't be fixed from Fullbeam.

A spec says "Leak found: edit before approving"

The spec gives away part of the answer. Select Show spec to see the finding, then Edit, remove the leaked text and select Save and re-check. A spec with a blocking leak is never approved.

Spec inference failed

• The AI provider did not respond or did not return a usable spec: try again.
• Every spec the AI drafted leaked the solution: try again, or write the spec by hand with Edit.
• The AI budget for this account is used up: contact Fullbeam.

Validation won't start

• Ask the primary workspace owner to review and approve this included validation. Only the primary owner can authorize it.
• Contact Fullbeam to confirm the included validation allowance for your agreement. No allowance is set up yet.
• The remaining included compute is not enough for this validation. Fullbeam never adds overage automatically.

Validation failing on one of its four checks usually means the verification command is wrong or the tests don't fail before the change. Open the task, fix the command with Correct setup or verification commands, and validate again.

"Start comparison" is disabled in Harness evaluations

Harness evaluations needs setup on Fullbeam's side during the preview. The page lists what is missing. Contact Fullbeam to enable it. Also check that you have at least one validated task and that your pull request changes only harness files.

My pull request was rejected by Harness evaluations

It changes files outside the harness, or GitHub returned an incomplete file list. The message names the files. Move application changes and anything under .github/ into a separate pull request. A change to reasoning effort alone is also refused.

No provider credential is offered in the setup comparison

The credential must match the agent's provider (OpenAI for Codex, Anthropic for Claude Code), and its limits must cover the run. Set the per-attempt limit to at least 2,000,000 micro-USD and an evaluation limit that covers every attempt. Only the primary owner can store a credential.

A number shows "Unknown", "Unavailable" or ends in *

Fullbeam did not measure that value, or measured only part of it. It is never treated as zero. Reasoning tokens are always Unknown in Harness evaluations.

I can't open an attempt's changes or create a draft review PR

Attempt files are kept only if you ticked the option before preparing the comparison, and only for seven days. The draft review PR also needs owner access, a completed attempt, and files without credential or workflow content.

FAQ

Which agents can I compare?

Codex in both flows. Claude Code in Compare your current and proposed setup. OpenCode is not supported for comparisons today.

Can I compare two models?

Yes, with Harness evaluations for Codex: open a pull request that changes the model in .codex/config.toml. The setup comparison requires both sides to use the same model.

Can I rerun the benchmark when a new model ships?

There is no one-click rerun of a finished benchmark. Start a new comparison on the same tasks. In the setup comparison, Review and run again retries a run that failed or was cancelled.

Whose model keys are used?

In the setup comparison, your own OpenAI or Anthropic key. In the Harness evaluations preview, Fullbeam's hosted runner account. Spec writing always uses Fullbeam's own AI provider.

Does the agent see the answer?

No. The agent gets the repository at the commit before the change and the approved spec. The pull request's code and tests are held back.

Does Fullbeam deploy the winning setup?

No. Fullbeam reports results. Your own tools roll out changes. A sealed rollout plan is a record for those tools, not a deployment.

Does Fullbeam approve or merge pull requests?

No. GitHub owns review and merge.

Can Fullbeam prove a setup caused a better result?

No. Results describe what happened on your tasks, with the sample size and the gaps. They are not proof of cause.

Can we pick only some repositories?

Yes. Choose them when you install the GitHub App and change the list later on GitHub.

How do I get access?

Fullbeam is in invitation-only private preview. Contact us.

What should I include in a support request?

The workspace, repository, pull request or run, and the exact message you saw. Never include provider keys or other secrets.