Build tasks from merged pull requests

What it does

A task is one merged pull request, replayed. It starts from the pull request's base commit, the commit before the change. The agent gets a short written spec of what to do. The pull request's test changes are held back and used only to grade the result.

You build tasks in three stages:

1. Import: Fullbeam's AI writes a spec for each pull request you pick and checks it for leaks. You approve the clean ones.
2. Prepare: you add the commands that set up the project and run the tests.
3. Validate: Fullbeam proves the tests can tell a real fix from a broken one. A task that passes is frozen and ready to use.

Before you start

• Connect GitHub and wait for pull request history to import.
• You need the Owner role to import and approve specs. Authorizing validation needs the workspace's primary owner.
• Pick pull requests that change both code and tests. The tests are what grade the agent.

Which pull requests work

A pull request can become a task only if it:

• is merged and already imported into Fullbeam;
• changes both tests and code;
• has no binary changes;
• has a diff GitHub can return;
• is not too large for one spec.

Fullbeam marks other pull requests Not usable and tells you why. It does not send them to the AI.

Steps

1. Import merged pull requests

1. Open Cases in the sidebar.
2. Find Import merged pull requests.
3. Choose a Repository. The list shows up to 300 of the most recent merged pull requests Fullbeam has imported.
4. Tick the pull requests you want, or use Select all. You can send up to 50 at a time.
5. Select Infer specs for N pull requests.

Fullbeam works through them a few at a time and shows Inferring… for each one.

2. Review the specs

Each pull request appears under Inferred specs with one of these states:

• Ready to approve: no blocking leak was found.
• Leak found: edit before approving: the spec gives something away.
• Not usable: the pull request doesn't meet the rules above.
• Inference failed or Request failed: try again. The reason is shown.

Select Show spec to read it. To fix a spec, select Edit, change the text and select Save and re-check. Fullbeam runs the leak check again. You can also use Edit to write a spec by hand.

3. Approve the clean specs

Select Approve all clean specs. Specs with a leak are never approved. Warnings stay visible on the task. Then select Open cases to prepare.

4. Prepare each task

1. In Cases, open the Needs review view.
2. Select the tasks and choose Prepare for review.
3. Enter a Verification command, the command that runs the tests. Add a Setup command if the project needs one.
4. Optionally, for a single task, add AI review criteria. These are hidden from the agent and only add to the test result. They do not replace it.
5. Select Prepare selected cases.

Preparing saves the task. It does not start validation.

5. Validate each task

1. Open the prepared task and find Included validation.
2. Select Authorize included validation. The confirmation shows the maximum compute it may use.
3. Wait for Validation completed. Use Refresh status to update.

Validation runs four checks:

• The saved environment rebuilds and setup completes.
• The tests fail on the code before the change.
• The merged change makes the tests pass.
• A known-bad version of the change is rejected by the same tests.

If all four pass, the task is frozen and ready. The Harness evaluations flow uses every ready task automatically.

6. Group tasks into a suite (setup comparison only)

The Compare your current and proposed setup flow runs a suite you choose.

1. From a validated task, select Add this case to a suite, or open Regression suite in the sidebar.
2. Select Create draft suite and add frozen tasks.
3. Validate the suite. It needs at least two frozen tasks.
4. Freeze it, then select Compare this suite.

A frozen suite can't change. Make a new version to add or remove tasks.

What the leak check looks for

These findings block approval:

• Quotes a line of the solution
• Names a file the PR created
• Quotes the hidden tests
• Mentions the pull request
• Mentions a commit
• Mentions the branch

This finding is a warning only:

• Names something the PR introduced (fine if the tests call it)

If the first draft leaks, Fullbeam repairs it automatically once and checks it again. A separate AI reviewer also reads each draft.

Other ways to add tasks

• Add a GitHub case: pair a GitHub issue with a merged pull request, then select Freeze pair for review. You confirm the issue text still describes the original task.
• Add a Linear or Jira case: the same, using an issue from a Linear or Jira connection set up in Connections.

Both create a task that goes through the same prepare and validate steps.

Limits

• The spec is written by Fullbeam's own AI provider and budget, not your provider keys. If that budget runs out, you see "The AI budget for this account is used up."
• The leak check proves a spec doesn't copy the answer. It does not prove the spec is clear enough for an agent to succeed. Read each spec before you approve it.
• Validation needs an included compute allowance agreed with Fullbeam. If none is set up, the page asks you to contact Fullbeam.
• The Templates tab in Cases is not available yet.

Next: Set up agents, models and provider keys.