FullbeamDocumentation

Fullbeam product guide

Evaluate a coding-agent change on private repository work and prepare an evidence-bound rollout plan.

Fullbeam evaluates changes to coding-agent skills, harnesses, models, MCP tools, instructions, permissions, and workflows before rollout. It compares the current and candidate setup on private repository work, shows the evidence for which workloads may move forward, and prepares a rollout plan for an external deployment system.

Start with the concrete change. Before execution, Fullbeam freezes the complete setup as a Stack Release. HarnessOps is the broader practice of versioning, qualifying, releasing, and governing these software-producing systems.

Private-preview execution boundary: Private qualification runs in isolated, ephemeral Fullbeam environments in Daytona's EU target, through the Fullbeam model gateway and with customer-owned provider credentials. No customer runner installation is required.

Current authority boundary: Managed-native private-preview results are advisory shadow evidence. They cannot authorize or deploy a release. Release authority remains closed until the required production controls and receipts are deployed and bound to the active platform release.

Fullbeam does not deploy a Stack Release. GitOps, MDM, your coding-agent vendor, or your internal platform remains the deployment authority.

Fullbeam compares current and candidate coding-agent setups on private repository work and returns advisory workload-level evidence.

Product map

The team workspace contains these main areas:

  1. Connections: connect GitHub, work sources, a Stack Repository, and provider credentials.
  2. Cases: curate versioned evaluation tasks from real engineering work.
  3. Regression suite: freeze the cases and workloads used for qualification.
  4. Stack Registry: create immutable current and candidate Stack Releases.
  5. Qualification: run repeated comparisons and read the release decision.
  6. Rollouts: prepare and seal a canary plan for an external deployment system.
  7. Audit trail: inspect access, evidence coverage, runtime identity, and security events.

The normal operating loop is:

Define → Qualify → Target → Hand off → Verify → Learn

1. Create or open a team workspace

Use a team workspace for shared release operations. The workspace owns its connections, cases, Stack Releases, policies, evaluations, and rollout plans.

Workspace owners can configure connections, credential profiles, policies, overrides, and rollout intent. Members can use the read and action surfaces allowed by their role. Repository access is checked separately. Team membership alone does not grant access to evidence from every connected repository.

To add a teammate:

  1. Open Settings → Members.
  2. Invite the person by email.
  3. Select the smallest role they need.
  4. Review repository access in the connected GitHub installation.

Billing and subscription controls are under Settings → Billing when team billing is enabled. See Billing for the standard workspace billing flow.

2. Connect engineering systems

Open Connections to see available source types, connection health, selected scope, repository mappings, sync history, and webhook status.

Connect GitHub

GitHub provides exact commit, pull-request, check, review, merge, and repository identity.

  1. Open Settings → GitHub.
  2. Select Connect GitHub.
  3. Choose the GitHub organization or account.
  4. Select only the repositories Fullbeam should use.
  5. Return to Fullbeam and confirm that the installation is healthy.

Fullbeam reads the evidence needed for qualification. It writes its own qualification check and one stable summary comment. GitHub remains responsible for branch protection, review, approval, and merge.

Read the detailed setup guide: Connect engineering outcomes.

Connect work sources

The current managed Golden Set path supports GitHub Issues paired with accepted GitHub pull requests. Linear and Jira connector foundations exist, but they do not yet create managed Golden Set cases; do not treat a connector record as a runnable qualification source.

  1. Open Connections.
  2. Select the provider.
  3. Complete the provider authorization flow when it is available.
  4. Choose projects, repositories, teams, or other supported scope.
  5. Map the source scope to the correct repositories.
  6. Run or queue a sync.
  7. Check imported, rejected, and unavailable counts.

An Activation required state means that the external OAuth application or tenant setup is incomplete. It does not mean the connection succeeded.

Connect the Stack Repository and provider credential

The Stack Repository versions the customer-controlled configuration around Claude Code, Codex, or OpenCode. Select it through the GitHub App and resolve each release to an exact commit, immutable tag, or PR head SHA.

From Qualification, an owner creates an encrypted OpenAI, Anthropic, or OpenRouter credential profile. Fullbeam validates the supported harness/provider/model combination before launch. Credential values never enter Stack Release manifests, reports, GitHub output, or persisted evidence.

At execution time Fullbeam injects one short-lived gateway token into a fresh Daytona environment. The customer provider credential remains encrypted behind the gateway and is resolved only for an authorized model request. Fullbeam destroys the token and workspace after evidence packaging. See Connect coding-agent execution.

3. Build the Case library

A Case is one versioned evaluation task. It can come from a user story, regression, incident, reviewer correction, migration, or an authored task. "User story" is a source type, not the universal data model.

Open Cases to work with:

  • candidate cases waiting for curation;
  • ready cases;
  • cases that need review;
  • rejected cases;
  • regression candidates;
  • archived versions.

For each case:

  1. Check the repository, source type, workload, and risk tier.
  2. Review the visible agent bundle and the hidden evaluator evidence separately.
  3. Check reconstruction, leakage, environment, grader, and acceptance evidence.
  4. Queue preflight when the case needs validation.
  5. Apply the curation decision or reject the candidate with a reason.
  6. Freeze a new version when the task, grader, environment, or policy changes.

Existing tenants with a legacy authored-task integration may continue to submit its versioned manifest through the compatibility API. There is no managed-setup UI for that path. It stores only opaque references, digests, and provenance, and cannot enter managed qualification until its exact agent-visible and hidden-verification assets pass the same managed preflight, isolation, and leakage checks.

Safe display titles appear only when a redacted title is explicitly available. Otherwise the UI shows Case identity unavailable and keeps UUIDs in secondary technical details.

4. Create a Regression suite

A Regression suite is the frozen set of cases used to qualify a candidate.

  1. Open Regression suite.
  2. Create a suite or open a draft version.
  3. Add the exact Case versions to evaluate.
  4. Assign each Case to a supported workload class.
  5. Set the repetition and aggregation rules.
  6. Review missing or invalid evidence.
  7. Freeze the suite version.

Frozen versions are immutable. Create a new version to change membership, workload assignment, repetitions, or evidence rules. This keeps old release decisions reproducible.

5. Create a Stack Release

A Stack Release identifies the complete coding-agent execution condition. It can include:

  • model and routing;
  • coding-agent client or harness;
  • instructions and AGENTS.md policy;
  • skills, tools, MCP servers, and hooks;
  • permissions and reasoning budget;
  • verification workflow;
  • runner or container identity;
  • environment and dependency digests.

To create one:

  1. Open Stack Registry.
  2. Create a draft.
  3. Add or import the component references and exact versions.
  4. Review unavailable identity fields instead of guessing them.
  5. Compare the draft with the current release.
  6. Freeze the draft.

The frozen Stack Release and digest are immutable. Use a new release for every material change. Do not reuse a release identity for different bytes.

6. Define the Release Policy

The Release Policy defines what evidence is required and which result is allowed. Create it before starting the evaluation.

In Qualification → Release Policy builder:

  1. Select the benchmark and workload scope.
  2. Set repetition, acceptance, reliability, regression, cost, and evidence rules.
  3. Decide whether a production canary is required.
  4. Set runtime-identity and production-evidence requirements.
  5. Review the decision contract.
  6. Freeze the policy.

Changing a threshold requires a new policy version. Fullbeam never changes a frozen policy after seeing the result.

7. Run a Qualification

Qualification compares the current and candidate Stack Releases on repeated, like-for-like private work.

  1. Open Qualification.
  2. Create an evaluation draft.
  3. Select the current release, candidate release, Regression suite, and Release Policy.
  4. Select a valid provider credential and confirm managed-execution readiness.
  5. Start the evaluation.
  6. Monitor queued, running, completed, cancelled, invalid, and infrastructure-failed work.
  7. Open the final decision report.

Every valid attempt keeps its measured cost, including failed attempts. Missing cost or acceptance evidence stays unavailable.

8. Read the decision report

The report keeps four questions separate:

  • Lifecycle: did the evaluation run complete correctly?
  • System recommendation: what did the frozen policy conclude?
  • Evidence state: what is complete, partial, insufficient, unknown, or invalid?
  • Allowed next action: what does the current authorization permit?

Case results are evidence. They do not authorize a rollout by themselves. The workload decision is the release-control output.

A Fullbeam qualification report example with repeated Case results, cost evidence, and a separate workload-level move-to-canary decision.

For each Case, inspect:

  • safe display identity and source type;
  • frozen Case version;
  • decision role: decision-driving, diagnostic, or excluded;
  • observed and expected repetitions;
  • baseline and candidate outcomes;
  • candidate-only regression or incomplete-evidence status;
  • the authenticated evidence link;
  • secondary technical IDs and digests.

For each workload, inspect whether the recommendation is Move to canary, Keep baseline, or Continue evaluation.

Read economics correctly

Measured model and execution-attempt cost per accepted-task equivalent includes the measured cost of every valid attempt and divides it by the frozen repetition-aggregated acceptance result.

This is not automatically total engineering cost. Steering, retries, corrective intervention, reviewer effort, and rework appear separately. When their coverage is missing, the report says Unavailable and excluded.

Raw scheduled-work cost and cost per accepted-task equivalent answer different questions. Read both with acceptance and regression evidence.

Download a report

Use the report download action to create a signed JSON, Markdown, or PDF report. The authenticated export contains the entitled Case identity and evidence links. The GitHub projection is deliberately smaller and aggregate-only.

9. Use pull-request qualification

When the GitHub release gate is configured, Fullbeam binds qualification to:

  • GitHub installation and repository;
  • pull request;
  • base and head commit SHA;
  • immutable Stack Release digest;
  • evaluation decision;
  • active override, when one exists.

The Fullbeam Qualification check enforces effective authorization. The stable PR comment explains the system decision, override state, repository-safe workload actions, evidence coverage, and current/candidate digests.

When the PR head or base changes, the old qualification becomes stale. Fullbeam must qualify the new exact revision. An old decision or unsealed rollout intent cannot authorize the changed PR.

To make the check required, add Fullbeam Qualification to the repository ruleset or branch-protection rule. GitHub owns merge enforcement.

Inspect the supporting PR Story

Fullbeam can also turn a large pull request into a reviewable story. The PR Story groups related changes into chapters, links claims to code, marks areas to inspect or skim, and saves the reviewer's checkpoint.

Use the PR Story as supporting evidence:

  1. Read the overview and review-focus list.
  2. Open each chapter and inspect its explanation beside the related code.
  3. Follow citations and file links to verify important claims.
  4. Mark a chapter reviewed only after checking the code.
  5. Hand off to GitHub when you are ready to approve or request changes.
  6. Recheck changed or newly risky chapters after a new PR revision.

The PR Story does not approve a pull request and does not replace Stack Release qualification. See Inspect supporting PR evidence for the complete review workflow.

10. Use an override safely

An override does not rewrite the immutable system recommendation. It adds a separate, time-bounded authorization record.

Owners can:

  1. Open the decision.
  2. Record the override decision and reason.
  3. Select eligible and restricted workloads for a scoped promotion.
  4. Set an expiry time.
  5. Confirm the effective authorization shown in the report and GitHub check.

An empty, unknown, duplicate, or overlapping scoped-promotion workload set is invalid and fails closed. Expired overrides no longer authorize the check or a canary plan.

11. Prepare a canary rollout plan

Select Prepare canary only from a current, exact PR qualification that authorizes promote or scoped_promote.

The guided form covers:

  • eligible workloads;
  • target exposure;
  • external system type;
  • guardrails;
  • required evidence;
  • observation window;
  • rollback conditions;
  • exact external desired-state identity.

The workflow has two artifacts:

  1. Rollout intent draft: mutable while the owner chooses scope and conditions.
  2. Sealed canary plan: immutable after it is bound to an exact Git commit, MDM policy version, vendor configuration version, or factory release ID.

A prepared intent remains editable until sealing. Before sealing, review every guardrail, evidence requirement, and rollback condition in detail.

If the PR, candidate release, override, release lifecycle, or qualification digest changes, Fullbeam marks the plan HOLD: do not deploy. The sealed record remains available as immutable history but is not actionable.

Fullbeam stops at the sealed plan in the current product. It does not send the release to GitOps, MDM, a vendor, or an internal deployment platform.

12. Verify runtime and production evidence

Assigned release and effective runtime are separate facts.

Use runtime evidence to check:

  • whether the observed release exactly matches the assignment;
  • whether it is modified, a subset, a superset, or unknown;
  • observation coverage and missing identities;
  • downstream CI, review, rework, merge, abandonment, and defect evidence;
  • canary maturity and comparison quality.

Do not treat an external acknowledgement as proof that the release executed. Runtime observation is separate evidence. Production comparisons are observational and do not prove universal causation.

Current rollout planning does not include outbound deployment adapters or an acknowledged-deployment receipt workflow. Those are later product steps.

13. Use the Audit trail

Open Audit trail to inspect:

  • data boundaries and retention;
  • runner health and capability evidence;
  • connection scopes and secret references;
  • organization roles;
  • immutable security and release events;
  • explicit unavailable or unsupported observations.

Private task bodies, repository contents, generated patches, hidden graders, provider credentials, and command output are available only inside the authorized ephemeral attempt when needed. Source checkouts and credential material are destroyed with the workspace. Fullbeam persists the frozen Golden Set material required for repeatability in encrypted tenant-scoped storage and an allowlisted evidence envelope of identities, digests, scores, coverage, usage, cost, runtime observations, and authenticated evidence references.

Read Security and access for repository entitlement, role, webhook, and data-handling details.

14. Handle unavailable evidence

Fullbeam uses unavailable states deliberately:

  • Unknown means the fact was not observed.
  • Partial means only part of the required coverage exists.
  • Insufficient means the evidence does not meet the frozen requirement.
  • Invalid means the evidence cannot be used safely.
  • Infrastructure failed means the execution system failed; it is not an agent failure.

Do not replace these states with zero, pass, or failure. Repair the source, managed-execution readiness, identity, or coverage problem and run the required work again.

15. Daily operating checklist

For each proposed Stack Release:

  1. Confirm Connections, credential validation, and managed-execution readiness.
  2. Curate and freeze representative Case versions.
  3. Freeze the Regression suite and Release Policy.
  4. Freeze the current and candidate Stack Releases.
  5. Run repeated qualification.
  6. Read Case evidence and workload decisions separately.
  7. Confirm the exact PR revision and required GitHub check.
  8. Prepare and seal a canary plan only for authorized workloads.
  9. Hand the plan to the external deployment owner.
  10. Observe runtime and production outcomes.
  11. Turn regressions, incidents, and reviewer corrections into new Cases.

For common setup and evidence failures, use Troubleshooting and FAQ.