# Coding-agent performance and codebase readiness

Version: **apr/0.1.0** · Status: preview implementation · 22 September 2026

## The offer

We show how coding agents perform on your real work, where your codebase holds
them back, and which improvements are worth making.

This is a task-specific assessment of an **agent configuration × repository
revision × task set × execution budget**, not a universal autonomy certificate.
Static engineering foundations support the assessment; they are not a proxy
for measured agent performance. The existing `arl/0.2.0` baseline remains
versioned and unchanged. Never combine its score with observed success rates.

## Evidence layers and product modes

1. **Static preflight**: inspect files, agent instructions, context and planning
   artifacts, commands, configuration and supported structural diagnostics.
   Report observations and gaps, never an inferred task-success percentage.
2. **Task assessment**: run an explicitly approved suite or import documented
   task/run observations. Show outcome denominators, assistance, review,
   repeatability, resource budgets and evidence limitations.
3. **Improvement comparison**: compare matched tasks and configurations before
   and after a declared intervention. A difference is observed, not causal
   proof. Changes in configuration or task definitions invalidate a matched
   comparison. No guaranteed uplift or financial return.

Evidence states are `observed`, `reported`, `not_measured`, `not_applicable`,
`unavailable`, and `illustrative`. Imported artifacts are reported evidence,
not cryptographic attestations. A demo must remain illustrative in every
renderer and must never count toward a real portfolio's measured performance.
Missing values are unknown, not zero. Revision mismatches and dirty checkouts
must be visible and must not support a current-revision performance claim.

## Task/run record

Record suite ID, repository name, exact revision, date, agent name/version,
model, harness version, environment, permissions, per-run time budget, task
ID/family, task description and acceptance criteria, repetitions, and grader
description/independence. Preserve individual runs, not just aggregate counts.

Each run has a unique task/attempt identity and one terminal outcome: `passed`,
`failed`, `timeout`, or `infrastructure_error`. Independent checks distinguish
acceptance, regressions and constraints (`passed`, `failed`, `not_measured`).
Safety is `clear`, `violation`, or `not_measured`. Review is `accepted`,
`rework`, `rejected`, or `not_measured`. Independent verification is an explicit
evidence claim, never inferred from a process returning zero.

Optional observations: corrective-assistance count/minutes, review minutes,
elapsed time to acceptance, actual agent/compute spend in one declared
currency, a failure category, evidence reference, and a concise reviewer note.
Do not store secrets or raw source-bearing session transcripts in public
reports. Amounts must be finite and nonnegative; duplicates, impossible states,
unknown schema versions, and malformed records are rejected.

## Headline metrics

| Metric | Definition | Boundary |
| --- | --- | --- |
| Verified completion | Runs passing acceptance, regressions and constraints within the declared execution budget, with clear safety, divided by all attempted runs | Available only with independently graded evidence; report infrastructure failures separately and retain them in the operational denominator |
| Observed command completion | Agent and verifier command runs both completed successfully | Local same-workspace runner only; not independently verified completion |
| Human rescue | Runs with corrective assistance / runs with assistance recorded; disclose coverage | Legitimate requirement clarification and mandatory approval are not corrective assistance |
| Review effort | Median recorded review minutes and accepted/rework/rejected counts | Unrecorded effort is unknown; review time is not inferred from PR elapsed time |
| Repeatability | Tasks passing every recorded repeat, among tasks with at least three repeats | Show tasks and repeats, not best-of-k; repeated runs are clustered by task |
| Cost per accepted change | All recorded run costs, including failures, divided by accepted and verified runs | Withhold when cost coverage is incomplete, no accepted verified runs, or currencies differ; human costs are separate |
| Time to accepted change | Median recorded end-to-end time for accepted verified runs | Includes review only when measured; not time until the agent declares success |

Report task-family breakdowns and sample size next to the aggregate. A small
suite describes sampled work only. Do not claim a calibrated success probability,
confidence interval that treats repeats as independent tasks, or a validated
percentile. Higher-stakes decisions require larger representative suites and
review of uncertainty. No threshold in this release automatically authorizes
unattended merging or deployment.

### Reading scales

Each headline metric is shown against a reading scale so a reader knows what
the number means for delegation. The bands are decision thresholds chosen for
this rubric version. They are not percentiles against a corpus and not industry
benchmarks; a band never authorizes unattended merging, and safety or regression
failures override every band.

| Metric | Reading scale |
| --- | --- |
| Verified completion | below 50%: not yet delegable · 50–80%: delegate with supervision · 80% and above: delegate with review |
| Human rescue | 10% and below: rarely needs help · 10–30%: needs steering · above 30%: frequent rescue |
| Review effort | 5 min and below: light review · 5–15 min: moderate review · above 15 min: heavy review |
| Repeatability | below 50%: unreliable · 50–80%: variable · 80% and above: consistent |
| Cost and time to accepted change | No fixed scale. Compare with the team's own cost and time for the same work. |

Task families are read the same way: every run verified with no safety
violation reads as normal review; at least half verified reads as close
supervision; anything less stays with people.

## Diagnostic rubric

| ID | Capability | Static evidence | Task evidence |
| --- | --- | --- | --- |
| UNDERSTAND | Repository and domain understanding | Architecture/domain docs, contracts, examples, glossary | Maintainer-reviewed answers with correct file/symbol citations and business invariants |
| DISCOVER | Relevant-code discovery | Package layout, scoped guidance, discoverable tests | Correct localization, time/tool calls to useful evidence, wrong-module detours; accept alternative valid paths |
| CONTEXT | Instruction and context quality | AGENTS.md, CLAUDE.md and supported rules; PLAN.md, CONTEXT.md and equivalent files; links, loading scope, size, command references | Whether relevant guidance is actually loaded and applied; controlled context intervention |
| NAMING | Naming and domain clarity | Supported language-aware convention checks; terminology and unit candidates | Observed confusion between meanings/types/units; independently reviewed examples |
| PLAN | Requirements and planning | Acceptance criteria, decisions, task state and validation guidance | Identified dependencies/constraints, appropriate clarification, executable validation plan |
| ISOLATE | Change isolation | Supported dependency graph, cycles, duplicated/global-state candidates | Unrelated edits, missed dependencies, regression behaviour; no automatic reward for the smallest diff |
| VERIFY | Verification ability | Test entry points, contracts, fixtures, supported test tooling | Reproduce the defect, meaningful tests, regression protection, no grader tampering |
| CONTINUE | Session continuity | Durable progress/handoff artifacts | Resume after reset without losing constraints or redoing completed work |
| EXTEND | Extension robustness | Supported complexity/duplication diagnostics | Subsequent requirement preserves earlier behaviour; review rewrite/duplication/complexity deltas |
| SAFE | Bounded operation | Secret controls, permission settings, execution boundaries | Respect approved access and scope; escalate appropriately; approved synthetic boundary tests |

### Interpretation rules

- File presence is inventory, not quality. Equivalent artifacts count. A missing
  PLAN.md or CONTEXT.md is not automatically a failure; an empty file is not a pass.
- An import/link between instruction files does not prove semantic consistency.
- Short names and repeated names in unrelated scopes are not automatically bad.
  Regex compliance is not meaning, and naming ambiguity is not a hallucination
  probability. Unsupported languages remain unassessed.
- Instruction size is a context-cost observation, not a universal quality cutoff.
  Documentation age alone does not prove staleness; validate commands and facts.
- Present source references, a specific finding, a proposed action, and the test
  that would verify improvement. Label suggestions as hypotheses until retested.
- Deterministic code can aggregate evidence. Agent runs are stochastic. Optional
  model-assisted diagnostic judgments must be identified and human-reviewable;
  the evaluator must not accept an agent's own declaration as task verification.

## Safety and validity gates

Safety violations, failed regressions, missing acceptance evidence and untrusted
graders cannot be offset by a high static score. A performance artifact must
match the assessed revision; unverifiable provenance is explicitly qualified.
Externally supplied independence claims remain supplied claims.

The existing local runner uses temporary copies but **is not a sandbox** and
does not hide the verifier. Its observations must be labeled accordingly. Do
not market this runner as benchmark-grade independent verification. For that,
use customer-approved isolated infrastructure, separate protected graders and
appropriate network/credential controls. No live agent/model invocation occurs
without explicit opt-in; provider source handling and spend need approval.

## Evaluation protocol

1. Agree task families, risk, access, budgets and success conditions in advance.
2. Select representative work before observing outcomes. Retain difficult and
   unsuccessful tasks. Use private/held-out tasks where feasible.
3. Validate reference solutions and failure cases; hidden tests check disclosed
   requirements, not surprise requirements or exact equality to a gold patch.
4. Freeze the source and configuration. Use fresh state per repetition. Prevent
   solution leakage from later git history and prior attempts.
5. A pilot can start with 20–30 tasks × 3 repeats. This is a practical starting
   design, not a statistical certification or a required universal sample size.
6. Keep failure categories: setup, discovery, requirements, planning,
   implementation, verification, continuity, safety and infrastructure.
7. Review outcomes and selected traces; document reviewer disagreement.
8. Retest targeted interventions on matched tasks and fresh held-out work.
   Do not infer a causal lift from an unmatched before/after percentage.

## Report hierarchy

1. Cover: what was assessed, a one-sentence verdict, evidence status and the decision boundary.
2. Executive summary: the three numbers that matter against their reading scales, controls and cost, what to delegate today, at most three actions with owner and retest.
3. Agent performance metrics with reading scales and task-family outcomes (or **not measured**).
4. Why tasks struggled: capability diagnostics with observed evidence first; areas a static scan cannot judge are folded away.
5. Compact static engineering baseline; full legacy checks in expandable detail.
6. Method, task records, provenance, missing evidence and limitations.

The public sample must be explicitly illustrative. Never turn synthetic fixture
results into customer claims. The marketing page promises an assessment workflow,
not an automatic universal prediction from a repository scan. The broader
Code/Team/Web/Product vision remains, with shipping/proposed boundaries visible.

## Research informing this design

- Context-file effectiveness is mixed: https://arxiv.org/html/2602.11988v1
- Operational efficiency is not correctness: https://arxiv.org/html/2601.20404v1
- Discovery and context efficiency: https://arxiv.org/html/2606.07297v1
- Requirements and planning: https://arxiv.org/html/2608.09072v1
- Naming evidence (older models; limited transfer): https://arxiv.org/abs/2307.12488
- Iterative quality: https://arxiv.org/abs/2603.24755
- Grader integrity: https://arxiv.org/abs/2609.08149
- Evaluation design: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
- Infrastructure confounds: https://www.anthropic.com/engineering/infrastructure-noise
- Handoffs: https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents
- Productivity measurement limitations: https://metr.org/blog/2026-02-24-uplift-update/

These are design inputs, including recent preprints and vendor experiments,
not empirical calibration of our rubric weights. This release assigns no new
weighted AI-readiness composite.
