Method / apr/0.1.0
What the evidence
can actually say.
One agent configuration, one repository revision, a defined set of tasks and a recorded budget. Not a universal certificate.
Read the complete versioned rubric → · Inspect the illustrative report →
Three distinct layers.
- Static preflight. File and configuration observations, context/planning inventory and supported structure checks. No agent-success prediction.
- Task performance. Attempt-level outcomes, independent checks, review and resource use where recorded. The grader and provenance remain visible.
- Improvement comparison. Matched work under the same configuration and grader before and after a change. Observed differences are not causal proof.
What each headline measures.
| Metric | Definition and limits |
|---|---|
| Verified completion | All acceptance, regression and constraint checks pass within budget, with clear safety. Divide by all attempted runs, retaining infrastructure failures. Requires independently graded evidence. |
| Human rescue | Runs with corrective assistance divided by runs where assistance was recorded. Normal clarification and mandatory approvals are separate. |
| Review effort | Median recorded review minutes, plus accepted, rework and rejected counts. Unrecorded effort is unknown. |
| Repeatability | Tasks with every run verified, among tasks with at least three repeats. Not best-of-k success. |
| Cost per accepted change | All run costs, including failures, divided by accepted verified runs. Withheld if any cost is missing or none were accepted. Human cost excluded. |
| Time to acceptance | Median recorded end-to-end time for accepted verified runs, including review. Not the time when the agent stopped. |
Context quality is not a file checklist.
We inventory agent guidance, context, plans, architecture, contracts and handoffs. Equivalent artifacts are valid. File presence does not prove that guidance is loaded, correct or useful. Naming conventions do not measure meaning or hallucination probability. Current language-aware static structure and naming diagnostics cover Go.
The ten diagnostic areas connect each observation to an action hypothesis and a task that could validate it. Understanding, discovery, semantic naming, context effectiveness, planning, isolation, verification, continuity, extension and runtime safety need task evidence or scoped review.
How a credible trial works.
- Select representative tasks and acceptance criteria before seeing results.
- Agree agent/model versions, harness, permissions, time budget, data handling and spend.
- Validate the grader against correct and incorrect solutions. Protect checks independently from the agent.
- Start each attempt from fresh state. Preserve unsuccessful runs and infrastructure errors.
- Record human review, interventions and costs rather than guessing them.
- Retest a targeted improvement on matched tasks and fresh held-out work.
What the product does today.
The CLI supports static scans, repeated local multi-task command suites, validated evidence imports, performance-first reports and matched comparisons. The assessment workflow supplies representative tasks, appropriate independent infrastructure, reviewer judgments and cost records. These do not happen automatically from a repository URL.
The built-in local runner is a temporary-copy process runner, not a sandbox or protected grader. Agent and verifier share workspace and privileges. Its successful commands remain local observations. External commands may access the network and provider services; execution and spend require approval.
What we will not imply.
- No blended score that hides safety failures or missing evidence.
- No customer claims from synthetic sample data.
- No current-revision claim from stale or dirty-checkout evidence.
- No claimed attestation of imported grader independence.
- No universal success probability, calibrated percentile or guaranteed ROI.
- No statistical confidence interval treating repeated attempts as independent tasks.
Small, selected task sets describe sampled work. The static baseline stays arl/0.2.0. Performance aggregation is separate under apr/0.1.0. The research sources and full interpretation rules are in the rubric.