packages/std/evals/regression-tracker.prose.md

name: regression-tracker kind: responsibility version: 0.15.0

Regression Tracker

Maintain a registry of "known good" baseline runs for each system. When a new run completes, compare it against the baseline and flag regressions. This is the continuous quality gate — it answers "did this change make things worse?" without requiring a human to review every run.

The registry is persistent at the project level, so baselines survive across runs and can be updated when a new run is confirmed as the new standard.

Goal

Maintain a standing registry of baseline runs per system and, on each new run, judge whether quality regressed, held, or improved against that baseline.

Continuity

Input-driven: woken by a new run to check, or by a request to set a baseline or list baselines.

Requires

subject: run — the new run to compare against baseline
system-name: the system name to look up the baseline for (e.g., "deep-research", "anomaly-detective")
action: "check" (compare against baseline), "set-baseline" (register this run as the new baseline), or "list" (show all registered baselines)

Maintains

The standing baseline registry — a persistent, project-level map of system name → baseline run, surviving across runs and updatable when a new run is confirmed as the new standard. Each system's entry carries its baseline_run_id, baseline_scores (contract_satisfaction, runtime_fidelity, task_effectiveness), and the timestamp it was registered; the last 5 previous baselines per system are kept for trend analysis. The registry is material truth; volatile request ids and read-time access timestamps are immaterial.

Alongside the registry, each handled request yields a report:

system: the system name
action: which action was performed
(for check) status: "pass" (no regressions), "regressed" (worse than baseline), or "improved" (better than baseline)
(for check) baseline_run_id: which baseline was compared against
(for check) new_run_id: the new run's ID
(for check) dimensions: per-dimension comparison (contract_satisfaction, output_quality, timing, error_rate) each with baseline value, new value, delta, and verdict
(for check) evidence: specific differences that support the overall status
(for check) recommendation: "promote to baseline" (if improved), "investigate regression" (if regressed), or "no action needed" (if pass)
(for set-baseline) confirmation: which run was registered as baseline for which system
(for list) baselines: list of all registered baselines with system name, run_id, timestamp, and scores

Postconditions:

if no baseline exists for this system: the report notes "no baseline — registering this run as initial baseline" and the registry is updated to register this run as the initial baseline (a "check" with no baseline auto-sets it).
a "check" status of "regressed" must cite the specific dimension that regressed with supporting evidence — e.g. "task_effectiveness dropped from 85 to 62 because the synthesizer's output is missing the sources section".
the report always includes the recommendation field — this is the actionable output that CI pipelines will read.
regression thresholds applied consistently: contract_satisfaction drop > 10 points, output quality materially worse (judgment call), timing > 50% slower, any new errors.

Errors

run-not-found: the subject run does not exist or is incomplete
baseline-corrupted: the registered baseline run no longer exists on disk

Invariants

when the baseline run no longer exists on disk: flag as corrupted and recommend setting a new baseline.
when the registry grows beyond 100 systems: warn but do not evict — regression tracking should not lose data silently.
comparison strategy: use contract-grader scores if available, otherwise use inspector scores, otherwise fall back to structural comparison (output existence and size).
when an improvement is detected: still compare carefully — an improvement in one dimension might mask a regression in another.

Execution

The render maintains the registry and produces the report by composing three internal steps. Order matters, so they run as ProseScript calls, not as separate mounted nodes.

const registryState = call registry(subject, system-name, action)

let comparison = null
if action == "check" and registryState.has_baseline:
  comparison = call comparator(subject, registryState)

const report = call reporter(registryState, comparison, action, subject, system-name)
return report

registry

Maintains the baseline registry. Maps system names to baseline run IDs and their scores. This is the persistent, standing truth that survives across runs.

Runtime

persist: project

Parameters

subject: the run binding
system-name: the system name
action: "check", "set-baseline", or "list"

Returns

registry-state: current state of the registry for this system, containing:
- has_baseline: boolean
- baseline_run_id: string or null
- baseline_scores: object with contract_satisfaction, runtime_fidelity, task_effectiveness scores (or null if no baseline)
- baseline_timestamp: when the baseline was registered
- all_baselines: (for "list" action) complete registry contents
The returned registry-state reflects the registry after this action: when action is "set-baseline" the registry entry for this system is updated and the returned state reflects the new baseline; when action is "check" the current baseline is returned without modifying the registry; when action is "list" all entries are returned. The registry stays compact (system_name, baseline_run_id, baseline_scores, timestamp) and keeps a history of previous baselines (last 5 per system) for trend analysis.

comparator

Compare the new run against the baseline across all quality dimensions. This reads artifacts from both runs.

Parameters

subject: the new run binding
registry-state: from registry (contains baseline run ID and scores)

Returns

comparison: structured comparison containing:
- baseline_run_id: string
- new_run_id: string
- dimensions: list of dimension comparisons, each with:
  - name: "contract_satisfaction", "output_quality", "timing", "error_rate"
  - baseline_value: number or descriptor
  - new_value: number or descriptor
  - delta: numeric difference (positive = improvement, negative = regression)
  - verdict: "improved", "stable", "regressed"
  - evidence: specific observations supporting the verdict
- overall_status: "pass", "regressed", or "improved" (regressed if any dimension regressed and delta exceeds threshold)
If no baseline: comparison is null (the registry handles auto-registration).

Errors

baseline-corrupted: baseline run directory does not exist or is missing critical files

Invariants

for contract_satisfaction: compare ensures clause satisfaction rates. Run contract-grader logic on both if needed, or use cached scores from registry.
for output_quality: compare final output content — is the new run's output as comprehensive, accurate, and well-structured as the baseline's?
for timing: compare durations from vm.log.md. A slowdown of more than 50% is a regression. A speedup is an improvement.
for error_rate: compare error markers in vm.log.md. Any new errors that the baseline did not have are regressions.
regression thresholds: contract_satisfaction drop > 10 points, output quality materially worse (judgment call), timing > 50% slower, any new errors.
when baseline has cached scores: use them rather than re-reading all baseline artifacts (the registry stores scores for this reason).

reporter

Synthesize the comparison into the final regression report. Handle all three actions (check, set-baseline, list).

Parameters

registry-state: from registry
comparison: from comparator (may be null for set-baseline and list actions)
action: the requested action
subject: the run binding
system-name: the system name

Returns

report: the final output matching the top-level Maintains report schema.

Invariants

for "check" with regression: lead with the most severe regression, include evidence, recommend investigation before promoting.
for "check" with improvement: lead with the improvement, recommend promoting to baseline, but note any dimensions that stayed the same or slightly degraded.
for "check" with pass: brief confirmation that the new run matches baseline quality.
for "set-baseline": confirm registration with the run ID and scores that were recorded.
for "list": format as a table with system name, baseline run ID, timestamp, and key scores.
always include the recommendation field — this is the actionable output that CI pipelines will read.