Use fresh no-change controls after drift

These examples and illustrative results are independently authored teaching materials, not measured model results.

Use case

A candidate scores 0.8 this week against last week’s0.7 baseline after serving conditions changed. Rerun the unchanged prompt now to distinguish drift from intervention.

Mechanism

Record code, lockfile, patches, tools, model/configuration fingerprints and historical differences. Repeat control and candidate under current matched cases/cache/state; rebuild stochastic intermediates rather than only rescore old artifacts. Retain historical records and gate on contemporary control with errors/variation. Control runs follow approved budgets.

Bad example

Attribute this week’s C0.8 minus last week’s B0.7 entirely to the prompt edit without environment checks.

Good example

Run unchanged B and C in the current environment with matched cases/state. Record fingerprints, identities, repetitions and failures. If B also becomes 0.8, C cannot claim a0.1 gain. Preserve historical 0.7 with its old conditions.

Why the change matters

No-change controls measure environmental movement for the same system so candidates do not inherit that credit. Identity and matched conditions prevent historical numbers becoming permanent standards.

Observable expectation

Teaching historical B0.7 and current B0.8/C0.8 show no distinguishable contemporary difference. Current B0.7/C0.8 still needs replication/noise checks. Confirm unchanged prompts and newly built stochastic controls. Values are illustrative.

Limits

Fingerprints miss hidden provider changes, and cache/state can confound. Frozen-output blind references differ from contemporary system controls and need separate metrics. Controls alone do not establish causation/general benefit; large temporal gaps warrant design reassessment.