Separate behavior success from intervention benefit

These examples and illustrative results are independently authored teaching materials, not measured model results.

Use case

Baseline and overlay both complete three teaching behaviors. ON success shows task capability, not added benefit; tools/cost can still regress.

Mechanism

Separate behavioral acceptance from matched ON/OFF evaluation with fixed cases/runtime/budget/definitions and raw rows. Verify intervention engagement and compute predefined quality/resource effects, labeling saturation, no detected gain, regression or incompleteness. Detector validity is also distinct from benefit of acting on signals, which requires outcome controls.

Bad example

ON solved all, so claim 20% improvement without OFF and hide higher resource use inside pass.

Good example

Test OFF/ON on the same fixture, accept behaviors separately and compare complete rows. Both 3/3 means baseline_saturated and no demonstrated marginal correctness benefit; retain resource changes. Missing comparisons are incomplete, not zeros.

Why the change matters

Task success and intervention benefit differ. Controls and discriminating metrics measure marginal effect; single-arm success credits preexisting capability to the change.

Observable expectation

Teaching OFF 3/3 and ON 3/3 yield delta 0; higher ON calls are a separate regression. Accurate detection without action-outcome comparison cannot establish useful intervention. These are virtual checks; free harness tests are not model efficacy.

Limits

Saturated fixtures lack discrimination, not proof of universal uselessness. Control state/sampling and define percentages, zero baselines and cost basis. Frozen 20% examples are not universal promises.