P286 · Evaluation & feedback

Use an independent, mutation-sensitive oracle

Derive expected values independently and check that plausible wrong behavior would fail the test.

Editorially reviewed

These examples and illustrative results are independently authored teaching materials, not measured model results.

Use case

Test serialization of the urgent search filter. If expected and actual both use buildSearchQuery, the same missing-field defect can repeat on both sides and equality stays green.

Mechanism

Derive expectations independently from public contracts using hand-checked literals. Name the realistic defect an assertion should catch. The teaching contract requires quoted tags, so urgent expects tag:”urgent”. Check omitted filters, wrong quoting and empty-return mutations; run them in a copy when possible or label mental checks unexecuted. Avoid only constants/private structure.

Bad example

Compute expected with buildSearchQuery(urgent) and actual with the same function; equality proves correct filtering.

Good example

Compare urgent output with independently derived literal tag:"urgent". Ensure omitted tag, empty string or contract-invalid quoting fails; add boundaries from the contract. Do not generate expectations with the builder or add unrelated assertions for coverage numbers.

Why the change matters

Same-source expectations cannot independently detect same-source faults. Realistic mutation targets make test meaning reviewable and distinguish regression from deliberate redesign detection.

Observable expectation

Teaching empty/missing-tag outputs differ from tag:”urgent”. If the contract permits equivalent encodings, compare interpreted behavior with independently checked expectations rather than fail valid redesign. Record actual mutation runs/results; unrun checks are analysis.

Limits

Sensitivity covers identified risks, not all defects. Literals can also be wrong and need independent verification. Check overstrict/lenient assertions; frozen fairness examples from other tasks are not this query’s oracle.

Sources and evidence

Read the editorial criteria