Use an independent, mutation-sensitive oracle
Derive expected values independently and check that plausible wrong behavior would fail the test.
These examples and illustrative results are independently authored teaching materials, not measured model results.
Use case
Test serialization of the urgent search filter. If expected and actual both use buildSearchQuery, the same missing-field defect can repeat on both sides and equality stays green.
Mechanism
Derive expectations independently from public contracts using hand-checked literals. Name the realistic defect an assertion should catch. The teaching contract requires quoted tags, so urgent expects tag:”urgent”. Check omitted filters, wrong quoting and empty-return mutations; run them in a copy when possible or label mental checks unexecuted. Avoid only constants/private structure.
Bad example
Compute expected with buildSearchQuery(urgent) and actual with the same function; equality proves correct filtering.
Good example
Compare urgent output with independently derived literal tag:"urgent". Ensure omitted tag, empty string or contract-invalid quoting fails; add boundaries from the contract. Do not generate expectations with the builder or add unrelated assertions for coverage numbers.
Why the change matters
Same-source expectations cannot independently detect same-source faults. Realistic mutation targets make test meaning reviewable and distinguish regression from deliberate redesign detection.
Observable expectation
Teaching empty/missing-tag outputs differ from tag:”urgent”. If the contract permits equivalent encodings, compare interpreted behavior with independently checked expectations rather than fail valid redesign. Record actual mutation runs/results; unrun checks are analysis.
Limits
Sensitivity covers identified risks, not all defects. Literals can also be wrong and need independent verification. Check overstrict/lenient assertions; frozen fairness examples from other tasks are not this query’s oracle.
Sources and evidence
- obra/superpowers · Independent expectations
File at this version8ca22dba9a94 - obra/superpowers · Mutation Check
File at this version8ca22dba9a94 - addyosmani/agent-skills · deliberate temporary mutation to test the oracle
File at this version9d0c60d406b4 - addyosmani/agent-skills · independent earliest-share fairness oracle
File at this version9d0c60d406b4 - ComposioHQ/awesome-claude-skills · Core Requirements / Complexity
File at this versionbe2a406907db - ComposioHQ/awesome-claude-skills · Stability / Answer Guidelines
File at this versionbe2a406907db - affaan-m/ECC · Grade the reference, shipped fixture and optional naive control through the same path
File at this versionef648e01899b - affaan-m/ECC · Preserve saturated prior corpus while revising discriminative tasks
File at this versionef648e01899b - affaan-m/ECC · Require a rejected control candidate but preserve actual-verification limits
File at this versionef648e01899b - anthropics/skills · Discriminating assertions
File at this version8a1541c4a3ff - anthropics/skills · Per-assertion and per-case analysis
File at this version8a1541c4a3ff - anthropics/skills · Whole-pipeline oracle and null
File at this version8a1541c4a3ff - anthropics/skills · Wrong-but-plausible grader controls
File at this version8a1541c4a3ff