P354 · Evaluation & feedback

Related-Input Invariant Oracle

Check defined relationships between transformed inputs and outputs when exact fixtures alone leave broad behavior unchecked.

Editorially reviewed

These examples and illustrative results are independently authored teaching materials, not measured model results.

Use case

A normalizer sorts without dropping records; teaching IDs are b,a,c. A single two-record example misses order dependence/loss, so add contract-supported relational predicates.

Mechanism

Hand-check a,b,c and declare idempotence, permutation invariance and multiset preservation. Test originals, permutations and larger boundaries. Treat duplicate IDs under contract rather than a set hiding lost duplicates. Save minimized failure/transformation while retaining exact anchors. Do not invent unsupported properties.

Bad example

Check only[a,b], or idempotence alone, calling constant-empty output correct.

Good example

Keep b,a,c→a,b,c as an independent fixture. Check permutation equality, second normalization stability and multiset preservation. Include duplicate counts under contract and ensure constant-empty cannot pass combined assertions.

Why the change matters

Relations cover input transformations, while correct anchors/preservation prevent degenerate passes. They test behavioral connections rather than one output label.

Observable expectation

Teaching b,a,c normalizes to a,b,c and remains unchanged afterward. Dropping c violates preservation; constant-empty is idempotent but fails the anchor/multiset. Order-dependent behavior violates permutation invariance. These are expectations, not measured runs.

Limits

Use only contractual relations; deduplication or input-order preservation changes them. Properties do not prove all-input correctness and generators can miss boundaries. Minimization must preserve failure meaning.

Sources and evidence

Read the editorial criteria