Keep answer keys outside agent reach

These examples and illustrative results are independently authored teaching materials, not measured model results.

Use case

Evaluate diagnosis of a fixture whose symptom is “the list stays stale after saving.” The grader knows the cause, but a reachable gold.json or history lets the system obtain it directly.

Mechanism

Expose realistic symptoms and allowed repository data while keeping answers in a separate grader environment with actual access restrictions, not merely outside cwd. Start clean trials and inspect files, Git history, tools, environment and leftovers. Avoid leaks in case names/examples/prompts. Retrieval tests need sufficiently unfamiliar or synthetic material; record isolation scope.

Bad example

Put gold.json in the checkout with only a do-not-read instruction and leave the previous repair log for the next trial.

Good example

Provide only the stale-list symptom and permitted repository. Keep gold and grader prompts genuinely inaccessible to agent tools, checking history/leftovers for hints. Recreate clean trials and record boundaries; outside the workspace is not automatically unreadable.

Why the change matters

Verbal prohibition does not hide information. Structural isolation and cleanup make scores reflect investigation rather than answers, previous logs or naming clues.

Observable expectation

A teaching access audit shows tools cannot read gold or old repair logs but can read permitted fixtures. Answers remain grader-only. Moving gold outside cwd while absolute-path access works fails isolation. Avoid private real data in fixtures.

Limits

Sandbox/tools/credentials define access, and public copies or model memory can leak answers. Isolation cannot prove no training contamination; realistic symptoms and difficulty still matter. No paid model or answer store is accessed by this article.