Grade the declared artifact kind

These examples and illustrative results are independently authored teaching materials, not measured model results.

Use case

One task promises a code fix with regression execution; another only interviews requirements. The first delivers files/results, the second conversational turns and clarified needs. Neither can be graded solely by a final process claim.

Mechanism

Declare artifact kind and expectations before running and check the actual promise rather than a kind label. Execution cases need usable fixtures/permissions and grading of edits, command results and artifacts. Dialogue cases assess questions, waiting and clarification without invented code requirements. Mixed tasks need both. Tool denials/missing files do not become execution success through narration.

Bad example

Grade fixes and interviews from final summaries. Give credit for claimed tests and fail interviews without file edits.

Good example

For a fix inspect actual failing-test, edit, rerun and final artifact evidence. For an interview inspect questions, awaited replies and unresolved needs. If it also promises a saved spec, verify that file separately. Match evidence to actual deliverables rather than using dialogue to exempt file requirements.

Why the change matters

Evidence must match promised artifacts. Traces separate performance from narration, while turns assess interviewing. Wrong labels create plausible grades of irrelevant behavior.

Observable expectation

Teaching claimed repair without edit/test evidence fails repair expectations; a sufficient interview can have no files. A promised unsaved spec still fails a mixed task. Record kind, canonical expectations and evidence, not dry-run plans as traces.

Limits

Traces can be incomplete; inspect raw materials and independently check artifacts. Dialogue and execution often mix, so avoid kind exemptions. Frozen tool/permission configuration describes an implementation, not authority to expand current permissions.