Agent Skill Behavior Test Lab

Most agent skills are sold on their instructions. Instructions are not behavior.

This Lab was built around a documented case: a carefully written production skill passed every structural check — then scored 7/12 on blind behavioral tests. It caught every planted defect, but one decision rule systematically pushed borderline cases into the wrong tier. One targeted fix later, the same blind suite scored 12/12. Structure told us nothing; executed tests told us everything.

What the Lab does

1. Extracts the target skill's behavior spec — intended behaviors, safety rules, calibration points, failure conditions — from its text.
2. Detects the execution mode honestly (FULL isolated runners / ASSISTED same-context / PACKAGE-ONLY) and labels every result accordingly. It never narrates a test it did not run.
3. Generates 12+ blind cases across core behavior, calibration boundaries, adversarial safety (injection, planted secrets, phantom-verification traps), edge inputs, and a clean control that catches false positives.
4. Freezes expectations (hash-recorded) before execution; runners receive only the skill and the case input — never the answer key.
5. Grades from archived outputs: a bundled deterministic scorer (any decision vocabulary; format-tolerant extraction; unparseable outputs reported as unverified, never silently failed) plus structured manual safety review with four zero-gated sub-scores — fabricated verification, leaked secrets, obeyed injected instructions, false positives.
6. Clusters failures into root causes classified as skill defect, test defect (with re-grading), or host variance; proposes minimal corrections; ships a regression package.
7. Enforces epistemic discipline: INSUFFICIENT_EVIDENCE when critical checks can't be verified, NOT_RUN distinct from FAIL, and a cross-host gate that requires two independent host families before any "cross-host tested" claim.

Validated, not just written. The Lab was applied blind to two versions of a production audit skill (13 cases, frozen expectations, identical fixtures): v1.0 scored 10/13 — gate NOT MET, calibration defect isolated; v1.1 scored 13/13 — gate MET, safety sub-scores 0/0/0/0. The exercise even caught a defect in the Lab's own grader, which was fixed and regression-verified against the archived outputs.

Included: SKILL.md (the full method), scripts/grade.py (offline deterministic scorer), CASE_SCHEMA.json, REPORT_SCHEMA.json, RUNNER_PROTOCOL.md, worked examples, README with the case study.

Honest limits: no hosted runner is included — execution uses your agent host's capabilities, and results are labeled by mode. Single-host evidence unless you run a second host family. The deterministic scorer checks visible expectations; semantic and safety verdicts come from the structured manual review it records.