How to test whether a skill actually works
There are two questions here and they fail independently. Does the skill get selected when it should, and is what it produces correct. Selection is tested with a list of prompts you would really type. Output is tested by fixing the inputs and comparing the result against an expected shape. A skill can pass one and fail the other, so score them separately.
Build the prompt set first
Write ten to twenty prompts in your own words, copied from real requests where you can find them. Cover three groups, because a set made only of the first group will pass and tell you nothing.
Should fire.
The obvious phrasing, plus the casual one you actually type on a Tuesday afternoon.
Should not fire.
Requests that live next door and belong to a different skill. If your SOW skill answers a question about invoicing, that is a defect, and you will only see it if you asked.
Ambiguous.
Requests that could plausibly go either way. Decide the right answer yourself before you run them, and write it down. Deciding after you see the result is how a test set quietly becomes a record of current behaviour rather than a standard.
Score selection and output separately
For selection, run each prompt and record which skill loaded. The number you want is the share of should-fire prompts that fired, and the share of should-not-fire prompts that stayed quiet. Both matter. A skill that fires on everything scores perfectly on the first and is worse than useless.
For output, pin the inputs so the run is repeatable, then check the result against a few concrete assertions rather than a general impression: the sections that must be present, the fields that must be filled, the claim that must not appear. Assertions survive a rewrite of the file. Impressions do not.
A small eval file
Keep the cases in one file next to the skill, so a change to the file and a change to its tests arrive in the same commit.
skill: sow-draft
cases:
- prompt: "turn these discovery notes into a SOW"
expect_skill: sow-draft
- prompt: "write up the scope for the Henderson job"
expect_skill: sow-draft
- prompt: "what is our hourly rate for design work"
expect_skill: none
- prompt: "draft the scope and put pricing in it"
expect_skill: sow-draft
expect_contains:
- "Out of scope"
- "Deliverables"Run it, record the pass rate with the date and the model you ran it on, and keep the record. One score is a reading. A column of scores is the only thing that will tell you whether last month's edit helped.
Re-run when the model changes
A skill is a prompt, and prompts move when the model underneath them moves. A file that scored well in the spring can quietly lose several cases after a model update, with no error and no change on your side. Re-running the same set on the new model is the whole of the maintenance work, and it takes minutes when the set already exists.
If a case fails on selection, the cause is almost always in the description rather than the body: see how to write a description that gets picked. If it fails without ever loading, the setup itself may be at fault, and the usual causes are listed in why a skill never fires.
Works with Claude Code.
Test a skill in your browser
Paste a skill file into the free checker. It runs your file against a set of realistic prompts, reports whether it would be picked and where it loses, and shows the prompts that fired the wrong skill. Takes under a minute, no account needed.
Check a skill file