Why a skill fires sometimes and not others

One run is not a measurement

The first thing to settle is which problem you have, because a skill that fires four times in five and a skill that never fires look identical on the run you happened to watch. Pick the prompt that failed, open a fresh session, and send it five times. Count how many times the skill answered.

• •0 of 5 is not this page. Either the file never loaded, or the selector reads your description and prefers another one: why a skill never fires.
• •5 of 5 means the failure came from something in that older session rather than from the skill. Read the rest of the thread before you touch the file.
• •Anything in between is the case this page is about, and the number itself is the useful part. 4 of 5 and 1 of 5 are the same symptom and very different distances from working.

Fresh sessions matter more than the count. A second prompt in the same conversation is not an independent try, because the first answer is now part of what the host is reading. So is anything you said while debugging.

Do this before editing, not after. An edit followed by one good run is the most common way people convince themselves they fixed something: on a skill that already fired three times in five, one good run afterwards is what you would expect from changing nothing at all.

Four things that move a borderline case

1. 1

   The prompt can be answered without opening anything.

   This is the big one, and it is invisible from the file. When a request can be satisfied straight out of the conversation, the host often just satisfies it, and a skill that would have handled it never loads. Ask for a brief with the notes pasted in and the skill tends to answer. Ask for a brief with nothing pasted, where the only sensible reply is a question, and the assistant often asks that question as itself. Same file, same description, and the reader never sees the skill they installed.

2. 2

   Something else in the folder changed.

   Selection is a comparison, so it is never about your description alone. Adding a skill, renaming one, or widening a neighbour's trigger list changes the comparison for every skill installed, including the ones you did not touch. A skill measured on its own is not measured in the set you actually ship, and the direction is not always the one you expect. If the flakiness started the week you added three skills, start there.

3. 3

   The session has already committed to a path.

   Once a conversation has gone a particular way, later prompts get read against it. Ask for something adjacent to what you have been doing for twenty minutes and the reply continues the thread instead of starting the skill. This is why bugs reported as intermittent often reproduce perfectly in a fresh session and never in the one where they were found.

4. 4

   The trigger sits at the end of a long list.

   A description that opens with a general summary and finishes with six trailing cases puts the least weight on the last clause, and the last clause is usually the awkward case somebody added after it went wrong once. If the situation you keep losing is named in the tail, it is named in the weakest position in the file.

Fix the borderline case, not the sentence you like

Once you know which prompt is losing, write the description so that prompt is the obvious match rather than a footnote to it. Move the situation up, state it as an instruction, and say what to do when the material is missing instead of implying the skill only applies once it is there.

---
name: client-brief
description: Turns discovery notes into a structured client brief.
  Use for call summaries, account handovers and scope outlines.
  Also useful when notes have not been pasted yet.
---

---
name: client-brief
description: Turns discovery notes into a structured client brief.
  Use when the user asks for a brief, a call summary, an account
  handover or a scope outline. If they ask for one and have pasted
  no notes, load this skill anyway and ask for the material.
  Not for writing the proposal itself.
---

Two things are doing the work there. The situation moved out of the tail, and it became an instruction with a verb instead of a hint. What is written inside the body of the file cannot help here, because the body is only read once the skill has been chosen, and being chosen is the thing that failed. Boundaries belong in the description too: a line saying what the skill is not for is what keeps a neighbour from drifting onto the same ground. More on that in writing a description that gets picked.

If a second skill also answers these prompts, editing one description moves the boundary between them and can push the problem sideways. That case has its own guide: when the wrong skill answers.

Hold the fix with a case, not a memory

An intermittent bug that is fixed and an intermittent bug that is having a good week look the same for several days. The only difference you can act on is whether the losing prompt is written down somewhere that gets run again. Keep it as a case: the prompt, a fresh session, and the number of tries out of the number you ran.

Two habits are worth more than the rest here. Repeat the same prompt several times and treat the case as failed unless every try passed, because a check that passes four times in five will tell you the day is fine when it is not. And run against the full set of skills you ship, not the one you are editing, since point two above is exactly the failure a single-skill check cannot see. How to test whether a skill actually works covers the mechanics.

We publish the same measurements for our own packs, misses included, on the results page. Cases that pass some of the time and not others are the ones worth reading there, because they are the ones that would otherwise be reported as fixed.