Optional ReadonlyallowedTools the agent may use. Default: Read Edit Write Bash Skill.
Optional ReadonlyconcurrencyHow many runs to execute in parallel across the whole prompts × trials grid.
Default 1 (serial, the politest to rate limits). Raise it to cut wall-clock on
a large prompt set or roster sweep — the spacingSec pause still applies per
run, so it stays best-effort polite. Mirrors EvalSpec.concurrency.
ReadonlyfiredDid the behaviour fire on this run? e.g. (t) => skillResolved(t, "x:y").
Optional ReadonlyfixtureFiles (path → contents) seeded into every run's cwd before the prompt — the
filesystem CONTEXT the skill is measured in. The default empty cwd is faithful
for opening-move skills ("describe a feature", "debug this") but biased-low for
skills whose trigger is a repo STATE ("in a git repo", "dirty tree"); seed that
state here so recall is honest instead of an artifact of the cold start. Mirrors
MeasureSpec.fixture. See research/plugin-behavioral-findings.md.
Optional ReadonlyinstallCo-install these skill sources ALONGSIDE the skill-under-test so it competes
for selection as it does in the real harness — the whole-harness tier.
Each entry is a plugin dir (skills/ or .claude/skills/) or a loose skills
dir (<name>/SKILL.md); their skills are merged into one install under the
under-test plugin's name (the under-test skill wins a name collision, so its
<name>:<skill> id still matches fired).
WHY: skill selection is competitive and Claude Code evicts the least-used
skill descriptions under a context budget, so an ISOLATED trigger-rate (the
default, installSet absent/empty) overstates recall and understates
false-positives. Isolated is the cheap authoring loop; populate the set for
a release gate. See research/isolated-vs-whole-harness-eval.md.
Optional ReadonlyirrelevantOptional irrelevant prompts the skill should not fire on — the
precision side of triggering. Firing on these is a false positive (a skill
whose description is too broad and hijacks unrelated work). When given, the
report adds TriggerRateReport.falsePositiveRate and
TriggerRateReport.precision; prompts alone measures recall only.
Optional ReadonlyminReject the run when two prompts in a set are closer than this in NCD (gzip-based distance, 0..1; 0 = identical) — near-duplicate prompts inflate a rate without testing varied phrasings. Default 0.3 (the rule-dup threshold).
Optional ReadonlyminMinimum model tier this eval may run on (haiku<sonnet<opus by family). The
run fails if the resolved model is weaker — trigger-rate under-measures
selection on a too-weak model, so this stops a cheap model from producing
false-negative recall. Default "sonnet". Lower it deliberately for a cheap run.
Optional ReadonlyminMinimum number of prompts each set (relevant + irrelevant) must have. A handful of prompts can't tell a real recall/precision rate from noise, so the run is rejected before it spends a token. Default 10; lower it deliberately for a genuinely narrow skill.
Optional ReadonlymodelModel alias/id. Default "sonnet" — the realistic selector most Claude Code
users run. NOT haiku: trigger-rate is a selection measurement and haiku is a
much weaker selector, so it under-reports recall (dogfooded: a skill scored
0.50 on haiku vs 0.90 on Sonnet). Override for a cheaper-but-pessimistic run.
Optional ReadonlypluginPlugin dir installed natively (--plugin-dir) so its skills/commands
activate. Provide this OR skillsDir, not both.
ReadonlypromptsThe varied prompts to test the trigger against.
Optional ReadonlyskillsA directory of LOOSE skills (<skillsDir>/<name>/SKILL.md, e.g. a repo's
.claude/skills) to trigger-test directly. vigiles packages them into a
throwaway --plugin-dir for you and removes it afterward — the one-liner
for repo-local skills that aren't a published plugin. Provide this OR
pluginDir, not both.
Optional ReadonlyspacingSeconds to wait between runs (avoid rate-limit bursts). Default 4.
Optional ReadonlystubReplace each skill's BODY with a no-op stub (keeping its frontmatter — name +
description). Trigger-rate is decided by the frontmatter ALONE (the model
selects a skill before its body loads), so stubbing can't change what's
measured — it just stops the run executing an expensive procedure once the
skill fires. All descriptions stay present, so selection competition is
faithful. Defaults to true here: testing a description never needs the
body, so it's automated, not a knob you must remember. Set false only in the
rare case you want the real body to run. See stubSkillBody.
Optional ReadonlytimeoutPer-run timeout ms. Default 240000.
Optional ReadonlytrialsTrials per prompt. Default 1.
Measure how reliably a skill/behaviour triggers. A skill's value is its description firing on the right task — the #1 documented skill-authoring pain — and that's a property of the real model, not the wiring (which the deterministic tier already proves). Install the plugin natively (
pluginDir), give a set of variedprompts, and afiredpredicate over the run'sTrace(reuse the bare predicates, e.g.(t) => skillResolved(t, "x:y")).