vigiles API reference
    Preparing search index...

    Interface EvalSpec<M>

    interface EvalSpec<M extends Metrics> {
        allowedTools?: readonly string[];
        arms: Record<string, EvalArm>;
        cache?: CacheMode;
        cacheDir?: string;
        concurrency?: number;
        ephemeralEnv?: boolean;
        fixture?: Record<string, string>;
        maxCostUsd?: number;
        measure: (ctx: RunContext) => M;
        model?: string;
        name?: string;
        rateLimitRetries?: number;
        retryBackoffMs?: number;
        spacingSec?: number;
        stubs?: readonly ToolStub[];
        task: string;
        timeoutMs?: number;
        trials?: number;
    }

    Type Parameters

    Index

    Properties

    allowedTools? arms cache? cacheDir? concurrency? ephemeralEnv? fixture? maxCostUsd? measure model? name? rateLimitRetries? retryBackoffMs? spacingSec? stubs? task timeoutMs? trials?

    Properties

    allowedTools?: readonly string[]

    Tools the agent may use. Default: Read Edit Write Bash.

    arms: Record<string, EvalArm>

    The arms to compare, by name.

    cache?: CacheMode

    Record/replay cache mode. Default "off" (always re-sample). "readwrite" records each trial (output + post-run files) and replays it on a matching re-run — so editing measure re-scores for free; the model is re-called only when a model-affecting input changes. "read" replays but never records. The cache key excludes measure, so changing your metric still hits.

    cacheDir?: string

    Where cache records live. Default .vigiles/eval-cache under cwd.

    concurrency?: number

    How many trials to run at once (across all arms × trials). Default 1 (fully sequential — the safe, no-surprise default). Raise it to cut wall-clock time; rate-limit bursts are absorbed by the retry/backoff below.

    ephemeralEnv?: boolean

    Opt-in, default OFF. Run each trial in an ephemeral run environment — a throwaway $HOME + scrubbed env, re-injecting only the harness's own auth (see ephemeralRunEnv). Running a model-driven skill/agent is itself a side effect (the model, not the author, chose the actions), so a git push / write to ~ should land in a disposable HOME, not the real ~/.gitconfig / ~/.ssh / ~/.aws. This is the cross-platform STATE-protection floor (no kernel features), orthogonal to the bubblewrap host-confinement in src/sandbox.ts.

    Ships default-OFF because a too-narrow auth allowlist would silently break the real claude CLI's authentication; leaving it off keeps every existing eval (including one running right now) authenticating exactly as before. When absent / false, the per-trial env is byte-identical to today ({ ...process.env, ...arm.env }). See docs/safety.md (ephemerality) and research/cross-platform-sandboxing.md.

    fixture?: Record<string, string>

    Base fixture files (path → contents), written fresh for every run.

    maxCostUsd?: number

    Abort the run once measured cost reaches this many USD. In-flight trials finish; remaining ones are skipped and report.aborted is set. Needs the model to report total_cost_usd (the eval tier does).

    measure: (ctx: RunContext) => M

    Compute this run's metrics from its outcome.

    model?: string

    Model alias. Default "haiku".

    name?: string
    rateLimitRetries?: number

    Retries on a detected rate-limit/overload before giving up. Default 3.

    retryBackoffMs?: number

    Base backoff ms (doubled each retry). Default 1000.

    spacingSec?: number

    Seconds to wait between runs (avoid rate-limit bursts). Default 4.

    stubs?: readonly ToolStub[]

    Tool stubs on PATH (rung R2). A list of fake binaries to shadow on PATH for every trial, so a skill/hook/agent that calls a CLI tool (gh, psql, redis-cli, z3, …) and works with its RESULT can be tested against a recorded / author-provided canned output — no live service. vigiles writes one executable stub per ToolStub into a bin dir under the trial cwd and PREPENDS that dir to the run's PATH (both the default and the ephemeral env path), so the fake wins over the real binary.

    The stubs are author/recorded fixtures, never model-synthesized — a synthesized tool output looks plausible but diverges from the real tool/version (false confidence). Absent → no change (the PATH is byte-identical to today). See ToolStub and research/eval-coverage-and-isolation.md.

    task: string

    The task prompt given to the agent.

    timeoutMs?: number

    Per-run timeout ms. Default 240000.

    trials?: number

    Trials per arm. Default 5.

    Settings

    Member Visibility

    On This Page

    Properties