Testing

What it is. A playbook is a test suite you declare beside your workflows: fixture inputs, assertions on outputs and timing, mocks for external side effects, and optional evals with LLM judges for stochastic checks.

Use it when you want confidence that a workflow still works after a change — without manually clicking through forms or firing live integrations on every PR.

How it runs. swirls test is hosted-only. There is no in-process runner. The CLI compiles your local workspace, creates a non-activating environment: test deployment, runs each case on the real platform executor (Temporal, sandboxes, node types), applies your mock policy, and reports results. Duration and path assertions reflect production infrastructure, not a local stub.

swirls doctor          validate mocks, playbooks, and cross-refs locally
       │
       ▼
swirls test            compile → test deployment → PlaybookRun → cases/evals
       │
       ▼
Cloud dashboard        PlaybookRun history on the test deployment detail page

Primitives

Playbooks live in the same multi-file .swirls workspace as everything else. Three top-level blocks work together:

Inside a playbook, case runs deterministic checks. eval adds stochastic scoring (builtin rubrics or free-form judges). Both require a workflow: target.

Example: smoke playbook

mock defaults {
  mode: all
  except: []
}

mock review_gate {
  node: approve_step
  review: {
    auto: approve
    formData: @json {
      { "approved": true }
    }
  }
}

playbook smoke {
  label: "Smoke tests"
  mocks: defaults
  tags: [ci]

  case happy_path {
    workflow: process_contact
    input: @json {
      { "email": "[email protected]", "name": "Test" }
    }
    timeout: "2m"
    tags: [smoke]

    expect {
      status: completed
      duration: { max: "2m" }
      output: @json {
        { "status": "processed" }
      }
    }
  }
}

Run it:

swirls auth login
swirls configure          # if you have not already
swirls doctor             # catch mock/playbook issues before hitting the API
swirls test               # all playbooks
swirls test --playbook smoke --tag ci

Test deployments

A test deployment is a first-class deployment environment alongside production, preview, and staging. It is created by swirls test (or deployments.deploy with environment: test).

That isolation means playbook runs never steal production traffic. Your live triggers keep firing against the active deployment; tests run against a snapshot of the workspace you just compiled.

Mocks

Mocks declare how external calls are intercepted for a PlaybookRun. They do not change production behavior.

Default policy

mock defaults {
  mode: all              // all | off | selective
  except: [safe_action] // when mode: all — pass these through (live creds)
}

A playbook selects its policy with mocks: defaults (or another named mock block).

Per-action and per-node stubs

Pin stubs to a specific integration action or workflow node. Resolution order: workflow+node-specific → action-specific → playbook mocks: → mock defaults → implicit off.

mock linkedin_create_post {
  action: linkedin_create_post
  output: @json {
    {
      "id": "urn:li:share:mock-001",
      "activity": "urn:li:activity:mock-001"
    }
  }
}

mock outbound_review {
  workflow: gtm_outbound_draft_lead
  node: review_draft
  review: {
    auto: approve
    formData: @json {
      { "approved": true, "editedSubject": "Test subject" }
    }
  }
}

mock social_review {
  workflow: gtm_social_draft_batch
  node: review_draft
  review: {
    auto: approve
    formData: @json {
      { "approved": true }
    }
  }
}

Node mocks require workflow: and node:. Node names are only unique within a workflow — review_draft in outbound and social are different gates. Pin both fields so the validator checks formData against the correct review schema and swirls test auto-approves with the right payload.

Action output: and review formData: are validated locally by swirls doctor and the LSP when schemas are available.

Cases and expectations

A case starts one workflow execution with a fixture input and asserts on the result inside expect { }.

Common assertions inside expect:

Each assertion may take severity: soft | gate (default gate). Gate failures fail the case; soft failures are recorded and only fail under --strict.

Evals and judges

An eval is for workflows where deterministic matchers are not enough — draft quality, summarization fidelity, and similar checks. Add a judge { } block with a rubric and optional minScore.

eval draft_quality {
  workflow: generate_draft
  input: @json { { "topic": "Testing harness" } }
  timeout: "5m"
  retries: 2

  judge: {
    rubric: criteria
    criteria: "Draft is coherent, on-topic, and suitable for LinkedIn. Score 0-1."
    minScore: 0.75
    severity: soft
  }

  expect {
    status: completed
    notCalled: [linkedin_create_post]
  }
}

Judge scoring uses a separate model from the workflow under test. Missing judge credentials skip judge-backed evals as skipped; deterministic cases still run.

Running tests

swirls test

Requires swirls auth login and a configured project (swirls.config.ts or --project).

swirls test                              # all playbooks, all cases/evals
swirls test --playbook smoke             # one suite
swirls test --case happy_path            # one case
swirls test --tag ci --exclude-tag slow  # tag filters
swirls test --list                       # discover without running
swirls test --reporter junit --strict    # CI reporters

Flags:

Exit code is non-zero when any gate assertion fails (or any soft assertion fails under --strict).

CI

Store Swirls credentials in your CI secret store, then:

{
  "scripts": {
    "test:swirls": "swirls test --reporter junit --strict --tag ci"
  }
}

Prefer mode: all on mocks in CI so integrations never fire with live credentials.

Validate before you run

swirls doctor is the first line of defense. It scans every .swirls file in the working directory and reports:

• Parse and validation errors (including playbook and mock cross-refs)
• Duplicate resource names across the workspace
• Mock action: / node: references to unknown targets
• Node mocks missing workflow: or with unknown workflow / missing node in that workflow
• Review formData and action output compatibility with declared JSON Schemas
• Warnings for unbound review mocks (no workflow: / node:)

The language server surfaces the same diagnostics in the editor as you type.

Doctor does not call the API. A clean doctor run is a strong signal that swirls test will compile the same workspace successfully.

Where results live

Each swirls test invocation creates a PlaybookRun record tied to the test deployment. View run history, per-case verdicts, and timing in the Cloud dashboard on the test deployment detail page.

Verdicts per case: passed, scored (soft threshold met), failed, or skipped.

Further reading

• CLI reference — swirls test
• Workflows: the DAGs playbooks exercise
• Reviews: human gates and review mock schemas
• Actions: integration stubs and output schemas
• Common mistakes: patterns that cause silent drops or validation failures