Testing

A playbook declares workflow inputs and expected results beside your .swirls definitions. swirls test compiles the local workspace, creates an environment: test deployment, starts each selected workflow, and records the results in Cloud.

Test deployments do not become active and do not receive live triggers. Their workflows still use project resources and credentials. External-call mocking is not implemented in the hosted execution path. Declaring mode: all does not prevent API calls, database writes, or paid model calls.

Example: smoke playbook

This complete example uses a code node with no external calls. Save it in your workspace, then run the commands below.

workflow process_contact {
  label: "Process contact"
  root {
    type: code
    label: "Normalize email"
    inputSchema: @json {
      { "type": "object", "required": ["email"], "properties": { "email": { "type": "string" } } }
    }
    outputSchema: @json {
      { "type": "object", "required": ["email"], "properties": { "email": { "type": "string" } } }
    }
    code: @ts {
      return { email: context.nodes.root.input.email.trim().toLowerCase() }
    }
  }
}

playbook smoke {
  label: "Smoke tests"
  tags: [ci]

  case normalize_email {
    workflow: process_contact
    input: @json { { "email": " [email protected] " } }
    timeout: "2m"
    expect {
      status: completed
      output: @json { { "root": { "email": "[email protected]" } } }
    }
  }
}

swirls auth login
swirls configure
swirls doctor
swirls test --playbook smoke

The project needs hosted execution access. doctor checks syntax and cross-references locally; it does not run the workflow or verify credentials.

Test deployments

The CLI creates a test deployment from your local source. Live forms, webhooks, and schedules continue using the project's active deployment. Test executions use the test snapshot, but persistent databases, disks, and credentials belong to the project. Use a separate project when tests need isolated resources.

The CLI creates a PlaybookRun for each selected playbook and updates it with case results. Creating a record through the playbooks.run API alone does not schedule executions; the CLI performs that work.

Cases and expectations

A case starts a workflow with fixture input and evaluates its expect block.

output is a subset match against the workflow's formatted output when present, otherwise its output. For a workflow without custom formatting, leaf results are keyed by node name, as in the smoke example.

Assertions normally fail the case. severity: soft records a failure without failing the case unless you pass --strict. Budget and similarity assertions are always soft. A skipped check does not fail under --strict; inspect the report's assertion messages.

Mocks

The language parses and validates mock blocks, including action outputs, review form data, and mode, only, and except policies. The current CLI applies mock mode to review automation only. It does not pass action or node output stubs into workflow execution, and only / except do not restrict external calls.

Review mocks target both a workflow and a node:

mock approve_draft {
  workflow: prepare_draft
  node: review_draft
  review: {
    auto: approve
    formData: @json { { "approved": true } }
  }
}

This is a fragment: prepare_draft must define review_draft with a compatible review schema. Select a named policy using the playbook's mocks: field. CLI --mock-mode overrides that policy, followed by the selected mock's mode, mock defaults, then implicit off.

Review automation currently queries pending reviews across the project, rather than just the current test execution. Run review tests in a dedicated test project so they cannot resolve production reviews.

Evals and judges

An eval runs a workflow and can score its output using a judge block. Declare it inside a playbook. This fragment assumes a generate_draft workflow exists:

playbook draft_checks {
  eval draft_quality {
    workflow: generate_draft
    input: @json { { "topic": "Workflow testing" } }
    timeout: "5m"
    retries: 2
    judge: {
      rubric: criteria
      criteria: "The draft explains how to test a workflow and includes a concrete example. Score 0-1."
      minScore: 0.75
      severity: soft
    }
    expect { status: completed }
  }
}

An eval's retries allows additional workflow executions, including repeated external side effects. Model-backed judges run from the CLI using OPENROUTER_API_KEY in the local environment. If that key is missing, the workflow still runs and the judge assertion is marked skipped. It does not fail the case.

Running tests

swirls test
swirls test --playbook smoke --case normalize_email
swirls test --tag ci --exclude-tag slow
swirls test --list
swirls test --reporter junit --strict

The CLI exits nonzero when a case fails. Cloud records case verdicts as passed or failed; the detailed CLI report includes skipped assertions and soft judge results. A passing report can contain skipped checks.

Validate before you run

swirls doctor validates the workspace, including mock and playbook references, node mock targets, and mock payload compatibility when schemas are available. Language acceptance does not imply that every declaration is enforced by the hosted runner; use the behavior tables above when deciding what a test proves.

See the CLI reference, workflow guide, and review guide for related configuration.

Test Jev routing and runtime model selection

Use the complete decision workflow as a starting point. Mock valid answers for automatic routing, an uncertain classification corrected by a reviewer, other, and decisive yes/no Noul. Exercise rejection/timeout and provider failure separately. Test inline/file-backed models, invalid returns, provider compatibility, different iterations, concurrent invocations, and retry reuse. Keep the original inference and human response assertions separate. Validate model quality on a labeled pilot set per pinned model: correctness, automation coverage, review burden, latency, and input-token usage. Mock tests cannot establish provider speed, cost, or accuracy.

Jev live smoke and pilot evaluation

Repository contributors can run the native SDK check with an explicit test vault. Create a private JSON file outside the repository containing the fixture's exact slot, decision_vendor::TYPESAFE_API_KEY, with a TypeSafe key authorized for the test project. This file is a test harness input; deployed workflows continue to use their declared project secret blocks. The runner never reads the SDK's environment key or searches other blocks.

bun packages/workflow-core/scripts/evaluate-decisions.ts \
  --vault /absolute/private/test-vault.json \
  --output /absolute/private/jev-smoke.json --smoke

Omit --smoke to evaluate the eight synthetic cases in packages/workflow-core/fixtures/support-triage-labels.json. The report records each native answer, requested/resolved models, latency, token usage, department accuracy, severity error, urgency Brier score, automation coverage, automated accuracy, and review burden. --threshold defaults to the example's unvalidated 0.9. Replace the starter cases with representative, independently labeled data before setting an automation policy. A provider failure aborts the evaluation instead of counting as a prediction.

The opt-in test uses the same runner: set SWIRLS_JEV_LIVE_VAULT to the test-vault path and run bun test packages/workflow-core/src/nodes/decision.live.test.ts. It is skipped without that explicit path. Contract tests use a mocked SDK transport and do not establish live quality, latency, or account-specific limits.

For calibration, pass --dataset /absolute/labeled-cases.json and --model jev-1.13.0 to the evaluation runner. The JSON array uses { "id": "case-id", "state": { "message": "..." }, "expected": { "department": "billing", "urgent": false, "severity": 0 } } records; department is billing, technical, or other, and severity is 0–2. Use representative, authorized examples including ambiguous, multilingual, urgent, and out-of-taxonomy cases. Each case makes one live request. Reports include a threshold sweep at 0.5, 0.7, 0.8, 0.9, 0.95, and 0.99 from the same inferences, so comparing thresholds does not multiply requests. Choose thresholds on a validation set and confirm them on held-out examples. The bundled synthetic examples are a plumbing check, not evidence of production accuracy.

Repository release checks include a real Temporal cancellation test and an opt-in Postgres suite. Set SWIRLS_TEST_POSTGRES_URL to a disposable localhost PostgreSQL server and run bun test apps/executor/src/__tests__/decision-postgres.test.ts; it creates and removes a uniquely named database. The suite exercises the deployment-to-executor round trip, model selection under concurrent connections and connection replacement, and managed quota isolation. CI runs it with a dedicated database service under Bun 1.3.11.

For a hosted release canary, repository contributors can use packages/language/examples/decisions-release.swirls in an isolated project with managed-key eligibility. Set the customer key in the declared release_customer secret block, retain release_managed as managed, and run swirls test --playbook jev_release. The two cases make live Jev requests and check all answer kinds, model/usage metadata, a dynamic model expression, and literal state through deployed execution. Inspect both execution records to verify customer-versus-platform key provenance. These cases perform no business actions.

Decision calibration

Use Cloud’s Decisions → Calibration to compare stored predictions with explicitly accepted labels, separated by definition, model, and deployment. The candidate Choice threshold performs a read-only simulation; it does not invoke TypeSafe or modify deployed routing. Keep tuning cases separate from held-out evaluation, inspect label coverage, and treat reviewed cases as a selected sample. See decision calibration for limits and measures.