Testing and benchmarks

There is no exported Benchmark primitive in the current SDK. Workflow benchmarking today combines ordinary tests, controlled workflow runs and recorded evidence. There is no Benchmark.make, automatic dataset replay or automatic workflow optimizer to configure.

Test the deterministic parts first

Extract scoring, validation and routing into ordinary functions and test them with bun:test. For a pure Effect workflow, use WorkflowEngine.layerMemory to test the workflow definition and result without a persistent Laufwerk server.

import { expect, test } from "bun:test";
import { Workflow, WorkflowEngine } from "@effect/workflow";
import { Effect, Schema } from "effect";

const workflow = Workflow.make({
  name: "sum-test",
  payload: { requestId: Schema.String, values: Schema.Array(Schema.Number) },
  success: Schema.Number,
  idempotencyKey: input => input.requestId,
});
const layer = workflow.toLayer(input =>
  Effect.succeed(input.values.reduce((sum, value) => sum + value, 0)),
);

test("returns the computed result", async () => {
  const result = await Effect.runPromise(
    workflow.execute({ requestId: "case-1", values: [2, 3] }).pipe(
      Effect.provide(layer),
      Effect.provide(WorkflowEngine.layerMemory),
    ),
  );
  expect(result).toBe(5);
});

This tests orchestration, not crash recovery, provider credentials, filesystem isolation or durable database behavior. An in-memory engine does not supply Laufwerk's Session/Human/Workspace runtime services. Use real registered runs to verify those integrations. Internal service mocks used in runtime development are not a stable public testing package.

Verify actual session commands

session.exec returns { exitCode, stdout, stderr }. A workflow should check exitCode and turn a failing check into an explicit failure or repair branch. An agent saying tests passed is not the command receipt. The Session example shows this boundary.

Compare workflow versions

A benchmark case is currently your own application data, not an SDK type. A useful case record includes:

Run baseline and candidate from equivalent fresh inputs. Use separate copied workspaces through separate runs. Keep external publication steps out of trials unless that side effect is explicitly part of the benchmark environment. Repeat model-dependent cases when needed to observe variability.

Editing workflow source affects future runs; existing runs retain their original bundle. Resuming the same run does not benchmark the new source. A repeated idempotency key can return old work instead of executing a fresh trial.

Inspect measurements

bunx laufwerk@0.0.1-alpha.13 status RUN_ID --json
bunx laufwerk@0.0.1-alpha.13 timings RUN_ID --json

Run status, recorded results, activity logs and timing information are available through CLI/Studio. Token usage depends on what the provider reports; missing usage is unknown, not zero. Overlapping activity times cannot simply be added to obtain total elapsed time. Token counts are not a billed-price receipt.

Historical accepted work can form a customer-owned comparison dataset, but reconstructing cases, judging new outputs and choosing a candidate remain explicit work today. Do not treat model novelty or a single successful trial as proof of workflow improvement.