Benchmark
Define variants and an evaluator, run native trials, and interpret saved results in alpha.14.
Import Benchmark, Finding and Findings from @laufwerk/sdk/benchmark.
Start with the runnable amount comparison.
Benchmark.define is the constructor; there is no Benchmark.make.
Benchmark.define(options)
Call at module scope, alongside the workflow definition.
| Property | Required | Meaning |
|---|---|---|
key: string | Yes | Nonblank benchmark identity; also names its output contract and trial dataset |
input: Schema.Schema<I, IE> | Yes | Input accepted by every variant |
output: Schema.Schema<O, OE> | Yes | Output produced by every variant |
variants | Yes | Array of { name: string, execute: (input: I) => Effect<O, E, R> } |
evaluatorVersion: string | Yes | Nonblank evaluator identity; change it for a changed judge |
evaluate | Yes | ({ input: I, output: O, reference: unknown }) => Effect<Findings, EE, ER> |
Variant names must be nonblank and unique. The returned object has layer, run
and evaluate. Merge layer with the parent workflow's registration layer using
Layer.mergeAll; it registers the child trial and evaluator workflows.
Definitions registered together need unique benchmark key/evaluator-version
pairs. Change the benchmark key when changing the output contract, because
<key>.output is stored as contract version 1.
benchmark.run(options)
| Property | Required | Meaning |
|---|---|---|
cases | Yes | Nonempty array of { exampleId: string, input: I, reference?: unknown } |
repetitions: number | No | Defaults to 1; integer from 1 through 100 |
Case IDs must be nonblank and unique. At least one variant is required to run.
exampleId is a caller-supplied case identity; passing an ID does not load its
input automatically. Map a snapshot to explicit cases in
your own code. The evaluator receives reference; the variant receives input.
Trials run sequentially in case → variant → repetition order in this release. There is no benchmark concurrency option. Each uses a native child execution identity derived from the benchmark, evaluator version, case, variant and zero-based repetition within its parent invocation.
Returned rows
The Effect returns an array with one of these shapes per trial:
| Field | Completed generation | Handled generation failure |
|---|---|---|
exampleId, variant | Strings identifying the case and variant | Same |
repetition | Zero-based number | Same |
status | "completed" | "failed" |
record | { recordId: string } for the saved output artifact | Reference to the saved failure record |
evaluation | { recordId: string } for the saved evaluation | Absent |
A completed row means generation returned a valid saved output. Read the evaluation record to establish quality or judge failure. The result links to records; it does not inline the findings.
Saved evaluation value
value.status | Other fields | Meaning |
|---|---|---|
"completed" | findings: Findings | The evaluator returned its judgments |
"failed" | error: string | The evaluator had a handled failure; there is no quality verdict |
Finding is an Effect Schema with:
| Field | Type | Required |
|---|---|---|
criterion | string | Yes |
verdict | "pass" | "fail" | "inconclusive" | Yes |
explanation | string | Yes |
score | Finite number from 0 to 1 | No |
Findings is an array of Finding. The API does not prescribe criteria or an
automatic promotion threshold.
Failure semantics
- A handled variant error becomes a
status: "failed"row; subsequent trials can run. - A handled evaluator error becomes a failed evaluation record. Its trial row can still be
completed. - A negative quality verdict is an ordinary completed evaluation.
- Schema/persistence failures and defects can fail the workflow. They are not all converted into negative reviews.
Trials contribute to dataset <key>.trials, version 1. A trial snapshot includes
its evaluator descendants and record links. A new parent run identity starts a
fresh comparison. Resuming the same run reuses its recorded work and saved bundle.
benchmark.evaluate(options)
Score an already saved output without generating it again:
| Property | Required | Meaning |
|---|---|---|
key: string | Yes | Fresh caller key for a new evaluation |
subject: RecordRef<O> | Yes | Saved output artifact to load and decode |
input: I | Yes | Original allowed input |
reference: unknown | No | Expected answer or other evaluator-only data |
Returns an Effect containing the evaluation record reference. A changed judge
needs a definition with a new evaluatorVersion and its layer registered too.
A rescore-only definition can have variants: []; run still requires variants.
New evaluations do not change previous snapshots.
Execution and isolation
Child executions isolate durable Activity, Human, clock and Session identities. They share the root run and do not allocate additional top-level workers. Files, credentials and external APIs require their own isolation choices.
Keep side-effecting generation and evaluation in durable Session/Activity
operations. Supply fake or restricted tools for actions a trial must not take.
Restore required starting files explicitly, using independent workspace ownership
where needed. Passing reference only to the evaluator does not prevent a variant
from reading the same information through an independently supplied tool.
For human gates, choose a generation-only path, simulated response, or a fresh human review in the variant. The benchmark API provides no general review bypass, automatic historical-environment restoration or optimizer.
Related: Evidence · Usage and cost · Deterministic testing