Compare workflow changes

A benchmark answers a narrow question: did this change improve these cases under these evaluation rules? It does not establish universal agent quality.

Availability: there is no @laufwerk/sdk/benchmark export in alpha.13. You can compare pure functions with Bun tests and compare agent workflows using separate recorded runs today. The README's dedicated benchmark API is not a runnable example for this published version.

A comparison you can run now

Save both files together in your consumer's tests/ directory. The example mirrors the README's amount-extraction comparison using ordinary functions.

export const cases = [
  { id: "integer", input: "42 EUR", expected: 42 },
  { id: "decimal", input: "42.50 EUR", expected: 42.5 },
] as const;
export const variants = {
  baseline: (input: string) => Number.parseInt(input, 10),
  candidate: (input: string) => Number.parseFloat(input),
};
export function compare() {
  return cases.flatMap(example => Object.entries(variants).map(([variant, run]) => {
    const output = run(example.input);
    return { caseId: example.id, variant, output, passed: output === example.expected };
  }));
}

import { expect, test } from "bun:test";
import { compare } from "./amount";

test("the candidate preserves decimal amounts on the fixed cases", () => {
  const results = compare();
  expect(results.filter(row => row.variant === "baseline" && row.passed)).toHaveLength(1);
  expect(results.filter(row => row.variant === "candidate" && row.passed)).toHaveLength(2);
});

Run bun test from that consumer directory.

Both functions execute successfully. Their quality differs on the decimal case. Neither is a validated money parser: currencies, separators, signs and malformed inputs need additional rules and cases.

Apply the same discipline to agents

1. Freeze cases and starting context. Keep reference answers out of prompts.
2. Give every case, variant and repetition a fresh workflow execution identity.
3. Isolate writable files with separate runs. Use fake or restricted tools for actions that should not publish, charge or send anything during trials.
4. Keep provider failures separate from completed but incorrect outputs.
5. Score all variants with the same evaluator; preserve its version.
6. Compare quality, elapsed time and available usage. Repeat uncertain cases.

Changing a prompt and resuming an old run does not test the new prompt: that run retains its bundle and recorded work. A human chooses whether to adopt a change. There is no automatic optimizer or promotion policy supplied by this example.

Reference: Testing workflows

← Evidence and reusable cases · Next: Applications, HTTP and ownership →