Evidence and reusable cases
Turn real work into reviewable examples while distinguishing published storage from development APIs.
A useful comparison case preserves the problem and the evidence needed to judge its answer. A human clicking “accept” is useful feedback, but may mean “good enough for this task,” not “every claim is correct.”
Availability: alpha.13 records runs, outputs, events and interactions. It does
not export Dataset or Evidence, or provide dataset snapshot commands. The
concepts here can be implemented in your application; dedicated APIs in the
development checkout are outside this published version.
Keep the candidate attached to its review
Input + source snapshot → candidate v1 → feedback on v1
↓
candidate v2 → acceptance of v2Do not attach a review of v1 to v2 as though it evaluated the revised content. For each case, retain the source revision, workflow version, candidate, checks and human feedback. Keep the actual starting files when they affect the answer.
An application-owned case format
import { Schema } from "effect";
export const ReviewCase = Schema.Struct({
caseId: Schema.String,
sourceRevision: Schema.String,
workflowVersion: Schema.String,
input: Schema.Struct({ topic: Schema.String, notes: Schema.String }),
candidate: Schema.Struct({ revision: Schema.Number, text: Schema.String }),
review: Schema.Struct({
candidateRevision: Schema.Number,
decision: Schema.Literal("accept", "revise", "decline"),
feedback: Schema.String,
}),
});This schema is your data contract, not a Laufwerk dataset primitive. Validate that
review.candidateRevision matches the reviewed candidate. Store the record in
your chosen system and manage its access, retention and immutable snapshots.
External writes still need idempotency or reconciliation after interruption.
Freeze a fair comparison
| Preserve | Why |
|---|---|
| Starting input and source content | Reproduce the problem |
| Workflow and model configuration | Know what produced the candidate |
| Outputs, failures and revisions | Avoid selecting only successful attempts |
| Human feedback and evaluator version | Know which judgment applied |
| Timing and available usage | Compare resources alongside quality |
Do not put the historical accepted answer in a new model's prompt when you intend to test whether it can solve the original case. Keep reference answers on the evaluator side. Redact or exclude sensitive data before reusing a real case in a broader evaluation environment.
Next use: compare alternatives in Benchmarks.