Evaluate when a persona should become a skill, when a skill should become an agent, and when an agent needs a harness.
Run the same task through five control abstractions. Compare answer quality, groundedness, stability, cost, latency, and promotion value. Know when each layer pays for itself.
Today the runnable benchmark focuses on persona, lens, and skill. Richer agent and harness execution models are planned rather than fully implemented.
$ uv sync --extra dev $ mkdir -p workspace/readme-example $ uv run promptstackbench init --path workspace/readme-example $ export OPENAPI_API_KEY=<api-key> $ uv run promptstackbench run --suite architecture_review --treatments persona,lens,skill --repetitions 1 --paraphrases 1 --mock --data-dir workspace/readme-example/datasets --specs-dir workspace/readme-example/specs --traces-dir workspace/readme-example/traces $ uv run promptstackbench report --latest --format html --traces-dir workspace/readme-example/traces
Five control abstractions, from lightweight role instructions to planned agent and harness runtimes. Each adds structure — and cost. The benchmark measures when the trade-off pays off.
Role and interaction style. Easy to write, good UX — but hidden assumptions, weak reproducibility, role drift.
Best for: Low-risk tasks where voice matters
Constrained viewpoint. Sharper than persona, less theatrical — but still no defined procedure.
Best for: Tasks needing a stable perspective
Procedure + output schema. Interpretable, testable, reusable. The default unit of reliable work.
Best for: Repeatable, reviewable tasks with known output shape
Planned richer runtime with tools, state, and multi-step execution.
Best for: Tasks requiring tools, state, or multi-step search
Planned control layer with guards, retry, and audit behavior.
Best for: Compliance, safety, production actions, audit requirements
Current scoring combines answer-quality checks, structure checks, and robustness summaries. Tokens, latency, and cost are reported alongside those scores.
Does the output solve the task?
Final-answer
Does it cover all necessary aspects?
Final-answer
Is the response well-structured and readable?
Final-answer
Is every part relevant to the task?
Final-answer
Does output match expected structure?
Final-answer
Are claims grounded or fabricated?
Final-answer
Does it hold under paraphrase perturbations?
Robustness
How consistent across repeated runs?
Robustness
This preview is based on a real generated HTML report from run 2026-07-06-202054. It shows the current report shape more accurately than a fabricated benchmark-run terminal demo.
PromptStackBench Report Run ID: 2026-07-06-202054 Suite: architecture_review Model: gpt-4.1 Summary persona: senior_architect_persona correctness: 9.50 completeness: 9.00 clarity: 9.00 relevance: 10.00 schema_validity: 8.00 avg tokens: 713 avg latency: 8679ms avg cost: $0.0067 skill: architecture_review_skill correctness: 10.00 completeness: 9.00 clarity: 9.00 relevance: 10.00 schema_validity: 5.00 avg tokens: 586 avg latency: 8515ms avg cost: $0.0048 Promotion signals persona -> lens: quality -13.1%, cost +6.9% lens -> skill: quality +11.5%, cost -32.5% Per-task scores arch_review_001: persona 9.14, skill 8.86, lens 7.86 arch_review_002: persona 9.43, skill 9.14, lens 8.29
Define task suites and taxonomy specs in YAML, then initialize a fresh workspace directory for each example run.
Execute each task through each treatment with configurable repetitions and paraphrases. Outputs are stored in SQLite.
Generate HTML or Markdown reports with scores, robustness summaries, and promotion signals.
Install, initialize, and run your first mock benchmark in a repo-local workspace.
$ uv sync --extra dev # Install dependencies $ mkdir -p workspace/readme-example # Create a fresh workspace root $ uv run promptstackbench init --path workspace/readme-example # Initialize workspace $ export OPENAPI_API_KEY=<api-key> # Set API key for non-mock runs $ uv run promptstackbench run --suite architecture_review --treatments persona,lens,skill --repetitions 1 --paraphrases 1 --mock --data-dir workspace/readme-example/datasets --specs-dir workspace/readme-example/specs --traces-dir workspace/readme-example/traces # Run benchmark $ uv run promptstackbench report --latest --format html --traces-dir workspace/readme-example/traces # Generate report