Runnable examples for the BenchFlow docs.
bench run benchflow-ai/skillsbench/tasks/edit-pdf \
--agent gemini --model gemini-3.5-flash-lite --sandbox daytonaSandboxes are docker (the default), remote-docker, daytona, modal, agentcore and apple-container. modal, agentcore and apple-container can't run BenchFlow's integrity check yet, so in the default mode a model agent's trial there is refused before its turn; use Docker or Daytona for model agents (Sandboxes).
- Hill-climbing a skill. An optimizer agent edits a skills folder each round. An edit stays only if the train score rises and the held-out test score does too. The optimizer only ever sees the train split. Based on Automating eval design and hillclimbing with Claude, with a SkillsBench recipe. See
hillclimb/. - Progressive disclosure on SWE-bench Pro. A simulated user gives the agent a short prompt first, then failing-test hints, then the full spec. Notebook:
swebench_pro_progressive_disclosure.ipynb. Guide: Progressive disclosure.
To give the agent a task's own skills (environment/skills), add --task-skills:
bench run tasks/my-task \
--agent gemini --model gemini-3.5-flash-lite --sandbox daytona --task-skillsFor a skills folder from elsewhere, pass it with --skills-dir and --skill-mode with-skill. See Architecture: skill loading.
python-sdk/ holds small scripts for the Python API. Start with python-sdk/quickstart.py, a tour in # %% cells.
| Script | Shows |
|---|---|
run-oracle.py |
one task with the oracle, and the typed result |
run-agent.py |
one task with a real agent, its trajectory and token usage |
run-many.py |
several agents and models on one task with bf.run_batch |
run-batch.py |
a folder of tasks as one Evaluation, with resume |
run-branch.py |
a run forked into two scored children with bf.branch |
run-with-manifest.py |
a task inside an environment manifest whose service BenchFlow starts |
compare-jobs.py |
two finished jobs compared with bf.compare; reads files only |
| Folder | What it is |
|---|---|
task-md/ |
task.md packages: runnable tasks and schema-only fixtures |
nanofirm-task/ |
a tiny task in the older task.toml layout, the input for bench import; keep it in that layout |
| File | Shows |
|---|---|
coder-reviewer-demo.py |
a single-agent baseline, then a coder-reviewer scene, on one task folder |
scene-patterns.md, scene-patterns.py, scene-patterns.ipynb |
single-agent, self-review, specialist-review and client-advisor scenes |
user_dogfood.py |
a rule-based FunctionUser progressive-disclosure loop |
swebench_pro_user_dogfood.py |
the same loop on SWE-bench Pro-style tasks |
swebench_pro_progressive_disclosure.ipynb |
the SWE-bench Pro walk-through as a notebook |
| File | Shows |
|---|---|
branch-agent-run.py |
an agent run on the hello-world task branched into two verifier-scored children, then the parent finished; see Composed checkpoints |
| GRPO recipe | a GRPO training workflow with TRL: task snapshots, a baseline eval, training, a final eval and the lift |
- Run your first task: Quick start.
- Several agents in one task: Scenes and simulated users.