Skip to content

Latest commit

 

History

History

examples

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Examples

Runnable examples for the BenchFlow docs.

bench run benchflow-ai/skillsbench/tasks/edit-pdf \
  --agent gemini --model gemini-3.5-flash-lite --sandbox daytona

Sandboxes are docker (the default), remote-docker, daytona, modal, agentcore and apple-container. modal, agentcore and apple-container can't run BenchFlow's integrity check yet, so in the default mode a model agent's trial there is refused before its turn; use Docker or Daytona for model agents (Sandboxes).

Featured

Skills

To give the agent a task's own skills (environment/skills), add --task-skills:

bench run tasks/my-task \
  --agent gemini --model gemini-3.5-flash-lite --sandbox daytona --task-skills

For a skills folder from elsewhere, pass it with --skills-dir and --skill-mode with-skill. See Architecture: skill loading.

Python SDK

python-sdk/ holds small scripts for the Python API. Start with python-sdk/quickstart.py, a tour in # %% cells.

Script Shows
run-oracle.py one task with the oracle, and the typed result
run-agent.py one task with a real agent, its trajectory and token usage
run-many.py several agents and models on one task with bf.run_batch
run-batch.py a folder of tasks as one Evaluation, with resume
run-branch.py a run forked into two scored children with bf.branch
run-with-manifest.py a task inside an environment manifest whose service BenchFlow starts
compare-jobs.py two finished jobs compared with bf.compare; reads files only

Tasks

Folder What it is
task-md/ task.md packages: runnable tasks and schema-only fixtures
nanofirm-task/ a tiny task in the older task.toml layout, the input for bench import; keep it in that layout

Scenes and simulated users

File Shows
coder-reviewer-demo.py a single-agent baseline, then a coder-reviewer scene, on one task folder
scene-patterns.md, scene-patterns.py, scene-patterns.ipynb single-agent, self-review, specialist-review and client-advisor scenes
user_dogfood.py a rule-based FunctionUser progressive-disclosure loop
swebench_pro_user_dogfood.py the same loop on SWE-bench Pro-style tasks
swebench_pro_progressive_disclosure.ipynb the SWE-bench Pro walk-through as a notebook

Branching and training

File Shows
branch-agent-run.py an agent run on the hello-world task branched into two verifier-scored children, then the parent finished; see Composed checkpoints
GRPO recipe a GRPO training workflow with TRL: task snapshots, a baseline eval, training, a final eval and the lift

Next steps