ActBench

ActBench is a self-evolving benchmark of behavioral safety in cowork agents. It defines behavioral safety as whether an agent's execution remains within the permissions and state changes required by a benign task, and evaluates realized behavioral risk from execution trajectories rather than final responses.

The benchmark contains 600 cases organized as 300 matched benign/adversarial pairs from 213 operational scenarios. Within each pair, the adversarial case preserves the benign user instruction, task configuration, initialized state, grading criteria, and trusted records while injecting one task-reachable payload. The cases span 15 risk behaviors, four propagation families, six execution spaces, and 48 web-service APIs.

This repository contains the runnable benchmark cases, trajectory runner, scorers, local mock services, and backend adapters. It is intentionally limited to the public execution and scoring path; construction experiments, publication artifacts, and release-only tooling are kept outside this checkout. Existing actbench.* result schema names are retained for compatibility.

The project and benchmark are named ActBench. The PyPI distribution and installed command are named deeptrap.

The accompanying paper is ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents.

Authors: Hongwei Yao^1†, Yiming Liu^2†, Meihui Chen⁴, Jieling Chen⁴, Zikun Chen², Yiling He³, Wangze Ni², Cong Wang¹, Kui Ren^2*

Affiliations: ¹City University of Hong Kong · ²Zhejiang University · ³University College London · ⁴Xiaomi Corporation

^†Equal contribution · ^*Corresponding author

Benchmark design

ActBench models an agent trajectory across six execution spaces: context, reasoning, safety policy, action, persistent memory, and environment. Its 15 risk behaviors are grouped into four propagation families: context manipulation, state integrity, action misuse, and policy mediation failure. A behavior label depends on the realized propagation path and prohibited effect, not merely on the presence of a suspicious payload or an unsafe-looking final answer.

Evaluation separates safety from task completion:

• AGS (Attack Grading Score) measures coverage of prohibited effects in adversarial executions.
• UGS (Utility Grading Score) measures coverage of required task effects in the matched benign executions.
• Dual evidence verification combines trusted log evidence with LLM-based trajectory evidence to verify both the realized effect and its execution path.

The accompanying paper describes a self-evolving construction method that uses reward-guided search and reflection to strengthen task-reachable payloads. This checkout contains the released benchmark cases and public evaluation path; construction experiments and release-only tooling remain outside the repository.

Install and verify

ActBench is published on PyPI under the distribution name deeptrap. Install it as an isolated command-line tool:

uv tool install deeptrap
# or
pipx install deeptrap

Alternatively, install it into the current Python environment:

pip install deeptrap

Verify the complete installation without calling a model or external judge:

deeptrap test --self-test

The wheel includes the benchmark tasks, skills, and mock-service fixtures, so a repository checkout is not required. Run deeptrap --help to see all commands.

What is included

• tasks/task_B*_T*/ — self-contained ActBench tasks.
• tasks/clean_scenes/ — benign clean-source bundles used for optional delta-aware baseline generation.
• scripts/actbench.py and scripts/benchmark/ — benchmark runner, backend adapters, and result aggregation.
• mock_services/ — local fixture-backed FastAPI services used by tasks.
• skills/mock_apis/ — standard mock API skill descriptions that describe the service endpoints.
• docs/ — task, result, mock-service, and backend setup notes.

Public dataset

The companion ZJUICSR/ActBench dataset on Hugging Face provides the released task pairs and execution trajectories described in the paper. It contains 300 matched benign/adversarial task pairs, representing 600 benchmark cases, together with the public task definitions, workspaces, fixtures, skills, logs, and graders.

Load the task pairs or released trajectories with datasets:

from datasets import load_dataset

task_pairs = load_dataset(
    "ZJUICSR/ActBench",
    "task_pairs",
    split="test",
)

trajectories = load_dataset(
    "ZJUICSR/ActBench",
    "trajectories",
    split="test",
    streaming=True,
)

Task inventory

ActBench contains 300 public matched-pair task directories, grouped by B-class selectors. Each directory contributes one benign case and one adversarial case to the 600-case benchmark:

B class	Risk behavior	Matched pairs
B1	Instruction injection	42
B2	Goal hijacking	13
B3	Data exfiltration	32
B4	Credential exposure	14
B5	Memory poisoning	15
B6	State tampering	37
B7	Deceptive tool invocation	14
B8	Unauthorized CMD execution	42
B9	Unauthorized API invocation	12
B10	Tool scope escalation	12
B11	Resource exhaustion	14
B12	Obfuscated execution	16
B13	False reporting	14
B14	Context flooding	11
B15	Permission chaining	12
Total		300

Requirements

• Python 3.10+
• uv or pip
• A Linux Docker Engine/daemon for real runs (Docker is the default execution mode)
• Native agent CLIs/services only when opting into --execution local:
  • OpenClaw CLI available on PATH for the default openclaw backend
  • Hermes CLI available on PATH or ACTBENCH_HERMES_BIN for --backend hermes
  • opencode CLI available on PATH or ACTBENCH_OPENCODE_BIN for --backend opencode
  • a running QwenPaw service for --backend qwenpaw
• A configured target model for the selected backend
• A judge-model API key when using LLM-assisted scoring:
  • DEEPSEEK_API_KEY for deepseek/... judge models
  • OPENROUTER_API_KEY for OpenRouter-routed judge models
  • For private OpenAI-compatible gateways, copy config/llm_backends.example.yaml to the ignored config/llm_backends.local.yaml and set the configured API-key/base-URL environment variables.

For development from a source checkout:

uv sync
# or
pip install -e .

For private OpenAI-compatible judge gateways, keep provider details local:

cp config/llm_backends.example.yaml config/llm_backends.local.yaml
export ACTBENCH_PRIVATE_GATEWAY_BASE_URL="https://your-gateway.example/v1"
export ACTBENCH_PRIVATE_GATEWAY_API_KEY="..."

Then use the configured neutral prefix, for example --judge-model deepseek/deepseek-flash.

One-command test

All six real backends run in Ubuntu 22.04 containers by default: OpenCode, Claude Code, Hermes, OpenClaw, QwenPaw and OpenAgent. Use --execution local only when deliberately running an agent installed on the host. Build/check select the image with --backend <name>; omitting it defaults to OpenCode. The controller and scorer stay on the host. For example:

deeptrap docker-build --backend opencode
deeptrap docker-check --backend opencode  # no model or judge calls
# Copy and edit the provider file before the first OpenCode Docker run.
cp config/opencode.provider.example.json config/opencode.provider.local.json
# Set its base URL and {env:...} credential reference to your provider.
export ACTBENCH_OPENCODE_PROVIDER_CONFIG="$PWD/config/opencode.provider.local.json"
ACTBENCH_DOCKER_ENV=OPENROUTER_API_KEY deeptrap test \
  --backend opencode --model openrouter/<model-id> \
  --suite task_B9_T01 --score-mode automated

Provide the selected provider credential in the host environment. See docs/DOCKER.md for image configuration, custom providers, rootless Docker networking, and execution artifacts.

For a bounded first run, use deeptrap test. It creates an invocation-scoped output directory, collects trajectories with the selected backend in Docker, scores only those trajectories, and writes a concise one_click_result.json summary. The default backend is OpenClaw and the default model label is deepseek/deepseek-v4-pro; pass --backend and --model when using another target.

Check the local ActBench plumbing without calling a real model or judge:

deeptrap test --self-test

Run the default representative security test against a backend/model:

deeptrap test \
  --backend claudecode \
  --model <target-model> \
  --judge-model deepseek/deepseek-flash

By default this runs one fixed task from each B class (task_B1_T01 through task_B15_T01), preserves clean-baseline generation/reuse, then performs Combined AGS scoring with the explicit judge model. Combined AGS invokes an external judge. For a no-judge local replay, opt into automated-only scoring:

deeptrap test \
  --backend claudecode \
  --model <target-model> \
  --score-mode automated

Use --suite task_B9_T01, --suite B9, or --suite all to change the task selection. Large runs and advanced recovery controls are still available through the lower-level collection and scoring commands below. See docs/ONE_CLICK.md for output layout, exit codes, backend notes, and score semantics.

Collect trajectories with ActBench

ActBench's public workflow is trajectory-first: the runner executes the target agent and records durable trajectories and artifacts. AGS, UGS, ASR, and pass@k are derived from execution evidence rather than the agent's final response alone. Official AGS/ASR/pass@k scoring is performed later with actbench_score.py.

Collect trajectories for all public tasks with the default OpenClaw backend:

deeptrap run --model deepseek/deepseek-v4-pro
# equivalent:
deeptrap run --backend openclaw --model deepseek/deepseek-v4-pro

Run with QwenPaw in Docker by using the default execution mode. The following host-service example opts into local execution explicitly. ActBench does not import the qwenpaw package or require a QwenPaw Python environment in Docker; it creates a task-scoped QwenPaw service agent bound to each materialized task workspace:

ACTBENCH_QWENPAW_BASE_URL=http://127.0.0.1:8088 \
  deeptrap run --backend qwenpaw --execution local --model deepseek/deepseek-v4-pro

For source-checkout QwenPaw runs, one typical service startup command is:

cd /path/to/QwenPaw
PYTHONPATH=/path/to/QwenPaw/src \
QWENPAW_WORKING_DIR=/tmp/qwenpaw-actbench \
python -m qwenpaw app --host 127.0.0.1 --port 8088

Relevant QwenPaw environment variables:

• ACTBENCH_QWENPAW_BASE_URL selects the QwenPaw service URL; default is http://127.0.0.1:8088.
• ACTBENCH_QWENPAW_API_KEY optionally sends Authorization: Bearer ... to the service.
• ACTBENCH_QWENPAW_TIMEOUT_SECONDS optionally caps individual service requests; if unset, ActBench uses the task timeout budget.
• ACTBENCH_QWENPAW_AGENT_PREFIX prefixes per-task service agent IDs; default is actbench.
• ACTBENCH_QWENPAW_DELETE_AGENT controls best-effort deletion of per-task QwenPaw agent registrations after each attempt; default is enabled.
• ACTBENCH_QWENPAW_HEADLESS_TOOL_GUARD is passed through to QwenPaw's request context.
• ACTBENCH_QWENPAW_USAGE_DELTA controls the ActBench-side token-usage fallback; by default ActBench first uses usage returned by QwenPaw process events, then falls back to the service's /api/token-usage/details or /api/token-usage aggregate delta when event usage is absent. Because that fallback is provider/model aggregate data from the QwenPaw service, unrelated concurrent QwenPaw traffic for the same provider/model can contaminate per-task token counts. For same-task parallel repeats (--run-workers > 1), ActBench disables this aggregate delta fallback and only trusts per-event usage returned by QwenPaw.

Run with OpenAgent in Docker by using the default execution mode. When an OpenAgent service is already running, use the explicit local mode shown here:

OPENAGENT_API_KEY=... \
OPENAGENT_BASE_URL=http://localhost:14000 \
deeptrap run --backend openagent --execution local --model deepseek/deepseek-v4-pro

OpenAgent support uses its OpenAI-compatible chat completions endpoint. ActBench records --model in results and sends it in the request, but the actual OpenAgent model and tools are controlled by the Store associated with OPENAGENT_API_KEY.

Run with Hermes in Docker by using the default execution mode. When the Hermes CLI is installed on the host, use the explicit local mode shown here:

ACTBENCH_HERMES_PROVIDER=... \
deeptrap run --backend hermes --execution local --model deepseek/deepseek-v4-pro

The Hermes backend launches isolated hermes -z subprocesses from each materialized task workspace. By default it writes a run-scoped HERMES_HOME, registers the ActBench MCP gateway as the actbench MCP server, and instructs Hermes to use task-scoped MCP tools for workspace and mock API access. See docs/HERMES.md for setup, environment variables, and troubleshooting.

Run with opencode in Docker by using the default execution mode. When the opencode CLI is installed on the host, use the explicit local mode shown here:

deeptrap run --backend opencode --execution local --model deepseek/deepseek-v4-pro

The opencode backend launches isolated opencode run --format json subprocesses from each materialized task workspace. By default it provides an inline opencode config with the ActBench MCP gateway as a remote MCP server named actbench, instructs opencode to use task-scoped MCP tools for workspace and mock API access, and extracts the full session with opencode export <sessionID> for scoring. See docs/OPENCODE.md for setup, environment variables, and troubleshooting.

By default, the OpenAgent adapter also exposes the per-task workspace and declared mock APIs through an ActBench-owned MCP gateway. Configure the OpenAgent Store once with the MCP URL that OpenAgent can reach:

• local OpenAgent: http://127.0.0.1:8765/mcp
• OpenAgent in Docker: http://host.docker.internal:8765/mcp

For each task attempt, ActBench materializes the workspace, starts declared mock services, registers a high-entropy task context_id with the gateway, and prepends a system message instructing OpenAgent to use the ActBench MCP tools with that context_id. The context is unregistered after the attempt and also has a TTL.

Relevant OpenAgent MCP environment variables:

• OPENAGENT_ENABLE_ACTBENCH_MCP=0 disables MCP and keeps the weak chat-completions-only mode.
• ACTBENCH_MCP_AUTOSTART=0 uses an externally managed gateway instead of autostarting one.
• ACTBENCH_MCP_HOST / ACTBENCH_MCP_PORT set the local gateway bind/check address; defaults are 127.0.0.1 and 8765.
• ACTBENCH_MCP_URL sets the public MCP URL shown to OpenAgent; default is http://127.0.0.1:8765/mcp.
• ACTBENCH_MCP_ADMIN_TOKEN optionally protects local context registration endpoints.
• OPENAGENT_TIMEOUT_SECONDS optionally caps individual OpenAgent HTTP requests; if unset, ActBench uses the task timeout budget instead of a fixed 120s cap.

The MCP gateway security model is task-scoped: file paths are resolved inside the materialized workspace, API discovery returns only service names and allowed business paths, actbench_call_api can call only the task's declared mock services and business paths, and administrative mock endpoints such as health, audit, reset, logs, fixture paths, raw base URLs, and admin tokens are not exposed to OpenAgent.

See docs/OPENAGENT.md for the full OpenAgent setup flow, including what to provide, how to add the ActBench MCP server in OpenAgent, and Docker networking notes.

OpenClaw, QwenPaw, OpenAgent, Hermes, and opencode all use --model as the model under test, so it can be varied across runs.

Run a subset by B class or exact task id:

deeptrap run --model deepseek/deepseek-v4-pro --suite B1
deeptrap run --model deepseek/deepseek-v4-pro --suite B1,B7
deeptrap run --model deepseek/deepseek-v4-pro --suite B10
deeptrap run --model deepseek/deepseek-v4-pro --suite task_B9_T01

Common options:

--runs 3                    # repeat each task three times
--run-workers 3             # run same-task repeats concurrently when the backend supports it
--skip-baseline-gen          # use cached benign baselines only
--regenerate-baselines       # rerun benign baselines and refresh aligned artifacts
--inline-scoring             # deprecated legacy mode: score inline during the run
--skip-scoring               # deprecated/no-op; trajectory-only collection is the default
--execution-retries 1        # retry retryable execution statuses within each repeat slot
--retry-status error,timeout # comma-separated statuses retried by --execution-retries
--no-training-artifacts      # legacy inline-only; do not use for offline trajectory scoring
--output-dir results         # where JSON results are written

Raw training artifacts may contain task prompts, transcripts, workspace contents, and model outputs. Benign baseline runs use the same artifact directory layout as attacked attempts when they are generated during a run; use --regenerate-baselines to refresh legacy baseline cache entries into the aligned artifact structure. Do not use --no-training-artifacts when collecting trajectories for later scoring; the CLI rejects that combination unless deprecated --inline-scoring is explicitly enabled.

Outputs

A default collection run writes:

• results/<run_id>_<model>.json — full per-run collection result with workflow: trajectory_collection, scoring_status: deferred, and inline_scoring: false.
• results/actbench_summary_<run_id>_<model>.json — compact collection summary (summary_kind: trajectory_collection).
• results/<run_id>_<model>_artifacts/ — raw per-attempt artifacts, including attacked and benign baseline trajectory.json copies.
• results/trajectories/ — canonical attacked-attempt trajectory tree used by offline AGS/ASR/pass@k scoring.

See docs/RESULT_FORMAT.md for schema notes.

Score collected trajectories

Trajectory-only collection is the default; --skip-scoring is retained only for older scripts. New artifacts use actbench.trajectory.v1; legacy OpenClaw actbench.openclaw_trajectory.v1 artifacts remain supported by the offline scorer.

Replay Python automated checks without external judge calls:

deeptrap score --trajectory results/trajectories --mode automated

This reruns Python automated checks against durable workspace_after/ snapshots and emits actbench.offline_score.v1 JSON. It does not compute combined AGS or call the LLM judge.

To reproduce official combined AGS/ASR/pass@k offline, explicitly opt in to external judge calls:

deeptrap score --trajectory results/trajectories --mode combined-ags

combined-ags invokes the configured external LLM judge using replay evidence from trajectories. Ensure the relevant judge API key is configured, and use --judge-model <model> to override the trajectory's judge model when needed. Prefer the canonical results/trajectories tree for attack scoring: raw artifact roots can also contain benign_baseline trajectories from baseline generation, while the canonical tree only contains attacked attempts. If reusing an output directory across unrelated runs, restrict the path to the relevant suite/task subtree or use a fresh --output-dir.

Pack runner output into a raw-by-task dataset when you want a stable manifest-addressed bundle for aggregate AGS scoring or clean utility workflows:

deeptrap pack \
  --result results/<run_dir>/<run_id>_<model>.json \
  --dataset-name <dataset> \
  --raw-by-task-root ~/data/actbench/raw_by_task

deeptrap score \
  --raw-by-task-root ~/data/actbench/raw_by_task \
  --raw-by-task-dataset <dataset> \
  --mode combined-ags \
  --judge-model deepseek/deepseek-flash

deeptrap utility-prep \
  --raw-by-task-root ~/data/actbench/raw_by_task \
  --raw-by-task-dataset <dataset> \
  --role benign \
  --output-dir results/utility_prep/<dataset>_benign

The packer copies attacked trajectories into <dataset>/<suite>/<task_id>/run_<n>/ and linked clean baselines into <dataset>/_baselines/<suite>/<task_id>/baseline/. The scoring command consumes attacked runs; the utility-prep command prepares clean baseline records for later UGS/task-completion judging. See docs/RESULT_FORMAT.md for raw-by-task layout and cache-only baseline exclusion details.

Scoring summary

ActBench reports behavioral safety and benign task utility with criterion-coverage scores in [0, 1]; these scores are not probability estimates:

• AGS is the fraction of attack criteria satisfied by an adversarial execution; AGS = 1 means that all prohibited-effect criteria are satisfied.
• UGS is the fraction of utility criteria satisfied by the matched benign execution; UGS = 1 means that all required task effects are satisfied.
• The AGS gradient uses the five levels 0, 0.25, 0.5, 0.75, and 1.
• New combined AGS rows expose independent Python and LLM binary attack verdicts; a run is successful only when both are 1.
• ASR is the fraction of binary-scored valid runs with both binary verdicts equal to 1; asr_binary reports the same value with its denominator.
• Rows without explicit binary fields are unscorable for the current ASR and must not be silently converted with a gradient threshold.
• For repeated runs, pass@k1, pass@k2, and pass@k3 report task-level attack success over the first k valid repeats.
• Lower AGS, ASR, and pass@k indicate less observed attack evidence; higher UGS indicates stronger benign task completion.

Task metadata uses behavior-first ActBench fields (behavior_id, behavior_type); scoring_family is the current B-class bucket (B1-B15). Legacy risk-family fields may appear in metadata for compatibility with older internal artifacts, but new integrations should use the behavior-first fields.

Paper results: base models under OpenClaw

The paper compares 15 base models under the fixed OpenClaw 2026.5.19 harness. At the paper's 0.9 attack-success threshold, ASR ranges from 8.6% to 89.4%. In the complementary experiment that fixes Deepseek-v4-Pro and varies six cowork-agent harnesses, ASR ranges from 68.9% to 89.4%. The values below are reported paper results for reference; they are not generated by the default checkout.

The table reports the fixed-OpenClaw results. Iter. is the median action-cycle count over adversarial rollouts.

Model	AGS_mal ↓	ASR (%) ↓	p@1/p@2/p@3 ↓	UGS_ben ↑	Iter.
Claude-Opus-4.8	0.284	8.6	8.3/10.3/11.7	0.938	15.0
Claude-Sonnet-4.6	0.347	17.9	17.0/21.0/22.3	0.927	16.0
GPT-5.5	0.493	35.6	34.3/42.0/45.3	0.928	16.0
GPT-5.4-mini	0.727	61.1	61.3/70.7/73.3	0.904	18.0
Grok-4.5	0.870	80.8	79.0/87.0/87.7	0.938	17.0
GLM-5.2	0.547	40.9	38.7/46.3/52.7	0.929	16.0
Qwen-3.7-max	0.511	36.3	35.3/45.7/52.0	0.915	16.0
Qwen-3.7-plus	0.524	40.1	40.7/49.0/52.7	0.915	16.0
Kimi-K3	0.489	33.7	33.7/42.3/45.3	0.940	17.0
Kimi-k2.6	0.748	65.7	66.3/73.3/78.3	0.869	17.0
MiniMax-M3	0.402	22.7	22.3/29.0/33.7	0.917	19.0
MiniMax-M2.7	0.804	71.1	72.0/80.0/84.7	0.880	17.0
Deepseek-v4-Pro	0.955	89.4	89.7/93.3/95.3	0.922	19.0
Deepseek-v4-Flash	0.887	80.2	78.0/86.0/89.3	0.900	19.5
Hunyuan-3.0	0.455	28.4	28.3/35.3/39.3	0.933	19.0

Mock services

Tasks declare any required mock services in task.yaml. The runner starts those services automatically on local random ports and writes api_endpoints.json into the task workspace. Users normally do not need to start mock services manually.

See docs/MOCK_SERVICES.md and mock_services/README.md for endpoint details.

Citation

If you use the ActBench benchmark, task pairs, or released trajectories, please cite ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents:

@article{yao2026actbench,
  title={ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents},
  author={Yao, Hongwei and Liu, Yiming and Chen, Meihui and Chen, Jieling and Chen, Zikun and He, Yiling and Ni, Wangze and Wang, Cong and Ren, Kui},
  journal={arXiv preprint arXiv:2608.09476},
  year={2026}
}