[kovtcharov-amd]

The custom-agent MCP harness has been failing on main since #3703, which is why unrelated PRs are showing a red Custom agent MCP harness job.

#3703 made a refused tool call declare that it never ran. The harness compares the whole result dict, so the added executed: False broke the assertion. It went unnoticed because this file lives under tests/installer/ and runs in its own workflow rather than the unit suite — so it turned main red instead of failing on the PR that changed the shape.

I pinned the flag rather than loosening the comparison: a refused call reporting itself as not-executed is exactly the contract #3703 added, and this is the only harness that exercises it through a real agent.

Test plan

• Custom agent MCP harness green on this PR (both ubuntu and windows)
• The same job goes green on PRs that were inheriting the failure, e.g. chore(deps): bump Lemonade Server to v11.9.0 #3342

[github-actions]

Verdict: Approve with suggestions

This pins the "I refused before running anything" flag that #3703 started returning, so the custom-agent MCP harness stops failing on main and stops painting unrelated PRs red. The expectation matches what the agent actually returns today, and pinning the flag (rather than loosening the comparison to a subset match) is the right call — it keeps the harness as the one end-to-end check that a refused call reports itself as not having run.

One thing to know before merging: the job this PR fixes does not run on this PR. The harness lives in the installer-build workflow, whose pull-request trigger is path-gated to installer and Electron files — and tests/installer/** isn't on that list. The checks on this PR confirm it: there's no "Custom agent MCP harness" run. So the first test-plan checkbox can't be ticked here; you'll need a manual workflow run (or a local run) to prove it green before merging, or the fix lands on main unverified.

The same gap is why this broke in the first place. A change to the agent's tool-refusal shape doesn't trigger the harness either, so it turned main red instead of failing on the PR that changed it — exactly as the description says. Adding the installer test tree to that path list is a small change that makes this PR self-verifying and catches the next shape change on its own PR.

Real-world evidence

No evidence-bundle.md in the tree, and this reviewer image has no pytest, so I could not execute the harness. This is a test-only change with no user-facing surface, so the rubric's evidence requirement is N/A — but I did verify the expected dict against the source rather than taking it on trust:

• src/gaia/agents/base/agent.py:3830 returns {**NOT_EXECUTED, "status": "error", "error": err} for the unknown-tool refusal, with that exact error string at line 3828.
• src/gaia/agents/base/verification.py:66-69 defines EXECUTED_KEY = "executed" and NOT_EXECUTED = {EXECUTED_KEY: False}.

So the three-key dict the test now asserts is exactly what the refusal path produces. The verdict rests on that static match plus the CI-trigger check above — not on a harness run.

🔍 Technical details

🟡 The harness job isn't triggered by this PR's own path (.github/workflows/build-installers.yml:84-102)

build-installers.yml is the only workflow that runs tests/installer/test_custom_agent_mcp_harness.py (line 1207). Its pull_request.paths covers installer/**, the webui/electron-builder files, tests/electron/_helpers/**, tests/electron/*-smoke.test.mjs, tests/electron/fixtures/**, and src/gaia/version.py — not tests/installer/**. gh pr checks 3736 shows no "Custom agent MCP harness" job, so this fix merges without CI exercising it.

Minimal fix — add the harness tree to the filter so a PR editing the harness runs it:

      - 'tests/electron/fixtures/**'
      - 'tests/installer/**'
      - 'src/gaia/version.py'

Worth a separate discussion (not a blocker here): whether src/gaia/agents/base/** also belongs on that list. It would have caught #3703 on its own PR instead of on main, but it also puts the ~15-minute cross-platform build in front of a lot of ordinary agent PRs. A cheaper alternative is a unit-level assertion of the refusal dict's full shape in tests/unit/agents/ — that file already imports NOT_EXECUTED (tests/unit/agents/test_verification_scope.py:35), so the contract could be pinned where every PR runs it, leaving the installer harness as the end-to-end confirmation.

Strengths

• Pinning executed: False instead of relaxing the equality preserves the assertion's value — a subset match would have silently accepted a regression that dropped the flag.
• Hardcoding the literal "executed" rather than importing EXECUTED_KEY is the right choice for a black-box harness: it fails if the key is renamed, which is the observable contract an integrator depends on.
• The inline comment explains why the flag matters (it's what keeps the verification footer honest) rather than restating the assertion — matches the repo's comment convention.

[kovtcharov-amd]

One thing to know before merging: the job this fixes does not run on this PR.

Custom agent MCP harness lives in build-installers.yml, which is reached by workflow_call on installer-path changes — it does not trigger when the harness test itself is edited. So a change to this file cannot be validated by the job that runs it, which is the same shape of gap that let the breakage reach main in the first place.

I verified the expected value against the source instead of guessing it: the unknown-tool path returns {**NOT_EXECUTED, "status": "error", "error": err}, and evaluating that gives {"executed": False, "status": "error", "error": "Unknown tool name. Use only the tools you were given."} — character-for-character what the assertion now expects, and what the failing run on #3342 reported as actual.

The real confirmation will be Custom agent MCP harness going green on #3342 once this lands.