Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/reference/agent-benchmarks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,9 @@ write that copy and nothing else. Scoring runs in two stages:
- **Verified** additionally checks the agent's own tool record for a `pytest` run
that passed after its last file edit — proof the "4 tests pass" in the answer
matches a command that was actually run, not just claimed.
- **After the answer**, any tool that ran on work the request never touched
fails the task: an agent that fixed the bug and then spent minutes on a job
nobody asked for did not finish cleanly, however good the fix.

**The judge for everything a probe can't check** — quality of the reasoning,
whether instructions were followed, and question tasks like "what does this file
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/cli.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ Agents stop after a maximum number of reasoning/tool steps. The default is **50*

An agent that runs out of steps before finishing still answers: it gets one more model call in which it can't use tools, and reports what it found, what it couldn't finish and why, and what you can do next. If that last call fails, you get the step-limit notice plus a line saying why the summary couldn't be written.

The same closing call runs when an agent is stopped for repeating one tool call with identical arguments (`max_consecutive_repeats`, default **4**): the model gets one correction first, and if it asks for the same call again the turn ends — a run that finished its work and then kept re-checking it reports what it did instead of a "can't confirm" notice. Repeats that failed or were refused still report the failure.
The same closing call runs when an agent is stopped for repeating one tool call with identical arguments (`max_consecutive_repeats`, default **4**): the model gets one correction first, and if it asks for the same call again the turn ends — a run that finished its work and then kept re-checking it reports what it did instead of a "can't confirm" notice. Repeats that failed or were refused still report the failure. The same applies to one tool that keeps failing with different arguments, and to an agent that, having answered, starts work the request never asked for: the off-request call is not run, and a second one ends the turn with an answer to what you asked.

- **Per-invocation:** pass `--max-steps <n>` to `gaia chat` (e.g. `gaia chat --max-steps 80`).
- **Fleet-wide:** set the `GAIA_AGENT_MAX_STEPS` environment variable to change the default for every agent at once, without per-command flags.
Expand Down
1 change: 1 addition & 0 deletions docs/reference/eval.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ It needs the flagship installed (`pip install -e hub/agents/gaia/python`), a run
- A task can name a **setup** (`"setup"` in `tasks.json`, defined in `src/gaia/eval/task_setups.py`) that changes its copy before the agent starts: an unrelated failing test, a test that passes only once its cache is warm, a test that never finishes, a 1.7 MB log generated from a fixed seed, a README that contradicts the code, or a git checkout with uncommitted edits. Everything works offline. The harness keeps a snapshot of the copy after its setup, outside the agent's reach, and measures the agent's changes against that snapshot.
- A **coding task** passes when every check in its `expect` block holds: the project's own tests pass (`tests_pass`), a probe run inside the finished project confirms the behavior (`probe`), and each file listed in `unchanged` is byte-identical to the snapshot. For example, the bug fix must accept lowercase `z`, and the weekly report must match an independent recomputation row for row.
- A **question task** is decided by the judge: it passes only if the answer establishes every one of the task's required points (`must_establish`), in any wording. Keyword matching was tried first and dropped: correct answers phrase the same finding in too many ways. A question has no `expect` block; loading one that does is an error.
- A task **fails when the agent kept working after its answer**: any tool that ran after the turn's first answer on something the request never touched (no file it named, read or changed, and not a test run) is listed in the task's `after_answer`, and the task fails whatever its checks or the judge say. GAIA harness only.
- **Verified** records, per task, whether the agent's own tool record shows a pytest run that passed after its last file edit. `pytest`, `python -m pytest`, and pytest run through `run_python` all count. A command run twice counts by its latest run, and a failure that a snippet or a pipe prints is a failure even when the command exits 0. Edits made through a shell command or a snippet are not seen.
- The **judge** (Claude, with no tools) reads the original project once, then each task's setup changes, answer and diff, and grades every task in a single call. It scores four things from 1 to 5: following the instructions, quality of the work, reasoning, and whether the answer claims anything the agent never did. It is never told whether a coding task passed.
- If the model backend cannot be reached, the task is **not measured**. That is reported as an infrastructure failure, never as the agent getting worse.
Expand Down
5 changes: 5 additions & 0 deletions docs/sdk/core/agent-system.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -578,6 +578,11 @@ def search_codebase(query: str) -> str:

**What the user gets:** at `max_consecutive_repeats` identical calls the model gets one correction in place of the call, and a repeat after that ends the turn. If the calls that ran worked, the agent makes one more model call with no tool calls allowed and answers from their results; if they errored or were refused, the answer names that failure.

Two loops that vary their arguments are stopped the same way:

- **One tool failing with new arguments each time.** After `max_consecutive_repeats` rejections of the same tool, with no file changed in between, the next call to it is not run and the model is told to change approach; calling it again ends the turn. A command that ran and exited non-zero (a failing test run) does not count.
- **New work after the answer.** Once the turn has produced an answer, a call that touches nothing the request touched (no file it named, read or changed, and not a test run) is not run, and the model is told to finish and offer the work instead. A second such call ends the turn with a closing answer to the original request.

**Solutions:**
```python
# 1. Limit max steps
Expand Down
8 changes: 8 additions & 0 deletions hub/agents/gaia/npm/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,14 @@ the terminal UI meant building it from source.

### Fixed

- **The agent no longer starts an unrelated job after answering.** A bugfix
request loaded the `coding` skill, whose "find every call site" tip switched
on document inventorying; after the fix was done and verified, the agent spent
minutes extracting every file it had read until the user cancelled. Only a
skill built on the extraction tool turns inventorying on now. Separately,
once a turn has answered, a tool call on nothing the request touched is not
run and a second one ends the turn, and a tool that keeps failing with new
arguments is stopped like an identical repeat.
- `/v1/gaia/query` now honours `provider` on an existing session. It used to
matter only when the session was created, so `provider: "lemonade"` could keep
sending a Claude session's conversation to Anthropic, and `provider: "claude"`
Expand Down
4 changes: 3 additions & 1 deletion hub/agents/gaia/npm/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -482,7 +482,9 @@ declares no tools, and its body measures 702 tokens (tiktoken `cl100k`).
**`document-extract` ships bundled but not enabled.** `gaia-voice` routes a
request for every item in a document to it, and it drives
`extract_document_items` and `save_extracted_items` so a long transcript yields
a complete, source-quoted inventory rather than a summary.
a complete, source-quoted inventory rather than a summary. A skill turns that
inventorying on only by listing `extract_document_items` in `tools_required`;
wording such as "find every call site" in another skill does not.

**No skill *set* loads.** `gaia-agent.yaml` ships its `skill_sets:` and
`default_skill_set:` blocks **commented out** — following the email agent's
Expand Down
123 changes: 118 additions & 5 deletions src/gaia/agents/base/agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,11 @@
from gaia.agents.base.project_map import resolve_project_root
from gaia.agents.base.step_timing import StepTimer
from gaia.agents.base.tools import _TOOL_REGISTRY
from gaia.agents.base.turn_scope import (
ANSWERED_MARKER,
POST_ANSWER_CLOSING_PROMPT,
TurnScopeGuard,
)
from gaia.agents.base.verification import (
NOT_EXECUTED,
VERIFY_AFTER_CHANGE_TAG,
Expand Down Expand Up @@ -1493,6 +1498,7 @@ def __init__(
"so below 2 every tool call is a repeat of itself and none runs."
)
self.max_consecutive_repeats = max_consecutive_repeats
self._turn_scope = TurnScopeGuard(max_consecutive_repeats)
self._current_query: Optional[str] = (
None # Store current query for error context
)
Expand Down Expand Up @@ -2081,17 +2087,21 @@ def read_tool_output(
del self._system_prompt_cache

def _extraction_skill_instructions(self, query, just_loaded=False):
"""A selected task skill may activate extraction; voice routing may not."""
"""A selected extraction skill may activate extraction; others may not."""
active = set(getattr(self, "_active_skill_filter", None) or ())
skills = getattr(self, "_loaded_skills", None) or {}
selected = active | {name for name in skills if name in query}
if isinstance(just_loaded, str):
selected.add(just_loaded)
selected -= self._always_on_skill_names
# Only a skill built on the extraction tool counts: coding's "find
# every call site" is advice, not a request for an inventory.
return "\n".join(
effective_skill_body(self, skills[name])
for name in sorted(selected)
if name in skills
and "extract_document_items"
in getattr(getattr(skills[name], "gaia", None), "tools_required", ())
)

def _read_validator(self):
Expand Down Expand Up @@ -4873,6 +4883,13 @@ def _execute_tool_timed(self, tool_name: str, tool_args: Dict[str, Any]) -> Any:
refused = self._refuse_inventory_hand_edit(tool_name, tool_args)
if refused is not None:
return refused
outermost = not getattr(self, "_tool_timing_depth", 0)
scope = getattr(self, "_turn_scope", None) if outermost else None
if scope is not None:
refused = scope.check(tool_name, tool_args)
if refused is not None:
logger.info("Turn scope refused %s: %s", tool_name, refused["error"])
return refused
evidence = getattr(self, "_completion_evidence", None)
before = (
evidence.snapshot(
Expand All @@ -4888,11 +4905,11 @@ def _execute_tool_timed(self, tool_name: str, tool_args: Dict[str, Any]) -> Any:
# Only the outermost call is timed. A tool body may call another tool
# (CodeAgent's orchestration does); timing both would count the inner
# one's seconds twice and push tool_s past the turn's own total.
if (recorder is None and step_timer is None) or getattr(
self, "_tool_timing_depth", 0
):
if (recorder is None and step_timer is None) or not outermost:
result = self._execute_tool(tool_name, tool_args)
self._note_verification_signal(tool_name, tool_args, result, before=before)
if scope is not None:
scope.record(tool_name, tool_args, result)
return result

started = time.perf_counter()
Expand All @@ -4905,6 +4922,8 @@ def _execute_tool_timed(self, tool_name: str, tool_args: Dict[str, Any]) -> Any:
result = self._execute_tool(tool_name, tool_args)
ok = not self._is_error_result(result)
self._note_verification_signal(tool_name, tool_args, result, before=before)
if scope is not None:
scope.record(tool_name, tool_args, result)
return result
finally:
self._tool_timing_depth = 0
Expand Down Expand Up @@ -6736,6 +6755,13 @@ def _process_query_impl(
self._extraction_ledger = ExtractionLedger(
user_input, os.getcwd(), available=self._read_validator() is not None
)
# A test agent built via __new__ never ran Agent.__init__, so it has
# no guard yet — create one rather than crash (mirrors _extraction_ledger).
if getattr(self, "_turn_scope", None) is None:
self._turn_scope = TurnScopeGuard(
getattr(self, "max_consecutive_repeats", 4)
)
self._turn_scope.begin_turn(user_input, os.getcwd())

# Orientation. Runs before the prompt is composed so anything it
# establishes is in the prompt on the turn that established it.
Expand Down Expand Up @@ -7063,6 +7089,16 @@ def _process_query_impl(
self._create_tool_message(tool_name, truncated_result)
)

if self._turn_scope.turn_should_end:
final_answer, steps_taken = self._end_turn_on_scope(
tool_name, messages, conversation, steps_taken
)
if final_answer is None:
cancelled_by_console = True
else:
loop_break_answer_unprinted = True
break

# Check for error (support multiple error formats)
is_error = isinstance(tool_result, dict) and (
tool_result.get("status") == "error" # Standard format
Expand Down Expand Up @@ -8234,6 +8270,17 @@ def _process_query_impl(
)
)

if self._turn_scope.turn_should_end:
final_answer, steps_taken = self._end_turn_on_scope(
tool_name, messages, conversation, steps_taken
)
if final_answer is None:
cancelled_by_console = True
else:
loop_break_answer_unprinted = True
fanout_repeat_break = True
break

# Track errors but DON'T break early — drain all N
# tool calls first so conversation history reflects
# the full set, per #944 acceptance criterion (b).
Expand Down Expand Up @@ -8496,6 +8543,16 @@ def _process_query_impl(
# Share tool output with subsequent LLM calls
messages.append(self._create_tool_message(tool_name, truncated_result))

if self._turn_scope.turn_should_end:
final_answer, steps_taken = self._end_turn_on_scope(
tool_name, messages, conversation, steps_taken
)
if final_answer is None:
cancelled_by_console = True
else:
loop_break_answer_unprinted = True
break

# For single-step plans, we still need to let the LLM process the result
# This is especially important for RAG queries where the LLM needs to
# synthesize the retrieved information into a coherent answer
Expand Down Expand Up @@ -8996,6 +9053,16 @@ def _process_query_impl(
"start GAIA with the `--sd` flag to enable it."
)

# An answer, not a plan or a narrated step: from here on, calls
# must serve the request rather than start new work.
if not self._turn_scope.answered:
self._turn_scope.mark_answered()
conversation.append(
{
"role": "system",
"content": {"type": ANSWERED_MARKER, "step": steps_taken},
}
)
# Changed code after the last test run: ask once for the run.
if not verify_after_change_reprompted and steps_taken < steps_limit - 1:
_correction = self._verify_after_change_prompt()
Expand Down Expand Up @@ -9099,7 +9166,12 @@ def _process_query_impl(
(
"[check:completion] "
+ " ".join(artifact_gaps)
+ " For incomplete extraction, call `extract_document_items` on each source file. Never replace enumeration with a summary. Use `write_file` for a missing requested save, then "
+ (
" For incomplete extraction, call `extract_document_items` on each source file. Never replace enumeration with a summary."
if extraction_gaps
else ""
)
+ " Use `write_file` for a missing requested save, then "
"`read_file` with offset=0 and limit=8000 to observe that exact output. Follow all "
"continuation pages. Report only contents observed in "
"tool results. An unrelated tool or file is not evidence."
Expand Down Expand Up @@ -9606,6 +9678,47 @@ def _answer_after_repeated_calls(
# Subclass hooks append corrections, so a second pass duplicates them.
return answer, steps_taken

def _end_turn_on_scope(
self,
tool_name: str,
messages: List[Dict[str, Any]],
conversation: List[Dict[str, Any]],
steps_taken: int,
) -> Tuple[Optional[str], int]:
"""End the turn once the turn-scope guard gives up, the loop-guard way.

After an answer the closing call restates it, because the answer is
what the user asked for; before one, a failing tool gets the loop
guard's failure summary.
"""
scope = self._turn_scope
self.console.print_repeated_tool_warning()
if not scope.answered:
return self._answer_after_repeated_calls(
tool_name,
scope.failures.get(tool_name, 0),
[{"status": "error", "error": scope.last_error.get(tool_name, "")}],
messages,
conversation,
steps_taken,
)
steps_taken += 1
if self._turn_recorder is not None and self.chat is not None:
self.chat.turn_step = steps_taken
self.execution_state = self.STATE_COMPLETION
try:
answer = self._closing_answer(
messages, conversation, POST_ANSWER_CLOSING_PROMPT, steps_taken
)
except Exception as e: # noqa: BLE001 - the reason goes in the answer
logger.warning("Could not write the closing answer: %s", e)
return (
"I stopped because I had started work this request did not ask "
f"for, and the closing answer couldn't be written: {e}",
steps_taken,
)
return answer, steps_taken

def _dedup_mutation_call(
self,
tool_name: str,
Expand Down
11 changes: 11 additions & 0 deletions src/gaia/agents/base/completion.py
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,17 @@ def _scan_paths(text: str, immediate: bool, modifiers: bool = False) -> list[str
return paths


def mentioned_paths(text: str) -> list[str]:
"""Every path *text* names, whether it is read from or written to."""
return list(
dict.fromkeys(
path
for match in _TARGET.finditer(text)
if (path := _path_token(match)) is not None
)
)


def destination_paths(text: str, prepositions: re.Pattern = _DESTINATION) -> list[str]:
"""Read a destination noun phrase, stopping at the next action."""
# A later 'email it to me' must not replace the save's destination.
Expand Down
Loading
Loading