[dianagrecu-uipath]

Problem

Agent evals that simulate tools failed with Gemini models in the tool simulator: the LLM call in mocks/_llm_mocker.py that generates the simulated tool response with the agent's model. There were two causes.

1. Type lists in tool schemas. MCP servers write nullable fields as "type": ["string", "null"]. The simulator sends that schema as the parameters of its forced submit_tool_response tool, and Gemini rejects it:

   400 Invalid JSON payload received. Unknown name "type" at 'tools[0].function_declarations[0].parameters.properties[0].value.properties[0].value.properties[0].value'
   

   Gemini's function declarations take a single type and mark nullable fields with nullable: true. OpenAI and Anthropic accept type lists.

2. Output token limit. The simulator never sets max_tokens, so every simulation call uses the chat service default of 4096. Gemini counts its reasoning tokens against that limit. When simulating a tool response of about 3,000 tokens, the failed calls used 2,600–3,900 reasoning tokens and returned no output. The run then failed with ValueError: LLM response contained no tool calls (content=None). Replaying the same request with max_tokens: 16384 used 5,658 tokens (2,657 reasoning + 3,001 output) and returned the expected response.

   Other reasoning models also count reasoning against this limit. gpt-6-luna's simulation calls in Agent Builder evals reach about 2,700 tokens today, close to the 4096 limit.

Fix

• _structured_output.py: GeminiStructuredOutput rewrites the schema with _to_gemini_schema() before the forced tool call. Other providers are unchanged.

  • {"type": [X, "null"]} becomes {"type": X, "nullable": true}.
  • Several non-null types become an anyOf of single types.
  • ["null"] alone becomes {"nullable": true}.

  It walks nested schemas (properties, items, anyOf, $defs, ...). It leaves data keywords (default, enum, const, examples, required) untouched and does not modify the caller's schema.

• _simulation_model.py: sets the max_tokens of every simulation call.

  • When the simulation uses the agent's model, the limit is the smaller of the agent's own max tokens and 16384. The agent already calls the model with its own value, so the model accepts it. The cap bounds a runaway reply.
  • When the agent's value is unknown (coded agents, or a model set in the simulation config), the limit is 16384.
  • An explicit maxTokens in the model settings is kept.
  • get_agent_max_tokens() reads the agent's value from schema.metadata["settings"], next to the model.

• Call sites: tool simulation and input generation in runtime/runtime.py, and uipath run / uipath debug simulation through build_mocking_context and load_simulation_config. Both gain an optional agent_max_tokens parameter.

• Placement of get_agent_max_tokens: it lives in mocks/_simulation_model.py, not in eval/helpers.py next to get_agent_model. cli_run.py already imports uipath.eval.mocks, while importing uipath.eval.helpers would add about 64 ms and 44 modules to every uipath run.

Tests

• test_structured_output.py:
  • Gemini rewrites type lists in an MCP-shaped schema: nullable object, string and array; several types to anyOf; a property named default; enum untouched.
  • An existing anyOf is kept.
  • The caller's schema is not modified.
  • Claude keeps type lists.
• test_simulation_model.py:
  • Default and explicit max_tokens.
  • The cap: agent 128000 gives 16384, 8192 gives 8192, none or 0 gives 16384.
  • Reading max_tokens and maxTokens from the schema metadata.
  • The debug simulation context.
  • max_tokens in the request body of an end-to-end tool simulation.

uv run pytest (2715 passed), ruff check, ruff format --check and mypy src tests are clean. Version bumped to 2.15.1.

Notes

• Checked against the gateway before this change, not with this code: Gemini accepted schemas written with nullable: true, and the 16384 replay above returned the expected response. This PR has unit tests only. The anyOf case for several types has not been checked against Gemini.
• Simulation cache: cache keys include the completion kwargs, so simulation responses cached locally miss once and are generated again.
• Existing test error: test_tool_simulation_sends_env_model errors at teardown when run alone, on an unregistered GET …/llm/api/discovery request. It does the same on main and passes in a full run.
• Release: low-code eval runtimes need their uipath pin bumped to pick this up.

[Copilot]

🟡 Changes recommended

Schemas combining a multi-type list with an existing anyOf still produce requests Gemini rejects, and the new test incorrectly asserts that incompatible payload.

1 open issue

• Normalize type lists when schemas already contain anyOf · New

What changed in this PR

This PR improves tool simulation and input generation in the UiPath evaluation framework by adapting schemas for Gemini and increasing simulation output-token limits.

Changes:

• Normalize JSON Schema type lists for Gemini while leaving other providers unchanged.
• Default simulation calls to 16,384 tokens, cap inherited agent limits, and preserve explicit model settings across evaluation and CLI paths.
• Add regression tests and bump uipath to 2.15.1.

File	Description
packages/​uipath/​uv.lock	Synchronizes the package version to 2.15.1.
packages/​uipath/​tests/​cli/​eval/​mocks/​test_structured_output.py	Tests Gemini schema conversion and provider isolation.
packages/​uipath/​tests/​cli/​eval/​mocks/​test_simulation_model.py	Tests token defaults, caps, metadata extraction, and request payloads.
packages/​uipath/​src/​uipath/​eval/​runtime/​runtime.py	Applies agent-derived limits to tool simulation and input generation.
packages/​uipath/​src/​uipath/​eval/​mocks/​_structured_output.py	Converts type lists before Gemini forced-tool calls.
packages/​uipath/​src/​uipath/​eval/​mocks/​_simulation_model.py	Defines token defaults and agent-settings helpers.
packages/​uipath/​src/​uipath/​eval/​mocks/​_mock_runtime.py	Threads agent token limits through simulation configuration.
packages/​uipath/​src/​uipath/​_cli/​cli_run.py	Passes agent token limits into run simulations.
packages/​uipath/​src/​uipath/​_cli/​cli_debug.py	Passes agent token limits into debug simulations.
packages/​uipath/​pyproject.toml	Bumps the package version to 2.15.1.

_{Review effort: Balanced}

[dianagrecu-uipath]

Fixed in ad96246. When a node has both, the listed types now apply to the anyOf branches: a branch without a type is copied once per type, and a typed branch is kept only when the list allows its type. No type list is left. The test now asserts that shape, and a new test checks that no type list reaches the Gemini tool schema. Checked against Gemini: the old shape returns the same 400, and the new one returns the tool call.

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
97.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud