feat(google): report Gemini Live reasoning token usage - #7326
rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| # Gemini includes thinking tokens in response_token_count. Keep None when | ||
| # omitted so a reported zero remains distinguishable from a missing count. | ||
| reasoning_tokens=usage_metadata.thoughts_token_count, |
There was a problem hiding this comment.
🟡 Gemini output totals omit reasoning
When Gemini reports thoughts, reasoning_tokens captures them while output_tokens excludes them. Usage summaries therefore undercount output tokens.
Learn more
Gemini Live maps the provider's candidatesTokenCount into response_token_count. The provider defines total_token_count as prompt, response candidates, tool-use prompt, and thoughts combined, so thoughts are separate from response_token_count. The shared RealtimeModelMetrics and LLMModelUsage contracts instead define reasoning as a subset of output tokens. Recording the separate count without folding it into output_tokens violates that contract and makes aggregate output usage too small.
Example: With 397 prompt tokens, 49 response tokens, and 26 thought tokens, the emitted output count is 49. The contract requires 75 output tokens, including 26 reasoning tokens, while the provider total also includes those thoughts.
Recommended fix: Set output_tokens to (usage_metadata.response_token_count or 0) + (usage_metadata.thoughts_token_count or 0), and calculate tokens_per_second from the same combined value. Update the test fixture's total_token_count to reflect the provider's separate thought count.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if metrics.input_token_details.cached_tokens_details | ||
| else 0, | ||
| "output_tokens": metrics.output_tokens, | ||
| "reasoning_tokens": metrics.reasoning_tokens or 0, |
There was a problem hiding this comment.
🟡 Logs erase missing reasoning counts
When a provider omits reasoning usage, log_metrics emits reasoning_tokens=0. Consumers cannot distinguish missing data from a reported zero.
Learn more
RealtimeModelMetrics.reasoning_tokens uses None specifically to represent an unreported value. The structured log converts that value to zero, unlike the telemetry path, which omits reasoning attributes when it is None. Any log-based usage pipeline therefore records a provider-reported zero even when no reasoning count was supplied.
Example: A realtime provider emits reasoning_tokens=None. The metric object and trace preserve that absence, but the metrics log contains "reasoning_tokens": 0.
Recommended fix: Preserve None in the log field or omit reasoning_tokens from extra when it is absent. Adjust the local metadata value type if needed.
Was this helpful? React with 👍 or 👎 to provide feedback.
Ports livekit/agents-js#2517.
Summary
thoughts_token_countwhile preserving reported zero versus omitted valuesSource diff coverage
.changeset/gemini-live-reasoning-tokens.md: not applicable. The Python repository does not use Changesets and has no counterpart release-note mechanism.agents/etc/agents.api.md: not applicable. Generated TypeScript API output has no Python counterpart; the corresponding Python public models are covered below.agents/src/inference/llm.ts: ported already inlivekit-agents/livekit/agents/inference/llm.py; gatewaycompletion_tokens_details.reasoning_tokensis already mapped.agents/src/llm/llm.ts: ported already inlivekit-agents/livekit/agents/llm/llm.py;CompletionUsage.reasoning_tokensand propagation intoLLMMetricsalready exist.agents/src/metrics/base.ts: adapted inlivekit-agents/livekit/agents/metrics/base.py;LLMMetrics.reasoning_tokensalready existed, and this PR adds optionalRealtimeModelMetrics.reasoning_tokenswith Python naming andNonesemantics.agents/src/metrics/model_usage.test.ts: adapted intests/test_metrics_usage.py; ports the realtime reasoning aggregation assertion using Python metric models.agents/src/metrics/model_usage.ts: adapted inlivekit-agents/livekit/agents/metrics/usage.py; LLM reasoning aggregation already existed, and this PR adds realtime reasoning aggregation.agents/src/metrics/utils.ts: adapted inlivekit-agents/livekit/agents/metrics/utils.py; realtime structured logging now includesreasoning_tokens.agents/src/telemetry/gen_ai.ts: adapted inlivekit-agents/livekit/agents/telemetry/utils.py, where Python builds realtime GenAI attributes; emits both standard and Langfuse-compatible reasoning attributes.plugins/google/src/realtime/realtime_api.test.ts: adapted intests/test_plugin_google_realtime.py; ports both source cases for mapping and zero-versus-omitted behavior.plugins/google/src/realtime/realtime_api.ts: adapted inlivekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.py; mapsthoughts_token_countdirectly without changing output totals.plugins/openai/src/realtime/gpt_live_model.ts: not applicable. The source removes a TypeScript intersection-type workaround; PythonLLMMetricsalready accepts the existingreasoning_tokensassignment directly.Validation
uv run pytest tests/test_metrics_usage.py: 10 passeduv run pytest tests/test_plugin_google_realtime.py: 28 passedenv -u LIVEKIT_API_KEY -u LIVEKIT_API_SECRET -u LIVEKIT_URL uv run pytest --unit --audio_eot --ignore=tests/test_room.py: 3493 passed, 3 skippedmake check: passed formatting, lint, and strict type checkinguv build --package livekit-agents: passeduv build --package livekit-plugins-google: passeduv run pytest --plugin google: all 76 assertions passed, but teardown reports two pending TTS tasks in the pre-existingtest_gemini_tts_success; the same error reproduces when that untouched test runs aloneThe unfiltered
make unit-testsadditionally requires Docker fortests/test_room.py; Docker is unavailable in this environment. Ambient invalid LiveKit credentials also caused unrelated cloud tests to run and return 401, so the successful hermetic rerun explicitly removed them.Ported from livekit/agents-js#2517
Original PR description
No description.