Repository navigation
fix(llm): keep earlier summaries when summarizing chat context again - #2701
rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: a182a68 The changes in this PR will be included in the next version bump. This PR includes changesets to release 41 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| } | ||
|
|
||
| if (it.type === 'function_call' || it.type === 'function_call_output') { | ||
| if (it.extra?.is_summary !== true) continue; |
There was a problem hiding this comment.
🟡 Repeated summaries exhaust the model context
After repeated task groups, _summarize retains every earlier summary and adds another. It excludes summaries from toSummarize, so later passes cannot compact them and the model eventually exceeds its context limit.
Learn more
Each call to _summarize replaces ordinary older messages with a new summary. The changed filter keeps all old summaries, while toSummarize excludes them from the next summarization request. TaskGroup calls this with keepLastTurns: 0 at completion, so repeated task groups add one summary each without ever compacting earlier ones. Those messages remain in the model input and can eventually exceed its finite context window.
Example: A session completes many task groups, each yielding a 200-token summary. Even when all raw messages have been compressed, 1,000 summaries alone occupy roughly 200,000 tokens instead of a bounded summary.
Recommended fix: Add a compaction strategy for older summaries, including their content in the new summary source before removing them, while retaining enough summary information to avoid losing earlier facts. Add a repeated-summarization test that checks both preservation of facts and bounded growth.
Was this helpful? React with 👍 or 👎 to provide feedback.
Ports livekit/agents#7721 so repeated chat-context summarization preserves earlier summary messages instead of dropping them while rebuilding the context.
Adds the source regression scenario and a patch changeset for
@livekit/agents.Source diff coverage
livekit-agents/livekit/agents/llm/chat_context.py: adapted toagents/src/llm/chat_context.ts. The PythonChatMessage.extra["is_summary"]preservation check maps to the TypeScriptitem.extra?.is_summaryrepresentation.tests/test_chat_ctx.py: adapted toagents/src/llm/chat_context.test.ts. The same prior-summary/resummarization scenario and assertions use Vitest, explicit timestamps for deterministic target ordering, and the targetFakeLLMexact-input fixture.No source files were omitted, and no additional infrastructure was required.
Testing
pnpm test agents/src/llm/chat_context.test.ts(94 passed)pnpm build(42 packages built successfully)pnpm test agents(3436 passed, 5 skipped, 6 failed in unrelated telemetry/STT tests; standalone rerun reduced this to 3 host-dependentloop_monitor.test.tsfailures because this environment exposes process-level rather than thread-level CPU accounting)pnpm lint(baseline failure across pre-existing files; no new lint finding in the added test or changed summarization block)pnpm format:check(baseline failure across 45 pre-existing files; touched additions were formatted directly)Source: livekit/agents#7721
Ported from livekit/agents#7721
Original PR description
Problem
ChatContext._summarizeleaves an earlier summary out of the text it sends to the LLM, to avoid summaries of summaries. When it rebuilds the context, it also drops that earlier summary, because the summary is an assistant message and the rebuild loop discards every user and assistant message from the head. The comment above the loop says prior summaries are kept.As a result, the second summarization of a context loses the first summary, and whatever it held, without ever passing it to the LLM.
TaskGroupwithsummarize_chat_ctx=Truecalls_summarize, so a context that already holds a summary loses it there.Change
Keep head messages marked
is_summarywhen rebuilding the context. The new summary is added after them, before the tail.Testing
test_summarize_keeps_prior_summaryintests/test_chat_ctx.pysummarizes a context that already holds a summary and expects both the earlier and the new summary in the result.chat_context.py:uv run pytest --unit -q tests/test_chat_ctx.pygives 1 failed, 82 passed (assert 1 == 2).uv run pytest --unit -q tests/ -k "summar or task_group"gives 9 passed, 1 skipped.ruff checkandruff format --checkpass on the changed files.Replaces #7522, which was closed before review.