Fit requests to a token window and summarize older turns

jacensekai commented

2026-10-13 19:09:13 -04:00

Owner

History was trimmed to a character budget that ignored the system
prompt and tool definitions, and old turns were simply dropped, so long
conversations lost context without notice and could still overflow the
model's window.

Requests are now fitted into context.windowTokens (system prompt, tool
definitions, history and room for the reply). Sizes are estimated in
characters with a chars-per-token ratio learned per model from the
usage the provider reports; streamChat() asks for usage and retries
without stream_options on endpoints that reject it.

Once a request passes compactAt of the window, everything but the most
recent keepRecent share is summarized by the model into a stored
summary message, cut at a user message and folding in the previous
summary. The summary goes into the system prompt in place of those
messages. If it still doesn't fit, older long tool results are replaced
by a note before whole turns are dropped. A failed summary falls back to
dropping turns and isn't retried within the run. Summaries can also be
made on request (POST /api/conversations/:id/compact), as a run.

The chat header shows a context meter with a breakdown and "Summarize
now", the timeline marks where a summary starts and dims what it
replaced, and Settings → Model gets the window and compaction settings.
A saved llm.contextChars is migrated to a token window unless it was
the old default.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

History was trimmed to a character budget that ignored the system prompt and tool definitions, and old turns were simply dropped, so long conversations lost context without notice and could still overflow the model's window. Requests are now fitted into context.windowTokens (system prompt, tool definitions, history and room for the reply). Sizes are estimated in characters with a chars-per-token ratio learned per model from the usage the provider reports; streamChat() asks for usage and retries without stream_options on endpoints that reject it. Once a request passes compactAt of the window, everything but the most recent keepRecent share is summarized by the model into a stored `summary` message, cut at a user message and folding in the previous summary. The summary goes into the system prompt in place of those messages. If it still doesn't fit, older long tool results are replaced by a note before whole turns are dropped. A failed summary falls back to dropping turns and isn't retried within the run. Summaries can also be made on request (POST /api/conversations/:id/compact), as a run. The chat header shows a context meter with a breakdown and "Summarize now", the timeline marks where a summary starts and dims what it replaced, and Settings → Model gets the window and compaction settings. A saved llm.contextChars is migrated to a token window unless it was the old default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jacensekai added 1 commit

2026-10-13 19:09:13 -04:00

Fit requests to a token window and summarize older turns

CI / Typecheck, build and test (pull_request) Successful in 16s

Details

CI / End-to-end (Playwright) (pull_request) Successful in 55s

Details

CI / Docker images and sandbox (pull_request) Failing after 11s

Details

CI / Compose stack smoke test (pull_request) Failing after 6s

Details

98ea273c14

History was trimmed to a character budget that ignored the system
prompt and tool definitions, and old turns were simply dropped, so long
conversations lost context without notice and could still overflow the
model's window.

Requests are now fitted into context.windowTokens (system prompt, tool
definitions, history and room for the reply). Sizes are estimated in
characters with a chars-per-token ratio learned per model from the
usage the provider reports; streamChat() asks for usage and retries
without stream_options on endpoints that reject it.

Once a request passes compactAt of the window, everything but the most
recent keepRecent share is summarized by the model into a stored
`summary` message, cut at a user message and folding in the previous
summary. The summary goes into the system prompt in place of those
messages. If it still doesn't fit, older long tool results are replaced
by a note before whole turns are dropped. A failed summary falls back to
dropping turns and isn't retried within the run. Summaries can also be
made on request (POST /api/conversations/:id/compact), as a run.

The chat header shows a context meter with a breakdown and "Summarize
now", the timeline marks where a summary starts and dims what it
replaced, and Settings → Model gets the window and compaction settings.
A saved llm.contextChars is migrated to a token window unless it was
the old default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jacensekai merged commit 007da54310 into master

2026-10-13 19:09:18 -04:00

jacensekai referenced this pull request from a commit

2026-10-13 19:09:20 -04:00

Merge pull request 'Fit requests to a token window and summarize older turns' (#5) from context-management into master

Rows
Columns

Fit requests to a token window and summarize older turns #5