When a conversation gets long

What happens when a conversation reaches its limits, and what carries over.

Every conversation has two limits

A model can only read so much at once. Everything you have said, everything it has said back, and every file and page and command result it looked at along the way are re-sent on every single turn, so a long conversation is not just approaching a limit. It is getting more expensive and slower with every exchange, all the way there.

Both limits are shares of the model’s context window rather than fixed numbers, so they follow the model you are using and are stated here as ratios, not token figures: the figures depend on the model.

• Soft (compaction.soft_ratio in the model configuration). The assistant is told to finish the step it is on and bring its handoff up to date. Nothing is cleared.
• Hard (compaction.hard_ratio). The conversation is written down and started fresh.

Soft must stay below hard. You can move both in Settings under Chat.

Both are counted on the conversation itself, after setting aside the fixed prompt and the tool definitions sent with every turn, since those two do not shrink when the conversation is cleared and counting them against a limit would either never trigger a boundary on a small-window model or trigger one on every turn.

They are checked before every request the assistant sends, not after the turn: the next request is estimated with a correction learned from what the provider actually billed, plus a margin for how wrong that estimate has been. A request that, together with the longest answer the model may give, would not fit the window at all counts as past hard, whatever the share says.

What happens when it is reached

The assistant does not start forgetting the middle, and it does not quietly compress what you said into something shorter. At hard it stops, writes down where the work stood, clears the conversation, and carries on from what it wrote.

That is one act, and it is the same act every time:

1. Write the record. The assistant itself, before the conversation is touched, writes down what the work was, what is done, what is left, what happens next, and what it decided. The runtime adds what it saw happen: the files read and changed, the calls that failed, the task list, anything still running. Both halves are kept up to date after every turn in one file per conversation, workstreams/<conversation>/handoff.md in your home folder, laid out the way config/handoff.md says. Edit that template and the next record follows it.
2. Reflect. A separate model turn reads the conversation and writes what it taught into memory as ordinary notes you can open. This is independent of the record above: it saves what was learned, not what the work is.
3. Archive. The whole transcript is copied and stays searchable.
4. Clear. The conversation empties. It keeps its name and its place in your list, so nothing moves under you.

Then one of two things happens, and it depends on whether there is work left.

Continue, or stop

If the work goes on, the cleared conversation opens with the record, written as a message you can read:

This conversation was consolidated and cleared, and the work carries on here.
What was said is archived and searchable with `recall_history`; what a past
boundary wrote down, including this one, is listed and read back with
`handoff_history`; what it taught is in memory. This is where the work stood.

Current through event N of this chat's record. The agent last wrote its judgement at event M.

# Handoff

## Objective
ship the export feature

## Constraints
- keep the receipt email out of the partial path until the operator decides

## Done
- the endpoint and its schema
- file_write src/export/handler.py (event K)

## Steps
- done: the failure path for a missing account: returns not found with the account id
- in progress: the test for a partial export
- pending: the changelog line

## Remaining
- the partial-export test, then the changelog line

## Blocked
- should a partial export still send the receipt email? The operator decides.

## Next action
write the partial-export test

If the work is finished, you are told the conversation ended, the record is left at the top of it, and nothing starts until you speak.

Either way you can read exactly what it thinks it was doing, and correct it if it is wrong. That is the point of writing it down rather than summarising: a summary is prose about a conversation, and this is a statement of where the work stands, in the assistant’s own words, which you can argue with.

Who decides

Two things can start it, and they are recorded differently so you can tell them apart afterwards.

• The hard limit. The conversation got too big. The runtime decides this and nothing else.
• The assistant. It finished a phase and judged that a clean start would serve the work better than carrying everything forward. It can reach this at any size.

Whether the work continues or stops is always the assistant’s answer, never the runtime’s. The one thing the runtime does refuse is a continue that nothing is owed for: if the record just written for this boundary says nothing is left to do and names no next step, there is nothing for the next conversation to pick up, so it stops instead and writes down why.

When the conversation reaches its hard limit and the assistant has not said where the work stood, the turn ends and the runtime asks once, with a request the model can only answer that way. If it answers, its answer stands. If it does not, the conversation starts fresh from the handoff the runtime kept along the way: what the assistant wrote there with handoff_update, and what the record saw it do. The work carries on from that handoff when it names something still to do, and waits for you when it does not. It is asked once per conversation, never twice.

And it can decline to do any of it. Carrying on is a real answer, not a failure, and it is what happens on most turns.

The assistant reads one line on every turn saying how full the conversation is and where both limits are:

Context: <used> of <window> tokens. Soft <soft>: reached. Hard <hard>: not reached.
Finish the step you are on, then bring the handoff up to date with `handoff_update`.

The second sentence appears only once soft is reached, and it is the only way soft reaches the assistant. The reading is taken as the turn starts, with your message already counted. So it can choose to stop at a clean point on its own, rather than always being the one interrupted at hard.

A turn is never cut in half

If hard would be crossed while the assistant is mid-answer, in the middle of a run of tool calls, nothing happens to the conversation at that moment. Clearing it there would take away the question it is answering and the results it has already gathered.

Instead the next request is not sent. The turn ends after the last tool result it already has, and the boundary runs then. One crossing, one boundary.

If your own message is what would carry the conversation past hard, the conversation is written down and started fresh first, and your message then goes into the fresh conversation exactly as you typed it.

There is a separate, harder limit underneath this: every request has to fit in what the model will accept at all, and a single turn with a lot of tool traffic can hit that before the turn is done. When it does, the request is not rejected. The runtime first clears the results of older tool calls from that same turn, oldest first, and leaves the most recent ones alone since those are the ones the assistant is still reasoning about; a cleared result can always be fetched again by calling the tool a second time. Only if that is still not enough does it start trimming the conversation itself to make the request fit. This is a last resort for one oversized turn, not the thing that bounds a conversation over time. That is the hard limit above, and it is recorded when it fires.

When it cannot happen

A boundary needs somewhere to put the transcript before it clears one. If that write fails, or the app cannot reach the conversation the turn ran in, the boundary does not happen and the conversation is left exactly as it was. Nothing is rewritten and nothing is lost. You get an error in the app saying so, and the next turn tries again.

Nothing else is tried in its place. There used to be a fallback that summarised the middle of the conversation instead, and the one branch it ran on was the branch where the copy had just failed to be written.

What this buys you

It costs less. A conversation at its limit re-sends everything on every turn. After a boundary it sends a page. On a long piece of work that is the difference between the cost climbing all afternoon and it staying flat.

It stays sharp. The alternative most tools use is to summarise the middle and keep going, then summarise the summary, and again. Detail drains out a little at a time and nobody can see it happening. Here the record is written once, by an assistant that still has the whole conversation in front of it, and it is never rewritten.

It survives a restart. The record is on disk. Close the app, come back tomorrow, and the work picks up from the same place.

Nothing is thrown away. The transcript is archived and searchable, so anything the record left out can still be found by asking for it.

What you can change

Both limits, in Settings under Chat. Lower hard and the assistant consolidates more often, holding less in front of it at any moment and spending less per turn. Raise it and it carries more at once and pays more for each exchange. Soft decides how early before that the assistant starts wrapping up its step and writing things down.

There is one more number, in the model configuration rather than Settings: how much of the record the next conversation’s opening message may spend, in tokens (boundary.continuity_budget_tokens). That is a ceiling on the message itself, not on the work behind it. How much survives a boundary is otherwise not a number any more: what carries over is the record, and the record is as long as the work needs it to be until it presses on that ceiling, at which point a list says how many items it had to drop rather than quietly leaving them out.

Parallel chats

Each Mirror chat has its own history, running turn and pending messages. Switching chats changes the view while other chats continue working and answering their queues. Returning to a running chat preserves its live reply, pending approvals and messages. Provider concurrency and rate limits still apply across the runtime.

Queued messages and redirects waiting for a tool boundary offer Edit and Delete directly on their bubbles. Once the assistant has picked up a message, an attempted deletion reports that it is already in use.

A recovery answer about an interrupted call resumes the conversation that owned the call, even when the answer arrives through Telegram. If its Mirror conversation is unavailable, the answer remains pending until that conversation reconnects. Repeating a call uses the usual permission checks.

The Monitor keeps all unresolved breakers visible. Closed history is grouped by date and limited by runtime.yaml::breaker_history_days and breaker_history_max. Older entries leave the display; canonical breaker records remain stored.

Gemini’s tokens per minute

Each model call sends the current context, including earlier tool results. For example, a context of a quarter of a million tokens uses about that much input for each tool round trip. A two-million-token minute quota allows roughly eight such calls before other traffic is counted. A context window and a minute quota measure different things: a conversation can fit the model’s window and still need pauses between calls.

The Google adapter learns input-token quotas from Google’s structured rate-limit responses. No TPM value is configured. It reserves input across roles and chats using the same API key and model in this backend, includes cached input in full, and leaves ten percent headroom for estimates. When the window is busy it shows the reported quota, local usage and wait, then continues the same request. Settings → Chains shows the last quota Google reported once one is known.

Quota observations expire periodically, allowing a project upgrade to be discovered without editing settings or restarting. A new throttle replaces the observation. Google’s retry delay governs recovery from external traffic too; other API keys and other processes are outside this backend’s local accounting.

You do not need to clear a chat to recover from a minute limit. Keeping a large context reduces calls per minute; the existing handoff boundary can carry the work into a smaller context when needed. Authentication, billing, daily and zero-quota refusals still fail promptly. Recovery has a bounded wait and attempt count, Stop cancels it, and partial replies are never replayed.

Where this is going

The right limits are measured, not argued. A quarter and a fifth are reasonable starting points and nothing more. The same task run at several settings, priced from the real ledger, is what should decide it, and until that has been run the number is an opinion.