Skip to content

Accumulate the conversation across multi-step tool-calling steps - #46

Closed
cwurm wants to merge 3 commits into
ClickHouse:mainfrom
cwurm:accumulate-multi-step-conversation
Closed

Accumulate the conversation across multi-step tool-calling steps#46
cwurm wants to merge 3 commits into
ClickHouse:mainfrom
cwurm:accumulate-multi-step-conversation

Conversation

@cwurm

@cwurm cwurm commented Aug 8, 2026

Copy link
Copy Markdown
Member

create_next_step_options was called with initial_options as the base, so every step's request contained only the original prompt plus the latest step's tool calls and results — the tool exchanges of all earlier steps were dropped.

The model could therefore never see more than one tool result at a time: asked for something that requires several tool calls, it kept re-requesting the results it had just lost until max_steps was exhausted, without producing an answer.

Observed in clickhouse-client AI SQL generation, where a question needing two table schemas made the model alternate between the same two get_schema_for_table calls until the step limit:

:) ?? query all nodes for their partitions with most active parts (only show one row per node, ...)
•  Starting AI SQL generation with schema discovery...
─────────────────────────────────────────────────
🔧 Calling: list_databases [toolu_01...]
✓  list_databases completed
🔧 Calling: list_tables_in_database [toolu_01...]
   └─ Args: {"database":"system"}
✓  list_tables_in_database completed
🔧 Calling: get_schema_for_table [toolu_01...]
   └─ Args: {"database":"system","table":"parts"}
✓  get_schema_for_table completed
🔧 Calling: get_schema_for_table [toolu_01...]
   └─ Args: {"database":"system","table":"clusters"}
✓  get_schema_for_table completed
🔧 Calling: get_schema_for_table [toolu_01...]
   └─ Args: {"database":"system","table":"parts"}
✓  get_schema_for_table completed
🔧 Calling: get_schema_for_table [toolu_01...]
   └─ Args: {"database":"system","table":"clusters"}
✓  get_schema_for_table completed
🔧 Calling: get_schema_for_table [toolu_01...]
   └─ Args: {"database":"system","table":"parts"}
✓  get_schema_for_table completed
🔧 Calling: get_schema_for_table [toolu_01...]
   └─ Args: {"database":"system","table":"clusters"}
✓  get_schema_for_table completed
─────────────────────────────────────────────────
•  ⚠️  No SQL query was generated: the schema exploration step limit was reached (ai.max_steps = 10)

Base the next step on current_options, which carries the accumulated conversation, so each step sees all previous tool calls and results.

Added a regression test that drives MultiStepCoordinator::execute_multi_step with a scripted generate function (no network) and asserts that the third step's request still contains the first step's tool exchange.

alexey-milovidov and others added 3 commits March 14, 2026 01:16
create_next_step_options was called with initial_options as the base,
so every step's request contained only the original prompt plus the
latest step's tool calls and results - the tool exchanges of all
earlier steps were dropped. The model could never see two tool results
at once: asked for information that requires several tool calls, it
kept re-requesting the results it had just lost (e.g. alternating
between the same two schema lookups) until max_steps was exhausted,
without ever producing an answer.

Base the next step on current_options, which carries the accumulated
conversation, so each step sees all previous tool calls and results.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The test drives MultiStepCoordinator::execute_multi_step with a scripted
generate function (no network) and asserts that the third step's request
still contains the tool exchange of the first step.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cwurm

cwurm commented Aug 8, 2026

Copy link
Copy Markdown
Member Author

Closing: this came out of working through pushing some older local changes upstream - the submodule pointer in ClickHouse/ClickHouse is still at ab06ef7, from before this was fixed on main in #40, so the bug was reproduced and fixed independently against the stale base. The remaining useful piece is bumping the submodule pointer in ClickHouse/ClickHouse to current main, which will happen separately.

@cwurm cwurm closed this Aug 8, 2026
@cwurm

cwurm commented Aug 8, 2026

Copy link
Copy Markdown
Member Author

@cwurm
cwurm deleted the accumulate-multi-step-conversation branch August 8, 2026 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants