Skip to content

Move the Python client to the result format API and defer row-returning statements - #650

Closed
evertlammerts wants to merge 7 commits into
duckdb:v2.0-cyanopterafrom
evertlammerts:new_results
Closed

evertlammerts wants to merge 7 commits into
duckdb:v2.0-cyanopterafrom
evertlammerts:new_results

Conversation

@evertlammerts

Copy link
Copy Markdown
Member

Builds on duckdb/duckdb#25801 and duckdb/duckdb#26314.

With this PR the client submits statements through QueryParameters::format. Arrow consumers are backed by a QueryResultStream<ArrowFormat>, so conversion runs in parallel on worker threads and arrays leave the engine as Arrow. The batch size is a maximum (arrays also end at row-group and producer boundaries); the default is 131,072, matching the C API.

execute() now describes a statement (BindStatement) instead of running it. A query or DML with RETURNING runs when its result is first read, and the first consumer fixes the shape: rows or Arrow, chosen once. Running another statement withdraws an unfinished result. Statements whose effects a rollback cannot undo (DDL, CALL, PRAGMA, COPY, and so on) still complete inside execute(). New complete() and abort() make the lifecycle explicit, and a RETURNING write is all-or-nothing with its rows: abandoned unread, it never happened, and it fails an open transaction. rowcount now reports changed rows, with executemany totalling its sets.

Existing bugs are fixed along the way:

  • Arrow after a row fetch silently dropped the partly read chunk. It now raises.
  • A reader scanned by a later query on its own connection hung, or returned a wrong count. It now reports the cancellation.
  • A failed bind in execute() ignored the transaction invalidation policy.
  • Lazy .pl() leaked an ArrowSchema per call.
  • A deferred query over a local DataFrame could resolve a different object when a rebind ran in the consumer's frame; resolutions are now pinned at bind time.

@evertlammerts evertlammerts added the submodule-bump Intentional DuckDB submodule pin move label Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

submodule-bump Intentional DuckDB submodule pin move

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant