Fix sampled planner statistics; add autoOptimize and Optimize()

[arotolo]

Fixes the sampled planner statistics that made indexed lookups on a large type read every row of the type (5.0.1–5.3.1), adds autoOptimize / Optimize() / OptimizeAsync() for deferring one-time upgrade maintenance, and keeps the planner able to rank a type's partial indexes against each other.

The bug

PRAGMA optimize on connect/disconnect (since 5.0.1) and the ANALYZE after CreateIndex (since 5.1.1) set analysis_limit = 400. The general (FullTypeName, Partition) index is ordered by type name and the types that sort first are tiny, so its sample credited every type with a handful of rows (81 on a 1.68M-row production store whose largest type held 586,986), while a partial index's sample sat inside one key and credited every key with hundreds (401). The planner preferred the general index and read the whole type on each indexed lookup: 82–366 ms per lookup on that store, under a millisecond with correct statistics. Results were never wrong; only speed.

The design

The branch's first design kept full statistics with PRAGMA optimize, and its second removed statistics entirely. Review found that the second one regresses two indexed filters ANDed — with no sqlite_stat1 the planner takes whichever partial index was created last — and that both suffer from the general index's own statistics row. What ships:

• Full statistics at index creation. CreateIndex runs PRAGMA analysis_limit = 0; ANALYZE JsonValue; after it builds an index, and again when it re-declares an index that was built on an empty type and has rows now. Rows recorded for an empty type are not kept. Nothing samples; PRAGMA optimize is gone from connect, disconnect and dispose.
• The general index's row is pinned to N N/2 N/2. ANALYZE writes it as the average rows per type, which is wrong for every type when one type dominates a store: it sends small types to a table scan when few types existed at ANALYZE time, and sends a big type's indexed lookups to the general index otherwise. Half the table keeps the general index cheaper than a scan and dearer than any partial index whose match is under half the rows, so the partial indexes' accurate rows decide between themselves. SQLite documents writing sqlite_stat1 for this ("Manual Control Of Query Plans", lang_analyze.html).
• Upgraded stores are repaired once, gated on PRAGMA user_version. With autoOptimize on (default) the first connect replaces sampled rows with full ones — or discards them when the store has no expression index — and stamps the store; a stamped store pays one header read at connect and writes nothing (schema_version is unchanged across reconnects; a test pins it). With autoOptimize: false the store keeps its previous plans until Optimize() / OptimizeAsync(), which also drop the four redundant 4.x indexes.

Full detail, including the plan table for every query shape on both engines and why the other two designs were rejected: docs/planner-statistics-sampling.md.

Benchmark

dotnet run --project TychoDB.Benchmarks -c Release -- --filter '*PlannerStatistics*' (new class; 40 small types written first, one type with 40,000 rows over 200 keys, indexed on a unique Seq and a 200-value GroupId; built, closed and reopened across three launches like an app). Apple M4 Max, .NET 9.0.17, SQLite 3.50.3, medians of 16 × 1 launch.

Shape	main (v5.3.1, sampled)	322be81 (no statistics)	This PR
Indexed equality, 200 rows of 40,000	10,108 µs	570 µs	551 µs
Two indexed equalities ANDed, selective index created first	22.7 µs	102.6 µs	22.8 µs
Two indexed equalities ANDed, selective index created last	23.6 µs	22.0 µs	24.5 µs
Whole read of a 10-row type	28.6 µs	31.5 µs	26.1 µs

The first row is the bug (18x). The second is the no-statistics regression this PR avoids; it grows with the rows matching the less selective property (4.6x here at 200 per key, 200x at 10,000 per key, and larger again on SQLCipher, where every extra page is decrypted).

Public API

• Tycho(..., bool autoOptimize = true) — source-compatible; assemblies that construct Tycho need recompiling.
• void Optimize(), ValueTask<bool> OptimizeAsync(CancellationToken = default).

Minor release: 5.4.0. Directory.build.props updated to match (the tag still sets the package version).

Tests

379 passed, 0 failed, 4 skipped in both Debug (SQLite 3.50.3) and Encrypted (SQLCipher, SQLite 3.39.2). PlannerStatisticsTests asserts the plan for every shape TychoDB emits (equality, non-selective equality, range, sort with limit, two indexed equalities in both creation orders, unindexed filter, whole type large and tiny, count), the two 5.3.1 failure scenarios, repair with autoOptimize on and off, a store without indexes, a fresh store, and that a current store is not written at connect.

🤖 Generated with Claude Code