Conversation
Benchmarks rot faster than tests, because nothing runs them. A previous attempt (PR #2203) shipped a bench that inlined a private function; the function was later deleted and the bench would not compile today, which nobody noticed because no job ever built it. Adds a blocking `bench-compile` job running `cargo bench --workspace --benches --no-run` — compiles every bench target, executes none. Timing on shared runners is too noisy to gate on; compilability is not. Also adds docs/benchmarking.md, the conventions every later bench in this series follows, and the bench-profile override that makes `cargo bench --workspace` build at all: feature unification leaks calimero-storage's dev-only `testing` feature into bench builds, and that crate's release-safety guard refuses to compile without debug assertions. The override is scoped to that one package; opt-level is untouched.
|
This pull request has been automatically marked as stale. If this pull request is still relevant, please leave any comment (for example, "bump"), and we'll keep it open. We are sorry that we haven't been able to prioritize reviewing it yet. Your contribution is very much appreciated. |
…tity id `row_counts_are_deterministic_across_calls` built its workload with `UnorderedMap::new()`, which mints a RANDOM entity id. A parent's children live in a hash trie (core#3633) whose descent depth follows the hash of that id, so two runs of an identical workload read a different number of trie nodes while writing exactly the same set. That is why only `rows_read` moved while `rows_written` sat at 313, and why it failed under `calimero-storage/testing` — which CI enables — roughly ALWAYS rather than occasionally: 0 pass / 12 fail locally before, 15/15 after. The sibling `byte_counts_are_not_reproducible` already records that ids are not deterministic; this test simply must not depend on them being so. `new_with_field_name` gives the collection a deterministic id, which is what the property under test actually needs. Measured: 15/15 under `--features calimero-storage/testing`, 10/10 under default features, and `./scripts/check-storage-cost.sh` still green on all 124 rows — the change touches only the test's own workload, never a measured row. This is the single failure shared by #3750, #3752 and #3754. It is not stale-base: the tool is byte-identical between `bench/05-gas-walls` and this branch, and it reproduces here. Claude-Session: https://claude.ai/code/session_014mxKpC4vXk9Bujuetq7KBk
…tity id `row_counts_are_deterministic_across_calls` built its workload with `UnorderedMap::new()`, which mints a RANDOM entity id. A parent's children live in a hash trie (core#3633) whose descent depth follows the hash of that id, so two runs of an identical workload read a different number of trie nodes while writing exactly the same set. That is why only `rows_read` moved while `rows_written` sat at 313, and why it failed under `calimero-storage/testing` — which CI enables — roughly ALWAYS rather than occasionally: 0 pass / 12 fail locally before, 15/15 after. The sibling `byte_counts_are_not_reproducible` already records that ids are not deterministic; this test simply must not depend on them being so. `new_with_field_name` gives the collection a deterministic id, which is what the property under test actually needs. Measured: 15/15 under `--features calimero-storage/testing`, 10/10 under default features, and `./scripts/check-storage-cost.sh` still green on all 124 rows — the change touches only the test's own workload, never a measured row. This is the single failure shared by #3750, #3752 and #3754. It is not stale-base: the tool is byte-identical between `bench/05-gas-walls` and this branch, and it reproduces here. Claude-Session: https://claude.ai/code/session_014mxKpC4vXk9Bujuetq7KBk
…tity id `row_counts_are_deterministic_across_calls` built its workload with `UnorderedMap::new()`, which mints a RANDOM entity id. A parent's children live in a hash trie (core#3633) whose descent depth follows the hash of that id, so two runs of an identical workload read a different number of trie nodes while writing exactly the same set. That is why only `rows_read` moved while `rows_written` sat at 313, and why it failed under `calimero-storage/testing` — which CI enables — roughly ALWAYS rather than occasionally: 0 pass / 12 fail locally before, 15/15 after. The sibling `byte_counts_are_not_reproducible` already records that ids are not deterministic; this test simply must not depend on them being so. `new_with_field_name` gives the collection a deterministic id, which is what the property under test actually needs. Measured: 15/15 under `--features calimero-storage/testing`, 10/10 under default features, and `./scripts/check-storage-cost.sh` still green on all 124 rows — the change touches only the test's own workload, never a measured row. This is the single failure shared by #3750, #3752 and #3754. It is not stale-base: the tool is byte-identical between `bench/05-gas-walls` and this branch, and it reproduces here. Claude-Session: https://claude.ai/code/session_014mxKpC4vXk9Bujuetq7KBk
…tity id `row_counts_are_deterministic_across_calls` built its workload with `UnorderedMap::new()`, which mints a RANDOM entity id. A parent's children live in a hash trie (core#3633) whose descent depth follows the hash of that id, so two runs of an identical workload read a different number of trie nodes while writing exactly the same set. That is why only `rows_read` moved while `rows_written` sat at 313, and why it failed under `calimero-storage/testing` — which CI enables — roughly ALWAYS rather than occasionally: 0 pass / 12 fail locally before, 15/15 after. The sibling `byte_counts_are_not_reproducible` already records that ids are not deterministic; this test simply must not depend on them being so. `new_with_field_name` gives the collection a deterministic id, which is what the property under test actually needs. Measured: 15/15 under `--features calimero-storage/testing`, 10/10 under default features, and `./scripts/check-storage-cost.sh` still green on all 124 rows — the change touches only the test's own workload, never a measured row. This is the single failure shared by #3750, #3752 and #3754. It is not stale-base: the tool is byte-identical between `bench/05-gas-walls` and this branch, and it reproduces here. Claude-Session: https://claude.ai/code/session_014mxKpC4vXk9Bujuetq7KBk
Bugbot is paused — on-demand spend limit reachedBugbot uses usage-based billing for this team and has hit its on-demand spend limit. A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue. |
|
Pushed the
Measured on this branch, same command: Applied as a cherry-pick rather than a rebase: the test exists on every branch in the stack including Not stale-base and not flaky, in case either is in your notes: the tool is byte-identical across the stack, and it fails essentially always under CI's feature set. |
|
This pull request has been automatically marked as stale. If this pull request is still relevant, please leave any comment (for example, "bump"), and we'll keep it open. We are sorry that we haven't been able to prioritize reviewing it yet. Your contribution is very much appreciated. |
Stack 1 of 5. Base:
feat/storage-cost-gates(#3660).Why
Benchmarks rot faster than tests, because nothing runs them. PR #2203 shipped a bench that inlined the body of a private function; the function was later deleted and that bench would not compile today. Nobody noticed, because no job ever built it.
What
bench-compilejob:cargo bench --workspace --benches --no-run. Compiles every bench target, executes none. Timing on shared runners is too noisy to gate on; compilability is not.docs/benchmarking.md— the conventions every bench in this series follows, including why criterion never gates and what to do with a comparison table.[profile.bench.package.calimero-storage] debug-assertions = true. Without itcargo bench --workspacedoes not build at all: feature unification leakscalimero-storage's dev-onlytestingfeature into bench builds, and that crate's release-safety guard (crates/storage/src/interface.rs:209) refuses to compile without debug assertions. Scoped to the one package;opt-leveluntouched.Verification
--no-runfails withE0425, reverted, confirmed clean.cargo clippy --workspace --all-targets --features calimero-storage/testing -- -D warningsclean — note this job newly lints every bench file.rustup run 1.88.0 cargo fmt --checkclean.Before merge
Benchmarks compileshould be in branch protection. TheCriterion trendandCompare against masterjobs added later in this stack must not be.