Skip to content

ci: compile every bench target on every PR - #3750

Open
xilosada wants to merge 2 commits into
feat/storage-cost-gatesfrom
bench/01-rot-gate
Open

xilosada wants to merge 2 commits into
feat/storage-cost-gatesfrom
bench/01-rot-gate

Conversation

@xilosada

@xilosada xilosada commented Sep 1, 2026

Copy link
Copy Markdown
Member

Stack 1 of 5. Base: feat/storage-cost-gates (#3660).

Why

Benchmarks rot faster than tests, because nothing runs them. PR #2203 shipped a bench that inlined the body of a private function; the function was later deleted and that bench would not compile today. Nobody noticed, because no job ever built it.

What

  • A blocking bench-compile job: cargo bench --workspace --benches --no-run. Compiles every bench target, executes none. Timing on shared runners is too noisy to gate on; compilability is not.
  • docs/benchmarking.md — the conventions every bench in this series follows, including why criterion never gates and what to do with a comparison table.
  • [profile.bench.package.calimero-storage] debug-assertions = true. Without it cargo bench --workspace does not build at all: feature unification leaks calimero-storage's dev-only testing feature into bench builds, and that crate's release-safety guard (crates/storage/src/interface.rs:209) refuses to compile without debug assertions. Scoped to the one package; opt-level untouched.

Verification

  • Gate proven to fire: injected a reference to a non-existent symbol into a bench, confirmed --no-run fails with E0425, reverted, confirmed clean.
  • cargo clippy --workspace --all-targets --features calimero-storage/testing -- -D warnings clean — note this job newly lints every bench file.
  • rustup run 1.88.0 cargo fmt --check clean.

Before merge

Benchmarks compile should be in branch protection. The Criterion trend and Compare against master jobs added later in this stack must not be.

Benchmarks rot faster than tests, because nothing runs them. A previous
attempt (PR #2203) shipped a bench that inlined a private function; the
function was later deleted and the bench would not compile today, which
nobody noticed because no job ever built it.

Adds a blocking `bench-compile` job running
`cargo bench --workspace --benches --no-run` — compiles every bench
target, executes none. Timing on shared runners is too noisy to gate on;
compilability is not.

Also adds docs/benchmarking.md, the conventions every later bench in this
series follows, and the bench-profile override that makes
`cargo bench --workspace` build at all: feature unification leaks
calimero-storage's dev-only `testing` feature into bench builds, and that
crate's release-safety guard refuses to compile without debug assertions.
The override is scoped to that one package; opt-level is untouched.
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

This pull request has been automatically marked as stale. If this pull request is still relevant, please leave any comment (for example, "bump"), and we'll keep it open. We are sorry that we haven't been able to prioritize reviewing it yet. Your contribution is very much appreciated.

@github-actions github-actions Bot added the Stale label Sep 8, 2026
xilosada added a commit that referenced this pull request Sep 9, 2026
…tity id

`row_counts_are_deterministic_across_calls` built its workload with
`UnorderedMap::new()`, which mints a RANDOM entity id. A parent's children live
in a hash trie (core#3633) whose descent depth follows the hash of that id, so
two runs of an identical workload read a different number of trie nodes while
writing exactly the same set.

That is why only `rows_read` moved while `rows_written` sat at 313, and why it
failed under `calimero-storage/testing` — which CI enables — roughly ALWAYS
rather than occasionally: 0 pass / 12 fail locally before, 15/15 after. The
sibling `byte_counts_are_not_reproducible` already records that ids are not
deterministic; this test simply must not depend on them being so.

`new_with_field_name` gives the collection a deterministic id, which is what
the property under test actually needs. Measured: 15/15 under
`--features calimero-storage/testing`, 10/10 under default features, and
`./scripts/check-storage-cost.sh` still green on all 124 rows — the change
touches only the test's own workload, never a measured row.

This is the single failure shared by #3750, #3752 and #3754. It is not
stale-base: the tool is byte-identical between `bench/05-gas-walls` and this
branch, and it reproduces here.

Claude-Session: https://claude.ai/code/session_014mxKpC4vXk9Bujuetq7KBk
xilosada added a commit that referenced this pull request Sep 9, 2026
…tity id

`row_counts_are_deterministic_across_calls` built its workload with
`UnorderedMap::new()`, which mints a RANDOM entity id. A parent's children live
in a hash trie (core#3633) whose descent depth follows the hash of that id, so
two runs of an identical workload read a different number of trie nodes while
writing exactly the same set.

That is why only `rows_read` moved while `rows_written` sat at 313, and why it
failed under `calimero-storage/testing` — which CI enables — roughly ALWAYS
rather than occasionally: 0 pass / 12 fail locally before, 15/15 after. The
sibling `byte_counts_are_not_reproducible` already records that ids are not
deterministic; this test simply must not depend on them being so.

`new_with_field_name` gives the collection a deterministic id, which is what
the property under test actually needs. Measured: 15/15 under
`--features calimero-storage/testing`, 10/10 under default features, and
`./scripts/check-storage-cost.sh` still green on all 124 rows — the change
touches only the test's own workload, never a measured row.

This is the single failure shared by #3750, #3752 and #3754. It is not
stale-base: the tool is byte-identical between `bench/05-gas-walls` and this
branch, and it reproduces here.

Claude-Session: https://claude.ai/code/session_014mxKpC4vXk9Bujuetq7KBk
…tity id

`row_counts_are_deterministic_across_calls` built its workload with
`UnorderedMap::new()`, which mints a RANDOM entity id. A parent's children live
in a hash trie (core#3633) whose descent depth follows the hash of that id, so
two runs of an identical workload read a different number of trie nodes while
writing exactly the same set.

That is why only `rows_read` moved while `rows_written` sat at 313, and why it
failed under `calimero-storage/testing` — which CI enables — roughly ALWAYS
rather than occasionally: 0 pass / 12 fail locally before, 15/15 after. The
sibling `byte_counts_are_not_reproducible` already records that ids are not
deterministic; this test simply must not depend on them being so.

`new_with_field_name` gives the collection a deterministic id, which is what
the property under test actually needs. Measured: 15/15 under
`--features calimero-storage/testing`, 10/10 under default features, and
`./scripts/check-storage-cost.sh` still green on all 124 rows — the change
touches only the test's own workload, never a measured row.

This is the single failure shared by #3750, #3752 and #3754. It is not
stale-base: the tool is byte-identical between `bench/05-gas-walls` and this
branch, and it reproduces here.

Claude-Session: https://claude.ai/code/session_014mxKpC4vXk9Bujuetq7KBk
xilosada added a commit that referenced this pull request Sep 9, 2026
…tity id

`row_counts_are_deterministic_across_calls` built its workload with
`UnorderedMap::new()`, which mints a RANDOM entity id. A parent's children live
in a hash trie (core#3633) whose descent depth follows the hash of that id, so
two runs of an identical workload read a different number of trie nodes while
writing exactly the same set.

That is why only `rows_read` moved while `rows_written` sat at 313, and why it
failed under `calimero-storage/testing` — which CI enables — roughly ALWAYS
rather than occasionally: 0 pass / 12 fail locally before, 15/15 after. The
sibling `byte_counts_are_not_reproducible` already records that ids are not
deterministic; this test simply must not depend on them being so.

`new_with_field_name` gives the collection a deterministic id, which is what
the property under test actually needs. Measured: 15/15 under
`--features calimero-storage/testing`, 10/10 under default features, and
`./scripts/check-storage-cost.sh` still green on all 124 rows — the change
touches only the test's own workload, never a measured row.

This is the single failure shared by #3750, #3752 and #3754. It is not
stale-base: the tool is byte-identical between `bench/05-gas-walls` and this
branch, and it reproduces here.

Claude-Session: https://claude.ai/code/session_014mxKpC4vXk9Bujuetq7KBk
@cursor

cursor Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@xilosada

xilosada commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

Pushed the storage-cost determinism fix — this is the Rust failure on this PR.

row_counts_are_deterministic_across_calls built its workload with UnorderedMap::new(), which mints a random entity id. A parent's children live in a hash trie (core#3633) whose descent depth follows that id's hash, so two runs of an identical workload read a different number of trie nodes while writing exactly the same set. That is the rows_written: 313 / rows_read: 1088 vs 1098 signature in the CI log. new_with_field_name gives the collection a deterministic id, which is what the property under test actually needs.

Measured on this branch, same command:

without fix:  0 pass / 6
with fix:     8 pass / 8

Applied as a cherry-pick rather than a rebase: the test exists on every branch in the stack including feat/storage-cost-gates, so the fix's natural home is the bottom — but cherry-picking onto each affected branch keeps your stack ordering untouched. When the stack is next rebased, git will drop the duplicates by patch-id. Same commit is on #3752.

Not stale-base and not flaky, in case either is in your notes: the tool is byte-identical across the stack, and it fails essentially always under CI's feature set.

https://claude.ai/code/session_014mxKpC4vXk9Bujuetq7KBk

@github-actions github-actions Bot removed the Stale label Sep 9, 2026
@github-actions

Copy link
Copy Markdown

This pull request has been automatically marked as stale. If this pull request is still relevant, please leave any comment (for example, "bump"), and we'll keep it open. We are sorry that we haven't been able to prioritize reviewing it yet. Your contribution is very much appreciated.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants