Skip to content

(12) test-clock - #1773

Draft
daniel-noland wants to merge 264 commits into
pr/daniel-noland/config-algebrafrom
pr/daniel-noland/driven-clock
Draft

(12) test-clock#1773
daniel-noland wants to merge 264 commits into
pr/daniel-noland/config-algebrafrom
pr/daniel-noland/driven-clock

Conversation

@daniel-noland

@daniel-noland daniel-noland commented Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

No description provided.

@coderabbitai

coderabbitai Bot commented Aug 28, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.57793% with 261 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
k8s-intf/src/bolero/mutate.rs 68.14% 59 Missing and 13 partials ⚠️
config/src/external/overlay/algebra.rs 94.78% 37 Missing and 25 partials ⚠️
config/src/external/overlay/vpcpeering.rs 91.69% 21 Missing and 4 partials ⚠️
acl-filter/src/nf_fuzz.rs 89.93% 11 Missing and 5 partials ⚠️
hardware/src/scan.rs 0.00% 15 Missing ⚠️
clock/src/virtual_time.rs 92.85% 9 Missing and 3 partials ⚠️
config/src/external/overlay/completeness.rs 95.15% 6 Missing and 5 partials ⚠️
k8s-intf/src/bolero/reduce.rs 85.07% 8 Missing and 2 partials ⚠️
hardware/src/nic/mod.rs 0.00% 7 Missing ⚠️
cli/bin/main.rs 0.00% 5 Missing ⚠️
... and 14 more

📢 Thoughts on this report? Let us know!

@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/driven-clock branch from fa0e8f2 to b164a0f Compare August 28, 2026 03:05
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/config-algebra branch from 5843caa to 6f106ac Compare August 28, 2026 03:05
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/driven-clock branch from b164a0f to 3cd5687 Compare August 28, 2026 03:34
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/config-algebra branch from 6f106ac to 320c6aa Compare August 28, 2026 03:34
@daniel-noland daniel-noland changed the title feat(clock): give a test one clock it drives, and fix what that exposed test-clock Aug 28, 2026
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/config-algebra branch from 320c6aa to d3c348d Compare August 28, 2026 04:22
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/driven-clock branch 2 times, most recently from 1ec0be0 to de8e45f Compare August 28, 2026 05:11
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/config-algebra branch 2 times, most recently from be78275 to 4969a2f Compare August 28, 2026 05:28
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/driven-clock branch from de8e45f to bea6549 Compare August 28, 2026 05:32
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/config-algebra branch from 4969a2f to 26e6361 Compare August 28, 2026 05:47
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/driven-clock branch from bea6549 to f850a96 Compare August 28, 2026 05:48
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/config-algebra branch from 26e6361 to 3ee9131 Compare August 28, 2026 06:18
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/driven-clock branch from f850a96 to 9398527 Compare August 28, 2026 06:21
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/config-algebra branch from 3ee9131 to fcd1d11 Compare August 28, 2026 06:40
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/driven-clock branch from 9398527 to d497a39 Compare August 28, 2026 06:40
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/config-algebra branch from fcd1d11 to b26928b Compare August 28, 2026 07:08
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/driven-clock branch from d497a39 to 3e43aea Compare August 28, 2026 07:08
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/config-algebra branch from b26928b to 84d66cc Compare August 28, 2026 07:31
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/driven-clock branch from 3e43aea to 610dcf1 Compare August 28, 2026 07:31
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/config-algebra branch from 84d66cc to a842024 Compare August 28, 2026 07:43
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/driven-clock branch from 610dcf1 to 698f663 Compare August 28, 2026 07:43
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/config-algebra branch from a842024 to c50fd96 Compare August 28, 2026 09:14
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/driven-clock branch from 698f663 to 26b967b Compare August 28, 2026 09:14
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/config-algebra branch from c50fd96 to ccbfc9c Compare August 28, 2026 17:16
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/driven-clock branch from 26b967b to 8968292 Compare August 28, 2026 17:16
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/config-algebra branch from ccbfc9c to 33377c4 Compare August 28, 2026 17:33
daniel-noland and others added 30 commits September 9, 2026 11:55
@daniel-noland @claude
test(dataplane): use endpoint-owned addresses for inbound traffic
92943be 
Inbound loads used a peer's public translation range as the sender address
and answered to the dialed address rather than the request's delivered
source. Both values happened to agree before translated inbound exposes were
generated; afterward the flow filter correctly rejected them as unowned.
Separate dial destinations from peer source addresses and reply to the
observed sender.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
feat(config): generate port-forwarding exposes
3814ec2 
Port forwarding was the last missing translated expose flavour and the only
source of generated port ranges, leaving several configuration fields
untested. Add
equal-width but distinct public and private ranges, forbid conflicting
directional translations on the opposite side, and require evidence that
inbound loads exercise the new flavour. Leave protocol and timeout variation
to separate changes.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
feat(config): generate peering ACLs
0310552 
Every generated peering permitted all traffic, so pipeline properties never
exercised a configuration intended to deny a packet. Add coarse ACL shapes
whose verdicts remain predictable without copying the production evaluator.
Update traffic derivation to omit denied loads and retain guards proving ACL
decisions are actually reached.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
test(config): measure ACL schema coverage
5620311 
The completeness survey stopped at whether a peering had an ACL, hiding
fixed rule, pattern, and scope fields behind a single present value. Extend
the census through the entire ACL schema. This reports eight pre-existing
fixed degrees of freedom without claiming the algebra gained new reach.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
feat(config): generate stateful peering ACLs
65bb9e5 
Packet-level ACL rules never exercise reverse-flow permission, where the
verdict depends on NAT state. Generate permit-by-flow peerings and verify
their replies through AclFilter's reverse lookup. The first run exposes and
pins a separate defect: port-forwarded flows retain generation one and lose
permission after any configuration change.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
feat(config): generate overlapping ACL rules
140edd4 
With at most one rule per direction, generated ACLs could not make precedence
affect a verdict. Add two predictable overlapping rules that partition a
peering around one masquerade expose; reversing them now breaks the denial
property. Track the named expose position and assign traffic variation before
filtering so independent derivations remain comparable.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
feat(config): vary gateway groups, ACL logging, and timeouts
0324e69 
Gateway-group names, ACL logging, and idle timeouts remained fixed solely
because the algebra never selected them. Vary all three and update the
census. Record the current limits: disjoint exposes do not exercise gateway
grouping semantics, and generated timeouts outlast these properties, so the
fields lower successfully without testing expiration.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
feat(config): narrow ACL rules by protocol, port, and destination
b24e110 
Generated ACLs still left protocol, port, and destination matching fixed, so
dropping a field or lowering it as a wildcard could pass. Add predictable
narrow rules whose target prefixes appear in exactly one traffic role,
avoiding a duplicate ACL evaluator. Include a protocol guard that must not
fire so both omission and wildcarding become observable.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
feat(config): vary port-forwarding protocols
357e5c3 
Every generated port-forwarding expose accepted any protocol, leaving its
protocol key and collision behavior untested. Cycle protocols across expose
slots so one manifest can contain several. Because derived traffic is UDP,
exclude TCP-only exposes from offered loads and make Draft::carries account
for both expose protocol and peering ACL.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
feat(config): generate address exclusions
01a7723 
Generated exposes had no exclusions, so matchers, LPM tables, and
RangeBuilder saw only contiguous prefixes. Remove a middle slice rather than
one half, forcing the effective set into two ranges. This change measures
reach and shape only; traffic derivation still selects from the effective set
and therefore does not yet test the excluded hole.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
feat(config): generate default-route exposes
f48b784 
The algebra could not express default-route exposes. Default routes
overlap every neighboring route, so their validity depends on more than the
operation's own arguments. Restrict them to otherwise empty peerings and let
Op::reads inspect the draft, keeping applicability local and restoring the
independence property. Skip loads whose expose protocol cannot carry their
traffic.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
test(dataplane): reject traffic to excluded addresses
63142f7 
Existing exclusion coverage drew traffic only from the effective address
set, so a matcher that ignored every exclusion still passed. Aim packets
directly at excluded addresses and require them to be unreachable. Exempt a
manifest with a default expose because its default route legitimately covers
the hole, and mark the census row as behaviorally enforced.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
feat(clock): reject wall-clock reads during virtual tests
6f8d490 
Tokio's paused clock belongs to one runtime. A worker without that
runtime silently reads wall time, so expiry assertions can compare two
timelines after a test advances time.

Introduce a shared `Paused` driver and refuse off-runtime reads while a
virtual clock is live. Scope the guard to the driver's lifetime so
ordinary readers remain valid afterward, and provide `wall_clock` so
the same properties can exercise real time. Move the NAT expiry suites
onto the shared driver.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
build(semgrep): forbid clock reads during Drop
914dfb9 
A destructor may run after Tokio's thread-local runtime context is
gone. If virtual time has been paused, reading the clock there panics
inside `Drop` and aborts the process without identifying the test.

Add an opengrep rule that rejects clock reads from `fn drop`. Match the
method itself because the Rust parser does not reliably constrain a
pattern to `impl Drop`.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
feat(tracectl): timestamp logs with the active test clock
cffdff6 
Logs used wall time while expiry code used virtual time, so events
could not be correlated with the deadlines a test observed.

Stamp test logs with an offset from the routed clock. Use a checked
read so logging cannot panic when a thread lacks the active clock;
mark those records `off-clock` instead.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
test(dataplane): check flows across scheduled time advances
a93c743 
The pipeline property never aged its flows, so it could not detect a
flow that expired too early. Advancing time had previously been
meaningless because waits and deadlines read different clocks.

Draw waits as part of the generated schedule and advance between
rounds, where the driver can move time without measuring thread
scheduling. Keep waits within the flow lifetime and require delivered
flows to retain their disposition. Limit strict clock enforcement to
process-isolated nextest runs until clock ownership follows threads.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
feat(clock): propagate test clocks to spawned threads
7bf8caf 
Clock enforcement was process-wide: concurrent `cargo test` cases
could trip each other's guard, while a worker spawned by the active
test could forget its runtime and fall back to wall time.

Use standard thread spawn hooks to inherit the active runtime and clock
membership through the thread tree. Enter the inherited runtime for
each read and keep the handle thread-local so later tests cannot reuse
an earlier clock. Probe hook support at build time; older toolchains
retain the check on the driving thread. Threads created outside
`std::thread` still fail rather than reading the wrong clock.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
fix(dataplane): poll timer tasks during fuzzing
cbabca0 
Bolero's synchronous case loop never yielded to the surrounding Tokio
test runtime. Flow timers therefore never ran, each retained its flow
table, and long fuzz runs grew until they exhausted memory. The same
properties also proved nothing about expiry.

Run each case through a driven runtime and poll spawned tasks before
moving on. This lets timer counts and memory settle instead of growing
with the corpus.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
fix(nat): make NAT properties reachable by the fuzzer
6f70748 
None of the thirteen NAT properties could run under `cargo bolero`.
Their vacuity guards rejected Bolero's target-selection pass, and
their undriven timer runtime accumulated flow tables until the process
ran out of memory.

Bypass case assertions during target selection and drive the timer
runtime around every fuzz case.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
build(just): compare the effective fuzz sanitizer
ceca5a0 
An empty `sanitize` setting still makes cargo-bolero use
AddressSanitizer. Treating it as no sanitizer allowed Rust and the
sysroot to use incompatible instrumentation, while the explicit
`NONE` setting was rejected even though it matched an uninstrumented
sysroot.

Resolve cargo-bolero's default before comparing the requested and
recorded sanitizer settings. Warn for the compatible legacy default
and reject explicit mismatches.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
fix(net): expose flow-info properties as fuzz targets
883fb0f 
Eleven flow-info properties placed `check!()` inside a closure, so
Bolero registered them under `{{closure}}`. No command-line target name
could select them.

Put each check at its named test site and scope the paused clock to one
case so timer tasks settle between inputs.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
build: enable Bolero's std feature explicitly
da90379 
Three packages built only because workspace feature unification exposed
`bolero_engine::any`. A package-only fuzz build lacked the `std` feature
that provides it and failed to compile.

Request Bolero's `std` feature in each affected package instead of
relying on an unrelated workspace dependency.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
fix(net): register each header shard as a fuzz target
6fa72da 
Twenty-four header shards delegated `check!()` to one helper. Bolero
registered the helper name repeatedly, leaving every shard impossible
to select.

Place the registration at each shard's test entry point while retaining
the shared property body.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
fix(fuzz): give shared properties selectable target names
e537369 
The remaining shared property helpers registered one target under the
helper name instead of the ACL, concurrency, config, and stats tests
that called them. Those advertised targets could never run.

Register `check!()` at each test entry point and adapt the config census
helper to the macro's early return. All 596 listed workspace targets are
now selectable.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
build(just): restore flags hidden by cargo-bolero
25d8b65 
cargo-bolero sets `RUSTFLAGS`, causing Cargo to ignore the workspace
flags. Fuzz builds lost `tokio_unstable` and the registered cfg names.
Sanitizer coverage also instrumented non-Bolero test binaries that had
no runtime symbols, so package builds failed without a sanitizer.

Prepend the configured workspace flags and link the local no-main
libFuzzer runtime when available. This removes the cfg warnings and
allows every package to fuzz with `sanitize=NONE`.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
build: compose instrumentation independently of profiles
391f974 
Modeling fuzzing as a Cargo profile made it mutually exclusive with
coverage, although both are compiler instrumentation that should
compose with each other and with sanitizers. Instrumented containers
could also replace clean images because instrumentation is absent from
their version tags.

Represent instrumentation as a normalized set parallel to sanitizers,
retain a `checked` profile for compiler safety settings, and refuse
container builds that carry diagnostic instrumentation.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
build: make fuzz-instrumented sysroots link
120ae1d 
Fuzz instrumentation broke the native-dependency sysroot. rdma-core's
build tools referenced sanitizer-coverage symbols without a runtime,
DPDK's ThinLTO discarded module constructors while retaining their
relocations, and the overlay could not see the selected instrumentation.

Allow unresolved symbols in throwaway rdma-core tools, disable LTO for
fuzz-instrumented C and C++, omit their fuzz link flags, and pass the
instrumentation set into the overlay.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@daniel-noland @claude
fix(clock): keep nested clock guards armed
0533c32 
Clock membership used a boolean even though one thread may hold nested
`Paused` drivers. Dropping the inner driver cleared the flag and let
off-runtime reads through for the rest of the outer driver's lifetime.

Count thread membership so the guard remains armed until the last
nested driver leaves. Update the module documentation to describe the
supported nesting behavior.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
@daniel-noland @claude
fix(concurrency): remove imports left by the fuzz refactor
785cfed 
Moving the shared fuzz registrations removed the last uses of
`RefUnwindSafe` from two integration tests, leaving warnings that fail
the all-targets Clippy run. Remove the stale imports.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
@daniel-noland
style(dataplane): stop bypassing the concurrency facade
5afdeeb 
The process-wide fuzz runtime still used `std::sync::LazyLock` directly,
violating the workspace rule that shared synchronization goes through
the concurrency facade.

Route that runtime through the facade. Keep paused-clock bookkeeping on
`std::sync`: it belongs outside model-checker scheduling, and replacing
it makes virtual-time properties sleep in real time.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@daniel-noland daniel-noland

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@daniel-noland