Repository navigation
feat(cli): opt-in --partialGraph for generate-hashes, deriving from --baseHashes without a bazel query - #589
Merged
Conversation
…-baseHashes without a bazel query Adopts serve's partial-graph derivation in the one-shot CLI. Under the flag, generate-hashes hashes the working tree's own git tree object (committed or not; a clean tree is HEAD's), records every rule's recipe and the tree id in the file, and with --baseHashes re-evaluates the base's recipes against that tree when `git diff` between the two trees only modifies regular file content. Anything else falls back to a full generation with the reason logged; either way the file is byte-identical to a full generation's, so derived files are bases for later runs. Details that fell out of making it usable: - the run's own output, --baseHashes and --depEdgesFile files are left out of the snapshot when they lie inside the workspace, otherwise a hashes file written into the repo forces a fallback every time; - partial-graph files always carry target types, and the dependency edges when --depEdgesFile is given, since recipes cannot be evaluated without kinds and a derivation cannot invent edges; --targetType is rejected under the flag; - get-impacted-targets and explain refuse a pair of files of which only one carries recipes, as the two formulas are not comparable. Refactor: git plumbing (resolution, ls-tree indexing, working-tree snapshot, checkout, classification) moves from server.rs to git.rs, and the configuration fingerprint and cache key to cache.rs, so a later CLI cache shares serve's key space. TreeIndex carries its tree id and hash files record it as metadata.tree in partial mode only. Tests: git.rs and cache.rs unit tests, CLI tests for the flags and the derive/fallback paths, and e2e core::generate_hashes_partial_graph_derives_from_base_hashes asserting byte-identical output and impacted-set equality with the default mode. README gains a "Partial Graph Updates" section. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ry copy The working-tree snapshot copies .git/index to a temporary index before `git add --all`. On Linux the copy gets a fresh mtime, which defeats git's racy-index rule: an edit made in the second the index was last written, keeping the file's size, is trusted as unchanged and left out of the snapshot. macOS's fs::copy preserves the mtime, which is why the CLI unit test only flaked on the Linux runner (Rust candidate, Bazel 9.x). Stamp the copy with the original's mtime so the snapshot re-reads exactly the entries git itself would, and add a deterministic regression test that models the race with core.checkStat=minimal and core.trustctime=false. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adopts
serve --partialGraph(#586) in the one-shot CLI.generate-hashes --partialGraphhashes the working tree's own git tree object, records every rule's hashing recipe and the tree id in the file, and with--baseHashes <file>re-evaluates that base's recipes against the workspace's tree whengit diffbetween the two trees only modifies regular file content: nobazel query, no Bazel at all. Anything else (graph-defining files, adds/deletes/renames, symlinks, repo-rule inputs, seed files,--partialGraphRebuildPattern, a base tree the clone lacks) falls back to a full generation with the reason logged. The file a derivation writes is byte-identical to the one a full generation in the mode writes, so derived files are bases for later runs.Phases 1-2 of a four-phase plan; phases 3 (a CLI cache:
--cacheDir, the shared S3/Nexus tiers, ancestor base search,--revision) and 4 (a CLI scenario in the nightly partial-graph bench) are not in this PR.Why
The query is the whole cost of
generate-hashes, and a source-only change cannot alter the graph. In the traditional two-file workflow this lets a PR job derive its hashes from the merge base's file without Bazel.Decisions a reviewer should know about
add --all,write-tree), so the repo's own index is never touched. A clean tree yieldsHEAD^{tree}, so CLI and serve entries for the same commit share an id; a dirty tree yields a real tree objectgit diff/ls-treeaccept. The workspace must be the repository root.--baseHashesand--depEdgesFilefiles are left out of the snapshot when they lie inside the workspace. Without this, a hashes file written into the repo shows up as an added path and forces a fallback on every run.--depEdgesFileis given: recipes cannot be evaluated without kinds, and a derivation cannot invent edges (a base without them falls back when edges are requested).--targetTypeis rejected under the flag; filter inget-impacted-targetsinstead.get-impacted-targetsandexplainrefuse a pair of files of which only one carries recipes, since the two formulas are not comparable.--baseHashesis an error (a misconfiguration); a base without recipes or tree, or a classification failure, is a logged fallback.metadata.treeis written only in partial mode, so default-mode files and serve's default-mode cache entries are byte-for-byte unchanged.Refactor
Git plumbing (revision/tree resolution,
ls-treeindexing, the working-tree snapshot, checkout, classification) moves fromserver.rstosrc/git.rsbehind aGitrunner that serve binds to the query's cancellation token; the configuration fingerprint and cache key move tosrc/cache.rsso a later CLI cache shares serve's<tree>.<fingerprint>key space.server.rskeeps one-line wrappers, so its tests did not move. The S3/Nexus tiers stay inserver.rsuntil the CLI uses them.Tests
src/git.rs: resolution, snapshot (clean = HEAD tree, dirty edits, untracked/ignored, deletions, excluded files, no index file, non-root workspace), checkout with a staleindex.lock, classification.src/cache.rs: fingerprint sensitivity, key shape.src/main.rs: flag parsing, every rejected combination, derive / same-tree / fallback paths against a real git repo with a never-called generator, the mixed-mode guard for both subcommands.core::generate_hashes_partial_graph_derives_from_base_hashes: derived file byte-identical to a full generation, dep-edges files equal, same impacted set as the default mode, uncommitted and committed edits yield the same file, mixed modes refused,BUILDedit falls back.//src:rust_tests,//src:cli_tests, the new e2e case plus both serve partial-graph cases,//tools/e2e:regen_check,//:rust_format_check, clippy aspect. The coverage gate was not run locally.README gains a "Partial Graph Updates" section (generated from
tools/readme_template.md).🤖 Generated with Claude Code