feat(cli): opt-in --partialGraph for generate-hashes, deriving from --baseHashes without a bazel query

README.md

@@ -68,6 +68,59 @@ See https://github.com/aspect-extensions/impacted
 
 * We run `bazel-diff` on the starting and final JSON hash filepaths to get our impacted set of targets. This impacted set of targets is written to a file.
 
+## Partial Graph Updates
+
+A full `generate-hashes` pays for a `bazel query` over the whole graph even when the change
+cannot have altered that graph: editing the body of a source file changes what Bazel builds,
+never which targets exist or how they depend on each other. `generate-hashes --partialGraph`
+skips that query when it can:
+
+* The hashes describe the workspace's *git tree* rather than the files on disk: the working tree
+  as it stands, committed or not, is written as a tree object (an untracked file `.gitignore`
+  does not exclude is part of it, an ignored one is not, and neither are the hash files the run
+  itself reads and writes: the output, `--baseHashes` and `--depEdgesFile`), and source content
+  is read from that tree. A clean working tree yields the commit's own tree. The file records
+  the tree's id under `metadata.tree`, and every rule's hashing *recipe* (the digest of
+  everything that feeds its hash except source content, plus its ordered source inputs and
+  dependencies) under `metadata.partialGraph`. The file always includes every target's type, as
+  `--includeTargetType` would.
+* With `--baseHashes <file>` naming a file written this way, the recipes it carries are
+  re-evaluated against the workspace's tree when `git diff` between the two trees only modifies
+  the content of regular files -- no `bazel query`, no Bazel at all. Anything else falls back to a
+  full generation: an added, deleted, renamed or type-changed path; a symlink or submodule on
+  either side; a change to a file Bazel reads while loading (`BUILD`, `BUILD.bazel`, `*.bzl`,
+  `*.scl`, `MODULE.bazel`, `MODULE.bazel.lock`, `WORKSPACE*`, `REPO.bazel`, `VENDOR.bazel`,
+  `*bazelrc`, `.bazelversion`, `.bazelignore`, `.bazeliskrc`, `tools/bazel`, `*.BUILD`,
+  `BUILD.*`, `*.patch`, `*.diff`); a file `MODULE.bazel` or a `WORKSPACE` file names by label;
+  anything a repository rule read from the workspace while the graph was generated; a
+  `--seed-filepaths` file; a path matching `--partialGraphRebuildPattern` (a glob: `*` within a
+  path segment, `**` across, a pattern without `/` matches the file name; repeatable); or a base
+  whose tree the local clone does not have. A fallback costs speed, never correctness, and the
+  log says which path each run took. A modified file no target reads impacts nothing, as it
+  would in a full generation.
+* The file a derivation writes is byte-identical to the one a full generation under the flag
+  writes for the same tree, so derived files are themselves bases for later runs. A base that
+  already describes the workspace's tree is written back as is. With `--depEdgesFile` the hash
+  file also carries the dependency edges, so a derivation can write them; a base without them
+  cannot feed `--depEdgesFile` and forces a full generation.
+* Hashes under the flag follow a slightly different formula from the default mode's, so
+  `get-impacted-targets` and `explain` refuse a pair of files of which only one carries recipes.
+  Not supported together with `--useCquery`, `--fineGrainedHashExternalRepos`,
+  `--alwaysAffectedTags`, `--contentHashPath`, `--modified-filepaths` or `--targetType` (a file
+  missing targets cannot be derived from; filter in `get-impacted-targets` instead): a derivation
+  reads only the git tree and keys recipes by bare label. The workspace must be the repository
+  root, and `--gitPath` names the git binary (default `git`).
+
+A typical CI job hashes the merge base once on the main branch and keeps the file; a pull
+request run then needs no Bazel for the common case of a source-only change:
+
+```terminal
+bazel-diff generate-hashes -w . -b bazel --partialGraph --baseHashes main_hashes.json pr_hashes.json
+bazel-diff get-impacted-targets --startingHashes main_hashes.json --finalHashes pr_hashes.json -o impacted.txt
+```
+
+The [query service](#query-service) runs the same derivation server-side behind a cache.
+
 ## Build Graph Distance Metrics
 
 `bazel-diff` can optionally compute build graph distance metrics between two revisions. This is
@@ -452,7 +505,9 @@ Notes and operational guidance:
   with `--useCquery`, `--fineGrainedHashExternalRepos` or `--alwaysAffectedTags` (the server
   refuses to start): a derivation reads only the git tree and keys recipes by bare label. The
   nightly [`partial-graph-bench`](.github/workflows/partial-graph-bench.yml) workflow checks the
-  mode's speed and its exact agreement with the default mode (see `tools/perf/README.md`).
+  mode's speed and its exact agreement with the default mode (see `tools/perf/README.md`). The
+  one-shot `generate-hashes --partialGraph` writes the same entries (see [Partial Graph
+  Updates](#partial-graph-updates)).
 
 * The service checks out revisions inside `--workspacePath`, so point it at a dedicated clone, not a
   working tree you edit. All workspace-mutating work (git checkout + `bazel query`) is serialized,
@@ -581,6 +636,14 @@ Options:
 
   -m, --modified-filepaths <MODIFIED_FILEPATHS>
 
+      --partialGraph[=<PARTIAL_GRAPH>]
+          Opt-in: hash the workspace's git tree (the working tree as it is, committed or not) and record every rule's hashing recipe, so a later run can derive a revision's hashes from this file with --baseHashes and no `bazel query` when the diff between the two only edits regular source files. The file always includes target types. Incompatible with --useCquery, --fineGrainedHashExternalRepos, --alwaysAffectedTags, --contentHashPath, --modified-filepaths and --targetType [default: false] [possible values: true, false]
+      --partialGraphRebuildPattern <PARTIAL_GRAPH_REBUILD_PATTERNS>
+          Glob (`*` within a path segment, `**` across, a pattern without `/` matches the file name) of workspace paths whose change always forces a full generation under --partialGraph
+      --baseHashes <BASE_HASHES>
+          A hash file written with --partialGraph to derive this revision's hashes from, when the diff between its tree and the workspace's allows it; otherwise a full generation runs
+      --gitPath <GIT_PATH>
+          Path to the git executable --partialGraph reads the workspace's tree with [default: git]
   -h, --help
           Print help
   -V, --version

src/cache.rs

@@ -0,0 +1,188 @@
+//! The cache key contract every producer of hash entries shares.
+//!
+//! `serve` and the one-shot CLI must agree on it exactly, so an entry one of them wrote is found
+//! and trusted by the other: the key is the revision's git *tree* (not its commit, so a rebase or
+//! an amended message shares the entry) joined with a digest of everything in the configuration
+//! that can change hash content. Retention, tiers and remote locations are deliberately not in
+//! it.
+
+use crate::hash::HashOptions;
+use crate::partial;
+use sha2::{Digest, Sha256};
+use std::collections::BTreeSet;
+use std::path::PathBuf;
+
+/// Digests the hash-affecting configuration: the package version, every query and hashing
+/// option, whether dependency edges are recorded, and, only when on, partial-graph mode (its
+/// entries hash under a different formula and carry recipes, so they are never served in place
+/// of a default-mode entry; folding the flag in only when enabled keeps existing keys put).
+pub fn configuration_fingerprint(
+    options: &HashOptions,
+    track_deps: bool,
+    partial_graph: bool,
+) -> String {
+    let mut hasher = Sha256::new();
+    hasher.update(env!("CARGO_PKG_VERSION").as_bytes());
+    hasher.update([0]);
+    for value in [
+        options.bazel.startup_options.join("\0"),
+        options.bazel.command_options.join("\0"),
+        options.bazel.cquery_options.join("\0"),
+        options.bazel.use_cquery.to_string(),
+        options.bazel.cquery_expression.clone().unwrap_or_default(),
+        options.bazel.keep_going.to_string(),
+        options.bazel.exclude_external_targets.to_string(),
+        options
+            .bazel
+            .exclude_targets_query
+            .clone()
+            .unwrap_or_default(),
+        options
+            .bazel
+            .fine_grained_external_repos
+            .iter()
+            .cloned()
+            .collect::<Vec<_>>()
+            .join(","),
+        options
+            .bazel
+            .label_only_module_extensions
+            .iter()
+            .cloned()
+            .collect::<Vec<_>>()
+            .join(","),
+        options
+            .ignored_attributes
+            .iter()
+            .cloned()
+            .collect::<BTreeSet<_>>()
+            .into_iter()
+            .collect::<Vec<_>>()
+            .join(","),
+        options
+            .seed_filepaths
+            .iter()
+            .map(|path| path.to_string_lossy())
+            .collect::<Vec<_>>()
+            .join("\0"),
+        options
+            .always_affected_tags
+            .iter()
+            .cloned()
+            .collect::<BTreeSet<_>>()
+            .into_iter()
+            .collect::<Vec<_>>()
+            .join(","),
+        track_deps.to_string(),
+        // Full-content entries hash git blob ids rather than file bytes; keep them apart from
+        // entries a build without this seeding wrote under the same package version.
+        String::from("sourceContent=git-blob"),
+    ] {
+        hasher.update(value.as_bytes());
+        hasher.update([0]);
+    }
+    if partial_graph {
+        hasher.update(format!("partialGraph={}", partial::RECIPE_VERSION).as_bytes());
+        hasher.update([0]);
+    }
+    hex::encode(hasher.finalize())[..12].to_owned()
+}
+
+/// The key of a revision's entry: its tree and the configuration fingerprint, plus a digest of
+/// the `modifiedFilepaths` set for a scoped entry so it is never served as the full one.
+pub fn cache_key(tree: &str, fingerprint: &str, modified: &BTreeSet<PathBuf>) -> String {
+    let base = format!("{tree}.{fingerprint}");
+    if modified.is_empty() {
+        return base;
+    }
+    let mut hasher = Sha256::new();
+    for path in modified {
+        hasher.update(path.to_string_lossy().as_bytes());
+        hasher.update(b"\n");
+    }
+    format!("{base}.{}", &hex::encode(hasher.finalize())[..12])
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::bazel::BazelOptions;
+    use std::collections::HashSet;
+
+    fn options() -> HashOptions {
+        HashOptions {
+            bazel: BazelOptions {
+                workspace: PathBuf::from("."),
+                bazel: PathBuf::from("bazel"),
+                startup_options: Vec::new(),
+                command_options: Vec::new(),
+                cquery_options: Vec::new(),
+                use_cquery: false,
+                cquery_expression: None,
+                keep_going: false,
+                fine_grained_external_repos: BTreeSet::new(),
+                label_only_module_extensions: BTreeSet::new(),
+                exclude_external_targets: false,
+                exclude_targets_query: None,
+                no_bazelrc: false,
+                verbose: false,
+                cancellation: None,
+            },
+            content_hashes: None,
+            ignored_attributes: HashSet::new(),
+            seed_filepaths: BTreeSet::new(),
+            modified_filepaths: BTreeSet::new(),
+            track_deps: false,
+            always_affected_tags: HashSet::new(),
+            partial_tree: None,
+        }
+    }
+
+    #[test]
+    fn fingerprint_moves_with_hash_affecting_options_only() {
+        let base = configuration_fingerprint(&options(), false, false);
+        assert_eq!(base.len(), 12);
+        assert_eq!(configuration_fingerprint(&options(), false, false), base);
+
+        let mut cquery = options();
+        cquery.bazel.use_cquery = true;
+        assert_ne!(configuration_fingerprint(&cquery, false, false), base);
+        let mut seeded = options();
+        seeded.seed_filepaths.insert(PathBuf::from("seed"));
+        assert_ne!(configuration_fingerprint(&seeded, false, false), base);
+        let mut ignored = options();
+        ignored.ignored_attributes.insert("visibility".into());
+        assert_ne!(configuration_fingerprint(&ignored, false, false), base);
+        assert_ne!(configuration_fingerprint(&options(), true, false), base);
+
+        // Partial-graph mode has its own key space; the default mode's keys stay put.
+        let partial = configuration_fingerprint(&options(), false, true);
+        assert_ne!(partial, base);
+        assert_eq!(configuration_fingerprint(&options(), false, true), partial);
+    }
+
+    #[test]
+    fn keys_join_tree_and_fingerprint_and_scope_modified_sets() {
+        let full = cache_key("abc", "fp", &BTreeSet::new());
+        assert_eq!(full, "abc.fp");
+        let scoped = cache_key(
+            "abc",
+            "fp",
+            &BTreeSet::from([PathBuf::from("b"), PathBuf::from("a")]),
+        );
+        assert!(scoped.starts_with("abc.fp."));
+        assert_eq!(scoped.len(), "abc.fp.".len() + 12);
+        assert_eq!(
+            scoped,
+            cache_key(
+                "abc",
+                "fp",
+                &BTreeSet::from([PathBuf::from("a"), PathBuf::from("b")])
+            )
+        );
+        assert_ne!(
+            scoped,
+            cache_key("abc", "fp", &BTreeSet::from([PathBuf::from("a")]))
+        );
+    }
+}