Skip to content

feat(cache): let a hit rebuild folders that hold only the task's outputs - #914

Open
wan9chi wants to merge 2 commits into
mainfrom
feat/cache-rebuildable-folders
Open

wan9chi wants to merge 2 commits into
mainfrom
feat/cache-rebuildable-folders

Conversation

@wan9chi

@wan9chi wan9chi commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Motivation

A task that stats or lists a folder holding only its own outputs pins that folder as an input. TypeScript 7's tsc, for example, stats dist while emitting into it. Removing the folder, as on every clean checkout in CI, is then a miss, although a hit restores every file in it.

Changes

After the run, vt climbs from the written paths through their parent folders and lists each folder once, deepest first. A folder is rebuildable when it isn't empty, every file or symlink in it was written by the task, and every folder in it is rebuildable. A folder that's gone by then, like Astro's dist/.prerender, plays no part. A rebuildable folder gets no inferred-input fingerprint; restoring its files recreates it. vt_path gets RelativePath::parent.

E2E cases: stat_own_output_folder (synthetic) and typescript_out_dir (TypeScript 7 emitting to dist). Removing the output folder misses today and hits with the folder restored after.

Part of a stack that lets automatic tracking cache tasks which read their own outputs and list the folders holding them, so they also hit on a clean checkout.

🤖 Generated with Claude Code


Note

Medium Risk
Changes post-run input fingerprinting and cache-hit rules for auto-tracked directory reads; incorrect rebuildable classification could hide real input changes or allow stale hits.

Overview
Tasks that stat or list an output directory (e.g. tsc on dist) no longer treat that folder as a tracked input when every entry in it is something the task wrote. Post-run fingerprinting skips those “rebuildable” directories, so a clean checkout with the folder gone can still cache hit and restore archived outputs, recreating the tree.

Rebuildable now walks parent paths from fspy’s unfiltered writes after the run, classifies directories whose live listing contains only written files/symlinks and already-known rebuildable subfolders (ignoring .DS_Store, ignoring dirs removed before classification), and exposes is_rebuildable_folder. PostRunFingerprint::create filters those paths out alongside prerun globs. RelativePath::parent supports the climb.

Docs/changelog and e2e fixtures cover a synthetic stat out case and TypeScript emitting to dist (typescript fixture marked ignore in snapshots.toml).

Reviewed by Cursor Bugbot for commit 9ad3c84. Bugbot is set up for automated code reviews on this repo. Configure here.

@wan9chi
wan9chi added this pull request to stack #920 October 30, 2026 03:17
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

fspy benchmark

linux

dynamic/launch             change  +0.84%  [ -5.19% ..  +8.09%]  overhead   +62.95%
dynamic/access             change  +0.37%  [ -8.61% .. +17.54%]  overhead   +23.59%
dynamic/access-relative    change  -0.21%  [ -6.52% ..  +5.94%]  overhead   +75.97%
dynamic/access-contended   change  +0.93%  [ -8.15% .. +11.44%]  overhead    +7.02%
static/launch              change  +0.66%  [ -5.07% ..  +6.39%]  overhead  +193.73%
static/access              change  +0.34%  [ -2.06% ..  +4.84%]  overhead +1477.63%
static/access-relative     change  -0.26%  [-12.65% .. +12.14%]  overhead +1904.69%
static/access-contended    change  +1.76%  [ -3.85% ..  +8.57%]  overhead +1161.38%

macos

dynamic/launch             change  +0.77%  [ -3.85% ..  +4.03%]  overhead  +229.11%
dynamic/access             change  -0.68%  [ -5.23% ..  +4.14%]  overhead    +4.94%
dynamic/access-relative    change  +1.17%  [ -5.43% ..  +6.79%]  overhead  +259.74%
dynamic/access-contended   change  -0.66%  [ -8.41% ..  +3.69%]  overhead    +3.03%

windows

dynamic/launch             change  +0.19%  [ -1.51% ..  +1.89%]  overhead   +25.35%
dynamic/access             change  +0.00%  [ -0.77% ..  +0.59%]  overhead    +0.96%
dynamic/access-relative    change  -0.18%  [ -0.94% ..  +0.56%]  overhead    +1.16%
dynamic/access-contended   change  +0.00%  [ -0.71% ..  +0.73%]  overhead    +1.65%

@wan9chi
wan9chi force-pushed the feat/cache-rebuildable-folders branch from 91b15b8 to b25c11e Compare October 30, 2026 03:19
@wan9chi
wan9chi force-pushed the feat/cache-rebuildable-folders branch from b25c11e to 4dff101 Compare October 30, 2026 04:07
@wan9chi wan9chi added the ci: full Run the slow CI jobs (macOS x64, musl) on this pull request label Oct 30, 2026
Base automatically changed from feat/cache-rewritten-inputs to main October 30, 2026 06:31
@wan9chi
wan9chi force-pushed the feat/cache-rebuildable-folders branch from 4dff101 to 9f74d41 Compare October 30, 2026 06:31
wan9chi and others added 2 commits October 30, 2026 15:48
Two cases for a task that reads a folder holding only its own outputs. In
`stat_own_output_folder`, the task makes `out`, writes `out/a.txt` and
stats `out`, through `vtt fs`. In `typescript_out_dir`, TypeScript 7's
`tsc` compiles `src` into `dist`: it stats `dist` and lists the package
root, where the hard-coded `dist` skip hides the folder.

Today removing the output folder, as on a clean checkout, is a cache miss
naming the folder, although the cached outputs hold everything in it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A task that stats or lists a folder holding only its own outputs, like
TypeScript 7's `tsc` statting `dist`, pinned that folder as an input, so
removing it, as on a clean checkout, was a cache miss although a hit
restores every file in it.

After the run, `Rebuildable` now climbs from the written paths through
their parent folders inside the workspace and lists each folder in the
real file system, deepest first. A folder is rebuildable when it isn't
empty, every file or symlink in it is a path the task wrote, and every
folder in it is rebuildable; climbing stops at the first folder that
isn't. A folder that is gone by then, like the `dist/.prerender` that
Astro writes to and deletes, plays no part, like a written file that is
gone: its parent is decided by the entries it holds. A rebuildable
folder the task read gets no inferred-input fingerprint. The archive is
unchanged: the written files in such a folder are archived already, and
restoring them recreates it. An empty folder isn't rebuildable yet, and
a listing of the folder's parent still names it.

`vt_path` gets `RelativePath::parent` for the climb.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wan9chi
wan9chi force-pushed the feat/cache-rebuildable-folders branch from 9f74d41 to 9ad3c84 Compare October 30, 2026 07:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci: full Run the slow CI jobs (macOS x64, musl) on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant