P3 (part 1): sidecar provenance, --check mode, in-repo Gate 1 figure, paper workflow (S-RP.1/3/4, S-RE.5, S-FD.4/5)

.github/workflows/paper.yml

@@ -0,0 +1,102 @@
+name: paper
+
+# Checks for the manuscript branch: the generated figures still reproduce
+# their committed sidecars, the survey bibliography regenerates identically,
+# and the PDF builds without unresolved references or the artefacts the
+# iteration-2 review caught. The pre-submission reviewer itself stays on
+# demand (review/REVISION_PLAN_ITER3.md, P3).
+
+on:
+  push:
+    branches: [docs/sign-convention-manuscript]
+  pull_request:
+    branches: [docs/sign-convention-manuscript]
+  workflow_dispatch:
+
+concurrency:
+  group: paper-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  sidecars:
+    name: figure sidecars reproduce
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    env:
+      OMP_NUM_THREADS: "1"
+      OPENBLAS_NUM_THREADS: "1"
+      MPLBACKEND: Agg
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+          cache-dependency-path: pyproject.toml
+      - name: install
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[test]"
+      - name: regenerate the fast figures in memory and diff against the sidecars
+        run: python -m codameter.figures --check --skip-slow --rtol 1e-5
+
+  survey:
+    name: survey bibliography regenerates identically
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: rebuild survey.bib and appendix_table.tex
+        run: python paper/build_survey.py
+      - name: no drift from the committed files
+        run: git diff --exit-code -- paper/survey.bib paper/appendix_table.tex
+
+  pdf:
+    name: PDF builds and passes the text checks
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - uses: quarto-dev/quarto-actions/setup@v2
+        with:
+          tinytex: true
+      - name: TeX packages the rendered manuscript loads
+        # quarto's TinyTeX installs missing packages on demand during the render;
+        # this pre-installs the ones the gji class, natbib and the tables need so
+        # the render does not stop on the first missing .sty.
+        run: |
+          tlmgr install latexmk natbib tabularx booktabs longtable caption \
+            lineno xcolor amsmath amsfonts siunitx float fontspec unicode-math \
+            lm-math upquote microtype etoolbox parskip footnote setspace \
+            iftex textcomp lmodern calc || true
+      - name: install codameter (for the version string)
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e .
+      - name: build
+        run: python paper/build.py --no-survey
+      - name: text checks
+        run: |
+          sudo apt-get update -qq && sudo apt-get install -y -qq poppler-utils
+          pdftotext -layout paper/manuscript_marine.pdf /tmp/paper.txt
+          n_unresolved=$(grep -c '??' /tmp/paper.txt || true)
+          n_commapct=$(grep -c ',%' /tmp/paper.txt || true)
+          n_lt=$(grep -c '<<' /tmp/paper.txt || true)
+          n_tilde=$(grep -cE '(Table|Fig\.|Section|eq\.)~' /tmp/paper.txt || true)
+          echo "pages: $(pdfinfo paper/manuscript_marine.pdf | awk '/Pages/{print $2}')"
+          echo "unresolved: $n_unresolved  commapct: $n_commapct  lt: $n_lt  tilde: $n_tilde"
+          test "$n_unresolved" -eq 0 && test "$n_commapct" -eq 0 && test "$n_lt" -eq 0 && test "$n_tilde" -eq 0
+      - name: upload the PDF
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: manuscript-pdf
+          path: |
+            paper/manuscript_marine.pdf
+            paper/manuscript_marine.tex
+          if-no-files-found: ignore

literature/figs/SOURCES.md

@@ -18,14 +18,29 @@ prints the registry. `paper/build.py --figures` runs the driver.
 | `demo_11_multiverse` | `codameter.deviations.multiverse` + `fig_multiverse_full` (slow) |
 | `demo_12_bayes` | `codameter.uq_bayes._build_bayes` + `_fig_bayes` (slow) |
 
+| `realdata_1_validation` | `codameter.gate1.fig_gate1_comparison`: needs the untracked daily products under `paper/data/gate1/dvv2y/` (skipped with a message where they are absent); every plotted array is in the sidecar |
+
+Every `.json` sidecar records `git_commit`; sidecars written since
+2026-09-13 also record `git_dirty` (true when tracked files under `src/`
+differed from that commit when the figure was made) and `generator_digest`
+(a digest of the package version and the figure-generating modules, so two
+sidecars with the same digest came from the same figure code). Older sidecars
+gain the two fields when their figure is next regenerated; the release step
+regenerates every figure from a clean tree at the tag.
+`python -m codameter.figures --check [--skip-slow]` regenerates the figures in
+memory and reports any array that differs from the committed sidecar; the
+`paper` workflow runs it on every pull request to the manuscript branch.
+
 ## Produced outside this repository
 
-`realdata_1_validation.png`, `realdata_2_interferograms.png` and
-`realdata_3_warmup.png` come from the noisepy-dvv-cloud Gate 1 run
-(CI.LJR / CI.RXH / CI.ARV, 2018-2019; see `paper/data/gate1/README.md`).
-Their inputs are the daily ensemble products under `paper/data/gate1/dvv2y/`
-(not tracked by git) and the published Clements and Denolle (2022) product
-under `paper/data/gate1/legacy_cd2022/`. The comparison script
-(`scripts/compare_cd2022.py`) and the figure scripts live in that repository;
-the commit they were run at is to be pinned here (audit finding REP-02).
-Until then these three figures cannot be regenerated from this checkout.
+`realdata_2_interferograms.png` and `realdata_3_warmup.png` come from the
+noisepy-dvv-cloud Gate 1 run (CI.LJR / CI.RXH / CI.ARV, 2018-2019; see
+`paper/data/gate1/README.md`). Their inputs are the daily correlations of that
+run, which are not archived here, so they are committed as produced. The
+comparison figure (`realdata_1_validation`) used to be produced there as well;
+it is now generated in this repository from the archived daily products and
+`paper/data/gate1/comparison.json`. The Gate 1 run commit and `--use-case`
+are to be pinned in `paper/data/gate1/README.md` (issue #46).
+Until then the two interferogram-based figures cannot be regenerated from
+this checkout; the comparison figure can, wherever the daily products are
+present.

literature/figs/realdata_1_validation.json

@@ -0,0 +1,256 @@
+{
+ "figure": "realdata_1_validation",
+ "generator": "codameter.gate1.fig_gate1_comparison (needs paper/data/gate1/dvv2y)",
+ "codameter_version": "0.4.0",
+ "git_commit": "319e57ac8e6c34e1833b5342d7a7e7d2b214c957",
+ "git_dirty": false,
+ "generator_digest": "7d5cd15e6b62",
+ "generated_utc": "2026-09-13T23:22:31+00:00",
+ "python": "3.12.13",
+ "numpy": "2.4.3",
+ "matplotlib": "3.10.9",
+ "axes": [
+  {
+   "axes": "ax0",
+   "title": "Gate 1, 2.0-4.0 Hz, 2018-2019: single-station ensemble against the published product (demeaned over the overlap)",
+   "xlabel": "",
+   "ylabel": "CI.LJR\ndv/v (%)",
+   "lines": [
+    "daily dv/v",
+    "codameter, trailing 90-day mean",
+    "Clements and Denolle (2022)",
+    "_child4"
+   ],
+   "images": 0,
+   "collections": [
+    "daily $\\pm1.96\\,\\sigma$ (corrected floor)"
+   ],
+   "patches": 0
+  },
+  {
+   "axes": "ax1",
+   "title": "",
+   "xlabel": "",
+   "ylabel": "CI.ARV\ndv/v (%)",
+   "lines": [
+    "daily dv/v",
+    "codameter, trailing 90-day mean",
+    "Clements and Denolle (2022)",
+    "_child4"
+   ],
+   "images": 0,
+   "collections": [
+    "daily $\\pm1.96\\,\\sigma$ (corrected floor)"
+   ],
+   "patches": 0
+  },
+  {
+   "axes": "ax2",
+   "title": "",
+   "xlabel": "date",
+   "ylabel": "CI.RXH\ndv/v (%)",
+   "lines": [
+    "daily dv/v",
+    "codameter, trailing 90-day mean",
+    "Clements and Denolle (2022)",
+    "_child4"
+   ],
+   "images": 0,
+   "collections": [
+    "daily $\\pm1.96\\,\\sigma$ (corrected floor)"
+   ],
+   "patches": 0
+  }
+ ],
+ "arrays": [
+  "ax0/collection0/offsets",
+  "ax0/collection0/path_lengths",
+  "ax0/collection0/vertices",
+  "ax0/line0/x",
+  "ax0/line0/y",
+  "ax0/line1/x",
+  "ax0/line1/y",
+  "ax0/line2/x",
+  "ax0/line2/y",
+  "ax0/line3/x",
+  "ax0/line3/y",
+  "ax1/collection0/offsets",
+  "ax1/collection0/path_lengths",
+  "ax1/collection0/vertices",
+  "ax1/line0/x",
+  "ax1/line0/y",
+  "ax1/line1/x",
+  "ax1/line1/y",
+  "ax1/line2/x",
+  "ax1/line2/y",
+  "ax1/line3/x",
+  "ax1/line3/y",
+  "ax2/collection0/offsets",
+  "ax2/collection0/path_lengths",
+  "ax2/collection0/vertices",
+  "ax2/line0/x",
+  "ax2/line0/y",
+  "ax2/line1/x",
+  "ax2/line1/y",
+  "ax2/line2/x",
+  "ax2/line2/y",
+  "ax2/line3/x",
+  "ax2/line3/y",
+  "data/arv/daily_dvv_pct",
+  "data/arv/daily_err_pct",
+  "data/arv/days",
+  "data/arv/published_days",
+  "data/arv/published_dvv_pct",
+  "data/arv/trailing_days",
+  "data/arv/trailing_dvv_pct",
+  "data/ljr/daily_dvv_pct",
+  "data/ljr/daily_err_pct",
+  "data/ljr/days",
+  "data/ljr/published_days",
+  "data/ljr/published_dvv_pct",
+  "data/ljr/trailing_days",
+  "data/ljr/trailing_dvv_pct",
+  "data/rxh/daily_dvv_pct",
+  "data/rxh/daily_err_pct",
+  "data/rxh/days",
+  "data/rxh/published_days",
+  "data/rxh/published_dvv_pct",
+  "data/rxh/trailing_days",
+  "data/rxh/trailing_dvv_pct"
+ ],
+ "gate1": {
+  "rules": {
+   "band_hz": "2.0-4.0",
+   "burn_in_days": 150,
+   "trailing_days": 90,
+   "trailing_min_finite": 45,
+   "centered_days": 45,
+   "centered_min_finite": 23,
+   "join": "inner join on calendar date; published product restricted to 2018-2019",
+   "r": "Pearson, on the overlap, after removing each series' overlap mean",
+   "slope": "OLS slope of the published product on the codameter series"
+  },
+  "stations": {
+   "CI.LJR": {
+    "station": "CI.LJR",
+    "daily_rows": 681,
+    "daily_first": "2018-01-01",
+    "daily_last": "2019-12-30",
+    "legacy_rows_2018_2019": 729,
+    "burn_in_until": "2018-05-31",
+    "matched": {
+     "n": 579,
+     "first": "2018-05-31",
+     "last": "2019-12-30",
+     "r": 0.9850624256185397,
+     "rms_diff": 0.04616500786170553,
+     "slope": 1.0691502357580198
+    },
+    "matched_no_burn_in": {
+     "n": 639,
+     "first": "2018-04-01",
+     "last": "2019-12-30",
+     "r": 0.9842861169792171,
+     "rms_diff": 0.04475023506362687,
+     "slope": 1.06604616277162
+    },
+    "centered": {
+     "n": 682,
+     "first": "2018-02-16",
+     "last": "2019-12-30",
+     "r": 0.8653456948495024,
+     "rms_diff": 0.11961154037286262,
+     "slope": 0.8767149750561822
+    },
+    "raw": {
+     "n": 681,
+     "first": "2018-02-15",
+     "last": "2019-12-30",
+     "r": 0.8482459139276761,
+     "rms_diff": 0.12926786732877105,
+     "slope": 0.8270457775608157
+    }
+   },
+   "CI.ARV": {
+    "station": "CI.ARV",
+    "daily_rows": 394,
+    "daily_first": "2018-01-01",
+    "daily_last": "2019-12-30",
+    "legacy_rows_2018_2019": 729,
+    "burn_in_until": "2018-05-31",
+    "matched": {
+     "n": 357,
+     "first": "2018-05-31",
+     "last": "2019-08-17",
+     "r": 0.9045574746252278,
+     "rms_diff": 0.1499582242219106,
+     "slope": 2.1497390972385637
+    },
+    "matched_no_burn_in": {
+     "n": 390,
+     "first": "2018-04-28",
+     "last": "2019-08-17",
+     "r": 0.8991749046935936,
+     "rms_diff": 0.14417421369707928,
+     "slope": 2.01388560869008
+    },
+    "centered": {
+     "n": 424,
+     "first": "2018-03-20",
+     "last": "2019-10-31",
+     "r": 0.36862022477672945,
+     "rms_diff": 0.21417922543897486,
+     "slope": 0.7964142472626492
+    },
+    "raw": {
+     "n": 394,
+     "first": "2018-02-15",
+     "last": "2019-12-14",
+     "r": 0.3050023982841881,
+     "rms_diff": 0.23344247992602937,
+     "slope": 0.48000644317005864
+    }
+   },
+   "CI.RXH": {
+    "station": "CI.RXH",
+    "daily_rows": 563,
+    "daily_first": "2018-01-01",
+    "daily_last": "2019-12-30",
+    "legacy_rows_2018_2019": 640,
+    "burn_in_until": "2018-05-31",
+    "matched": {
+     "n": 446,
+     "first": "2018-05-31",
+     "last": "2019-12-30",
+     "r": 0.8288596923048149,
+     "rms_diff": 0.020031985799333026,
+     "slope": 0.8061950785354097
+    },
+    "matched_no_burn_in": {
+     "n": 507,
+     "first": "2018-03-31",
+     "last": "2019-12-30",
+     "r": 0.7639616357430739,
+     "rms_diff": 0.024834967608089075,
+     "slope": 0.6416129386867837
+    },
+    "centered": {
+     "n": 554,
+     "first": "2018-02-15",
+     "last": "2019-11-28",
+     "r": 0.620432199974238,
+     "rms_diff": 0.03468605759921402,
+     "slope": 0.5227569651774386
+    },
+    "raw": {
+     "n": 560,
+     "first": "2018-02-15",
+     "last": "2019-12-30",
+     "r": 0.4911444632042665,
+     "rms_diff": 0.045858139775157306,
+     "slope": 0.35447454696414116
+    }
+   }
+  }
+ }
+}