Skip to content

Reuse catalog preparation across historical import pages - #356

Draft
abhi1693 wants to merge 2 commits into
masterfrom
perf/importer-catalog-budgets
Draft

abhi1693 wants to merge 2 commits into
masterfrom
perf/importer-catalog-budgets

Conversation

@abhi1693

@abhi1693 abhi1693 commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

What changes and why?

Retained-scope CPU/preparation reductions require caching to be enabled. With DEVFEED_CACHE_ENABLED=false, the added long-lived scope is bypassed and the existing per-stage/page SQL path is retained; no retention benefit is claimed for that mode. An unavailable cache transport is distinct from disabled caching: the enabled-cache fallback already pays revision checks, and local retention avoids repeated catalog reads.

Historical card import rebuilt the full approved catalog and candidate-ranking index for every page of up to 200 identities. Retain those immutable inputs per importer, validate transactional catalog revisions on every page, and release the old generation before replacing it. Current article/job inputs and publication decisions are still rebuilt from each bounded read-only source snapshot.

Cancellation preserves exclusive ownership of the native builder until it exits; abandoned build results never publish or advance source checkpoints. Concurrent build/publication is rejected, failed writes replay with unchanged baseline identities, and finite import completion releases retained memory.

The recurring current-state reconciler already has retained scopes on master. This PR benchmarks that existing behavior and adds reuse to the missed historical caller. The production audit confirmed recurring reconciliation activity; it did not establish that historical import was active, so this does not claim to explain all observed analytics-container CPU.

Before and after measurements

Metric Before: cold historical pages After: retained historical importer Measurement context
Catalog loads 3 1 55,000 tags, 4,450 active topics, 600 cards in three 200-row pages
Ranking index preparations 3 1 Same catalog and card outputs; caching enabled, transport unavailable; real revision triggers enabled
Python process CPU 14.652 s 10.077 s One local x86_64 sample, includes builder threads; excludes PostgreSQL CPU
Builder wall time 15.178 s 10.567 s In-memory acknowledged sink; excludes ClickHouse transport and fixture insertion
SQL statement count 54 48 Enabled-cache fallback; retained pages avoid repeated catalog reads and their before/after fences
Whole-container CPU, peak memory, native ARM64 latency Pending Pending Not established by this fixture

The recurring reconciler already retained catalog/index inputs on master: the same cold control used 14.652 s Python CPU versus 9.668 s for that existing reconciler behavior. This PR benchmarks it; the new implementation change applies to the previously missed historical importer. The historical result is about 31% less builder CPU in one sequential sample with warm database pages, without alternating trials or a confidence interval. Retention keeps one catalog/index generation plus existing bounded candidate caches, releases the previous generation on change, and clears importer memory on completion. Exact memory cost and production savings remain unmeasured.

PostgreSQL impact

One sequential local x86_64 PostgreSQL 18 pair per mode, 2 CPUs/2 GiB: 55,000 tags,4,450 active topics and600active article cards in three200-row pages; cold direct builder versus retained historical importer. Values below are before → after. Cache-disabled scopes are explicitly withdrawn; existing revision/publication checks are preserved. The tested statement/row/transaction budgets do not increase. Timing correlations alone are not treated as proof of PostgreSQL CPU neutrality.

Cache mode Server SQL calls Server rows Application transactions Shared hits/reads Physical WAL bytes Server plan + execution ms
disabled 58 → 58 181954 → 181954 3 → 3 54525/29 → 54520/0 8048 → 58 575.6 → 578.1
cold 58 → 58 63062 → 63062 3 → 3 53386/27 → 53362/0 146744 → 58 446.4 → 401.4
warm 55 → 55 3610 → 3610 3 → 3 51683/27 → 51664/0 7848 → 0 291.1 → 287.4
outage 64 → 58 181966 → 63062 3 → 3 56777/27 → 53362/0 146744 → 58 573.8 → 399.5
churn 72 → 72 181968 → 181968 5 → 5 58937/27 → 58834/0 148046 → 1611 604.7 → 607.8
disabled_churn 66 → 66 181956 → 181956 5 → 5 56712/27 → 56670/0 157460 → 1512 601.1 → 591.2

Peak application checkouts are one in the serial fixture. Every ordinary-analysis specialist callback asserts zero application checkouts; historical card construction releases source transactions before sink acknowledgement. Server totals retain one fixed reset-observer call per sample. Physical buffer/WAL values are deliberately shown, including increases. They vary with fixture writes, checkpoints and checksum/hint-bit effects; there is no zero-WAL, statistically established CPU, production or ARM64 claim.

Method, exact scope and reproduction, raw server/client measurements. Cache outage is injected at the cache transport boundary; database work and worker/card paths are native.

PostgreSQL rollout gate

The tested statement, row and transaction budgets pass. Overall PostgreSQL CPU, WAL and database-slot neutrality are not certified by these sequential samples. Raw physical-WAL increases remain visible and must not be assumed to be explained entirely by checkpoint effects.

Keep this PR draft: do not merge, enable or roll out until matched-checkpoint repeat controls and native ARM64/production canary evidence meet PostgreSQL CPU, WAL-rate, backend-slot and lock/transaction-duration budgets at fixed arrivals and unchanged admission/replicas. For the importer, which has no separate enable flag, this is a merge/rollout gate. The measured cache-enabled CPU benefit is not approval to shift pressure to PostgreSQL.

Validation

  • Current PostgreSQL-guard validation: 70 importer/backfill units passed with 100% statement/branch coverage for the importer; 7 production-size native cases passed, including the full card-fidelity check and six paired cache modes at 55k tags / 4,450 topics / 600 cards. Python checks and commit hooks passed.
  • Native PostgreSQL 18 / Redis 8 / ClickHouse 26.8 import and checkpoint suites: 71 passed.
  • Focused importer, reconciler, and native-card unit suites: 70 current importer/backfill unit cases passed, 4 integration cases deselected for that unit invocation.
  • Selected importer unit statement and branch coverage: 100%.
  • Native compact catalog regression passed, then a production-size run passed with 55,000 tags, 4,450 active topics, and 600 active article cards in three 200-row pages, with caching enabled and a fault injected at the cache transport boundary. Real migrated database triggers stayed enabled. Both retained paths match every exported card field and semantic hash from the cold control.
  • Catalog loads/ranking preparations fall from 3 to 1. In one local x86_64 sample, historical builder Python CPU was 14.65s → 10.08s, wall time 15.18s → 10.57s. Client SQL statement count fell from 54 to 48 in the enabled-cache/outage fixture. This is a single construction sample, excludes PostgreSQL CPU/ClickHouse transport/telemetry, and does not predict whole-container savings. Reproduction, raw counters, and limits are in benchmarks/analytics-card-catalog.{md,json}.
  • Regression coverage checks committed raw-SQL invalidation, rollback, old-generation collection, canceled/failed build replay without checkpoints, publication ownership, source cards with zero catalog loads, and private feed-query redaction.
  • Python lint, formatting, complexity, mypy, workspace versions, and pre-commit checks passed.
  • Hosted CI and native ARM64/released-image profiling remain pending. No frontend behavior changes.

Rollout notes

No schema, cache key, policy, source-owned checkpoint, live image pin, or resource request changes. Historical card inserts keep baseline revision 1 and the existing per-policy backfill checkpoint. Existing completed installations do not replay history. The retained working set exchanges bounded process memory for repeated catalog loading/preparation, clears on revision change, and releases at import completion. No production deployment was performed.

Ready for review

  • The description explains the change and the validation results, including any gaps.
  • Relevant documentation and regression tests are updated, or the description explains why they aren't needed.

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Backend unit coverage

PASS · minimum combined coverage 98.00%.

Tests: 11967 passed, 0 failed, 0 skipped. Source files: 400.

Workspace Lines Branches Combined
All backend code 98.68% 96.05% 98.10%
apps/admin-api 96.26% 88.28% 94.93%
apps/aggregator 98.04% 94.90% 97.30%
apps/api 99.22% 98.90% 99.16%
apps/article-enrichment-worker 80.00% 50.00% 71.43%
apps/cli 99.39% 90.63% 98.13%
apps/images-worker 80.00% 50.00% 71.43%
apps/mcp 99.72% 98.17% 99.44%
apps/notifications 100.00% 100.00% 100.00%
apps/partner-api 98.70% 95.51% 98.22%
apps/search-indexer 85.48% 85.71% 85.53%
apps/source-discovery-worker 80.00% 50.00% 71.43%
apps/user-api 99.20% 97.95% 98.99%
packages/core 99.20% 97.12% 98.70%
packages/http 99.18% 97.99% 98.93%

Unit coverage includes every backend workspace. PostgreSQL/Redis integration tests run separately.

Commit: 6973265 · HTML, JSON, XML reports and job logs · attempt 1

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant