Repository navigation
Conversation
Backend unit coveragePASS · minimum combined coverage 98.00%. Tests: 11967 passed, 0 failed, 0 skipped. Source files: 400.
Unit coverage includes every backend workspace. PostgreSQL/Redis integration tests run separately. Commit: 6973265 · HTML, JSON, XML reports and job logs · attempt 1 |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



What changes and why?
Retained-scope CPU/preparation reductions require caching to be enabled. With
DEVFEED_CACHE_ENABLED=false, the added long-lived scope is bypassed and the existing per-stage/page SQL path is retained; no retention benefit is claimed for that mode. An unavailable cache transport is distinct from disabled caching: the enabled-cache fallback already pays revision checks, and local retention avoids repeated catalog reads.Historical card import rebuilt the full approved catalog and candidate-ranking index for every page of up to 200 identities. Retain those immutable inputs per importer, validate transactional catalog revisions on every page, and release the old generation before replacing it. Current article/job inputs and publication decisions are still rebuilt from each bounded read-only source snapshot.
Cancellation preserves exclusive ownership of the native builder until it exits; abandoned build results never publish or advance source checkpoints. Concurrent build/publication is rejected, failed writes replay with unchanged baseline identities, and finite import completion releases retained memory.
The recurring current-state reconciler already has retained scopes on master. This PR benchmarks that existing behavior and adds reuse to the missed historical caller. The production audit confirmed recurring reconciliation activity; it did not establish that historical import was active, so this does not claim to explain all observed analytics-container CPU.
Before and after measurements
The recurring reconciler already retained catalog/index inputs on master: the same cold control used 14.652 s Python CPU versus 9.668 s for that existing reconciler behavior. This PR benchmarks it; the new implementation change applies to the previously missed historical importer. The historical result is about 31% less builder CPU in one sequential sample with warm database pages, without alternating trials or a confidence interval. Retention keeps one catalog/index generation plus existing bounded candidate caches, releases the previous generation on change, and clears importer memory on completion. Exact memory cost and production savings remain unmeasured.
PostgreSQL impact
One sequential local x86_64 PostgreSQL 18 pair per mode, 2 CPUs/2 GiB: 55,000 tags,4,450 active topics and600active article cards in three200-row pages; cold direct builder versus retained historical importer. Values below are before → after. Cache-disabled scopes are explicitly withdrawn; existing revision/publication checks are preserved. The tested statement/row/transaction budgets do not increase. Timing correlations alone are not treated as proof of PostgreSQL CPU neutrality.
Peak application checkouts are one in the serial fixture. Every ordinary-analysis specialist callback asserts zero application checkouts; historical card construction releases source transactions before sink acknowledgement. Server totals retain one fixed reset-observer call per sample. Physical buffer/WAL values are deliberately shown, including increases. They vary with fixture writes, checkpoints and checksum/hint-bit effects; there is no zero-WAL, statistically established CPU, production or ARM64 claim.
Method, exact scope and reproduction, raw server/client measurements. Cache outage is injected at the cache transport boundary; database work and worker/card paths are native.
PostgreSQL rollout gate
The tested statement, row and transaction budgets pass. Overall PostgreSQL CPU, WAL and database-slot neutrality are not certified by these sequential samples. Raw physical-WAL increases remain visible and must not be assumed to be explained entirely by checkpoint effects.
Keep this PR draft: do not merge, enable or roll out until matched-checkpoint repeat controls and native ARM64/production canary evidence meet PostgreSQL CPU, WAL-rate, backend-slot and lock/transaction-duration budgets at fixed arrivals and unchanged admission/replicas. For the importer, which has no separate enable flag, this is a merge/rollout gate. The measured cache-enabled CPU benefit is not approval to shift pressure to PostgreSQL.
Validation
benchmarks/analytics-card-catalog.{md,json}.Rollout notes
No schema, cache key, policy, source-owned checkpoint, live image pin, or resource request changes. Historical card inserts keep baseline revision 1 and the existing per-policy backfill checkpoint. Existing completed installations do not replay history. The retained working set exchanges bounded process memory for repeated catalog loading/preparation, clears on revision change, and releases at import completion. No production deployment was performed.
Ready for review