Skip to content

Reuse source tag identities across worker processes - #354

Draft
abhi1693 wants to merge 3 commits into
masterfrom
perf/source-tag-shared-identities
Draft

abhi1693 wants to merge 3 commits into
masterfrom
perf/source-tag-shared-identities

Conversation

@abhi1693

@abhi1693 abhi1693 commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

What changes and why?

RQ work horses repeatedly fetch and normalize the same source-tag catalog. Add an opt-in, bounded shared identity projection, with a dedicated namespace and PostgreSQL transactional revisions as the authority. Only name/slug/alias→UUID identities are cached; article/editorial decisions remain current SQL reads.

Ingestion prefetches Redis bytes and decodes them before opening its final SQL transaction. A scoped context carries the candidate and is cleared on every exit. Inside SQL, matching revisions are checked before/after reuse. Callers without prefetch use baseline SQL and perform no shared GET inside their transaction. Borrowed-Connection Sessions skip publication; explicitly Connection-bound ingestion factories skip prefetch too, because their external transaction can outlive the Session. Cold compression/publication runs after the root transaction and connection check-in; explicit rollback cancels it and an open breaker skips serialization. Cache failures do not fail an already committed delivery, while RQ deadlines propagate.

Validation

Metric / workload Before After Measurement basis
Resolver SQL / rows / shared buffers, fresh-worker hit 3 / 55,002 / 11,703 2 / 2 / 74 Actual baseline/current local x86-64 PostgreSQL 18/Redis fixture, 55,000 tags with one alias each; one paired sample, slot time 1,539.9→7.3ms.
Resolver cold/outage/churn/corrupt fallback 3 SQL / 55,002 rows / 1 transaction Same No replacement PostgreSQL calculation/scan/transaction/lock; complete buffer, elapsed and lifetime observations linked below.
Whole tagged ingestion on shared hit: SQL / rows 30 / 114,473 29 / 59,473 Same feed and taxonomy fixture; shared buffers 16,068→4,582, slot 3,384.3→1,779.5ms.
Whole-ingestion transactions / checkouts / row-lock statements 2 / 2 / 7 2 / 2 / 7 Cold, warm, cache outage, alias edit and corrupt-payload conditions.
Earlier resolver app CPU / normalizations 1.879 s / 110,001 0.328 s / 1 Earlier one-sample cold versus fresh-worker warm observation; not total-worker CPU or production sizing. Final lifecycle pressure audit is separate below.
Shared payload / decoded serialization Empty before publication 2,424,003 / 6,291,220 bytes One latest-generation payload at 55,000 tags; these bytes are not Python RSS.

Negative and variable observations remain explicit:

Condition / metric Before After Measurement basis
Whole-ingestion cold PostgreSQL-container CPU 528.4 ms 627.9 ms Single ordinary sample with background maintenance and fixed read probes; matching SQL work is not a CPU-equivalence certification.
Whole-ingestion corrupt-payload PostgreSQL-container CPU 224.5 ms 236.7 ms Same ordinary-cohort caveat; slot time fell 3,391.8→3,242.6ms.
Quiet-cohort outage median CPU 86.31 ms 86.87 ms Three samples/side, autovacuum disabled only in disposable PostgreSQL, checkpoint before each target; probe/background costs remain.
Quiet-cohort corrupt-payload median CPU 87.18 ms 92.86 ms Three samples cannot establish equivalence.
Quiet-cohort alias-churn median slot occupancy 1,466.3 ms 1,483.7 ms Upward 17 ms observation retained; matching fallback work is not a universal zero-cost claim.
  • A 46-case disposable-service integration suite and a final four-case owned/borrowed connection selection pass, including 55k-tag projection reuse, current manual classifications, raw SQL/rollback invalidation and concurrent failed-fence fallback within the baseline three-read budget. Native checks assert no checked-out SQL connection during ingestion Redis GET/decode or deferred publication, including warm/corrupt/unavailable transports.
  • 136 selected source-tag/ingestion/language unit cases pass. Python lint, formatting, types and versions pass across 400 modules; Compose contracts pass.
  • 22 controlled baseline/current pressure scenarios plus 30 quiet-cohort samples record actual SQL/rows/buffers, planning/execution elapsed, connection/transaction lifetime, locking and container CPU. See raw counters including negative observations and full tables, protocol and limits.
  • Hosted current-head CI/CodeQL, native ARM64 CPU/RSS and sustained realistic PostgreSQL/slot canary validation remain pending. No blanket PostgreSQL CPU-neutrality claim is made.

Rollout notes

DEVFEED_SOURCE_TAG_SHARED_CACHE_ENABLED remains false. Do not enable until native ARM64 concurrent-import canaries establish PostgreSQL CPU/slot and worker-memory budgets, shared hit ratio, catalog edit rate and Redis payload/connection limits. Frequent edits can erase reuse and add application publication cost. The dedicated flag restores process-local behavior without disabling public API caching. No migration, production configuration change or deployment performed.

GETRANGE and bounded decompression enforce 4 MB wire/16 MB decoded/100,000-tag limits; malformed, stale, oversized and unavailable data falls back to baseline SQL. The feature never replaces current article approval/classification reads.

Ready for review

  • Correctness, cache failure/race coverage and production-count local PostgreSQL audit included.
  • Current-head required CI/CodeQL and native workload CPU/slot/RSS gates pass before moving out of draft or enabling.

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Backend unit coverage

PASS · minimum combined coverage 98.00%.

Tests: 11995 passed, 0 failed, 0 skipped. Source files: 400.

Workspace Lines Branches Combined
All backend code 98.68% 96.04% 98.10%
apps/admin-api 96.26% 88.28% 94.93%
apps/aggregator 98.05% 94.90% 97.31%
apps/api 99.22% 98.90% 99.16%
apps/article-enrichment-worker 80.00% 50.00% 71.43%
apps/cli 99.39% 90.63% 98.13%
apps/images-worker 80.00% 50.00% 71.43%
apps/mcp 99.72% 98.17% 99.44%
apps/notifications 100.00% 100.00% 100.00%
apps/partner-api 98.70% 95.51% 98.22%
apps/search-indexer 85.48% 85.71% 85.53%
apps/source-discovery-worker 80.00% 50.00% 71.43%
apps/user-api 99.20% 97.95% 98.99%
packages/core 99.19% 97.09% 98.68%
packages/http 99.18% 97.99% 98.93%

Unit coverage includes every backend workspace. PostgreSQL/Redis integration tests run separately.

Commit: 60104b9 · HTML, JSON, XML reports and job logs · attempt 1

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant