Skip to content

Avoid unused ingestion taxonomy ORM loads - #347

Draft
abhi1693 wants to merge 2 commits into
masterfrom
perf/ingestion-lazy-catalogs
Draft

abhi1693 wants to merge 2 commits into
masterfrom
perf/ingestion-lazy-catalogs

Conversation

@abhi1693

@abhi1693 abhi1693 commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

What changes and why?

AI-enabled feed ingestion hydrated every active Topic and every Tag before skipping heuristic classification. Repeated HTTP 200 feeds paid the same ORM cost even when every origin was already retained. Load these catalogs lazily only when a new entry reaches the heuristic classifier, and share that one pair of loads across the eligible feed entries.

Publisher-supplied tag resolution and restoration remain independent: repeated imports still restore explicit labels, and manual/AI classifications retain ownership. No classification or publication policy changes.

Validation

PostgreSQL audit against the actual baseline/current path, using a separate local x86-64 fixture with 4,450 active topics and 55,000 tags:

PostgreSQL metric / workload Before After Measurement basis
AI new import: app SQL / returned rows 18 / 59,460 16 / 10 pg_stat_statements and SQLAlchemy events, two matching feed entries; SQL execution 106.10→29.68ms.
Retained import: app SQL / returned rows 4 / 59,452 2 / 2 AI and heuristic modes; no catalog scan remains. AI sample buffers 1,412→145, SQL 70.35→0.05ms.
Eligible heuristic import: SQL / returned rows 18 / 59,460 18 / 59,460 Same SQL work. One paired SQL elapsed sample 95.36→114.60ms; no speedup claim for this path.
Transactions / checkouts / existing article row-lock statements 1 / 1 / 2 new, 0 retained Same AI retained open-transaction time 1,332.3→6.6ms; complete checkout/lifetime counters recorded.

Raw PostgreSQL statements and counters and protocol/limits include server buffers, planning/execution elapsed and observed container CPU. These are single paired samples with warm analyzed data; cgroup read-probe/background cost is retained, not presented as isolated backend CPU. The final two-case native regression also asserts actual catalog SQL row counts. This PR adds no cache, invalidation read, transaction or replacement PostgreSQL computation.

Metric Before After Measurement basis
Heuristic catalog ORM objects, AI-enabled import 20,000 from the prior unconditional loads 0 observed Local x86-64 PostgreSQL fixture: 10,000 active topics, 10,000 tags, two entries; SQLAlchemy load events. Before count follows the old loads, not a timed old-head run.
Heuristic catalog ORM objects, repeated known-origin import 20,000 from the prior unconditional loads 0 observed Same fixture, both AI and heuristic modes; repeat creates no articles.
Heuristic catalog ORM objects, eligible first import 20,000 20,000 observed, loaded once per feed AI disabled; both new entries still reach the classifier.
Total worker CPU / peak RSS Not measured Not measured; native ARM64 comparison pending Object-load counts establish removed work, not measured worker CPU or memory savings. No retained cache is introduced; peak RSS remains unmeasured.
  • 32 disposable-service integration cases pass across ingestion, source-tag restoration/concurrency and multilingual handling.
  • 102 source/language unit cases pass. Python lint, formatting, types and workspace versions pass across 400 modules.
  • A PostgreSQL fixture with 10,000 active topics and 10,000 tags observes zero heuristic catalog ORM hydration for AI ingestion and retained origins. AI-disabled new entries load 20,000 objects once per feed and preserve existing classification behavior. See measurement.
  • Local validation is x86-64. Hosted CI, native ARM64 CPU/RSS comparisons and production before/after throughput remain pending; object-load counts are not a total-worker savings claim.
  • Prior-head (f2eb29cd) hosted integration shard 1 failed in the unchanged 100,000-history migration test: analytics cutover refused an exclusive NOWAIT lock on article_analysis_jobs. The hosted lock holder is unknown. That focused test passed in two local disposable-service runs, including an aggressive-autovacuum run with lock diagnostics. A targeted failed-job rerun was unavailable while the workflow remained active. The new audit/test head has fresh CI pending. This PR remains draft; no speculative migration or production-lock change was made.

Rollout notes

No migration or cache is added. This change removes unused reads; image, source authority, leases, rollback/retry and editorial guards remain in the existing ingestion transaction. Kept draft for native resource validation. No production deployment performed.

Ready for review

  • Regression coverage and measurement describe the changed behavior.
  • Current-head required CI and CodeQL pass; native workload validation is recorded before moving out of draft.

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Backend unit coverage

PASS · minimum combined coverage 98.00%.

Tests: 11961 passed, 0 failed, 0 skipped. Source files: 400.

Workspace Lines Branches Combined
All backend code 98.68% 96.04% 98.10%
apps/admin-api 96.26% 88.28% 94.93%
apps/aggregator 98.05% 94.84% 97.29%
apps/api 99.22% 98.90% 99.16%
apps/article-enrichment-worker 80.00% 50.00% 71.43%
apps/cli 99.39% 90.63% 98.13%
apps/images-worker 80.00% 50.00% 71.43%
apps/mcp 99.72% 98.17% 99.44%
apps/notifications 100.00% 100.00% 100.00%
apps/partner-api 98.70% 95.51% 98.22%
apps/search-indexer 85.48% 85.71% 85.53%
apps/source-discovery-worker 80.00% 50.00% 71.43%
apps/user-api 99.20% 97.95% 98.99%
packages/core 99.20% 97.12% 98.69%
packages/http 99.18% 97.99% 98.93%

Unit coverage includes every backend workspace. PostgreSQL/Redis integration tests run separately.

Commit: f7669f3 · HTML, JSON, XML reports and job logs · attempt 1

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant