Skip to content

Separate search projection epochs from public content cache invalidation - #353

Draft
abhi1693 wants to merge 2 commits into
masterfrom
perf/search-cache-epochs
Draft

abhi1693 wants to merge 2 commits into
masterfrom
perf/search-cache-epochs

Conversation

@abhi1693

@abhi1693 abhi1693 commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

What changes and why?

PostgreSQL assessment: this is not approved as a zero-overhead optimization under the user's constraint. The new provenance boolean adds approximately one WAL byte per newly inserted outbox event, and migration 0041 requests a transient exclusive DDL lock with a five-second wait deadline. Normal-workload SQL reductions do not erase those costs. Keep this PR in draft and DEVFEED_SEARCH_CACHE_ISOLATION_ENABLED=false; merging or enabling is blocked pending review of that tradeoff and a representative canary.

Every completed search-index batch currently invalidates the same public generation used by article, feed, and catalog responses. Split index publication into its own epoch, and make Typesense search-result cache entries depend atomically on both the public-data and search-index epochs.

The outbox now records conservative provenance: existing, trigger-created, manual and unknown events default to public_change=true. Those batches retain public-data invalidation, including the existing fallback for raw SQL writers. Only indexer-generated projection/fanout pages explicitly carry false. Repair backfills invalidate public data once after enqueue commit and mark their pages projection-only. This preserves source withdrawal and unknown-maintenance safety while removing repeated unrelated cache eviction from index-only work.

The default-off rollout gate preserves old public API replicas during a mixed-version upgrade. Updated search routes always understand both epochs; enable isolation only after every public API and indexer replica is compatible.

Before/after PostgreSQL measurements

Local x86-64 PostgreSQL 18, session-mode PgBouncer and Redis; three samples per
isolation mode, alternating order. Each sample includes the producer insertion,
indexer processing/acknowledgement and public API read for 20 projection-only pages.
The baseline is isolation off, which retains the old broad invalidation behavior.
A separate observer role captures PostgreSQL utility/nested statements, driver
configuration, transaction commands and pooler resets. Warm-up is excluded. All
samples use the same hydrated response and preserve the raw-writer/publication fences.

Metric Before After Measurement basis
Public API cache outcomes 20 MISS 20 HIT 20 projection-only pages per mode, three samples.
API data SELECTs / all recorded PG commands 20 / 360 0 / 260 Entire producer + indexer + API stage; no extra indexer query replaces the saved reads.
Rows returned/affected / shared buffer hits 100 / 720 80 / 660 Includes inserted/acknowledged outbox rows.
PG planning + execution elapsed 28.449 ms 24.192 ms Medians; elapsed time is not database CPU.
PostgreSQL-container CPU 152.606 ms 131.571 ms Median cgroup delta, includes logging, background work and probes.
Read transaction rollbacks / pooler resets 20 / 60 0 / 40 Producer/indexer retain 40 BEGIN and 40 COMMIT in both modes.
Real Redis outage: commands / rows / buffer hits 360 / 100 / 720 360 / 100 / 720 Owned Redis paused; 20 API bypasses in each mode, three samples.
Real Redis outage: PG-container CPU range 150.297–172.076 ms 152.390–164.025 ms Overlapping ranges with identical statement/row/buffer work.
Public churn: commands / rows / buffer hits 360 / 100 / 720 360 / 100 / 720 Public epoch invalidated before every read; 20 misses in both modes, three samples.
Public churn: PG-container CPU range 149.617–163.599 ms 149.616–166.850 ms Overlapping ranges; no fallback SQL added by isolation.
Total workload WAL 4,440–4,539 B 4,440–4,539 B Observed range in all three scenarios on the new schema, including outbox writes and pruning.
Backend startups / remaining locks / idle transactions 0 / 0 / 0 0 / 0 / 0 Warmed session pool; same in-transaction advisory/table lock footprint sampled during remote sync.

These gate-off/on results isolate runtime invalidation behavior on the new schema.
They do not hide the additive schema cost below, establish an across-the-board
PostgreSQL CPU reduction under failure, or predict whole-production container savings.
The degraded cases retain equal SQL work; they are not an outage capacity guarantee.

Migration and ongoing outbox storage

A fresh disposable database at revision 0040 held 103,000 events before the actual
Alembic upgrade. Before/after checks matched every ID/xmin checksum, relation filenode
and the 7,888,896-byte heap. All old rows read public_change=true. No heap rewrite
or row-by-row backfill occurred. A deliberately held reader made the requested
AccessExclusiveLock wait and fail at the migration's five-second lock deadline;
a subsequent uncontended upgrade succeeded.

Metric Before After Measurement basis
Retained event heap / IDs / xmin 7,888,896 B / 103,000 unchanged Same One actual 0040→0041 upgrade; filenode and row checksum identical.
Migration commands / WAL Not applicable 8 / 2,153 B Actual Alembic upgrade; ALTER TABLE metadata used 2,020 B and version-row update 133 B.
Migration lock Reader holds AccessShareLock AccessExclusiveLock requested; blocked attempt timed out at 5.038 s Deliberate contention in the disposable database; this is not a production lock-duration estimate.
INSERT commands / rows 1 / 1,000 1 / 1,000 Three 1,000-event samples before/after, raw SQL, no new lookup or backfill.
INSERT WAL 172,250 B median 173,250 B median Approximately one extra WAL byte per new event; both true and false provenance measured. B-tree/sequence page boundaries also affect individual samples.
INSERT planning + execution elapsed 4.676 ms 4.709 ms true / 4.743 ms false Three samples per form; small timing differences are not proof of zero overhead.
INSERT PG-container CPU 24.045 ms 23.528 ms true / 23.890 ms false Medians, probe/background-inclusive; no whole-production CPU inference.

The boolean provenance has a concrete storage/WAL cost, even while the fixed runtime
workload avoids API SQL. Keep the PR in draft and isolation disabled until migration
contention, native ARM64 and a representative production canary have been reviewed.
Do not weaken raw-SQL or source-withdrawal fences to remove this byte. Rollout still
requires every public API and indexer to support both epochs before enabling isolation.

Validation

  • 266 cache/outbox/API-policy/contract/observability unit cases pass; one existing integration-marked case is skipped in this unit invocation.
  • 44 broad disposable-service integration cases pass across cache behavior, schema history, partnership schema, article-maintenance migrations and a populated search-provenance migration. A further focused cache rerun passes all 17 cases including raw-SQL source rejection.
  • Current focused regressions pass 21 cache integration cases through session PgBouncer and the strengthened no-heap-rewrite migration case through direct PostgreSQL.
  • Twenty projection-only pages cause twenty unrelated public API misses and SQL reads with the gate off; the same reads remain cached and perform zero SQL with isolation enabled. This is a controlled avoided-work demonstration, not an estimate that all observed production misses would disappear.
  • Redis Lua checks both epochs and loader ownership atomically; tests cover index/public invalidation, raw SQL name and approval changes, old-publisher rejection, explicit repair backfills and unchanged retained event IDs/xmin across a 10,000-row upgrade/downgrade.
  • Python lint, formatting, types and versions pass across 400 source modules; Compose contracts, whitespace and commit hooks pass. Hosted CI, native ARM64 and representative production CPU/cache/freshness measurement remain pending.

Rollout notes

Requires the single additive migration 0041_search_cache_publication.py. Keep DEVFEED_SEARCH_CACHE_ISOLATION_ENABLED=false until all public APIs and indexers are upgraded; then enable and compare cache hit rate, API SQL/CPU and search freshness. Source business events remain conservative, so this does not remove every indexer-originated public invalidation.

Disable the gate before an older API rollback. Keep the additive column until compatible image rollback is complete. No TTL, replicas, polling interval or production deployment changes are included. The cache epoch guide documents provenance, raw-writer fallback and rollout boundaries.

Ready for review

  • Implementation, correctness regressions and conservative rollout are documented.
  • Native ARM64 workload and freshness gates pass before leaving draft.
  • Current-head required CI and CodeQL pass before merge.

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Backend unit coverage

PASS · minimum combined coverage 98.00%.

Tests: 11971 passed, 0 failed, 0 skipped. Source files: 400.

Workspace Lines Branches Combined
All backend code 98.68% 96.05% 98.10%
apps/admin-api 96.26% 88.28% 94.93%
apps/aggregator 98.04% 94.90% 97.30%
apps/api 99.22% 98.90% 99.16%
apps/article-enrichment-worker 80.00% 50.00% 71.43%
apps/cli 99.39% 90.63% 98.13%
apps/images-worker 80.00% 50.00% 71.43%
apps/mcp 99.72% 98.17% 99.44%
apps/notifications 100.00% 100.00% 100.00%
apps/partner-api 98.70% 95.51% 98.22%
apps/search-indexer 85.48% 85.71% 85.53%
apps/source-discovery-worker 80.00% 50.00% 71.43%
apps/user-api 99.20% 97.95% 98.99%
packages/core 99.20% 97.12% 98.70%
packages/http 99.18% 97.99% 98.93%

Unit coverage includes every backend workspace. PostgreSQL/Redis integration tests run separately.

Commit: e68cfb4 · HTML, JSON, XML reports and job logs · attempt 1

@abhi1693 abhi1693 mentioned this pull request Oct 31, 2026
1 of 2 tasks
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant