Skip to content

Improve JSON.MERGE ancestor filtering - #132

Open
gkorland wants to merge 1 commit into
valkey-io:unstablefrom
gkorland:gkorland-json-merge-performance
Open

gkorland wants to merge 1 commit into
valkey-io:unstablefrom
gkorland:gkorland-json-merge-performance

Conversation

@gkorland

Copy link
Copy Markdown

JSON.MERGE previously performed pairwise path_is_ancestor comparisons across all matched JSON pointers (O(P^2 * L)), causing quadratic scaling when wildcard or recursive-descent paths matched many targets.

Replace the nested loop with filter_topmost_merge_targets, which sorts matched pointers with '/' ordered before all token bytes and filters descendants in a single linear prefix scan (O(P * L * log P)). Because literal slashes inside JSON Pointer tokens are escaped as ~1 per RFC 6901, ordering '/' first keeps every ancestor contiguous with its entire subtree even when sibling keys share a prefix (for example "a!" or "a-" alongside "a").

Regression and scaling tests in dom_test.cc cover tricky prefix boundaries (escaped slashes/tildes, empty keys, embedded NULs) and deterministic comparison-count bounds across both disjoint targets and multi-level overlapping subtrees.

Local benchmark: End-to-end dom_merge_value median timings (5 runs after warmup, matching -O3 -DNDEBUG Release builds on arm64):

Workload Matched paths (P) unstable PR Speedup
Disjoint ($.users[*].profile) 1,000 4.97 ms 1.72 ms 2.9x
Disjoint ($.users[*].profile) 10,000 107.81 ms 24.32 ms 4.4x
Disjoint ($.users[*].profile) 25,000 765.79 ms 54.23 ms 14.1x
Overlapping ($..a) 4,000 35.30 ms 5.47 ms 6.5x
Overlapping ($..a) 10,000 212.78 ms 14.23 ms 15.0x
Overlapping ($..a) 20,000 472.47 ms 29.51 ms 16.0x

Timings are local microbenchmarks; actual gains may vary by workload and environment.

Replace pairwise ancestor checks in dom_merge_value with a sorted-prefix scan over matched JSON pointers, ordering '/' before token bytes so each ancestor precedes its entire subtree.

Add unit tests covering prefix boundaries, escaped keys, and O(P log P) comparison scaling across disjoint targets and overlapping subtrees.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Guy Korland <gkorland@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 15, 2026 06:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The JSON.MERGE performance optimization has no active review issues.

No open issues 🎉

What changed in this PR

Improves JSON.MERGE performance when paths match many targets, while preserving ancestor-wins behavior for overlapping selections.

Changes:

  • Replaces quadratic ancestor comparisons with separator-aware sorting and a prefix scan, reducing filtering complexity to O(P * L * log P).
  • Adds regression tests for pointer boundaries, including escaped characters, empty keys, and embedded NULs.
  • Adds comparison-count scaling checks for disjoint targets and overlapping subtrees.
File Description
tst/​unit/​dom_test.cc Adds boundary-case, merge-correctness, and scaling tests.
src/​json/​dom.cc Introduces topmost-target filtering, integrates it into merge processing, and updates complexity documentation.

Review effort: Balanced

@coderabbitai

coderabbitai Bot commented Nov 15, 2026

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

dom_merge_value now filters matched merge targets to retain only topmost paths before applying updates. Unit tests cover path boundaries, comparison scaling, and merge results for disjoint and overlapping targets.

Changes: Merge Target Filtering

Layer / File(s) Summary
Filter targets and apply merges
src/json/dom.cc
dom_merge_value sorts matched paths, removes paths covered by retained ancestors, and applies updates to the remaining targets. The documented complexity now includes sorting and scanning.
Validate filtering and merge behavior
tst/unit/dom_test.cc
Tests cover punctuation, empty keys, embedded NULs, comparison scaling, and patching only topmost overlapping targets.

Suggested reviewers: patrickn2, roshkhatri

Change: Refactor

Merge Risk: Minimal · up to 214cd

JSON.MERGE now filters matched targets to the topmost paths with a sort and prefix scan instead of pairwise checks. The merge results for retained targets are unchanged, and the new tests cover path boundaries, escaped keys, and scaling. No merge-blocking risk was identified.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: JSON.MERGE ancestor filtering is improved. The PR replaces pairwise ancestor checks with filter_topmost_merge_targets, a sort-and-prefix-scan filter. The title is spec…
Description check Passed The description explains the change. It states the old O(P^2 * L) pairwise ancestor check, the new filter_topmost_merge_targets sort-and-scan approach, the escaped-slash ordering rationale, the regres…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: JSON.MERGE ancestor filtering is improved. The PR replaces pairwise ancestor checks with filter_topmost_merge_targets, a sort-and-prefix-scan filter. The title is specific and matches the summary and objectives.

Full details: Description check

Explanation

The description explains the change. It states the old O(P^2 * L) pairwise ancestor check, the new filter_topmost_merge_targets sort-and-scan approach, the escaped-slash ordering rationale, the regression and scaling tests in dom_test.cc, and local benchmark tables. It matches the changeset.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 74cecbf5-7be7-4898-bd93-a3346f1f52cf
Commits

Reviewing files that changed from the base of the PR and between b0b6b63 and 214cddd.

Files selected for processing: 2

  • src/json/dom.cc
  • tst/unit/dom_test.cc

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants