Skip to content

review-prs: link a finding to the rule it breaks, and propose a missing rule - #197

Merged
bbondy merged 1 commit into
masterfrom
review-prs-cites-rules-and-proposes-practices
Oct 20, 2026
Merged

bbondy merged 1 commit into
masterfrom
review-prs-cites-rules-and-proposes-practices

Conversation

@netzenbot

Copy link
Copy Markdown
Collaborator

User impact: a bravebot review comment from the bug pass or the project-guidance pass now links the best-practice rule it breaks, and a review can open an issue proposing a rule the project lacks.

The problem

Bravebot's bug pass and project-guidance pass are told to write findings with no rule link, and those are the High comments. Several break a documented rule: a test that passes with the fix reverted, a spec the change no longer matches. Of the reviewer's last 49 comments, 1 of 18 High comments links a rule, against all 31 Medium and Low ones. A review that sees a rule the project is missing has nowhere to say so.

Reproduce

gh api "repos/brave/bravebot/pulls/comments?per_page=100&sort=created&direction=desc" --jq '[.[] | select(.user.login=="netzenbot-reviewer") | select(.body|startswith("**Severity"))] | group_by(.body|capture("Severity: (?<s>[A-Za-z]+)").s) | map({severity: (.[0].body|capture("Severity: (?<s>[A-Za-z]+)").s), total: length, linked: (map(select(.body|test("best-practices/[a-z-]+\\.md#"))) | length)})'

Run from any directory with gh logged in. Observed today: High 1 linked of 18, Medium 11 of 11, Low 20 of 20. Expected: a High comment that breaks a documented rule links it too.

tests/test_review_rules_and_proposals.py covers the new behaviour and cannot run on the parent commit, which has none of the modules it loads.

The fix

review.linkRules has the validator, which reads the source, name the rule a finding breaks. It is given an index of the rules that PR was checked against and writes the URL into a new field, cited_rule_link, not rule_link: post-review merges two findings that share a rule_link in one file, so reusing it would turn two real findings into one comment. The collector keeps the link only if the anchor exists and takes the link text from the docs, not from the model.

**Severity: High**

This wiring is untested. The new test still passes if it is reverted to the old call. ([Show that a regression test rejects the bug](https://github.com/brave/bravebot/tree/main/docs/best-practices/tests.md#TS-003))

review.proposeBestPractices lets the validator describe one rule the project lacks. An automatic run files it as an issue after posting. A sample, rendered by the real code and not filed:

Best practice: Assert the fault a test claims to reject

<!-- review-prs best-practice proposal -->
Proposed while reviewing [PR #2768](https://github.com/brave/bravebot/pull/2768). A maintainer decides whether this becomes a rule; nothing has been added to the docs.

## Proposed rule

A test whose name or comment says it rejects a fault asserts on the observable that fault changes, so restoring the fault makes it fail.

## Why a reviewer has to decide it

A reviewer has to read the assertion against the change to see whether it would fail with the fault put back; a passing run cannot show that.

Suggested home: `tests.md`.

## Where it showed up

- [`ui/src/app.ts:3`](https://github.com/brave/bravebot/blob/0123456789abcdef/ui/src/app.ts#L3): the test builds the joined text itself, so reverting the call keeps it green

One issue per PR and three per run, none when a rule heading or an earlier issue already says it, mentions and links off the repo removed, and a failure is logged and never fails the review. Without --auto it only prints. Both keys are set in bravebot's profile and not in brave-core's.

Test plan

  • make lint — passed
  • python3 -m pytest tests/test_review_rules_and_proposals.py — 51 passed
  • python3 -m pytest tests on the rebased branch — 1451 passed, 8 failed; the same 8 fail on master without this change, and 4 of them only fail with this host's local config.json
  • make check-reviewdog — no findings from opengrep on this branch; it and pip-audit report "could not run cleanly" on this host (no landrun), so the dependency audit did not run
  • A live scheduled review on bravebot — not run here, it needs model sessions; the seams between the steps are tested, not the model's behaviour
  • CI passes cleanly
Faults put back one at a time, and the test that caught each
Fault Caught by
the validator's link stored as rule_link test_a_cited_link_that_names_a_rule_is_kept_with_the_docs_heading
a cited link kept without checking it names a rule test_a_cited_link_that_names_no_rule_is_dropped_and_the_finding_stays
the index lists every doc, not the ones the PR was checked against test_the_index_lists_the_rules_the_pr_was_checked_against
mentions in model text not defused test_mentions_are_defused_and_foreign_links_removed
no cap on issues per run test_at_most_one_per_pr_and_a_few_per_run
the bravebot profile loses linkRules test_bravebot_asks_for_both_and_brave_core_for_neither

…ng rule

The bug pass and the project-guidance pass write findings with no rule link, so
on bravebot 18 of 19 High comments carried none even where a documented rule
covered them. With review.linkRules the validator gets an index of the rules the
PR was checked against and names the rule in cited_rule_link. It is a separate
field because rule_link is a dedup key in post-review.py: two findings cited to
the same rule in one file would collapse into one comment. collect-results.py
keeps a cited link only when it names a rule in the docs and takes the link text
from them.

With review.proposeBestPractices the validator may describe one rule the project
lacks, and an automatic run files it as a "Best practice:" issue through
file-proposals.py: one per PR, three per run, skipped when a rule heading or an
earlier issue already says it, with mentions and foreign links removed.

Both keys are set for bravebot only.
@bbondy
bbondy merged commit 770a001 into master Oct 20, 2026
3 checks passed
@bbondy
bbondy deleted the review-prs-cites-rules-and-proposes-practices branch October 20, 2026 23:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants