Skip to content

Comment on a pull request what failed in CI and how to fix it, and welcome a first pull request - #1203

Merged
felipesauer merged 5 commits into
mainfrom
comment-what-failed-on-pull-requests
Nov 12, 2026
Merged

felipesauer merged 5 commits into
mainfrom
comment-what-failed-on-pull-requests

Conversation

@felipesauer

Copy link
Copy Markdown
Member

What

When CI is over for a pull request, one comment on the pull request says what failed and how to fix it, rewritten in place after every run, and a contributor's first pull request is welcomed.

  • what-failed job in ci.yml (pull_request, read-only token: actions: read, contents: read; no secret). It waits for every other job and writes a small JSON artifact, what-failed: the failing jobs and steps (from the API), the why-it-went-red verdict of each cell, the failing tests with what each said (from the vitest JSON reports; the guards job now writes one too), and, when a Lint step failed, the files biome check --write changes and the diff (cut at 30,000 characters). It is continue-on-error, so it never turns a run red.
  • .github/workflows/what-failed.yml (workflow_run on CI completed; pull-requests: write is its only write, plus actions: read to download the other run's artifact and contents: read for the checkout of the default branch). It never checks out or runs the pull request's code: it checks out the default branch only (no ref, no repository, no credential kept), installs nothing, downloads the artifact under $RUNNER_TEMP, and runs .github/what-failed/comment.mjs on the event file and the report, with no ${{ }} in the shell line.
  • comment.mjs reads the report with JSON.parse only, refuses it whole above 256 KiB or if a field is off its shape or limit (unknown fields included), and puts every text of it inside a fenced block longer than any backtick run in it, with control, zero-width and bidi characters removed and a zero-width space after every @. It finds the pull request by the run's head SHA through the API (workflow_run.pull_requests is empty for forks) and keeps one comment, found by a hidden marker and by being authored by github-actions[bot]. On FIRST_TIME_CONTRIBUTOR / FIRST_TIMER it opens with a welcome naming the author by the login the API gave (the only mention), pointing at CONTRIBUTING.md and pnpm check:contribution. A green run updates an existing comment to say so; it opens a new one only to welcome a first pull request.
  • Documented in .github/what-failed/README.md, with what it does not cover; a bullet in CONTRIBUTING.md and a changelog entry.

Every action is pinned by commit SHA. No secret and no pull_request_target anywhere.

Tests

  • the-comment-runs-nothing-of-the-fork.test.ts (reads the workflow files): the only workflow_run workflow is started by nothing else, holds no permission at file level and only actions: read, contents: read, pull-requests: write in its job; no checkout with ref/repository, credentials not kept; the only shell line is the one node .github/what-failed/comment.mjs call; the artifact goes outside the workspace; actions pinned; comment.mjs parses the report through readReport and runs no code (no eval, Function, dynamic import, only gh as a child); the CI job holds only reads and needs every other job. A last case feeds the reader each defect (checkout of the head, a write, a second trigger, a pasted expression, an install, a download into the workspace, a tag).
  • the-pull-request-is-told-what-failed.test.ts: the reader refuses non-JSON, oversize, unknown fields, an over-limit message, a climbing or dashed path; the builder, given an @everyone, HTML, a </details><script>, a fence breaker and the marker itself, keeps all of it inside fences, lets no @ of the report notify, carries no HTML outside a fence, and stays under 60,000 characters; tell with a fake API finds the pull request by head SHA, rewrites its own comment and never one somebody else forged with the marker, skips a moved head and a cancelled run; the collector's output is accepted by the reader and stays within its limits.

Mutation

Put a checkout of the head back in .github/workflows/what-failed.yml (ref: ${{ github.event.workflow_run.head_sha }} on the checkout step): the-comment-runs-nothing-of-the-fork goes red with .github/workflows/what-failed.yml: job comment checks out something else. Restored, green.

Not proven here

GitHub starts a workflow_run workflow only from the file on the default branch, so the comment is first seen working on the first pull request after this merges.

…lcome a first pull request

CI's last job writes a small report with the run's read-only token: the failing
jobs and steps, the why-it-went-red verdict, the failing tests with what each
said, and for a failed lint the biome --write command and its diff. A separate
workflow on workflow_run reads it as data, without checking out or running
anything of the pull request, finds the pull request by the run's head SHA and
keeps one comment on it, rewritten after every run.
…every control and format character, and skip Dependabot's runs

Two open pull requests can share a head SHA; the run belongs to one head
repository and branch, and only that pull request is commented on. The text of
the report loses every Unicode Cc and Cf character (tag characters and the
Arabic letter mark included) and the line and paragraph separators, keeping the
tab and the line end. A run Dependabot started gets a read-only token, so the
comment job is skipped for it.
… kept tab, the kept line end and the event with no head branch

The report's text also loses U+E0000 to U+E01EF whole, unassigned points and
supplementary variation selectors included; U+FE00 to U+FE0F stay for emoji.
A case holds that a tab and a line end survive into the lint diff, and one that
an event naming no head repository or branch comments nothing. The comments
and the page say the pull request is found by head SHA, repository and branch,
and the page names the limit of one branch open against two bases.
@felipesauer

Copy link
Copy Markdown
Member Author

Review: approved at e5e301e.

An independent review read the change against a hostile fork: nothing of the fork is checked out, installed or run in the job that holds pull-requests: write; the report is parsed under a strict schema and size limit; text from it is fenced, mentions are neutralised, control, format and tag characters are stripped, and the comment is found by its marker and its author. The pull request is matched by head SHA, repository and branch. Each guard was checked by putting the defect back and watching it fail.

This merges after #1204, which adds the pnpm check:contribution command the welcome points to.

@felipesauer
felipesauer merged commit 315f5bb into main Nov 12, 2026
28 checks passed
@felipesauer
felipesauer deleted the comment-what-failed-on-pull-requests branch November 12, 2026 05:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant