Skip to content

feat: add community assessment tooling and workflow proposal - #4489

Open
dajiaohuang wants to merge 6 commits into
github:mainfrom
dajiaohuang:feat/4410-community-assessment-pilot
Open

feat: add community assessment tooling and workflow proposal#4489
dajiaohuang wants to merge 6 commits into
github:mainfrom
dajiaohuang:feat/4410-community-assessment-pilot

Conversation

@dajiaohuang

@dajiaohuang dajiaohuang commented Sep 9, 2026

Copy link
Copy Markdown

Description

Adds the assessment extension and retrospective baseline tooling for #4410, with the GitHub Agentic Workflow retained as an inactive proposal. No assessment or label-cleanup automation is activated by this PR: the generated lockfile and the proposed pull_request_target cleanup workflow are absent. Running the fork-PR pilot still requires maintainers to approve a trusted execution and publication context consistent with the issue's pull_request boundary.

The extension captures the assessed PR head, gathers policy/scope/validation/disclosure evidence without executing PR content, and writes a SHA-qualified local assessment. The proposed publisher checks freshness before every mutation and leaves newer revisions' outcome labels untouched. Its behavior is covered with mocked GitHub API tests; this is not evidence of a deployed pilot.

The baseline collector records a deterministic, stratified 100-PR sample with cutoff-bounded review/check evidence. Maintainer triage time and clarification rounds remain unknown. The review stage remains gated on the eight-week/50-PR assessment pilot criteria in #4410.

Testing

  • This checkout's .venv/Scripts/python -m pytest tests/test_community_assess_baseline.py tests/contract/test_wheel_core_pack_scripts.py tests/test_github_workflows.py tests/extensions/test_community_assess_extension.py -q: 20 passed, 3 skipped.
  • node --test tests/community_assess_publish.test.mjs: 6 passed, including a delayed old publisher preserving a newer head's labels.
  • Added a wheel script-discovery regression that failed before the correction and passed afterward; empty/docs/cache directories are excluded.
  • Full suite before the final test-helper correction: 7,177 passed, 497 skipped, 64 failed. The failures reported Windows symlink privilege errors (WinError 1314); this is not an all-green suite.
  • Python compile checks and git diff --check passed. No current hosted CI result or maintainer acceptance is claimed.

AI Disclosure

OpenAI Codex agents authored and reviewed the implementation, tests, workflow proposal, and audit artifacts, and executed the validation above. AI assistance included code generation and PR description drafting. No human-review or pilot-success claim is made.

Assisted-by: OpenAI Codex (model: GPT-6, autonomous)
@dajiaohuang
dajiaohuang requested a review from mnriem as a code owner September 9, 2026 12:47
@dajiaohuang dajiaohuang changed the title feat: add community contribution assessment pilot feat: add community assessment tooling and workflow proposal Sep 9, 2026
@mnriem

mnriem commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the substantial effort here — this is a thoughtful implementation, and the care around the pull_request (not pull_request_target) boundary, freshness checks, and read-only assessment is appreciated. But #4410 is still in design (feature-needs-clarification), gated on the 8-week/50-PR pilot criteria, and this is my own governance process where the design decisions need to be settled before a full implementation lands — a 1300-line PR ahead of that conversation is hard to accept as-is, per the "large changes need prior agreement" guidance in CONTRIBUTING. Could we move the discussion to #4410 first: agree the assessment schema, the label vocabulary, and the pilot mechanics, and then land an implementation matching that? I'd also want a dedicated security review of any workflow before it activates, given it runs against community-PR content. Happy to have you involved in building it once the shape is agreed. Also — please add the AI-disclosure (the body has a placeholder section but I don't see the statement filled in).

@mnriem mnriem added triage-can-wait Verdict: valid and in-scope but deprioritized; held behind the evidence gate author-awaiting Waiting on author response labels Sep 9, 2026
@dajiaohuang

Copy link
Copy Markdown
Author

Posted on behalf of @dajiaohuang by OpenAI Codex (model: GPT-5, autonomous).

Review-round update for c0912fdfa86669b5989d3f9dfa2f14f764289314: no code changes were made. The current PR body contains the requested explicit AI-disclosure statement. Focused validation remains green: 20 passed and 3 skipped across the four Python test files; 6 passed in tests/community_assess_publish.test.mjs; git diff --check passes.

The branch remains intentionally unchanged while #4410's assessment schema, label vocabulary, and pilot mechanics are settled and the requested dedicated security review is completed. No hosted checks are currently reported, and this update makes no pilot-deployment or maintainer-approval claim.

@mnriem mnriem removed the author-awaiting Waiting on author response label Sep 10, 2026
@mnriem

mnriem commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Thanks @dajiaohuang — disclosure noted, and freezing the branch while we settle #4410 is exactly the right call. The gating item is on my side: #4410 is still in clarification, so the assessment schema and the pilot decision need to land there before this gets a full review. I'll drive that on the issue; no further changes needed here in the meantime. Appreciate the care on the pull_request boundary and the read-only, no-pilot-claim framing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

triage-can-wait Verdict: valid and in-scope but deprioritized; held behind the evidence gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants