feat(Classifier): Initialize with skill and defined agent

.claude/skills

@@ -0,0 +1 @@
+../skills

.cursorrules

@@ -0,0 +1 @@
+AGENTS.md

.github/copilot-instructions.md

@@ -0,0 +1 @@
+../AGENTS.md

.opencode/agents/flaky-test-investigator.md

@@ -0,0 +1,80 @@
+---
+description: Investigates a failing Juju charm integration test from a GitHub Actions run/job link and reports whether it is flaky or a real failure. Invoke with a GitHub Actions URL.
+mode: subagent
+temperature: 0.1
+permission:
+  edit: deny
+  read: allow
+  glob: allow
+  grep: allow
+  # Allowlist: deny everything, then allow read-only commands.
+  # opencode applies the last matching rule, so "*" must come first.
+  bash:
+    "*": deny
+    "gh run view*": allow
+    "gh run list*": allow
+    "gh pr view*": allow
+    "gh pr diff*": allow
+    "gh api *": allow
+    # gh api switches to POST when fields are passed; keep it read-only.
+    "gh api *-X*": deny
+    "gh api *--method*": deny
+    "gh api *-f *": deny
+    "gh api *-F *": deny
+    "gh api *--field*": deny
+    "gh api *--raw-field*": deny
+    "gh api *--input*": deny
+    "git log*": allow
+    "git show*": allow
+    "git diff*": allow
+    "git blame*": allow
+    "git remote -v": allow
+    "grep *": allow
+    "rg *": allow
+    "cat *": allow
+    "head *": allow
+    "tail *": allow
+    "wc *": allow
+    "ls*": allow
+    "jq *": allow
+  webfetch: deny
+  skill: allow
+---
+
+You investigate failing Juju charm integration tests and classify them as flaky or real failures.
+
+Input: a GitHub Actions URL (run, job, or failed step) of a failing integration test.
+
+## Procedure
+
+1. **Load the diagnosis skill** — use the `skill` tool to load `integration-test-diagnosis` and follow its procedure for this investigation.
+2. **Fetch the CI run with `gh`** — always use the `gh` CLI, never the web. Assume `gh` is installed and authenticated; do not verify its existence or auth status (no `which gh`, `gh --version`, or `gh auth status`) — just use it directly:
+   - Parse the run ID from the URL and the repo from the URL or git remotes.
+   - `gh run view <run-id> --repo <owner/repo>` for the run overview (workflow, branch, commit, event).
+   - `gh run view <run-id> --repo <owner/repo> --log-failed` for the failing step's logs.
+   - `gh api` for job metadata if you need the runner image or labels.
+3. **Gather context** per the skill's step 1: the triggering commit or PR diff (`gh pr diff`, `git log`), the failing test name, and environment details from the logs (juju/charm versions, runner image, spread setup).
+4. **Check flakiness signals** — a failure is flaky when it is not deterministic:
+   - Did the same test pass on the same commit in another run or re-run?
+   - Does the failure pattern (timeout, resource exhaustion, transient network error, service not yet ready) point to environment or timing rather than logic?
+   - Is the triggering change unrelated to the failing test's functional area?
+   - Does this test have a history of intermittent failures on this branch?
+5. **Hypothesize the root cause** using the root cause categories in `docs/taxonomy.md` (Axis 2).
+
+## Report
+
+Classify using the taxonomy in `docs/taxonomy.md` (source of truth for all
+values). Report verdict and root cause as **independent** fields — do not
+collapse them. A `FLAKY` verdict with `code_regression` root cause is still
+a real bug (recommend `block_merge`), not something to dismiss.
+
+Return a single report with:
+
+- **Verdict**: `REAL_FAILURE` / `FLAKY` / `INCONCLUSIVE`
+- **Root cause category**: `code_regression` / `code_preexisting` / `test_defect` / `workload_defect` / `environment_infra` / `dependency_drift` / `external_service` / `unknown`, with a one-line explanation
+- **Confidence**: a percentage (0-100) plus its bucket (`high`/`medium`/`low`), computed from evidence completeness per the formula in `docs/taxonomy.md` — e.g. `65% (medium)`. Do not report a percentage that isn't traceable to which signals were present, missing, or contradictory.
+- **Recommendation**: derived from the (verdict, root cause) combination table in `docs/taxonomy.md` — `block_merge` / `open_issue` / `fix_test` / `quarantine_test` / `retry_safe` / `escalate_infra` / `needs_human_review`
+- **Evidence**: the concrete signals supporting the verdict and root cause (log excerpts, run history, diff analysis)
+- **Suggested fix**: a short, concrete pointer to how the root cause could be fixed (file/function and the change in prose), or `none` when the recommendation is `retry_safe` / `needs_human_review`. Follow the skill's rule on not weakening tests.
+
+Be evidence-driven: if the signals are insufficient, report `INCONCLUSIVE` / `unknown` / `needs_human_review` and list what additional data (e.g., a re-run, node metrics) would settle it. Do not modify any files.

AGENTS.md

@@ -0,0 +1,27 @@
+Flake0 is a project to detect and classify flaky integration tests in the GitHub CI runs of Juju charms. It is scoped specifically to Juju Charms — not general-purpose CI flakiness detection.
+
+## Architecture
+Flake0 detects flakiness by using as inputs the following items categorized by:
+1. Code Source
+    - Repository source code.
+    - Changes if the run is after a PR or commit.
+2. Environment
+    - The runner's info e.g image, size.
+    - The packages versions e.g snap, juju.
+    - Spread environment setup logs.
+    - Node exporter metrics.
+3. Logs
+    - Integration tests logs e.g pytest, juju debug-log.
+    - Workload logs.
+    - Prometheus exporter metrics.
+
+The output would be for each integration test saying if this is a flaky integration test or not.
+Classification uses the taxonomy defined in `docs/taxonomy.md` (source of truth for verdict, root cause, confidence, and recommendation values) — see `skills/integration-test-diagnosis/SKILL.md` for the procedure.
+
+## Project Structure
+The project is structured as follows:
+- `docs` - Project documentation and diagrams.
+- `scripts` - Scripts used for infrastructure provisioning or for log collection.
+- `skills` - A set of skills used by the AI agent.
+- `snaps` - Snaps that are used in the log collection or infrastructure provisioning.
+- `tests` - Unit / Integration tests of the project.

CLAUDE.md

@@ -0,0 +1 @@
+AGENTS.md

docs/taxonomy.md

@@ -0,0 +1,115 @@
+# Failure Classification Taxonomy
+
+This is the canonical taxonomy used by Flake0 to classify a failing Juju charm
+integration test. Any agent, skill, or downstream consumer (GitHub PR label,
+OpenSearch document) that classifies or reads a classification must use these
+exact values.
+
+Classification uses **two independent axes**. They must be reported and
+reasoned about separately, never collapsed into a single field: a failure can
+be non-deterministic (`FLAKY`) *and* still be caused by a real bug in the
+code under test (e.g. a race condition introduced by the PR). Treating that
+combination as "just flaky" and waving it off is the most common and most
+costly classification error, so the two axes exist specifically to prevent it.
+
+## Axis 1 — Verdict (determinism)
+
+Whether the failure is reproducible given the same code and environment.
+
+| Value | Definition | Evidence needed |
+|---|---|---|
+| `REAL_FAILURE` | Reproducible given the same code + environment. | Fails consistently on retry/same commit, or the root cause is structural (bad assertion, missing wait, genuine bug). |
+| `FLAKY` | Same code, same environment, inconsistent outcome across runs. | History shows pass/fail alternation on unchanged commits; no structural cause found. |
+| `INCONCLUSIVE` | Not enough signal to decide. | No history, missing logs, single occurrence, ambiguous or contradictory evidence. |
+
+## Axis 2 — Root cause category
+
+Why the failure happened, scoped to the evidence Flake0 actually collects
+(source diff, spread setup logs, node exporter metrics, pytest/juju
+debug-log, workload logs/sosreport, prometheus scrape — see `AGENTS.md`).
+
+| Category | Definition | Primary evidence source |
+|---|---|---|
+| `code_regression` | Bug in the charm code introduced by this PR/commit. | Source diff; failure maps to a changed code path. |
+| `code_preexisting` | Bug in charm code, unrelated to this PR. | Same failure reproduces on `main`/unrelated PRs; no relevant diff. |
+| `test_defect` | Bad assertion, missing wait/retry, bad fixture/teardown, timing assumption in the test itself. | pytest traceback points at test code, not charm/workload code. |
+| `workload_defect` | The managed application (e.g. MongoDB, OpenSearch) misbehaves, not the charm logic. | Workload logs/sosreport show app-level crash/error; juju debug-log shows the charm handled it correctly. |
+| `environment_infra` | Runner resource exhaustion, network blip, kernel/image issue. | Node exporter metrics show CPU/mem/disk pressure at failure time; spread setup log errors. |
+| `dependency_drift` | snap/juju/charm-lib version change breaks behavior, unrelated to the PR's own diff. | Package version diff between last-good and failing run; no correlation with the PR's code diff. |
+| `external_service` | GitHub Actions, Charmhub, snap store, or cloud provider outage. | Failure timing correlates with a known external incident; error is a connection/5xx to an external endpoint. |
+| `unknown` | Insufficient evidence to pick a category. | — |
+
+## Confidence
+
+Confidence is a percentage (0-100), reported alongside a bucket label. Both
+are required: the percentage lets the OpenSearch record and dashboards
+sort/filter/trend on a continuous value; the bucket label is what gates the
+recommendation logic and human-review routing, so a single stray "73%
+instead of 74%" can't flip a decision.
+
+The percentage must be **derived from evidence completeness**, not a raw
+LLM self-reported number — an unanchored "I'm 85% confident" from the model
+is false precision and not comparable across runs. Compute it from how many
+of the expected signals for the chosen root cause category (see the
+"Primary evidence source" column above, plus run history) are actually
+present and consistent:
+
+- Start at 50%.
+- +15% for each independent corroborating signal that agrees with the
+  chosen root cause (e.g. diff correlation, run history, resource metrics,
+  workload logs) — cap the total contribution from corroborating signals.
+- -20% for each signal that is missing entirely (not checked/unavailable).
+- -30% for any signal that is checked and contradicts the chosen root cause.
+- Clamp to [0, 100].
+
+| Bucket | Percentage range | Meaning |
+|---|---|---|
+| `high` | 80-100 | History, diff, and logs all agree. |
+| `medium` | 50-79 | One supporting signal is missing (e.g. no history available, single occurrence). |
+| `low` | 0-49 | Evidence is circumstantial or contradictory — this should push the verdict toward `INCONCLUSIVE` regardless of how certain the reasoning sounds. |
+
+Report both, e.g. `confidence: 65 (medium)`. If the bucket derived from the
+percentage doesn't match the evidence narrative, trust the percentage
+formula and adjust the narrative, not the reverse — the formula exists so
+confidence isn't just a vibe.
+
+## Combining the axes → recommendation
+
+The (verdict, root cause) pair maps to a recommended next step. This is what
+drives the PR label and the "recommendation for next step" published to
+OpenSearch.
+
+| Verdict × Root cause | Meaning | Recommendation |
+|---|---|---|
+| `REAL_FAILURE` × `code_regression` | New bug introduced by this PR. | `block_merge` |
+| `REAL_FAILURE` × `code_preexisting` | Known/existing bug, not this PR's fault. | `open_issue` (don't block this PR) |
+| `REAL_FAILURE` × `test_defect` | The test itself is broken. | `fix_test` (don't block on product code) |
+| `FLAKY` × `code_regression` | Race condition introduced by this PR — still a real bug, just intermittent. | `block_merge` (do **not** auto-quarantine) |
+| `FLAKY` × `code_preexisting` | Pre-existing race condition/bug that surfaces intermittently. | `open_issue`, consider `quarantine_test` |
+| `FLAKY` × `workload_defect` | The underlying managed application is flaky. | `open_issue` against workload; consider `quarantine_test` |
+| `FLAKY` × `environment_infra` | Classic infra flake. | `retry_safe`, no code action |
+| `FLAKY` × `dependency_drift` | A new dependency version is unstable. | `escalate_infra` / pin version |
+| `*` × `external_service` | Outside the project's control. | `retry_safe`; note the incident; exclude from flaky-rate metrics |
+| `INCONCLUSIVE` × `unknown` | Not enough data. | `needs_human_review` |
+
+Any (verdict, root cause) pair not listed above should default to
+`needs_human_review` rather than guessing a recommendation.
+
+## Recommendation values
+
+- `block_merge` — do not merge; a real code defect exists.
+- `open_issue` — file/link a tracking issue; does not need to block this PR.
+- `fix_test` — the test needs fixing, not the product code.
+- `quarantine_test` — mark as known-flaky so it doesn't block merges going forward.
+- `retry_safe` — safe to re-run; no code or test change implied.
+- `escalate_infra` — notify infra/ops (runner, dependency, environment problem).
+- `needs_human_review` — evidence is insufficient for an automated decision.
+
+## Reporting format
+
+Any classification (agent report, structured LLM output, OpenSearch
+document) must report `verdict`, `root_cause`, `confidence`, and
+`recommendation` as independent fields — never as one combined string.
+`confidence` itself is two sub-fields: `confidence_pct` (0-100, int) and
+`confidence_bucket` (`high`/`medium`/`low`, derived from `confidence_pct`
+per the ranges above).

skills/integration-test-diagnosis/SKILL.md

@@ -0,0 +1,73 @@
+---
+name: integration-test-diagnosis
+description: Diagnosis of a failing integration test. Use this skill to investigate and classify the root cause of a failing integration test.
+---
+
+Identify the root cause of a failing integration test and classify it using
+the taxonomy defined in `docs/taxonomy.md`. That file is the source of
+truth for all label values — treat it as authoritative and keep this
+procedure in sync with it.
+
+Classification has two **independent** axes, reported separately:
+- **Verdict** (determinism): `REAL_FAILURE` / `FLAKY` / `INCONCLUSIVE`.
+- **Root cause**: `code_regression` / `code_preexisting` / `test_defect` /
+  `workload_defect` / `environment_infra` / `dependency_drift` /
+  `external_service` / `unknown`.
+
+Do not collapse these into one label. A failure can be `FLAKY` and still be
+caused by `code_regression` (e.g. a race condition introduced by the PR) —
+that combination must still route to `block_merge`, not be dismissed as
+"just flaky." See the combination table in `docs/taxonomy.md`.
+
+Do not treat "change the test so it passes" as a fix. The tests under
+investigation were reviewed and accepted as stable when they were merged, so
+the default assumption is that the test is correct. Never suggest loosening
+assertions, increasing timeouts or retries, skipping, or `xfail`-ing a test
+to hide a failure. Use `test_defect` / `fix_test` only when the evidence
+points at a specific defect in the test itself (e.g. a missing wait on a
+status the charm is documented to reach asynchronously), and cite that
+evidence.
+
+## Procedure
+1. **Gather Information**: Collect all relevant information about the failing integration test. All what you need will be available locally, this includes code source change, test logs, environment configuration and metrics about resources.
+
+2. **Analyze Test Logs**: Review the test logs and error messages to identify any patterns or specific errors that can indicate the nature of the failure.
+
+3. **Investigate Test Environment**: Check the test environment for any issues that may be affecting the test execution. This includes verifying:
+   - Environment variables and configurations
+   - Dependency versions and compatibility
+   - Resource availability (CPU, memory, disk space)
+   - Network connectivity and access to required services
+
+4. **Check Recent Changes**: Examine recent code changes or deployments that may have introduced the failure. Look for:
+   - Code commits that affect the functionality being tested
+   - Changes in dependencies or configurations
+   - Any recent merges or pull requests
+
+5. **Determine the verdict**: decide `REAL_FAILURE`, `FLAKY`, or
+   `INCONCLUSIVE` based on run history and whether the failure reproduces
+   deterministically. See `docs/taxonomy.md` Axis 1 for the exact
+   definitions and required evidence.
+
+6. **Determine the root cause category**: independently of the verdict,
+   assign one of `code_regression`, `code_preexisting`, `test_defect`,
+   `workload_defect`, `environment_infra`, `dependency_drift`,
+   `external_service`, or `unknown`. See `docs/taxonomy.md` Axis 2 for
+   definitions and the primary evidence source expected for each.
+
+7. **Assign confidence**: compute a percentage (0-100) from evidence
+   completeness using the formula in `docs/taxonomy.md` (start at 50%,
+   adjust per corroborating/missing/contradicting signal), then derive the
+   bucket (`high`/`medium`/`low`) from that percentage. Do not report a
+   free-floating "gut feeling" percentage — it must trace back to which
+   signals were present, missing, or contradictory.
+
+8. **Derive the recommendation**: look up the (verdict, root cause) pair in
+   the combination table in `docs/taxonomy.md` to get the recommendation
+   (`block_merge`, `open_issue`, `fix_test`, `quarantine_test`,
+   `retry_safe`, `escalate_infra`, or `needs_human_review`). Any
+   uncovered combination defaults to `needs_human_review`.
+
+9. **Document Findings**: Record verdict, root cause, confidence,
+   recommendation, and the supporting evidence for each — these must be
+   reported as independent fields, not merged into a single string.