Skip to content

fix(ontogeny): prevent empty EvalSuite from passing - #105

Open
zlh123123 wants to merge 1 commit into
hugegraph:ontogenyfrom
zlh123123:codex/fix-empty-evalsuite
Open

zlh123123 wants to merge 1 commit into
hugegraph:ontogenyfrom
zlh123123:codex/fix-empty-evalsuite

Conversation

@zlh123123

Copy link
Copy Markdown

Problem

EvalRunner.run() initialized every report as passed and returned passed=True when an EvalSuite contained no query, replay, or agent cases. The evaluation API and promoter only checked that a suite result existed, so a suite with no executable evaluation could be treated as green.

This is a vacuous evaluation result. It weakens the intended fail-closed evaluation gate, although this change does not claim a separately verified exploitable security vulnerability.

Reproduction

An EvalSuite with empty queries, replays, and agents produced:

{"cases": [], "passed": true}

The same occurred when only the currently unimplemented coverage configuration was present.

Solution

  • Mark a report as failed when no executable evaluation cases were run.
  • Include the explicit error no executable eval cases.
  • Add regression coverage for both an empty suite and a coverage-only suite.

The change does not alter the EvalSuite model or add new DSL validation rules. Suites containing executable cases retain their existing behavior.

Tests

  • uv run --directory ontogeny --group dev pytest tests/test_evolve.py -q (63 passed)
  • uv run ruff check ontogeny/server/ontogeny/evolve/evalrunner.py ontogeny/tests/test_evolve.py
  • uv run ruff format --check ontogeny/server/ontogeny/evolve/evalrunner.py ontogeny/tests/test_evolve.py

Base branch: ontogeny

@coderabbitai

coderabbitai Bot commented Nov 11, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: ab29a372-ab78-434b-af38-54efa55300f1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

@codecov-ai-reviewer review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant