Skip to content

Release v0.2.0: ARS, registry, eval packs, 438 tests - #6

Merged
alepot55 merged 8 commits into
mainfrom
feat/v0.2.0-audit-fixes
Feb 6, 2026
Merged

alepot55 merged 8 commits into
mainfrom
feat/v0.2.0-audit-fixes

Conversation

@alepot55

@alepot55 alepot55 commented Feb 6, 2026

Copy link
Copy Markdown
Owner

Summary

  • P0: Fixed agenteval → agentrial references, expanded pricing to 45 models (+21 tests)
  • P1: VS Code extension marketplace-ready (runner, resultsTree, icon, README, CHANGELOG)
  • P2: 50 new tests for CLI, reporters, OTel, dashboard store
  • P3: Phase 4 modules — ARS (0-100 score), eval packs (entry points), benchmark registry (SHA-256)
  • Release: Version bump to 0.2.0, rewritten README, updated CHANGELOG

Stats

  • 438 tests passing (up from 349)
  • 15 CLI commands (5 new: ars, publish, verify, packs list)
  • 29 files changed, +2305 / -113 lines
  • 0 lint errors (ruff + tsc)
  • VS Code extension: VSIX builds successfully (19 KB)

Test plan

  • pytest -x -q — 438 passed, 1 skipped
  • ruff check agentrial/ tests/ — all checks passed
  • npm run compile in vscode-extension — no errors
  • agentrial --help — all 15 commands listed
  • npx @vscode/vsce package — VSIX built successfully

🤖 Generated with Claude Code

alepot55 and others added 8 commits February 6, 2026 15:49
- action.yml: fix default config agenteval.yml → agentrial.yml
- config.py, github_action.py: rename remaining agenteval references
- pricing.py: expand from 20 to 45 models (Anthropic, OpenAI, Google,
  Mistral, Meta/Llama, DeepSeek) with pattern matching for all families
- Add tests/test_pricing.py with 21 tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix runner.ts compareSnapshot() → real agentrial snapshot check call
- Fix resultsTree.ts hardcoded 85% → reads agentrial.defaultThreshold
- Add defaultThreshold setting to package.json
- Update package.json: publisher, icon, repository, homepage
- Add README.md, CHANGELOG.md, icon.png (128x128)
- Compiles cleanly with tsc

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
P2: 50 new tests covering:
- CLI integration tests (help for all commands, init, run)
- Terminal and JSON reporter formatting and export
- OpenTelemetry span capture and context manager
- Dashboard store CRUD operations and persistence

Total test count: 408

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
P3: Three new modules with CLI commands and tests:

- agentrial/ars.py: Agent Reliability Score (0-100 composite from
  6 weighted components: accuracy, consistency, cost, latency,
  trajectory quality, recovery). CLI: `agentrial ars <results.json>`

- agentrial/packs.py: Eval Packs discovery via Python entry points
  plus runtime registration API. CLI: `agentrial packs list`

- agentrial/registry.py: Local benchmark registry with SHA-256
  integrity verification. CLI: `agentrial publish` / `agentrial verify`

Total test count: 438

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix line too long in CLI ars command
- Fix import ordering in packs.py (collections.abc.Callable)
- Remove unused typing.Any import

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…n, 438 tests

- Bump version to 0.2.0 (first stable release)
- Rewrite README with all Phase 4 features, VS Code extension, updated
  comparison table, 15 CLI commands, 45+ model pricing
- Add CHANGELOG entry for v0.2.0
- Add .vscodeignore for clean VSIX packaging
- Add LICENSE to VS Code extension

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@alepot55
alepot55 merged commit cfe1bb8 into main Feb 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant