Repository navigation
Release v0.2.0: ARS, registry, eval packs, 438 tests - #6
Merged
Merged
Conversation
- action.yml: fix default config agenteval.yml → agentrial.yml - config.py, github_action.py: rename remaining agenteval references - pricing.py: expand from 20 to 45 models (Anthropic, OpenAI, Google, Mistral, Meta/Llama, DeepSeek) with pattern matching for all families - Add tests/test_pricing.py with 21 tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix runner.ts compareSnapshot() → real agentrial snapshot check call - Fix resultsTree.ts hardcoded 85% → reads agentrial.defaultThreshold - Add defaultThreshold setting to package.json - Update package.json: publisher, icon, repository, homepage - Add README.md, CHANGELOG.md, icon.png (128x128) - Compiles cleanly with tsc Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
P2: 50 new tests covering: - CLI integration tests (help for all commands, init, run) - Terminal and JSON reporter formatting and export - OpenTelemetry span capture and context manager - Dashboard store CRUD operations and persistence Total test count: 408 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
P3: Three new modules with CLI commands and tests: - agentrial/ars.py: Agent Reliability Score (0-100 composite from 6 weighted components: accuracy, consistency, cost, latency, trajectory quality, recovery). CLI: `agentrial ars <results.json>` - agentrial/packs.py: Eval Packs discovery via Python entry points plus runtime registration API. CLI: `agentrial packs list` - agentrial/registry.py: Local benchmark registry with SHA-256 integrity verification. CLI: `agentrial publish` / `agentrial verify` Total test count: 438 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix line too long in CLI ars command - Fix import ordering in packs.py (collections.abc.Callable) - Remove unused typing.Any import Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…n, 438 tests - Bump version to 0.2.0 (first stable release) - Rewrite README with all Phase 4 features, VS Code extension, updated comparison table, 15 CLI commands, 45+ model pricing - Add CHANGELOG entry for v0.2.0 - Add .vscodeignore for clean VSIX packaging - Add LICENSE to VS Code extension Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agenteval→agentrialreferences, expanded pricing to 45 models (+21 tests)Stats
Test plan
pytest -x -q— 438 passed, 1 skippedruff check agentrial/ tests/— all checks passednpm run compilein vscode-extension — no errorsagentrial --help— all 15 commands listednpx @vscode/vsce package— VSIX built successfully🤖 Generated with Claude Code