Issues

• Would you take Faber Romanus as a comparison language?

  aallan/vera-bench#126 · Filter by author ianzepp opened on Oct 20, 2026

• An expected-Tier-1 result passes when every obligation was demoted to tier3_unguarded

  bug
  aallan/vera-bench#122 · Filter by author aallan opened on Sep 17, 2026

• Under-specified ADT problems are graded inconsistently across languages

  aallan/vera-bench#121 · Filter by author aallan opened on Jul 28, 2026

• A nullary Python class is read as unverifiable, so the shape guard renders a call it cannot honour

  aallan/vera-bench#119 · Filter by author aallan opened on Jul 28, 2026

• baselines destroys its output before it has a replacement

  aallan/vera-bench#118 · Filter by author aallan opened on Jul 28, 2026

• TypeScript declaration scanning is not literal-aware

  aallan/vera-bench#115 · Filter by author aallan opened on Jul 27, 2026

• Deduplicate the Python and TypeScript wrapper builders across runner and baseline_runner

  aallan/vera-bench#111 · Filter by author aallan opened on Jul 26, 2026

• Expose the LLM client timeout as a --timeout flag

  aallan/vera-bench#105 · Filter by author aallan opened on Jul 25, 2026

• Tier the test suite (unit vs integration) so the merge gate is fast and hermetic

  aallan/vera-bench#102 · Filter by author aallan opened on Jul 24, 2026

• run_sweep.sh re-runs whole targets to clear a few transient failures — retry per-problem instead

  aallan/vera-bench#101 · Filter by author aallan opened on Jul 24, 2026

• Track Harness/Codecov acquisition impact on vera-bench CI

  infrastructurequality
  aallan/vera-bench#80 · Filter by author aallan opened on Jun 3, 2026
  1

• Add MoonBit as a comparison language

  languages
  aallan/vera-bench#49 · Filter by author aallan opened on Apr 13, 2026

• Refactor models.py to a provider registry

  enhancement
  aallan/vera-bench#45 · Filter by author aallan opened on Apr 9, 2026
  1

• Automated scheduled benchmark runs with structured storage

  infrastructure
  aallan/vera-bench#31 · Filter by author aallan opened on Mar 30, 2026
  5

• Results dashboard (GitHub Pages or veralang.dev)

  publishing
  aallan/vera-bench#30 · Filter by author aallan opened on Mar 30, 2026
  1

• Multi-turn and agentic evaluation modes

  evaluation
  aallan/vera-bench#29 · Filter by author aallan opened on Mar 30, 2026
  2

• Hugging Face dataset export

  publishing
  aallan/vera-bench#27 · Filter by author aallan opened on Mar 30, 2026

• Generate paper-quality figures (matplotlib/seaborn)

  analysis
  aallan/vera-bench#26 · Filter by author aallan opened on Mar 30, 2026
  3

• Expand to 75+ problems (15 per tier)

  problems
  aallan/vera-bench#25 · Filter by author aallan opened on Mar 30, 2026
  8

• Run benchmark against multiple models (Opus, GPT-4o, DeepSeek, Gemini)

  evaluation
  aallan/vera-bench#24 · Filter by author aallan opened on Mar 30, 2026
  5

• Add Go as a fourth comparison language

  golanguages
  aallan/vera-bench#21 · Filter by author aallan opened on Mar 30, 2026

• Increase test coverage to >90%

  enhancementqualitytesting
  aallan/vera-bench#5 · Filter by author aallan opened on Mar 29, 2026
  2