Runnable evaluation kits for Karate Agent — the governance layer for
AI-generated code. It joins requirements, tests, business rules, coverage and evidence into one
traceability graph in git. An LLM drives it; a human audits it. Requirements live as code in markdown.
A Requirements Traceability Matrix (RTM) is built from real run evidence. API and UI test coverage
land in the same report. One engine, many surfaces: a JavaScript API, curl, MCP (so any AI agent
can drive it), and a served console.
Why it exists. AI now writes more code than any team can review by hand. These kits show the deterministic answer. Every requirement carries EARS acceptance criteria, and every test attests to a criterion. The report then says — from evidence, not from an AI's opinion — what is covered, what is failing, what was never verified, and whether it is safe to ship.
Each kit's
README.mdsays how to run it. Either drop the engine jar (karate-async-<version>.jar) into the kit, or run the one-shotkarate-agentcontainer (no jar to stage). We sent the product sheet, the QUICKSTART and yourkarate.licseparately; the license governs the engine.
The traceability-demo kit publishes its real CI output to GitHub Pages on every run. You see
the actual report in a browser, with nothing to install and no license needed:
https://karatelabs.github.io/karate-agent-examples/traceability-demo/
- Coverage report —
.../traceability-demo/ext/coverage/pages/coverage.html - Traceability matrix (RTM) —
.../traceability-demo/ext/traceability/pages/traceability.html
| kit | what it demonstrates |
|---|---|
traceability-demo |
git-first requirements traceability for a loan-decision engine: requirements → business rules → run evidence in one RTM. The same kit runs two ways — ALM-linked on Azure Pipelines (each requirement id links to its Azure DevOps User Story) and pure-git / spec-driven on GitHub Actions. Two environment variables switch the mode. |
policy-api |
one insurance API across three protocols — REST (OpenAPI) + gRPC + Kafka — in a single coverage report: live probe, durable suite, method coverage, input dimensions, the rich-error path, and the gap worklist. |
kiro-demo |
"done" is a claim, not evidence. The requirements come straight from an AI coding tool's own spec folder, with its task list ticked complete. Every scenario passes, yet the verdict is NOT READY: one criterion a completed task claims is implemented, but no test exercises it. Rules + REST only, no browser, about half a second per run. |
store-api |
start-from-scratch benchmark. A bare OpenAPI spec and nothing else. The engine stands up a stateful mock from the spec, your AI agent authors the suite, and the gap lists define "done" deterministically. Includes a cheat-sheet for timing your own agent environment against a clean reference. |
fleetquote |
the rulebook and its twin, end to end. A fleet-auto rating engine as one executable rulebook — readable spec, oracle with a plain-English audit trail, the mock's brain, and the model every view derives from — plus a lifecycle twin walked to "the defect only order finds". Start with TUTORIAL.md; rulebooks/rating/ holds the twin and its pinned sequences, and the live report shows the result. |
checkout |
consumer-side contract testing. A checkout service depends on a payments API another team owns. The kit holds the mock the team builds against, an independently implemented real provider, and the paired run that proves the mock can stand in — including one deliberate divergence the suite's assertions cannot see, which the pair reports anyway. Consumer-driven contracts with a readable functional suite instead of pact files, and git instead of a broker. |
Plain, self-contained examples of testing a non-HTTP protocol. Each kit ships the service under test, so
it runs standalone, and each runs off the karate-async engine jar; MCP needs no protocol libraries, so
it runs off the container image too. All four run on every push and publish their reports:
grpc ·
kafka ·
websocket ·
mcp.
| kit | what it demonstrates |
|---|---|
grpc |
unary, server-streaming, client-streaming and bidirectional calls, request/response metadata, and asserting a gRPC error status. No generated stubs on the test side — the engine reads the .proto at run time, so there is nothing to regenerate when the contract changes. |
kafka |
produce and consume, as JSON and as Avro through a Schema Registry, with message headers and a filtering consumer. docker compose starts the broker and the registry. |
websocket |
raw text, JSON, and collecting a stream of messages — plus a custom frame-based protocol (STOMP) handled by a codec, so protocol handling stays out of the checks. Both demo servers compile against the engine jar alone: no build tool. |
mcp |
discover an MCP server's tool catalog, call tools, and assert results and side effects — with the two kinds of failure kept apart: a tool's own error is a result you assert, a protocol error is a failed step. The client negotiates the protocol era, and a saved tools/list catalog gives the run tool coverage, so an uncalled tool is reported. Runs the official conformance suite alongside as the companion step. |
quick-meetings-karate |
property-based testing, measured against. The InfoQ "Beyond Accidental Quality" sample — a Spring Boot meeting scheduler with five seeded bugs and the jqwik property tests that find them — with a rulebook and twin as a thin overlay that re-finds all five deterministically: a deck, a bounded walk, live replay with shrink, and negative probes. The original source is untouched, so the two approaches sit side by side. |
Every kit can be driven over the Model Context Protocol. Serve the project, point any MCP client (Claude Code, Cursor, VS Code, or your own agent) at it, and ask in plain language — "run the loan rules oracle and show me the requirement coverage gaps." Each kit README carries the exact command.
- Karate Agent — the AI-driven agent, served console and traceability engine: https://karatelabs.io/agent
- Karate Enterprise — the commercial platform, licensing and support: https://karatelabs.io/karate-enterprise
The example code in this repository is provided under the MIT License. It is not the
karate-agent license — your karate.lic is sent separately and governs the engine.