AI-powered medical image analysis with temporal evolution tracking.
Powered by GABBE. All project generation flow documents saved in docs
Website: https://andreibesleaga.github.io/AgenticMedicalImagingHelper/
A local TypeScript CLI tool that uses Google Gemini/OpenRouter AI and LangGraph.js to analyze series of medical images, detect findings, and track how conditions evolve over time across multiple imaging sessions.
docs/COMPLIANCE.md §0 for the full scope statement.
See docs/PRODUCT.md for the unified PRD / SPEC / user stories / architecture for AgenticMedicalImagingHelper.
This repository is the reference implementation behind two publications. It is research/educational software — see docs/COMPLIANCE.md for scope and regulatory positioning.
Andrei N. Besleaga, "Agentic Multimodal Architectures for Medical Imaging: Orchestration, Deterministic Fairness Probing, and Governance," accepted at the IEEE International Conference on Smart Innovations for Medicine and Engineering (SIME 2026), Sousse, Tunisia, 2–4 Nov. 2026. To appear in IEEE Xplore.
- Preprint DOI: 10.5281/zenodo.20762929 (access restricted until publication)
This project is featured as a reference implementation in the forthcoming Wiley book "Agentic AI Architectures" by Andrei Besleaga.
@inproceedings{besleaga2026agentic,
author = {Besleaga, Andrei N.},
title = {Agentic Multimodal Architectures for Medical Imaging:
Orchestration, Deterministic Fairness Probing, and Governance},
booktitle = {Proceedings of the IEEE International Conference on Smart
Innovations for Medicine and Engineering (SIME 2026)},
address = {Sousse, Tunisia},
month = nov,
year = {2026},
publisher = {IEEE},
doi = {10.5281/zenodo.20762929},
note = {to appear}
}
Machine-readable citation metadata is in CITATION.cff — GitHub renders it under "Cite this repository".
The experiment scripts, selection manifests and result logs used for the paper live in experiments/sime2026/; see the README in that directory for the exact reproduction steps. The dataset is the public NIH Clinical Center ChestX-ray14 collection (Wang et al., CVPR 2017) — de-identified frontal PA chest radiographs, used here as a 224-px derivative with no accuracy claim attached. See also docs/EXPERIMENTS.md for a short index of the individual experiments (E1–E4, payload, scanner stress).
Follow-up after the paper: experiments/sime-full/ repeats the longitudinal experiment (E4) on the original 1024 × 1024 NIH images of the same patients, with seven models including a local MedGemma run on a laptop GPU, then the 20-patient cohort, the scaling test and repeat runs (320 runs in all, every input checked against the original files' hashes, every run manifest verified, USD 11.19). In short: the paper's results reproduce, the better/same/worse verdict changes between identical runs for many patients, and full resolution changes some models' findings but does not fix the cases that get worse. RESULTS.md sets out what matches the paper and what differs; REPLICATE.md lists every command.
Research/educational software — see docs/COMPLIANCE.md for scope and regulatory positioning.
- Multi-series analysis — processes multiple series of images (CT, MRI, X-ray, Ultrasound, etc.) in parallel
- Fan-Out/Fan-In architecture — LangGraph StateGraph with p-limit concurrency control
- Temporal evolution tracking — compares series across time and reports progression (Improving/Stable/Worsening)
- Context integration — reads
.txtfiles alongside images for additional clinical context - Research grounding — Gemini built-in Google Search for literature citations
- Structured reports — per-image JSON + per-series Markdown + combined evolution report
- Providers — default Google Gemini (
gemini-3.8-flash; Google limits the 2.5 models to projects that already used them, so new keys cannot callgemini-2.5-flashdirectly). Optional OpenRouter second provider viaAI_PROVIDER=openrouter+OPENROUTER_API_KEY+OPENROUTER_MODEL(defaultgoogle/gemini-2.5-flash), which runs the unchanged pipeline on any vision-capable OpenRouter model. Note: the OpenRouter path has no Google Search grounding. See ADR-006. Optional local provider (AI_PROVIDER=local): a model served on this machine by Ollama, vLLM or llama.cpp through an OpenAI-compatible endpoint, defaultmedgemma1.5:4b, no key and no API cost — see LOCAL-INFERENCE.md.
input/
├── patient_context.txt # Optional: overall patient context
├── series_1/ # First imaging session
│ ├── image_001.png
│ ├── image_002.jpg
│ └── clinical_notes.txt # Optional: series-specific context
├── series_2/ # Second imaging session (later date)
│ ├── image_001.png
│ └── image_002.png
└── series_n/
└── ...
output/
├── series_1/
│ ├── image_001_analysis.json # Per-image AI analysis
│ ├── image_002_analysis.json
│ └── series_summary.md # Aggregated series report
├── series_2/
│ ├── image_001_analysis.json
│ └── series_summary.md
├── evolution_analysis.json # Temporal comparison data
├── combined_diagnostic_report.md # Full evolution narrative
└── run_manifest.json # Hash-sealed audit record of the run
run_manifest.json is written on every run that reaches the model — including
the failing ones — and is described under Governance controls.
- Node.js 22+
- Google Gemini API key (default model
gemini-3.8-flash; override withGEMINI_MODEL), or an OpenRouter key, or a local OpenAI-compatible model server (no key)
# Clone the repository
git clone https://github.com/andreibesleaga/AgenticMedicalImagingHelper.git
cd AgenticMedicalImagingHelper
# Install dependencies (exact versions from the committed package-lock.json)
npm ci
# Copy environment template, then edit .env in your editor to set
# GOOGLE_API_KEY (and optionally GEMINI_MODEL and LOG_LEVEL).
cp .env.example .env
$EDITOR .env
# Build
npm run build
# Analyze images in ./input (output directory defaults to ./output)
npm run dev -- analyze ./input
# With options
npm run dev -- analyze ./input ./output --concurrency 3 --verbose
# Analyze only specific series
npm run dev -- analyze ./input --series series_1 series_2
# Using the built binary
node dist/main/index.js analyze ./input --verbose
medical-imaging [options] [command]
Options:
-V, --version Show version
-h, --help Show help
analyze <inputDir> [outputDir]
Arguments:
inputDir Path to input directory with series sub-folders
outputDir Output path (default: ./output)
Options:
-s, --series <ids...> Process only specified series IDs
-c, --concurrency <n> Max parallel model API calls (default: 5)
--max-cost-usd <n> Abort the run once estimated cost exceeds this
USD figure (client-side soft cap; exit code 5)
--manifest-chain <file> Append this run's manifest hash to an
append-only ledger, linked to the previous run
--reviewer <name> Record a human-oversight attestation in the
run manifest (Art. 14; no clinical effect)
--allow-phi Acknowledge PHI-scan findings and continue
--strict-phi Refuse to upload anything if the PHI scan
finds something (exit code 7)
--secure Refuse to upload anything if the input guard
finds instruction-like context text or an image
whose bytes do not match its extension (exit 9)
--fail-on-probe Exit 8 (outputs and manifest still written) if
any governance probe fires
--expect-hashes <file> JSON map of image file name to SHA-256; refuse
to upload anything unless every image matches
the expected original (exit code 10)
-v, --verbose Print progress to stderr
-h, --help Show help
verify-manifest <outputDir>
Arguments:
outputDir Output directory of the run to verify
Options:
--chain <file> Ledger file the run should be linked into
--concurrency <n>bounds how manyanalyzeImagecalls are in flight at once viap-limit; higher values finish faster but increase the chance of hitting provider rate limits (seeAI_MAX_RETRIESbelow).--max-cost-usd <n>is a client-side estimate-based cap, not a substitute for a provider-side billing quota; see Data handling & privacy.--manifest-chain,--reviewer,--allow-phi,--strict-phiand theverify-manifestsubcommand are described under Governance controls.
| Code | Meaning |
|---|---|
| 0 | All images analyzed successfully |
| 1 | Missing or invalid API key |
| 2 | Input directory not found or unreadable |
| 3 | No image series found in input directory |
| 4 | Partial failure — some images could not be analyzed |
| 5 | --max-cost-usd cap exceeded — run aborted (CostCapExceededError) |
| 6 | verify-manifest: run manifest missing, altered, or its ledger link broken |
| 7 | --strict-phi: PHI/PII found in a context file — nothing was uploaded |
| 8 | --fail-on-probe: a governance probe fired — outputs and manifest written |
| 9 | --secure: the input guard found a problem — nothing was uploaded |
| 10 | --expect-hashes: an image is missing or changed — nothing was uploaded |
| 99 | Unexpected internal error |
Code 2 also covers the input refusals added with the governance controls:
DICOM input (by extension or by magic bytes), a file over MAX_IMAGE_BYTES,
and a run over MAX_IMAGES_PER_RUN. Code 1 also covers an unknown AI_PROVIDER
value and an invalid --max-cost-usd value; code 10 also covers an
--expect-hashes file that cannot be read as a JSON object.
Codes 1, 2, 3, 7, 9 and 10 are pre-flight rejections: they happen before any model call and before the output directory is created, so they leave no artefacts. Codes 0, 4, 5 and 8 each write a run manifest recording that exit code, and so does an unexpected error (99) raised after the run has started.
| Variable | Required | Description |
|---|---|---|
GOOGLE_API_KEY |
Yes (Google path) | Google Gemini API key |
GEMINI_API_KEY |
Yes (alternative) | Alternative env var name |
GEMINI_MODEL |
No | Gemini model id (default gemini-3.8-flash) |
AI_PROVIDER |
No | google (default), openrouter, or local (a model served on this machine through an OpenAI-compatible endpoint: Ollama, vLLM, llama.cpp) |
OPENROUTER_API_KEY |
Yes when AI_PROVIDER=openrouter |
OpenRouter API key |
OPENROUTER_MODEL |
No | OpenRouter model id (default google/gemini-2.5-flash); any vision-capable model works |
LOCAL_LLM_BASE_URL |
No | Base URL of the local OpenAI-compatible server (default http://127.0.0.1:11434/v1, Ollama); see LOCAL-INFERENCE.md |
LOCAL_LLM_MODEL |
No | Local model id (default medgemma1.5:4b); zero pricing, so --max-cost-usd never trips |
LOCAL_LLM_API_KEY |
No | Sent as a bearer token only when set (servers started with an API key) |
AI_MAX_RETRIES |
No | Max retry attempts on HTTP 429/5xx/transport errors, shared by both provider adapters (0 disables retries); see src/infrastructure/retry.ts |
MAX_IMAGES_PER_RUN |
No | Hard ceiling on images accepted in one run (default 500); exceeded ⇒ exit 2 |
MAX_IMAGE_BYTES |
No | Hard ceiling on a single image file in bytes (default 52428800, 50 MB); exceeded ⇒ exit 2 |
See .env.example for the full, commented list.
# Run the mocked test suite (default, no network) — see CI or `npm test`
# for the current pass count; do not hard-code a test count here, it drifts.
npm test
# Same, with coverage report
npm run test:coverage
# Run the opt-in live test against the real Gemini API (real network)
# Requires GOOGLE_API_KEY (or GEMINI_API_KEY) exported in your environment.
# Defaults GEMINI_MODEL=gemini-2.5-flash (free-tier quota); override if needed.
GOOGLE_API_KEY=... npm run test:live
# Type check
npm run typecheck
# Lint (treat warnings as errors)
npm run lint -- --max-warnings 0
# Format (prettier)
npm run format
# Build
npm run build
Test layout:
tests/
├── unit/ # Per-module unit tests (mocked)
├── e2e/ # Pipeline + CLI scenarios (mocked Gemini, real file I/O)
├── live/ # Opt-in real-API smoke (excluded from `npm test` by default)
└── fixtures/ # Synthetic PNGs, mock responses, demographic-skewed context
# Build image
docker build -t medical-imaging .
# Run (mount input/output directories)
docker run --rm \
-e GOOGLE_API_KEY=your-key \
-v $(pwd)/input:/app/input:ro \
-v $(pwd)/output:/app/output \
medical-imaging analyze /app/input /app/output --verbose
src/
├── domain/ # Business entities, types, error classes
├── application/ # Use cases (analyze-image, aggregate-series, analyze-evolution)
├── infrastructure/ # External adapters (Gemini API, file scanner, report writer)
├── adapters/ # LangGraph StateGraph orchestrator
└── main/ # CLI entry point (Commander.js)
The system uses the LangGraph Fan-Out/Fan-In pattern:
scanInputDirectorydiscovers all image seriesanalyzeImagesnode fans out — analyzes all images concurrently (p-limit)aggregateSeriesnode fans in — synthesizes per-series summariesanalyzeEvolutionnode — compares series for temporal progressionwriteReportswrites structured output files
Diagram currency. The rendered C4 diagrams (docs/context.png,
docs/components.png, docs/runtime.png, embedded in
docs/PRODUCT.md §5) were generated from docs/PLAN.md
when the default model was gemini-2.5-pro and Gemini was the only provider.
They are not regenerated here. The current default model is
gemini-3.8-flash (Google limits the 2.5 models to earlier users)
with optional OpenRouter and local-model adapters — read the diagrams as
predating ADR-006.
- API key required via environment variable; never logged (every provider key is redacted by value and by field name)
- Path traversal protection on all file operations
- DICOM input refused; per-image and per-run size limits enforced pre-upload
- Context truncated to 2000 characters to prevent prompt injection, and scanned for PHI/PII before the first upload
- Images bounded before API submission by a model-aware long-edge cap (ADR-003): a file over the cap is resized; only an 8-bit PNG or JPEG within the cap, with no alpha channel and no embedded metadata, is sent byte-for-byte; anything else is re-encoded, which drops the metadata
- All output includes the mandatory medical disclaimer and a clinician-review block
- Every run leaves a hash-sealed, verifiable manifest
See Governance controls for what each of those does.
See docs/SECURITY_CHECKLIST.md for the full security audit of v1.0.0 (2026-02-26), and SECURITY.md for the vulnerability-disclosure policy and Gemini API-key handling guidance.
The controls below exist because a research artefact that produces medical-looking findings should be auditable by someone who was not in the room when it ran. They are engineering controls, not a conformity assessment: the scope statement in docs/COMPLIANCE.md §0 is unchanged, and this remains an educational research tool that is not a medical device.
- Input validation — path-traversal protection, an image extension
allow-list, and hard pre-upload limits:
MAX_IMAGES_PER_RUN(default 500) andMAX_IMAGE_BYTES(default 50 MB). All refusals, all exit code 2. - DICOM refusal —
.dcm/.dicomfiles, and any file carrying a DICOM Part-10 preamble whatever its extension, are refused. DICOM carries PHI in its tags and often burned into its pixels; converting it silently would strip neither. Convert to PNG/JPEG after de-identifying. DICOM ingestion with tag-level de-identification is future work. - PHI/PII scan — every context
.txtis scanned before the first upload for MRN-like identifiers, SSNs, phone numbers, emails, dates of birth, patient-name lines and postal addresses. The default is a stderr warning plus a record in the run manifest;--allow-phiacknowledges,--strict-phiexits 7 before anything leaves the machine. Findings carry masked excerpts only (123-45-6789→1**-**-****), so the warning cannot become a second disclosure. It is a heuristic: a clean scan means "nothing obvious", never "de-identified", and it does not look inside images. - Input integrity —
--expect-hashes <file>checks every image against a list of SHA-256 digests of the expected originals before anything is uploaded (exit 10 on a missing or changed file), so a run can prove it used the published dataset files unaltered.experiments/sime-full/selection-hashes.jsonis such a list for the NIH originals. - Input guard (OWASP LLM01/LLM02) — before any upload: each image's leading
bytes must match its extension (a renamed PDF or DICOM is caught); embedded
image metadata (PNG
tEXt/zTXt/iTXt/eXIf/tIME, JPEG EXIF/XMP/IPTC/ comment) is recorded and always stripped by the image pre-flight, which re-encodes any file that carries it; context files are scanned for instruction-like text ("ignore previous instructions", role tags, requests to reveal the prompt, instructions to report "normal"). Default: warning plus manifest record;--secureexits 9 instead. Heuristics, not a guarantee. - Output sanitisation (OWASP LLM05) — every model-authored field is cleaned
before it enters a Markdown report: tag-shaped text is escaped (
<script>→<script>),javascript:/data:/vbscript:/file:link targets are removed, control characters are dropped. - Bias probe v2 — runs on every completed run's generated text and records
each sentence that uses a demographic or social attribute (race and ethnicity,
sex and gender stereotypes, religion, socioeconomic status, sexual orientation
and gender identity, disability, nationality and migration, language, body
size, age stereotypes) as the justification for a diagnostic statement, with
negation handling. Measured and pinned in
tests/unit/domain/fairness-v2.test.tson a 96-item held-out benchmark (tests/fixtures/fairness-benchmark-v2.json): precision 0.929, recall 0.565; indirect proxies (surname, tribe, shelter, accent) are not caught. Version 1, whose numbers the SIME 2026 paper reports, is kept unchanged. - Unsupported-measurement probe — PNG/JPEG carry no pixel spacing, so every absolute size in the output ("4 cm") is a model estimate stated as a measurement; each is recorded, and sizes on abnormalities reported at ≥ 90 % confidence (the SIME 2026 failure case F1) are listed first.
--fail-on-probe— turns any probe finding (context consistency, bias, measurement) into exit 8 while still writing every output and the manifest.- Mandatory disclaimer — a required field on every output type, enforced at the TypeScript type level, on every JSON and Markdown artefact.
- Clinician-review block — every Markdown report opens with "Requires review by a qualified clinician before any use", before the title.
- Cost cap —
--max-cost-usd <n>aborts the run (exit 5) once the token-based estimate crosses the cap. A client-side guard-rail, not a substitute for a provider-side billing quota. - Bounded retry — HTTP 429/5xx and transport failures only, with exponential
backoff and full jitter, honouring
Retry-After; never other 4xx, never a validation failure.AI_MAX_RETRIES=0disables it. - Run manifest, ledger and verification — see below.
- Deterministic exit codes — 0–10 and 99, documented in Exit Codes and SPEC §2.3.
Every admitted run writes <outputDir>/run_manifest.json, sealed with a SHA-256
over the canonical JSON of its own content
(ADR-007):
{
"manifestVersion": "1.0",
"toolVersion": "1.0.0",
"startedAt": "2026-09-08T10:00:00.000Z",
"finishedAt": "2026-09-08T10:01:12.481Z",
"provider": "google",
"model": "gemini-2.5-flash",
"pricing": { "inputUsdPerMillion": 0.3, "outputUsdPerMillion": 2.5 },
"settings": { "concurrency": 5, "maxCostUsd": null, "retries": 3, "imageQuality": "auto" },
"inputDir": "/abs/path/input",
"outputDir": "/abs/path/output",
"inputs": [{ "path": "series_1/img_01.png", "sha256": "…", "bytes": 40213 }],
"contextFiles": [{ "path": "series_1/context.txt", "sha256": "…", "bytes": 512 }],
"calls": [
{
"index": 1,
"stage": "image",
"timestamp": "2026-09-08T10:00:03.117Z",
"tokensIn": 1284,
"tokensOut": 902,
"estimatedUsd": 0.0026,
"providerUsd": 0.0027,
"retries": 0,
},
],
"outputs": [{ "path": "series_1/series_summary.md", "sha256": "…", "bytes": 2144 }],
"totals": {
"calls": 4,
"tokensIn": 5136,
"tokensOut": 3608,
"estimatedUsd": 0.0105,
"providerUsd": 0.0108,
"images": 2,
"imagesSucceeded": 2,
"imagesFailed": 0,
"series": 1,
},
"exitCode": 0,
"warnings": [],
"humanReview": { "required": true, "reviewedBy": null },
"prevManifestHash": null,
"manifestHash": "9f3c…", // SHA-256 over the canonical JSON of everything above
}
tokensOut includes the model's thinking tokens, because that is how they are
billed. providerUsd appears only when the provider reports its own charge
(OpenRouter does; the Gemini SDK does not). stage is image, series or
evolution.
medical-imaging is the package's bin name; the package is not published to
npm, so from a clone run node dist/main/index.js in its place.
# Link each run into an append-only ledger and attest who reviewed it
medical-imaging analyze ./input ./output \
--manifest-chain ./audit/chain.jsonl \
--reviewer "Dr A. Reviewer"
# Re-hash the run and check its ledger link — exit 0 ok, 6 tampered/missing
medical-imaging verify-manifest ./output --chain ./audit/chain.jsonl
verify-manifest re-computes the manifest hash, re-hashes every input and
output file still on disk, and validates the chain link, printing PASS / FAIL /
SKIP per check. Files that no longer exist are reported as skipped rather than
failed — inputs live outside the tool's control — while a changed file fails.
Honest limit. This is tamper-evidence, not tamper-proofing. The ledger sits on the same disk as the manifests it seals, so a local attacker with write access who rewrites a manifest and every subsequent ledger entry leaves no trace (THREAT_MODEL T9). Signing over an off-box key, or an external timestamping log, is the upgrade path and is deliberately not claimed here. Keep the ledger on separate append-only storage if you need more.
Images you provide are sent to the configured model provider for analysis — the
Google Gemini API by default; OpenRouter and the upstream vendor it routes to
with AI_PROVIDER=openrouter; the server at LOCAL_LLM_BASE_URL (the loopback
interface by default) with AI_PROVIDER=local — and they leave your machine
only via that API call. The tool stores nothing remotely and keeps no
telemetry.
- You must de-identify inputs. Strip PHI / patient identifiers, including EXIF/DICOM metadata, before running. The tool strips embedded PNG/JPEG metadata before upload and refuses DICOM, but it does not de-identify image pixels or context text for you.
- Compliance is the operator's responsibility. If you process real patient data you — not this project — are responsible for HIPAA (45 CFR Part 164), GDPR Art. 9 (special-category health data), and local equivalents. Gemini's data-retention terms are governed by your contract with Google (and, on the OpenRouter path, by OpenRouter's and the upstream vendor's terms).
- Cost control. A single run can fan out to many paid Gemini calls. Set a
hard ceiling with a Google Cloud billing quota, and optionally
--max-cost-usd <n>as a client-side soft cap.
This software is not a medical device and must not drive clinical decisions.
This project treats governance under regulation as a first-class concern. The project-local artefacts behind that posture:
- docs/COMPLIANCE.md — 13-row EU AI Act article matrix + 19-row NIST AI RMF function matrix + ISO/IEEE/HIPAA cross-reference + gap roadmap + review cadence.
- docs/ROADMAP.md — everything still open: next-version engineering items, OWASP Top 10 for LLM Applications mapped to this system (including prompt injection through text inside images), the next research study, and the path to any real deployment.
- docs/architecture/decisions/ — seven ADRs (ADR-001…007). ADR-004 names seven trigger conditions (T1–T7) for migrating from single-model Gemini to a hybrid SLM router (single-model monoculture defense); ADR-006 records the OpenRouter second-provider adapter added when trigger T1 fired (the default Gemini model was retired for new API keys, September 2026); ADR-007 records the per-run manifest and hash-chained audit ledger that closes the Art. 12 / MANAGE 4.1 gaps, together with the PHI pre-flight and the input refusals.
- src/domain/fairness.ts — allocative-harm probe: demographic-token list + diagnostic-justifier window heuristic. Exercised by tests/e2e/fairness.test.ts (mocked) and tests/live/cli-full.test.ts (real Gemini output).
- Mandatory
DISCLAIMERfield on every output type, enforced at the TypeScript type level in src/domain/types.ts and asserted as a walk-the-tree hard test in Scenario 8 of tests/e2e/full-analysis.test.ts.
| Gate | Result |
|---|---|
npm test (default, 40 suites of mocked tests) |
876 / 876 pass, 4 snapshots — coverage 98.86 % stmts · 97.38 % branches · 98.52 % funcs · 99.12 % lines (floor enforced in jest.config.js: 97 / 92 / 94 / 97) |
npm run test:live (opt-in, real Gemini API) |
requires GOOGLE_API_KEY; skipped by default |
tsc --noEmit |
clean |
eslint src tests |
clean |
These numbers are a point-in-time snapshot; run npm test / npm run test:coverage for the current figures rather than trusting this table as
tests are added. Skip behaviour: with no GOOGLE_API_KEY / GEMINI_API_KEY
exported, the live suite passes a guard test and console-warns how to enable
the real-API run. Default npm test excludes tests/live/ via
testPathIgnorePatterns.
GPL v3 — see LICENSE for details.
Copyright © 2026 Andrei N. Besleaga, Independent Researcher and Software Architect. See NOTICE for copyright and attribution, DISCLAIMER.md for use restrictions (research and education only; not a medical device; no warranty), and docs/PRIOR-ART.md for the dated public disclosure of the system's technical elements.

