Ourios

οὔριος · the fair, following wind that fills a ship's sail. Logs flow forward, under fair wind, with minimal friction.

Ourios is a log storage and query backend built on three off-the-shelf parts:

• Apache Parquet — columnar storage on S3-compatible object storage, with predicate pushdown via row-group statistics, bloom filters, and page indexes.
• A Drain-derived online template miner — collapses log lines to (template_id, params) at ingest time: a logical 50–200× reduction that drives query pruning (not a bytes-vs-codec claim; see RFC 0011).
• Apache DataFusion — a production-ready vectorised SQL execution engine. We hand it logical plans; it does the work.

The thesis: this combination collapses the inverted index, the compression layer, the storage tier, and the query engine into one stack of off-the-shelf parts plus thin glue. (Compression here is Parquet's byte codec plus the miner's logical template reduction — the query-pruning win noted above, not bytes that beat a codec.) The novel work is gluing the parts together honestly and handling the failure modes of template mining rather than pretending they don't exist.

---

Note

Status: pre-release, under active RFC-driven development. The full path — OTLP in; WAL, miner, Parquet on object storage; logs-DSL / DataFusion queries out — is implemented and tested behind RFC acceptance gates, and the performance thesis is measured (including against Grafana Loki) in docs/benchmarks.md. Signed pre-release binaries, container images, and a Helm chart are published with every release, but interfaces and the on-disk schema can still move: treat everything as pre-1.0.

---

What Ourios is for

Ourios is built for logs specifically, by leaning on two facts about how logs behave:

1. Log lines are generated from a small number of printf templates. Mine the template once, key every occurrence by a small, stable template_id, and selective queries prune to the row groups that can contain their answer.
2. The most common log query shape is "filter by time + attributes, fetch a bounded result" — exactly what Parquet + DataFusion is already good at when the file layout is right.

If your logs and queries look like that, it may be a good fit. The workload-fit section below is the honest version of when it isn't.

What the template model makes possible

• Template-exact queries. template_id == 42 filters by the mined template, not by string match — "every occurrence of the login-error template, across every service" prunes to the row groups whose bloom filters can contain it. resolves_to(42) expands an operator-asserted alias set, so a template that was renamed across a deploy still answers as one query.
• Drift as a first-class query. drift from -7d to now reads the miner's audit stream and reports the templates that appeared, widened, or changed in the window. A deploy that changes a log line from "user logged in" to "user authenticated" shows up here — at the time of the change, not three weeks later when your alert stops firing.

Workload fit: when to choose Ourios — and when not

Ourios is not "better than Loki" (or anything else); it is built for a specific workload shape, and the honest comparison (RFC 0031; every benchmark query pair — the same question posed to both systems — is machine-checked before it counts: row-level answers must be multiset-identical, and the aggregation pair is checked within a documented completeness margin for the reference system's own under-counting) is evidence of fit, not a ranking.

Choose Ourios when this is your workload:

• Your logs arrive as OTLP (a Collector already in front) and you want them stored with exact OTLP fidelity — every field back out, bit-identical bodies.
• Your bread-and-butter queries are selective: needles by template, trace-ID correlation, severity outliers, counts by template — the shapes that prune to a handful of row groups instead of scanning.
• You care about predictable query cost over long retention. The measured profile is flat: 39–86 ms across every latency-measured comparative pair at 4.9 M records — needles, severity, and window browses alike — where the grep-architecture reference spans 13.8 ms to 24.1 s depending on how much it must scan (dataset, environment, and the pair definitions: docs/benchmarks.md §9.13). Flatness, not any single ratio, is the headline: cost tracks the result, not the corpus.
• You want one binary on object storage: no index tier, and no fleet of distinct services — the receiver, querier, and compactor are roles of the same ourios-server binary, run together in one process or (as the Helm chart does) as separately scaled workloads.

Choose something else when:

• Arbitrary substring hunts over recent data are the daily shape. A label + time index (Loki's design) is genuinely better at "the most recent few thousand rows matching this string, fast" — our time-window browse is a documented storage-bytes loss (docs/benchmarks.md §9.13), and we publish it rather than argue with it.
• You live in the Grafana ecosystem and its integrations are the point. Ourios has a Grafana datasource for its queries, but the rest of that ecosystem is built around Loki.
• You need years of operational hardening today. Ourios is pre-release; the incumbents have production miles we simply do not.

One honesty note on the numbers: the comparison measured the architecture's antithesis (grep-shaped storage). Against columnar engines closer to our own design — ClickHouse-family, VictoriaLogs-style — the margins would be thinner; that comparison is a possible future RFC, and until it runs we claim nothing about it.

Non-goals

• Metrics and traces. OTLP logs only. Traces are linked by trace_id but not stored.
• Managed-service features. We ship a binary, container images, and a Helm chart.
• Being a SIEM. No correlation engine, no detection rules, no alerting DSL beyond what Grafana/Prometheus already provide.
• Being a Loki / Mimir / ClickHouse replacement in their full scope. If "just use $X" covers your use case, do that.

---

Architecture, in one picture

                  ┌──────────────────────────────────────────────────┐
  OTel SDK /      │  OTLP gRPC (:4317) / HTTP (:4318)                │
  Collector ────► │  TLS/mTLS · bearer/OIDC · per-tenant binding     │
                  │       │                                          │
                  │       ▼                                          │
                  │  ┌───────────┐   ┌─────────────┐   ┌───────────┐ │
                  │  │   WAL     │──►│ template    │──►│  Parquet  │ │
                  │  │ (fsync,   │   │ miner       │   │  writer   │ │
                  │  │ then ack) │   │ (per-tenant)│   └─────┬─────┘ │
                  │  └───────────┘   └─────────────┘         │       │
                  │                    receiver              │       │
                  └──────────────────────────────────────────┼───────┘
                                                             ▼
                ┌──────────────┐              ┌──────────────────────┐
                │  compactor   │ ◄──────────► │ S3-compatible object │
                │ (background  │              │ store — per-tenant,  │
                │  merge)      │              │ hour-partitioned     │
                └──────────────┘              │ Parquet + blooms +   │
                                              │ audit stream         │
                                              └──────────┬───────────┘
                                                         │
                  ┌──────────────────────────────────────┼──────────┐
   logs DSL /     │  ┌──────────────┐  ┌──────────────┐  │          │
   JSON (:4319) ► │  │  DSL parser  │─►│  DataFusion  │◄─┘          │
   + MCP (/mcp) ◄ │  │ (own grammar)│  │  logical plan│             │
                  │  └──────────────┘  └──────────────┘             │
                  │                      querier                    │
                  └─────────────────────────────────────────────────┘

One binary, three roles — receiver, querier, and the background compactor. Two orderings are load-bearing: a batch is fsync'd to the WAL before it is acknowledged and before the miner touches it (WAL-before-ack, AGENTS.md §3.4), and object storage is the source of truth — in the S3-backed topology local disk holds only the WAL and caches (§3.6).

Receivers and queriers scale out; the compactor does not. Run exactly one compactor per store. On the local backend two compactors can lose rows, and with erasure a second compactor can bring erased graph tuples back (RFC 0047 §3.6). The Helm chart runs one compactor per release, but it cannot see other releases, so install one release per store.

All storage seams run against S3-compatible object storage (RFC 0013, RFC 0019); the local filesystem remains the zero-dependency development backend, and the WAL stays on local disk by design.

What is implemented

Everything below is built RFC-first: each RFC pins Given / When / Then acceptance scenarios, which land as failing (red-gate) tests before the code that turns them green. Statuses live in each RFC's frontmatter (docs/rfcs/).

• Ingest — OTLP gRPC + HTTP receiver with the full compliance-scenario set (RFC 0003, RFC 0018); WAL with fsync-before-ack and a SIGKILL crash-recovery test in CI (RFC 0008); WAL reclamation and quiesce recovery (RFC 0052); the record-sink write path (RFC 0014).
• Template miner (RFC 0001) — per-tenant Drain-derived trees, three-zone confidence matching (a line below the floor mints its own template), whole-token masking of UUIDs, IPv4 addresses, timestamps and numbers (including decimals), byte-identical reconstruction (lossy lines flagged and retained verbatim), audited merges and aliases, bounded memory at corpus scale (RFC 0023), template ids that stay unique across restarts (RFC 0059), and ourios.miner.* telemetry defined through an OTel weaver registry.
• Storage — per-tenant, hour-partitioned Parquet on S3-compatible object storage (RFC 0005, RFC 0013, RFC 0019); bloom filters on template_id / trace_id / span_id; promoted queryable attribute columns (RFC 0022); background compaction (RFC 0009); an audit-event stream.
• Query — a pipe-composable logs DSL with its own grammar (no DataFusion or SQL leaking through): service == "api" and severity >= error | count by template_id | sort count desc | limit 10, resolves_to(42), half-open range(...), plus a structured JSON surface (RFC 0002, RFC 0007); template-registry body rendering (RFC 0017); the drift from <t1> to <t2> audit query (RFC 0010); the HTTP query endpoint (RFC 0016); a cached template-map artifact (RFC 0033).
• Security & tenancy — enforced multi-tenant binding on every surface (RFC 0026), with the tenant named out of band by the credential and request header, never by the data (RFC 0046); static bearer tokens and OIDC (RFC 0029); TLS/mTLS with hot certificate reload on the listeners (RFC 0030); an OpenFGA (ReBAC) resolver that can bind tenants and scope visibility inside a tenant (RFC 0047); and the ourios-server graph {erase,erasures,backfill} operator verbs for erasure and graph backfill (RFC 0048).
• Operations — YAML config file with ${env:VAR} substitution or pure OURIOS_* environment variables (RFC 0020); signed releases with offline-verifiable provenance (cargo-dist); container images including -static / -scratch variants; a Helm chart validated by a deploy test; dogfooded telemetry (Ourios ships its own logs to an OTLP endpoint and exposes OTel-defined metrics); nightly fuzzing (RFC 0015); SHA-pinned CI and a cargo-deny supply-chain gate.

Quick start

Grab a signed release (or cargo build --release -p ourios-server) and run one process with the receiver and querier roles on localhost (background compaction also runs by default; OURIOS_COMPACTION_ENABLED=0 turns it off):

mkdir -p /tmp/ourios/data /tmp/ourios/wal

OURIOS_BUCKET_ROOT=/tmp/ourios/data \
OURIOS_WAL_ROOT=/tmp/ourios/wal \
OURIOS_RECEIVER_ENABLED=1 \
OURIOS_RECEIVER_GRPC_ADDR=127.0.0.1:4317 \
OURIOS_RECEIVER_HTTP_ADDR=127.0.0.1:4318 \
OURIOS_QUERIER_ENABLED=1 \
OURIOS_QUERIER_HTTP_ADDR=127.0.0.1:4319 \
./ourios-server   # ./target/release/ourios-server if built from source

Point any OpenTelemetry SDK or Collector at the OTLP ports — on the ingest side Ourios speaks OTLP and nothing else (queries use the plain HTTP API below). Every export names its tenant out of band with the X-Ourios-Tenant header / x-ourios-tenant gRPC metadata (OTEL_EXPORTER_OTLP_HEADERS=x-ourios-tenant=checkout on an SDK, an exporter headers: entry on a Collector); resource attributes such as service.name describe the producer and never choose the tenant. Then query that same tenant with the logs DSL:

curl -s http://localhost:4319/v1/query \
  -H 'X-Ourios-Tenant: checkout' \
  -H 'Content-Type: text/plain' \
  -d 'severity >= error | count by template_id'

That returns one row per log template with its count. New records become queryable once the receiver flushes them to Parquet, within about five and a half minutes on a quiet tenant (the quickstart guide shows how to see them sooner). sort and limit after an aggregation are part of the query DSL but not implemented by the querier yet.

The quickstart guide walks this end to end, configuration covers the --config ourios.yaml file form, and Kubernetes (Helm) is the production topology on object storage. Read authentication before any listener leaves localhost.

Agents: the MCP surface

The querier can serve the Model Context Protocol at /mcp — opt-in via querier.mcp.enabled / OURIOS_QUERIER_MCP_ENABLED=1, off by default (RFC 0027): the query_logs, list_templates, and template_drift tools, plus the ourios://dsl-grammar resource and the ourios://query-schema cost-model resource (RFC 0032) that tells an agent which query shapes prune and which scan before it spends a query. Same auth, same tenant binding as the HTTP API.

How it measures

docs/benchmarks.md is the scoreboard: §7 defines the thesis gates, §9 records every run with its run IDs, hardware class, and caveats. Read the caveats before quoting anything. Where things stand:

• Pruning gates (B1/B2): pass on the hardware baseline (8 dedicated vCPU / 32 GiB). Predicate-needle queries clear the ≥ 10× gate against a zstdcat | grep reference with exactly matching row counts, and windowed template-exact queries stay flat as the corpus doubles — result-bound, not corpus-bound — at 16 GiB scale.
• Reconstruction (C1): 1.000000 — every non-lossy row reconstructs bit-identically; lossy rows are flagged and retained.
• On-disk compression vs. zstd (A1): a recorded diagnostic, not a gate (RFC 0011) — a whole-stream codec captures the same redundancy; the miner's reduction pays in pruning, and the doc says so plainly.
• Against Grafana Loki (RFC 0031; shared CI runner, so indicative — both systems ingest the identical OTLP stream and every counted pair is machine-checked for result-set equivalence): the two classes the thesis stakes itself on hardest pass their frozen 10× storage-bytes must-wins — template-exact lookup at ~77× and trace correlation at ~22× fewer bytes fetched — and on the program's latency channel the needle classes answer in ~75 ms against 23–24 s for Loki (308–323×). Time-window browses are an honest, documented loss on the storage-bytes channel (Loki's label + time index is genuinely better at "give me k recent rows"), while the RFC's latency floor for them passes as written. The frozen gates now assert on every comparative-bench dispatch, as a regression gate.

Repository layout

crates/      # Rust workspace: ourios-{core,config,miner,wal,parquet,
             #   ingester,querier,server,serving,df-otel,telemetry,
             #   semconv,testgen,bench}
deploy/helm/ # the Helm chart
docs/        # mdBook source: guides, architecture, hazards, RFCs,
             #   benchmarks, verification, roadmap, glossary, talks
fuzz/        # cargo-fuzz harness (its own nightly-only workspace)
semconv/     # REGISTRY_REF: the pinned ref of the external ourios-semconv
             #   registry repo that crates/ourios-semconv is generated from
testdata/    # anonymised seed corpora; the multi-GB bench corpora are
             #   captured by CI workflows and published as corpus/* tags

Documentation

The docs are an mdBook, published at https://jensholdgaard.github.io/ourios/ (source under docs/):

• Guides — quickstart, configuration, Docker, Kubernetes, authentication.
• docs/rfcs/ — how we make decisions, and the RFCs that specify everything implemented so far.
• docs/hazards.md — where projects in this space die, and how we won't.
• docs/benchmarks.md — the thesis gates and the measurements against them.
• docs/verification.md — how an RFC criterion becomes a red-gate test becomes a green one.
• docs/glossary.md — terms of art (Parquet, Drain, row group, …).
• docs/talks/ — lecture-length explanations, starting with the template miner.

Development process

Every change that touches a pillar, invariant, or hazard goes through an RFC with pinned acceptance scenarios; RFCs climb a maturity ladder (drafted → specified → red → green → validated → accepted) defined in docs/rfcs/README.md, and the scenarios land as failing tests before the implementation.

Ourios is also an experiment in how software gets built: development is intentionally fully AI-assisted (Claude), with a human maintainer owning direction, review, and every merge. AI-authored commits carry Co-Authored-By trailers and clear the same CI, review, and RFC bar as any human contribution — see AGENTS.md §9 and the "tests are specifications" discipline in §6.2 for how the known failure modes are handled.

Governance and contributing

• CONTRIBUTING.md — how to contribute (including the RFC process).
• CODE_OF_CONDUCT.md — how we work together; technical conduct only.
• SECURITY.md — vulnerability reporting.

Licensing

Apache License 2.0. See LICENSE.

Acknowledgements

• The Parquet project and the Apache Arrow ecosystem for making columnar storage a solved problem.
• Pinjia He et al. for the original Drain paper (IEEE ICWS 2017) and IBM for Drain3.
• The DataFusion and InfluxData teams for proving that a fast, vectorised, pluggable query engine can ship as an open-source library.