Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"name": "data-engineering-knowledge-capture",
"source": "./",
"description": "Capture data-platform knowledge \u2014 schemas, lineage, medallion, semantic layer, business glossary \u2014 into OKF. Extends PKC; depends on OKF.",
"version": "0.5.1",
"version": "0.5.2",
"author": {
"name": "Rick Hightower"
},
Expand Down
2 changes: 1 addition & 1 deletion .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "data-engineering-knowledge-capture",
"version": "0.5.1",
"version": "0.5.2",
"description": "Data Engineering Knowledge Capture (DEKC) \u2014 extends PKC + OKF for schemas, lineage, medallion layers, SQL/DAX, semantic models, business objects, and an indexed second brain. Agents walk data lakes. Works in Claude Code, Grok Build, Codex, and OpenCode.",
"author": {
"name": "Rick Hightower",
Expand Down
2 changes: 1 addition & 1 deletion .codex-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "data-engineering-knowledge-capture",
"version": "0.5.1",
"version": "0.5.2",
"description": "Data Engineering Knowledge Capture (DEKC) \u2014 schemas, lineage, medallion layers, semantic models, business glossary. Extends PKC; depends on OKF. Codex-native skills + hooks port.",
"author": {
"name": "Rick Hightower",
Expand Down
2 changes: 1 addition & 1 deletion .cursor-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "data-engineering-knowledge-capture",
"version": "0.5.1",
"version": "0.5.2",
"description": "Data Engineering Knowledge Capture: schemas, lineage, medallion, semantic models, and glossary into a durable OKF knowledge graph. Multi-host bindings and write isolation.",
"author": {
"name": "Rick Hightower"
Expand Down
4 changes: 2 additions & 2 deletions .grok-plugin/marketplace.json
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
{
"name": "dekc-plugin-marketplace",
"description": "Optional native Grok marketplace metadata. Grok Build loads Claude plugins with zero config; this pins identity for listings.",
"version": "0.5.1",
"version": "0.5.2",
"plugins": [
{
"name": "data-engineering-knowledge-capture",
"source": ".",
"description": "DEKC \u2014 data engineering knowledge capture on PKC + OKF. Claude-compatible.",
"version": "0.5.1",
"version": "0.5.2",
"compatibility": {
"claude_plugin": true,
"zero_config": true
Expand Down
2 changes: 1 addition & 1 deletion .opencode/plugin/dekc.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "data-engineering-knowledge-capture",
"version": "0.5.1",
"version": "0.5.2",
"description": "OpenCode port of DEKC \u2014 same skills/agents/scripts as Claude & Codex. Policy via AGENTS.md.",
"skills": "../../skills",
"agents": "../../agents",
Expand Down
13 changes: 12 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,16 @@ Plugin root: `${CLAUDE_PLUGIN_ROOT}`.
| **data-lake-walker** | Default orchestrator: walk → produce → adversarial grade → index |
| **reverse-engineering-orchestrator** | Multi-cloud RE (Fabric/AWS/GCP), strict LoopPolicy + fan-out |

## Query-time retrieval

Orthogonal to reverse engineering. Search, score, pack, and deepen stay in the sub-agent. The parent sees a **summary card only**.

| Agent | Role |
|-------|------|
| **data-retriever** | Query-time data-plane context. Spawn via `/dekc-retrieve`. Never dump hit lists or pack markdown into the parent. |

When the question also needs project decisions or system topology, fan out PKC **knowledge-retriever** and/or SAC **architecture-retriever** in parallel. Do not run `dekc_search.py` / `dekc_pack.py` / `dekc_brain.py` in the parent for Q&A.

## Producer workers

| Agent | Role |
Expand Down Expand Up @@ -131,7 +141,8 @@ python3 tests/test_dekc.py
| medallion health | layer-auditor / dekc-doctor |
| wireframe / architecture / ERD / job diagrams | dekc-diagram / dekc_diagram.py |
| lake / mart / catalog / stream / DQ / ingestion job | dekc-platform / dekc_platform.py |
| search the second brain | dekc-search / dekc-index |
| retrieve / what's this table / metric / impact Q&A | data-retriever / dekc-retrieve |
| search the second brain (raw hits, not parent Q&A) | dekc-search / dekc-index |
| multi-agent loop authoring | okf-agent-graph `/ager-*` + DEKC KnowledgeBind |

<!-- worklog:policy:start -->
Expand Down
21 changes: 21 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,27 @@

## Unreleased

## 0.5.2 — 2026-09-12

Query-time retrieval sub-agent, matching the PKC/SAC parity pattern: search +
score + pack + deepen stay off the parent; the parent gets a summary card only.

### Added

- **`data-retriever`** agent — retrieval-only. No capture, walk, reverse
engineer, or knowledge writes. Returns a retrieval card (seed, fit, engine,
pack stats, lead nodes, lineage note, gaps, next).
- **`/dekc-retrieve`** skill + slash command. Parent must spawn `data-retriever`
and must not run `dekc_search.py` / `dekc_pack.py` in-process for Q&A.
Orthogonal fan-out: PKC `knowledge-retriever`, SAC `architecture-retriever`.
- `dekc_pack.py --summary` — compact card-friendly output, bodies off (root
included), fail-closed token budget. Retriever prefers `--tiny --summary`.

### Notes

- Reverse-engineering agents (`data-lake-walker`, `lineage-tracer`, …) are
unchanged. Query-time is a separate section on the RE orchestrator.

## 0.5.1 — 2026-08-31

Correctness patch for catalog rendering and the rg-backed reverse index. Both
Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ DEKC **extends [Project Knowledge Capture (PKC)](https://github.com/SpillwaveSol
|---|---|
| **Plugin name** | `data-engineering-knowledge-capture` |
| **Repo** | [SpillwaveSolutions/data-engineering-knowledge-capture](https://github.com/SpillwaveSolutions/data-engineering-knowledge-capture) |
| **Version** | 0.5.1 |
| **Version** | 0.5.2 |
| **License** | MIT |
| **Hosts** | Claude Code · Grok Build · Codex · OpenCode · Agent Plugins 1.0 · Grok Bot · LangChain Deep Agents |

Expand Down Expand Up @@ -208,12 +208,14 @@ python3 scripts/dekc_doctor.py --repo . --bundle knowledge
python3 scripts/dekc_search.py "revenue" --repo . --bundle knowledge
```

Slash / skill entry points: `/dekc-init` · `/dekc-walk` · `/dekc-lineage` · `/dekc-business-object` · `/dekc-glossary` · `/dekc-semantic` · `/dekc-context` · `/dekc-search` · `/dekc-index` · `/dekc-doctor`
Slash / skill entry points: `/dekc-init` · `/dekc-walk` · `/dekc-lineage` · `/dekc-business-object` · `/dekc-glossary` · `/dekc-semantic` · `/dekc-retrieve` · `/dekc-context` · `/dekc-search` · `/dekc-index` · `/dekc-doctor`

## Agent loop (AGER-shaped): producers + adversarial judges

**Orchestrators:** `data-lake-walker` (default) · `reverse-engineering-orchestrator` (Fabric/AWS/GCP).

**Query-time:** `data-retriever` (`/dekc-retrieve`) — search + pack stay in the sub-agent; the parent gets a summary card only. Not an RE worker.

Producer workers fan out, then **adversarial skeptics grade reverse engineering with rubrics** before index/pack. Lead judge: **re-adversary-judge** (threshold **0.75**). Fail → re-plan or **retract** unproven claims — never invent edges to pass.

```text
Expand Down
72 changes: 72 additions & 0 deletions agents/data-retriever.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---
name: data-retriever
description: Retrieve DEKC data-plane context without contaminating the parent. Use for Table, Metric, LineagePath, IngestionJob, Transformation, Dashboard, DataProduct, GlossaryTerm, BusinessObject queries. Search, score fit, pack, optionally deepen/lineage. Returns a summary card only.
---

You are **Data Retriever** — the query-time retrieval sub-agent for DEKC (Data Engineering Knowledge Capture).

You are **not** a reverse-engineering agent. Do not confuse this role with `data-lake-walker`, `lineage-tracer`, `schema-scout`, or other capture/RE workers. You read the knowledge graph. You do not grow it.

This is the PKC `knowledge-retriever` / SAC `architecture-retriever` parity role for the data plane.

## Contract

- **Retrieval-only.** No capture, walk, reverse-engineer, promote, materialize, index rebuild, or knowledge writes. No `brain_session`.
- **Never** return full hit lists or full pack markdown to the parent.
- Search, score, pack, and deepen stay **inside this sub-agent**.
- The parent receives a **summary card only**.
- Fiction samples only in public trees (Northstar / Lumenfield). Never name a private remote.

## Card shape (only thing the parent sees)

```markdown
## Retrieval card
- Query: …
- Seed: `/path` (`Type`) — why chosen
- Fit: high|medium|low — one sentence
- Engine: index|rg|scan
- Pack: hops=N nodes=N tokens=N/budget
- Lead nodes: (5–8 bullets)
- Lineage note: (upstream/downstream one-liner) or none
- Open gaps: or none
- Next: stay|deepen-2hop|lineage|try-alt-seed `/other`
```

## Scripts (read-only)

```bash
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_search.py" "<query>" --repo . --bundle knowledge --limit 5
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_search.py" "<query>" --type Table,Metric,LineagePath,DataProduct --limit 5 --json
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_pack.py" <seed> --repo . --bundle knowledge --tiny --summary
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_pack.py" <seed> --repo . --bundle knowledge --hops 2 --summary
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_lineage.py" --repo . --bundle knowledge upstream <seed> --hops 2
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_lineage.py" --repo . --bundle knowledge downstream <seed> --hops 2
```

Prefer `--summary` on every pack. Fall back to `--tiny` without `--summary` only if the flag is missing. Do **not** run `dekc_brain.py` for the parent — those packs stay here if you need an intent hint, and they still must not leak.

Forbidden: `dekc_capture.py`, `dekc_walk.py`, `dekc_lineage.py materialize`, `dekc_business.py`, `dekc_pack.py --write`, `brain_session.py`.

## Workflow

1. **Seed or search.** If the caller already named a concept path, use it. Otherwise `dekc_search.py --limit 5`.
2. **Score fit** (no LLM in Python — you score). Prefer **Table / Metric / LineagePath / DataProduct**. Then IngestionJob, Transformation, Dashboard, BusinessObject, GlossaryTerm.
- **high** — preferred type and title/path match the question
- **medium** — preferred type *or* a strong keyword hit
- **low** — glossary-only, wrong layer, or weak lexical match
3. **Pack** `--tiny --summary`. Bodies stay off. Read hops / nodes / tokens / engine / lead nodes from the summary.
4. **Deepen or lineage** only if the card would otherwise be thin, or the question is impact / blast-radius. Use `--hops 2 --summary` or the lineage upstream/downstream helpers. **Max 2 deepen steps.**
5. Return the **card only**. One card. No appendix.

## Fit and Next

- `stay` — seed answers the question
- `deepen-2hop` — neighborhood is thin; parent may re-spawn you with a 2-hop ask
- `lineage` — impact / upstream / downstream still open
- `try-alt-seed /other` — better seed exists; name it

Open gaps: missing gold BO, no landing job, broken link, empty search. Write `none` when the pack is enough.

## Orthogonal fan-out

If the question also needs project decisions or system topology, tell the parent to spawn PKC **knowledge-retriever** and/or SAC **architecture-retriever** in parallel. Do not impersonate those agents. Do not pack their nouns.
13 changes: 13 additions & 0 deletions agents/reverse-engineering-orchestrator.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,3 +88,16 @@ Full topologies: `docs/designs/current_design_doc.md`.
- Pass/fail vs RE threshold **0.75**
- Explicit revise list if fail
- Never claim reverse engineering complete without judge pass

## Query-time (not reverse engineering)

**data-retriever** is query-time only. It is **not** an RE sub-agent. Do not put it in producer or skeptic fan-out.

When someone asks about an existing Table, Metric, LineagePath, IngestionJob, Transformation, Dashboard, DataProduct, GlossaryTerm, or BusinessObject:

1. Spawn **data-retriever** (via `/dekc-retrieve`).
2. Do **not** run `dekc_search.py`, `dekc_pack.py`, or `dekc_brain.py` in this orchestrator for that Q&A.
3. Keep the **retrieval card** only — never a hit list or full pack.
4. If the question also needs project decisions or system topology, fan out PKC **knowledge-retriever** and/or SAC **architecture-retriever** in parallel.

See `agents/data-retriever.md` and `skills/dekc-retrieve/SKILL.md`.
10 changes: 10 additions & 0 deletions commands/dekc-retrieve.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
name: dekc-retrieve
description: Query-time retrieval. Spawn data-retriever; parent keeps a summary card only.
---

Run the **dekc-retrieve** skill.

User request: `$ARGUMENTS`

Follow `${CLAUDE_PLUGIN_ROOT}/skills/dekc-retrieve/SKILL.md` completely. Spawn **data-retriever**. Do not run `dekc_search.py` or `dekc_pack.py` in the parent. Return the retrieval card only.
4 changes: 3 additions & 1 deletion docs/GROK_BOT.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,9 @@ Writes fail closed without `--author` or `SECOND_BRAIN_IDENTITY`. Successful kno

Default ContextPack: **2 hops / ~20 nodes**.

Pack before answering or writing. Do not dump the entire tree.
For Q&A, spawn **data-retriever** (`/dekc-retrieve`). Search, score, pack, and deepen stay in that sub-agent. The parent keeps a **retrieval card** only — never a hit list, full pack, or `dekc_brain.py` dump. Pack `--summary` (bodies off) is the card-friendly path.

Do not dump the entire tree. When the question also needs project or architecture context, fan out PKC **knowledge-retriever** and/or SAC **architecture-retriever** in parallel.

## Skill binding

Expand Down
2 changes: 1 addition & 1 deletion docs/ONBOARDING.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ See [GROK_BOT.md](GROK_BOT.md) for the binding contract.

1. State your identity: `Grok Bot: Data Engineering Knowledge Capture`.
2. Confirm the knowledge root (`SECOND_BRAIN_ROOT` or the target bundle).
3. Pack the relevant subgraph (2 hops) before answering or writing.
3. For Q&A, spawn **data-retriever** (`/dekc-retrieve`) and keep the retrieval card only. Do not dump a 2-hop pack or `dekc_brain.py` output into the parent. Pack before writing.
4. Persist only through skills + deterministic scripts inside an isolation session when writing a shared brain.
5. Report path + validation result, not a dumped graph.

Expand Down
2 changes: 2 additions & 0 deletions docs/designs/current_design_doc.md
Original file line number Diff line number Diff line change
Expand Up @@ -314,6 +314,8 @@ Compensation is rare (knowledge is additive); “undo” = revert Git commit of

Orchestrator and workers load **2-hop packs** (~20 nodes) via RetrievalBinding, not the full lake graph. Judge sees doctor JSON + gap list only. This matches OKF progressive disclosure and AGER ContextIsolationPolicy for subagents.

Query-time Q&A is a different path: spawn **data-retriever** (`/dekc-retrieve`). Search, score, pack (`--summary`), and deepen stay in that sub-agent; the parent keeps a retrieval card only. Do not confuse this with RE producers.

### 3.10 Optional formal AGER bundle

Teams can author an explicit AGER bundle (via `/ager-init`) that **KnowledgeBinds** the DEKC `knowledge/` root and **uses** DEKC scripts as Tools. DEKC agents remain usable without that bundle; the AGER bundle is the portable runtime config for LangGraph/CrewAI adapters.
Expand Down
1 change: 1 addition & 0 deletions docs/designs/retrieval-ladder.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ the tree. The curate hook does not install rg and does not build the index.
|---|---|---|
| `dekc_search.py` | yes | candidate prefilter |
| `dekc_pack.py` inbound | yes | reverse index via literal path + self SQL |
| `dekc_pack.py --summary` | n/a | card-friendly render; bodies off; fail-closed budget |
| `dekc_validate.py` / `dekc-curate.sh` | **no** | must parse frontmatter + resolve links |
| orphans / doctor listings | **no** | need the parsed graph |
| mermaid (`build_graph`) | **no** | leftover full scan; accepted |
Expand Down
13 changes: 10 additions & 3 deletions docs/user_guide/user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,9 @@ python3 scripts/dekc_brain.py "silver.orders" --intent impact --repo . --bundle

Each intent returns: design checklist, ranked schema-typed concepts, progressive disclosure pack, lineage snippet.

Skills: `dekc-second-brain`, `dekc-design-report`, `dekc-land-data`.
For Q&A, spawn **data-retriever** (`/dekc-retrieve`) instead of dumping a brain pack into the parent.

Skills: `dekc-retrieve`, `dekc-second-brain`, `dekc-design-report`, `dekc-land-data`.

Patterns: `patterns/design-report-from-gold.md`, `patterns/land-stream-to-bronze.md`.

Expand Down Expand Up @@ -221,6 +223,9 @@ python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_pack.py" tables/gold-order-daily.md
# tiny pack for chat focus
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_pack.py" tables/gold-order-daily.md \
--repo . --bundle knowledge --tiny
# card-friendly summary (bodies off) — prefer this from data-retriever
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_pack.py" tables/gold-order-daily.md \
--repo . --bundle knowledge --tiny --summary
```

If `okf-graph-eng` is available, prefer its pack/impact for the same paths.
Expand All @@ -237,8 +242,9 @@ If `okf-graph-eng` is available, prefer its pack/impact for the same paths.
| `dekc-business-object` | Promote technical → business + glossary |
| `dekc-glossary` | Author glossary terms |
| `dekc-semantic` | Semantic models, metrics, dashboards |
| `dekc-context` | Context packs |
| `dekc-search` | Second-brain search |
| `dekc-retrieve` | Query-time retrieval — spawn `data-retriever`; card only |
| `dekc-context` | Context packs (retriever internals / known seed) |
| `dekc-search` | Second-brain search (raw hits; not parent Q&A) |
| `dekc-index` | Rebuild index |
| `dekc-grade` | Rubric grade + adversarial RE protocol |
| `dekc-doctor` | Health: coverage, orphans, validation |
Expand All @@ -247,6 +253,7 @@ If `okf-graph-eng` is available, prefer its pack/impact for the same paths.

| Agent | Use when |
|-------|----------|
| **data-retriever** | Query-time Q&A — spawn via `/dekc-retrieve`; parent keeps a card only |
| **data-lake-walker** | Full reverse-engineer loop (orchestrator) |
| **schema-scout** | Only structure (schemas/tables/columns) |
| **lineage-tracer** | Only SQL/job lineage and promotions |
Expand Down
2 changes: 1 addition & 1 deletion hosts/grok-bot/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,6 @@ Read `docs/ONBOARDING.md` first, then follow `docs/GROK_BOT.md`.

1. Identity: `grok-bot/data-engineering-knowledge-capture`
2. Open an isolation session before knowledge writes (`scripts/brain_session.py open`) unless the human already pointed `SECOND_BRAIN_ROOT` at a session worktree.
3. Pack 2 hops, then write owned types only via this plugin's scripts.
3. For Q&A, spawn **data-retriever** (`/dekc-retrieve`); keep the card only. Pack 2 hops before writing owned types via this plugin's scripts.
4. Close the session to PR. Report path + validation result.
5. Never document a private remote. Never write raw Markdown into the tree.
2 changes: 1 addition & 1 deletion marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"name": "data-engineering-knowledge-capture",
"source": "./",
"description": "Data lake second brain: schemas, lineage, medallion, semantic layer, glossary. Depends on PKC + OKF.",
"version": "0.5.1",
"version": "0.5.2",
"author": {
"name": "Rick Hightower"
},
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,6 @@
"vite": "^8.2.0"
},
"description": "Data Engineering Knowledge Capture (DEKC) \u2014 Claude/Grok/Codex/OpenCode plugin + explorer",
"version": "0.5.1",
"version": "0.5.2",
"license": "MIT"
}
2 changes: 1 addition & 1 deletion plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
"name": "data-engineering-knowledge-capture",
"version": "0.5.1",
"version": "0.5.2",
"description": "Data Engineering Knowledge Capture: schemas, lineage, medallion, semantic models, and glossary into a durable OKF knowledge graph. Multi-host bindings and write isolation.",
"author": {
"name": "Rick Hightower",
Expand Down
2 changes: 1 addition & 1 deletion public/data/catalog.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"title": "Retail Lake Sample Knowledge",
"version": "0.5.1",
"version": "0.5.2",
"depends_on": [
"project-knowledge-capture",
"okf-graph-eng"
Expand Down
Loading
Loading