Record a job on screen, turn it into a playbook

[ldbumble]

Mail tells Taskuary what the job is about. It does not tell it how the job is done: which system, which screen, which field, what you check before you click, what you do when the number does not match. Today that knowledge gets into a playbook only after an agent has done the job once and proposed one at close.

The idea

Opt-in local screen recording while the owner performs a procedure once. Taskuary watches the recording and writes the playbook.

• The owner presses Record in the Docs section (Playbooks shelf), does the thing (approve the bills in the portal, run the census export, file the weekly report), presses Stop.
• The recording never leaves the box until the owner says so. Frames are sampled, the model reads them (plus window titles and the UI Automation tree where a native app exposes one), and extracts: the systems touched, the steps in order, the decision points, the exceptions, what "done" looked like.
• The output is a draft playbook in the existing six-line shape (when / uses / steps / alone / ask first / done when) plus prose, filed as a proposal. Nothing is saved without the owner's click, like every playbook today.
• A second recording of the same job refines the playbook rather than creating another.

Why this one is near-term, not far roadmap

Everything downstream exists. Playbooks are read by triage (the when line), ride into the agent's seed, and are proposed at close by playbooks.draft. The browser pane already records a walk the owner performed and replays it tomorrow (the needs:browser workflow repeat). This extends recording from one browser to the whole desktop, and the output from "replay" to "a written procedure an agent can follow and a human can read."

It also feeds #178 (computer use in a sandboxed desktop): a playbook written from a recording is exactly the instruction set an agent in a sandbox needs.

What to decide

• Capture: a lightweight local recorder (Windows Graphics Capture) at about 1 fps, plus the active window title per frame. No audio by default.
• Redaction before the model sees anything: the credential scrub (redact.py) runs on the extracted text, and the owner can blank regions.
• Where it shows: Docs, Playbooks, Record a job, next to the existing editor. The proposal lands on the Assistant like any other.
• Scope of the first version: one recording, one proposal, no refinement loop.

Status: good candidate to build next.

[ldbumble]

The first buildable slice is now #253: record on the task page, write the playbook from the recording, approve it like any playbook proposal. Replay and computer use (#178) stay out of it.