Add Pipecat voice traces and opt-in audio by delner · Pull Request #836 · braintrustdata/braintrust-sdk-python

David Elner (delner) · 2026-10-01T14:27:16Z

Inspect conversations, tool calls, and audio through the existing setup_pipecat() API. Supports cascade and OpenAI Realtime voice pipelines.

Readable conversations: transcripts, native turns, model responses, and tool calls.
Optional audio: call recordings and speech clips, with sample-based playback selections.
Progressive export: upload clips and call segments during conversation; configurable limits preserve completed audio.
Opt-in recording: no recording buffers or encoding when disabled; codecs install separately with braintrust[audio].

setup_pipecat(capture_audio_attachments=True)  # Ogg/Opus; keep the existing pipeline

Example trace

pipecat.pipeline                    call recording segments
├─ pipecat.user_turn                 transcript + caller clip
│  └─ pipecat.stt
├─ pipecat.assistant_turn
│  └─ pipecat.llm_response
│     └─ lookup_order                arguments + result
└─ pipecat.assistant_turn             spoken continuation
   ├─ pipecat.llm_response
   └─ pipecat.tts_response            transcript + generated clip

Design

Pipecat lifecycle/media hooks
  ├─ native spans ───────────────┐
  └─ opt-in recorder → worker ───┤
                                 ↓
                       Existing SDK exporter

Preserve native turns, IDs, and payloads; map speech to recorded samples.
Encode off the voice event loop with bounded buffers/queues; workers share application CPU and memory.
Update owning spans as attachments finish; cleanup drains recording work before final flush.
Pipecat 1.12.0 native hooks; unsupported versions/pipelines retain existing tracing. Output selections require equal-rate mono audio without mixing.

Performance

53-second cascade replay, 24 kHz audio, 30-second rotation, eight attachments. Means of two fresh-process trials on Apple M5 Pro; baseline is tracing with recording disabled.

Metric	Recording off	Ogg enabled	Added cost
Local response delay¹	20.92 ms	20.99 ms	+0.06 ms
CPU during call	2.84 s	4.16 s	+1.31 s (2.48% of one core over the call)
Peak RSS growth²	2.73 MiB	6.29 MiB	+3.56 MiB
Final recording drain	5 ms	395 ms	+390 ms
Audio / final trace JSON	0 / 30.6 kB	609 kB / 41.6 kB	+609 kB / +10.9 kB

¹ Provider-audio schedule to local output write, not caller-heard latency. ² Startup-to-final-drain RSS growth. Offline replay excludes network/upload; production latency and concurrency remain unmeasured. Ogg uses 10.7× fewer audio bytes than WAV in this call.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: babe2621bc

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "Codex (@codex) address that feedback".

chatgpt-codex-connector · 2026-10-01T14:32:27Z

    async def on_process_frame(self, data: Any) -> None:
+        if self._voice is not None:
+            return


Preserve LLM usage metrics in native voice traces

When a supported Pipecat 1.12 voice pipeline is bound, this early return prevents the legacy _capture_metrics path from seeing any MetricsFrame. NativeObserver.on_push_frame only stores those frames as pipecat.events; it never adds LLMUsageMetricsData or TTFB values to the pipecat.llm_response span. Consequently normal voice calls lose token/cost and time-to-first-token metrics even when Pipecat usage metrics are enabled. Extract and log the metrics in the native observer before bypassing the existing handler.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-10-01T14:32:27Z

 braintrust = "braintrust.integrations.harbor:HarborPlugin"

 [project.optional-dependencies]
+audio = ["numpy>=1.26", "soundfile>=0.13.1"]


Include audio dependencies in the all extra

This adds the audio optional extra, but the existing all extra still omits both NumPy and SoundFile. Users following the documented pip install "braintrust[all]" path and then enabling Pipecat audio recording will hit the encoder's missing-dependency error unless they separately discover and install braintrust[audio]; add these dependencies to all so it continues to install every optional feature.

Useful? React with 👍 / 👎.

Abhijeet Prasad (AbhiPrasad)

most of these tests seem not necessary and use a lot of mocks/fakes. Can we just use vcr tests instead? We can probably combine some redundant tests or remove lower value tests in py/src/braintrust/integrations/pipecat/test_pipecat.py as well

Abhijeet Prasad (AbhiPrasad) · 2026-10-07T14:54:04Z

+def test_prepare_recording_preserves_disabled_result():
+    assert prepare_recording("call", lambda: None) is None


I'd take another pass to see if we can remove low value tests like this, or tests that use excess mocks/fakes.

also asking the agent to combine redundant tests is also pretty effective.

a lot of these unit tests don't give me a huge amount of confidence, but maybe that is fine.

Yeah I tried to do a couple of passes on this, not sure if it was very effective at reduction. I might have to do some more careful combing.

Abhijeet Prasad (AbhiPrasad) · 2026-10-07T14:55:25Z

can we remove these added readmes? The code/file structure should self document itself just fine, these are going to go stale.

Abhijeet Prasad (AbhiPrasad) · 2026-10-07T14:56:06Z

+from .options import RecordingOptions
+
+
+__all__ = ["RecordingOptions"]


why is this the only export from this folder? Does everything else get exported directly?

Abhijeet Prasad (AbhiPrasad) · 2026-10-07T14:59:29Z

+from .worker import RecordingBusy
+
+
+encoded_budget = ByteBudget(32 * 1024 * 1024)


IMO we don't need a separate budget for audio. We can just use the generic attachment budget.

Abhijeet Prasad (AbhiPrasad) · 2026-10-07T15:06:29Z

I find it hard to reason about all the unit versions (sec/ms, byte offsets, etc.). I wonder if we can have some stronger named helpers to reason about this

the classes here feel like they mix reponsibilities too much. class Alignment does both mapping to recording and calling span.log. I'd rather those two be explicitly two different steps / functions / classes

I did some vibing about the class structure, and I think I'd like to see something like so (following is AI GENERATED):

Class Owns / does

RecordingSession Call-level state: accepts capture while active, rotates segments, handles limits, drains on finish. Owns the active segment and tracks pending segment jobs.

AudioSegment One segment’s state and timeline: OPEN → PENDING → READY / OMITTED. When submitted, it holds an immutable PCM snapshot; once the worker finishes, it holds the encoded result or omission reason.

AudioEncoder Converts a segment snapshot into encoded bytes and metadata. Keeps codec details out of recording lifecycle code.

RecordingWorker Runs encoding off the event loop, enforces concurrency limits, and keeps PCM alive until native work actually stops.

AlignmentResolver Purely maps sample ranges to ready segments and produces selection and clip timeline records. It does not know about spans or loggers.

RecordingExporter Turns encoded results into Braintrust attachments and publishes descriptors and selections through the SDK logger. Owns the encoded-byte lease for as long as the exporter retains the attachment.

PipecatRecordingAdapter ├─ capture PCM ─────────────→ RecordingSession │ ├─ active AudioSegment │ └─ submitted AudioSegment │ ↓ │ RecordingWorker │ ↓ │ AudioEncoder │ ↓ │ RecordingExporter → SDK logger │ └─ turn sample ranges ──────→ AlignmentResolver ───→ RecordingExporter

…t tests

Abhijeet Prasad (AbhiPrasad) · 2026-10-08T14:29:05Z

+from collections import deque
+from contextvars import ContextVar
+
+from braintrust.audio.timeline import pcm_bytes_to_ms, samples_to_ms


So I realized that braintrust.audio is now public API. I think we should make it _audio (the folder name) so that users know it's private/internal to the SDK. Unfortunately we don't have a better way in python to do this.

also I would really prefer if everything in these files were imported from braintrust._audio, helps me see what exactly the public API of the audio module is in py/src/braintrust/audio/__init__.py

Abhijeet Prasad (AbhiPrasad) · 2026-10-08T14:31:49Z

+class Alignment:
+    """Publish resolved selections and clip timelines to their owning spans."""
+
+    def __init__(self, root, recording):


feel like all of these should be typed?

Abhijeet Prasad (AbhiPrasad) · 2026-10-08T14:32:46Z

+                        "audio.selection": selections[0] if len(selections) == 1 else None,
+                    }
+                )
+        self.root.log(metadata={"braintrust.alignment.ranges_omitted": self.omitted})


The fact that the Alignment class also logs feels confusing. I would like this to be a separate responsibility. But open to discussion.

Abhijeet Prasad (AbhiPrasad) · 2026-10-08T14:33:46Z

+        def release_once():
+            nonlocal released
+            if not released:
+                released = True
+                release()


there feels like a better way we can do this with some of the python concurrency mechanisms built in.

Abhijeet Prasad (AbhiPrasad) · 2026-10-08T14:34:06Z

+    segment_duration_seconds: float = 60
+    max_duration_seconds: float = 1800
+    max_buffer_bytes: int = 32 * 1024 * 1024
+    flush_fraction: float = 0.5


why did we pick these? Ditto with some of the other byte numbers we picked.

Abhijeet Prasad (AbhiPrasad) · 2026-10-08T14:34:45Z

+        max_bytes=8 * 1024 * 1024,
+        max_duration_ms=120000,
+        max_packets=16000,


feel like there are enough of these constants around that we should have a constants.py file and write comments about why they exist.

Abhijeet Prasad (AbhiPrasad) · 2026-10-08T14:35:05Z

+    def __del__(self):
+        if getattr(self, "bytes", 0):
+            self.clear()


why do we do this?

Abhijeet Prasad (AbhiPrasad) · 2026-10-08T14:36:07Z

+        self.bytes = 0
+        self.packets.clear()
+
+    def capture(self, channel, pcm, sample_rate, channels, *, observed_ns=None, observed_unix_ms=None):


this capture logic is pretty confusing, with a lot of conditionals. I think we need to codify the state machine better here.

I was hoping the usage of AudioSegment would make this better, but maybe not.

David Elner (delner) requested a review from Abhijeet Prasad (AbhiPrasad) October 1, 2026 14:27

David Elner (delner) self-assigned this Oct 1, 2026

David Elner (delner) added the feature label Oct 1, 2026

chatgpt-codex-connector Bot reviewed Oct 1, 2026

View reviewed changes

Abhijeet Prasad (AbhiPrasad) reviewed Oct 1, 2026

View reviewed changes

David Elner (delner) added 15 commits October 6, 2026 22:14

Added: Common audio recording with progressive export

ede702f

Added: Pipecat voice tracing and opt-in recording

e5737d4

Added: turn_metrics to Pipecat

27ec415

Added: TTFB routing to Pipecat

56c94aa

Fixed: Preserve LLM usage and timing metrics in native Pipecat traces

4cb5a2b

Fixed: timing-dependent Pipecat attachment test

7d0dfd0

Fixed: Missing llm_tool_calls in Pipecat

4308a3e

Changed: Pipecat instrumentation fields to generic names

30bfb41

Changed: Pipecat tests to use VCR and recording behavior during shutdown

20f1011

Added: Pipecat runnable examples

622399e

Fixed: Realtime audio test cassette mismatch

8104157

Changed: Restructure Pipecat events, move metadata to contrib.pipecat.*

6295eeb

Fixed: TTS early arrival, buffer limits

ae0a630

Added: Pipecat state and timeline mappings

b9257d8

Refactored: Recording-job cleanup, synthesis recording, alignment

20b9392

David Elner (delner) force-pushed the delner/pipecat-voice-recording branch from 8130ef7 to 20b9392 Compare October 7, 2026 02:19

Abhijeet Prasad (AbhiPrasad) reviewed Oct 7, 2026

View reviewed changes

David Elner (delner) added 2 commits October 7, 2026 22:51

Refactored: Simplify audio recording lifecycle and consolidate Pipeca…

bd7df8d

…t tests

Fixed: Audio segments stay pending until uploaded

8ce91f4

Abhijeet Prasad (AbhiPrasad) reviewed Oct 8, 2026

View reviewed changes

Refactored: Audio export and alignment ownership

6d51557

		def test_prepare_recording_preserves_disabled_result():
		assert prepare_recording("call", lambda: None) is None

		from .options import RecordingOptions


		__all__ = ["RecordingOptions"]

		from .worker import RecordingBusy


		encoded_budget = ByteBudget(32 * 1024 * 1024)

Class	Owns / does
RecordingSession	Call-level state: accepts capture while active, rotates segments, handles limits, drains on finish. Owns the active segment and tracks pending segment jobs.
AudioSegment	One segment’s state and timeline: OPEN → PENDING → READY / OMITTED. When submitted, it holds an immutable PCM snapshot; once the worker finishes, it holds the encoded result or omission reason.
AudioEncoder	Converts a segment snapshot into encoded bytes and metadata. Keeps codec details out of recording lifecycle code.
RecordingWorker	Runs encoding off the event loop, enforces concurrency limits, and keeps PCM alive until native work actually stops.
AlignmentResolver	Purely maps sample ranges to ready segments and produces selection and clip timeline records. It does not know about spans or loggers.
RecordingExporter	Turns encoded results into Braintrust attachments and publishes descriptors and selections through the SDK logger. Owns the encoded-byte lease for as long as the exporter retains the attachment.

Conversation

David Elner (delner) commented Oct 1, 2026

Example trace

Design

Performance

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

Abhijeet Prasad (AbhiPrasad) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants