Return short reads from runToInputStream - #505
Merged
Merged
Conversation
Port of jox #364. Bulk reads filled the whole buffer, blocking for further chunks until it was full or the stream ended, which stalled consumers reading in large blocks from live sources. Bulk reads now block only until at least one byte is available, copy what is buffered from the current chunk, and return the count. Claude-Session: https://claude.ai/code/session_01QKfjf299iNreqRt32n2p5D
adamw
added a commit
that referenced
this pull request
Sep 14, 2026
Port of jox [#351](softwaremill/jox#351), using jsoniter-scala instead of Jackson. Adds a `flow-json` module (package `ox.flow.json`) with four extension methods on `Flow`: - `parseNdjson[T]` – LF-delimited records, blank lines skipped, CRLF, a final unterminated record and one leading BOM accepted, configurable max record size (32 MiB by default) - `parseJsonArray[T]` – one top-level array, elements emitted as they are parsed - `renderNdjson` – one value per line - `renderJsonArray` – one array, `[]` for an empty flow Values are converted with a `JsonValueCodec[T]` in scope; the examples derive it with `derives ConfiguredJsonValueCodec`. ## Why not use jsoniter's streaming directly jsoniter-scala-core already streams: `scanJsonValuesFromStream` and `scanJsonArrayFromStream` call back for each value of a whitespace-separated sequence or of one array. Two differences make it worth wrapping: - **Push vs pull.** The callback runs inside the parse loop over a blocking `InputStream` and can only return `false` to stop. A flow is pulled, so values can be filtered, mapped, buffered, merged with other flows and cancelled. - **Framing.** `scanJsonValuesFromStream` splits on whitespace, not lines, so two values on one line and one value spanning several lines are both accepted. It has no per-record size limit, and no jsoniter reader skips a leading BOM. `parseNdjson` adds all three, and parses each record out of the incoming chunk rather than running the flow into an `InputStream` first. Plain jsoniter stays the better choice when no flow is involved: a whole document in memory, or a plain `InputStream` to scan. The docs page says this too. ## Compared to jox No reader/writer/type-reference overloads (the codec covers them), no null policy (derived codecs reject JSON `null`; use `Option[T]`), and parse errors are not wrapped. NDJSON framing works on the chunk's backing arrays, so a record within one chunk is parsed without copying, and the framer reuses one buffer for records that span chunks. Array parsing goes through `runToInputStream`, so it depends on the short-read fix. Docs: `doc/streaming/json.md`. Stability: experimental. Reviewed for correctness, performance, API shape and structure. Two known points, both removed by #507, which drops the element channel: parsing an array crosses a channel per element, and elements parsed before a failure may be dropped from that channel's buffer. Merge order: after #505 (merged), before #507. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QKfjf299iNreqRt32n2p5D
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port of jox #364.
The
InputStreamfromrunToInputStreamfilled the whole buffer on bulk reads, blocking for further chunks until the buffer was full or the stream ended. Consumers reading in large blocks (e.g. JSON parsers) stalled on live sources.Bulk reads now block only until at least one byte is available, copy what is buffered from the current chunk, and return the count. End of stream still returns -1. Two existing tests changed to expect one chunk per read.
Merge order: this PR first, then #506, then #507.
🤖 Generated with Claude Code
https://claude.ai/code/session_01QKfjf299iNreqRt32n2p5D