Skip to content

fix(batch-wrapper): restore batch execution with concrete dummy storage - #547

Open
rikkurea-boop wants to merge 2 commits into
OpenDCAI:mainfrom
rikkurea-boop:publish/p04-01-batch-wrapper-dummy-storage
Open

rikkurea-boop wants to merge 2 commits into
OpenDCAI:mainfrom
rikkurea-boop:publish/p04-01-batch-wrapper-dummy-storage

Conversation

@rikkurea-boop

Copy link
Copy Markdown
Contributor

Summary

Fix BatchWrapper failing before the first batch can run because its internal DummyStorage is abstract.

DataFlowStorage requires get_keys_from_dataframe(), while DummyStorage does not implement that method. BatchWrapper.run() constructs DummyStorage for both cached and non-cached batching, so Python raises a TypeError during storage construction before the wrapped operator executes.

This change introduces a concrete batch-local dummy storage adapter that implements the missing key-enumeration contract while preserving the existing DummyStorage read/write and cache behavior.

Regression coverage

The regression exercises the real boundary through PipelineABC.compile() rather than instantiating DummyStorage directly:

PipelineABC.compile()
→ BatchWrapper.run()
→ internal dummy storage construction
→ batch read
→ wrapped operator
→ batch write
→ final FileStorage write

It verifies both batch_cache=False and batch_cache=True.

For three input rows with a batch size of two, the test verifies that:

  • the wrapped operator executes twice;
  • data is read and written through the batch storage;
  • all three transformed rows reach the final output;
  • non-cached batching creates no per-batch cache files;
  • cached batching creates exactly the expected two per-batch cache files.

Validation

Pre-fix RED:

  • the real repository regression fails on the current-main baseline after dependencies install successfully.

Post-fix GREEN:

  • Ubuntu / Python 3.10: targeted regression, CPU/API suite, and py_compile pass.
  • Ubuntu / Python 3.12: targeted regression, CPU/API suite, and py_compile pass.
  • Windows / Python 3.10: targeted regression, CPU/API suite, and py_compile pass.

The production diff is limited to dataflow/wrapper/batch_wrapper.py plus the focused CPU regression test.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant