[ArthurZucker]

Hey!

Tried to play a bit with the data, think they belong well on the HF hub!

The updated code fetches test fixture on demand from a Hugging Face bucket instead of the 3.2 GB Zenodo zip

pytest tests/test_roi.py now needs 1.8 MB and 7 s instead of a 3.22 GB download, which takes ~2.9 h at the 0.31 MB/s I measured pulling the archive single-stream from Zenodo.

The bucket (ArthurZ/pyiconeus-test-data) is public, it's byte-identical to the Zenodo archive (same MD5), and I'd be glad to transfer it to an Iconeus org .

[ArthurZucker]

cc @simoncampredon-lang !

[danieleTrimarchi]

Hi @ArthurZucker,

Thanks for the PR, and for contributing to pyIconeus! You’re right that downloading the full 3.2 GB archive was a real pain point. Per-file downloads with a local cache are a promising direction, and making the tests independent of the working directory is a nice bonus.

We’re still evaluating the PR, and we’d like two changes at this stage:

1. Download robustness. The old script verified the MD5 against Zenodo; the new fixture checks nothing. If a download is interrupted, a partial file stays on disk, and since local.exists() is then true, the corrupted file is served indefinitely and tests fail in confusing ways. HF buckets are also mutable and unversioned, unlike Zenodo, so the source can change over time. Could you download to a .tmp file (unique per worker) and then os.replace() it atomically? That also avoids a race between workers when running under pytest-xdist. An optional size/hash check would be a good addition too.

2. CI caching. The prepare-data job and the actions/cache blocks are removed. A full CI run exercises the whole test suite, so without a cache it would re-download the entire dataset (about 3.2 GB) from HF on every run, and make CI depend on HF availability each time. Could you restore caching in the workflow, for example by caching the test-data directory with actions/cache? The cache key could be based on a version string or a hash of the file list, so it gets invalidated when the data changes.

Once these are addressed, we may come back with a few smaller remarks. Also, since the bucket currently lives in a personal namespace, hosting the test data under an Iconeus-owned HF organization will be a prerequisite for any merge. We’ll take care of that at the end of the review, so no action is needed from you on that point for now. Keeping PYICONEUS_TEST_BUCKET as an override will help with that.

Would you be willing to update these? Thanks again!

[ArthurZucker]

Hey!

Thanks for the review! Both points addressed:

• moved from a bucket to a HF dataset (ArthurZ/pyiconeus-test-data), which gets the robustness and versionning.
• hf_hub_download writes to a .incomplete file under a file lock and only moves it into place once the size matches the server's, so a partial file is never served, and pytest-xdist workers don't race.
• Restored the prepare-data job

Hosting: if you add me to an Iconeus org on the Hub, I can transfer the dataset there directly (history and commit hashes are kept, so the pinned REVISION stays valid).

You can also generate it yourself:

       hf auth login
       hf repos create Iconeus/pyiconeus-test-data --repo-type dataset
       hf upload Iconeus/pyiconeus-test-data ./tests/data . --repo-type dataset