Repository navigation
Make Low Level Binary File I/O Independent of Host System's Byte Order - #5543
Conversation
|
jenkins build this please |
There was a problem hiding this comment.
Note
Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.
🟡 Changes recommended
The change still has a moderate active issue in opm/io/eclipse/ByteOrder.hpp where readFileOrder() writes through a FileScalarRange that can be const, casting away const via reinterpret_cast<char*>, so the range constraint should be tightened to prevent unsafe misuse.
1 open issue
What changed in this PR
This PR modernizes the low-level binary ECL I/O byte-order handling by introducing a central ByteOrder.hpp utility layer (based on C++20 <bit> endian detection) and migrating binary reads/writes to use host↔file (big-endian) conversions consistently across the I/O stack.
Changes:
- Added
opm/io/eclipse/ByteOrder.hppwithbyteswap(),toFileOrder()/fromFileOrder(), andreadFileOrder()/writeFileOrder()utilities plus platform assumption assertions. - Updated binary readers/writers (
EclUtil,EclOutput,EGrid,ESmry) to use file-order conversion instead offlipEndian*()and fixedLOGIhandling to convert likeINTE(withtrue_value_ixnow1). - Expanded/strengthened tests to validate exact on-disk byte sequences (“golden bytes”) and round-trips independent of host endianness.
| File | Description |
|---|---|
| tests/test_EclIO.cpp | Updated as part of this pull request. |
| opm/io/eclipse/ESmry.cpp | Updated as part of this pull request. |
| opm/io/eclipse/EGrid.cpp | Updated as part of this pull request. |
| opm/io/eclipse/EclUtil.hpp | Updated as part of this pull request. |
| opm/io/eclipse/EclUtil.cpp | Updated as part of this pull request. |
| opm/io/eclipse/EclOutput.cpp | Updated as part of this pull request. |
| opm/io/eclipse/EclIOdata.hpp | Updated as part of this pull request. |
| opm/io/eclipse/ByteOrder.hpp | Updated as part of this pull request. |
| CMakeLists_files.cmake | Updated as part of this pull request. |
Review effort: Balanced
0e34afb to
d10b249
Compare
|
jenkins build this please |
d5157b6 to
a744395
Compare
There was a problem hiding this comment.
Note
Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.
🟡 Changes recommended
Moderate-severity active issues remain around binary I/O correctness—specifically potential partial-read garbage values and non–fixed-width integer sizing (e.g., LOGI using unsigned int) that can break the on-disk format or desynchronize streams on some platforms—so the change needs fixes before approval.
4 open issues
1 resolved since last review
Review effort: Lite
65a3524 to
c7ffd26
Compare
|
jenkins build this please |
c7ffd26 to
295185d
Compare
This commit introduces a new header, ByteOrder.hpp, with utilities for
converting numeric values between the host's native byte order and
the big-endian byte order used in "unformatted" result files. The
utilities are
* byteswap() -- Unconditionally reverse the bytes of a four or eight
byte integer or floating-point value.
* toFileOrder(), fromFileOrder() -- Convert between native and
big-endian byte order. Identity operations on big-endian hosts.
* readFileOrder(), writeFileOrder() -- Read or write a single value,
or a contiguous range of values, in big-endian byte order.
We implement byteswap() in terms of std::bit_cast<>() to a same-sized
unsigned integer type, and use std::byteswap() from C++23 if
available. Otherwise we fall back to a shift-and-mask implementation
which mainstream compilers recognise and translate into a single byte
swap instruction. This replaces the need for compiler intrinsics and
reinterpret_cast<>()-based byte reversals.
The header also asserts the platform assumptions of the binary file
I/O layer, i.e., 8-bit bytes, a purely little or big-endian host,
4-byte 'int', and IEEE 754 floating-point types.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit switches the low-level binary file I/O in EclUtil, EclOutput, EGrid, and ESmry over to using the byte order utilities from ByteOrder.hpp instead of the unconditional flipEndian*() functions. In turn, this removes the compiler specific byte swapping intrinsics and makes the binary file I/O layer correct on big-endian hosts. As part of this, we also fix the handling of LOGI arrays. These were previously read and written without byte order conversion, with the IX convention's 'true' value encoded as the little-endian representation of a big-endian '1' (0x1000000). LOGI elements now pass through the same byte order conversion as INTE elements and the 'true_value_ix' constant is now 1. The on-disk representation is unchanged on little-endian hosts. The callback of readBinaryArray<>() now receives numeric elements in native byte order. Its only outside user, ESmry, now uses readBinaryInteArray() instead. Keep the flipEndian*() functions for now, but mark them as deprecated. They are implemented in terms of byteswap() and remain unconditional byte swaps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit adds unit tests for the byte order utilities and for the on-disk representation of binary INTE, REAL, DOUB, and LOGI arrays. The latter compare the output of EclOutput against hand-written, big-endian byte sequences--including LOGI arrays using the IX convention--and read the same byte sequences back through EclFile. These tests are independent of the host's byte order and are intended to catch byte order errors if we ever run on big-endian hosts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The FileScalarRange concept admits immutable ranges such as std::span<const T> and 'const std::vector<T>&'. That is appropriate for writeFileOrder(), which only reads the in-memory values, but not for readFileOrder() which stores the values loaded from the file into the range. Passing an immutable range to readFileOrder() would fail to compile, but with a diagnostic from inside the function body. This commit introduces a new concept, FileOutputScalarRange, which additionally requires that the range be an output range for its own value type, and uses that to constrain readFileOrder(). Misuse is now reported as an unsatisfied constraint at the call site. Add compile-time tests of which range types readFileOrder() and writeFileOrder() accept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
295185d to
e494a23
Compare
|
jenkins build this please |
|
I've tested this as much as I'm able to so I am, with much trepidation, marking it "ready for review". |
akva2
left a comment
There was a problem hiding this comment.
LGTM. Killing all those nasty reinterpret_casts makes for a good work day.
It was indeed very satisfying. As the PR's approved and the build check is green, I'll merge this into the master branch now. |

This PR replaces the bottom layer of the binary file I/O stack, represented by the functions
flipEndian*(), with a more portable alternative. We leverage C++20's<bit>header, and especially itsendiandetection support, to determine whether we need to swap byte order at all. We still require that the binary builds and runs on the same host platform, but at least we're now able to portably read and write files that can be correctly interpreted on systems of both byte orders. We don't support mixed byte orders however.To this end we introduce a new header,
ByteOrder.hpp, with utilities for converting numeric values between the host's native byte order and the big-endian byte order used in "unformatted" result files. The utilities arebyteswap()– Unconditionally reverse the bytes of a four or eight byte integer or floating-point value.toFileOrder(),fromFileOrder()– Convert between native and big-endian byte order. Identity operations on big-endian hosts.readFileOrder(),writeFileOrder()– Read or write a single value, or a contiguous range of values, in big-endian byte order.We implement
byteswap()in terms ofstd::bit_cast<>()to a same-sized unsigned integer type, and usestd::byteswap()from C++23 if available. Otherwise we fall back to a shift-and-mask implementation which mainstream compilers recognise and translate into a single byte swap instruction. This replaces the need for compiler intrinsics andreinterpret_cast<>()-based byte reversals.The header also asserts the platform assumptions of the binary file I/O layer, i.e., 8-bit bytes, a purely little or big-endian host, 4-byte 'int', and IEEE 754 floating-point types.
We use these new utilities in
EclUtiland other facilities. As part of this, we also fix the handling ofLOGIarrays. These were previously read and written without byte order conversion, with the IX convention's 'true' value encoded as the little-endian representation of a big-endian '1' (0x1000000).LOGIelements now pass through the same byte order conversion asINTEelements and thetrue_value_ixconstant is now 1. The on-disk representation is unchanged on little-endian hosts.The callback of
readBinaryArray<>()now receives numeric elements in native byte order. Its only outside user,ESmry, now usesreadBinaryInteArray()instead. We reimplement theflipEndian*()functions in terms ofbyteswap()as unconditional byte swaps, but mark these as "deprecated".This work is partly inspired by long-standing Issue #3014. That issue alluded to using C++20's
endiansupport and this PR finally makes that step. More immediately though, I wanted to get away from using compiler intrinsics for byte swapping and reduce the number ofreinterpret_cast<>()invocations in the code base.