Repository navigation
detect_text_encoding has no EBCDIC branch, so EBCDIC text is reported as binary #3507
Description
Summary
libmagic recognizes EBCDIC text by translating the buffer to ASCII through a lookup table and
re-testing it. PolyFile's detect_text_encoding has no EBCDIC branch, so EBCDIC text is classified
as binary and falls through to application/octet-stream.
| input | file 5.48 |
PolyFile |
|---|---|---|
7f 45 4c 46 |
text/plain; charset=ebcdic |
application/octet-stream |
Reproducer
$ printf '\x7fELF' > elf.bin
$ ./file/src/file -m tests/magic.mgc -i --keep-going elf.bin
elf.bin: text/plain; charset=ebcdic
$ ./file/src/file -m tests/magic.mgc --keep-going elf.bin
elf.bin: ELF\012- , International EBCDIC text, with no line terminatorsfrom polyfile.magic import MAGIC_DEFS, MagicMatcher, MatchContext, detect_text_encoding
matcher = MagicMatcher.parse(*MAGIC_DEFS)
print(detect_text_encoding(open("elf.bin", "rb").read()))
print({mt for m in matcher.match(MatchContext.load("elf.bin", only_match_mime=True))
for mt in m.mimetypes})None
{'application/octet-stream'}
Expected an EBCDIC encoding name and {'text/plain'}.
Cause
file_encoding tries EBCDIC after the ASCII, UTF-8 and UCS families fail
(file/src/encoding.c:159-169):
from_ebcdic(buf, nbytes, nbuf);
if (looks_ascii(nbuf, nbytes, *ubuf, ulen)) {
DPRINTF(("ebcdic %" SIZE_T_FORMAT "u\n", *ulen));
*code = "EBCDIC";
*code_mime = "ebcdic";
...
} else if (looks_latin1(nbuf, nbytes, *ubuf, ulen)) {
DPRINTF(("ebcdic/international %" SIZE_T_FORMAT "u\n", *ulen));
*code = "International EBCDIC";
*code_mime = "ebcdic";from_ebcdic (file/src/encoding.c:657) is a byte-for-byte table translation. Two tables back it:
ebcdic_to_ascii (file/src/encoding.c:599) and ebcdic_1047_to_8859 (file/src/encoding.c:633),
256 bytes each. The comment at encoding.c:583-597 notes that the five documented EBCDIC variants
agree on the characters that matter for this test, so one table is enough to identify EBCDIC text.
detect_text_encoding (polyfile/magic.py:3031) implements the ASCII, UTF-8, UCS and 8-bit
branches and stops there, returning None. MagicMatcher.match then takes the OctetStreamTest
branch (polyfile/magic.py:3453-3456).
Note the ordering constraint: EBCDIC is tried only after the other families fail, because most
EBCDIC buffers are also valid 8-bit buffers. Inserting the branch anywhere earlier would reclassify
ordinary Latin-1 text.
Impact
pocs-master/mocks/elf.bin in the corkami corpus, currently skipped by KNOWN_BAD_FILES in
tests/test_corkami.py. Its skip comment claimed PolyFile was the more accurate of the two, which
no longer holds after #3468.
Worth weighing before picking this up: the payload is two 256-byte tables plus a translation pass,
and EBCDIC input is rare outside mainframe data. It is a real fidelity gap against file rather
than an urgent one, and the corpus entry can stay skipped until someone wants the coverage.
Independent of #3488, which changes when the plain-text result is emitted rather than whether these
bytes count as text. Related to #3506, the missing NUL-trimming step in the same function.
Activity
ESultanik commentedon Sep 11, 2026
Anchor correction
Verified against
masterat906d916.detect_text_encodingis now atpolyfile/magic.py:4034-4057, not:3031. The EBCDIC branchgoes at the tail of the dispatch chain, after
_eight_bit_encodingat:4057, mirroringlibmagic's order — EBCDIC is tried only once the ASCII, UTF-8, UCS and 8-bit families have failed,
because most EBCDIC buffers are also valid 8-bit buffers.
The
OctetStreamTestfall-through isMagicMatcher.matchat:4818-4822, not:3453-3456.Two more sites this needs beyond the function body:
LIBMAGIC_ENCODING_NAMESat:4060-4069, for bothEBCDICandInternational EBCDIC_EIGHT_BIT_ENCODINGSat:4076, and a translation helper alongside_eight_bit_encodingat:4025Sequencing
Blocked by #3506, which rewrites the same ten-line body for trailing-NUL trimming. Branch from
master after #3506 merges, not from #3506's branch.
Verification
Remove this issue's
KNOWN_BAD_FILESentry fromtests/test_corkami.py:25-28(elf.bin, MD5d1531b1622de54fe3a0187c3344600e9) and show the Corkami suite green. That list is a plainskipTestwith no unexpectedly-passing guard, so the removal is the proof.The two 256-byte tables are
ebcdic_to_ascii(file/src/encoding.c:599) andebcdic_1047_to_8859(:633). The comment atencoding.c:583-597explains why one table suffices.