Skip to content

detect_text_encoding has no EBCDIC branch, so EBCDIC text is reported as binary #3507

Closed

Description

@ESultanik

Summary

libmagic recognizes EBCDIC text by translating the buffer to ASCII through a lookup table and
re-testing it. PolyFile's detect_text_encoding has no EBCDIC branch, so EBCDIC text is classified
as binary and falls through to application/octet-stream.

input file 5.48 PolyFile
7f 45 4c 46 text/plain; charset=ebcdic application/octet-stream

Reproducer

$ printf '\x7fELF' > elf.bin
$ ./file/src/file -m tests/magic.mgc -i --keep-going elf.bin
elf.bin: text/plain; charset=ebcdic
$ ./file/src/file -m tests/magic.mgc --keep-going elf.bin
elf.bin: ELF\012- , International EBCDIC text, with no line terminators
from polyfile.magic import MAGIC_DEFS, MagicMatcher, MatchContext, detect_text_encoding

matcher = MagicMatcher.parse(*MAGIC_DEFS)
print(detect_text_encoding(open("elf.bin", "rb").read()))
print({mt for m in matcher.match(MatchContext.load("elf.bin", only_match_mime=True))
       for mt in m.mimetypes})
None
{'application/octet-stream'}

Expected an EBCDIC encoding name and {'text/plain'}.

Cause

file_encoding tries EBCDIC after the ASCII, UTF-8 and UCS families fail
(file/src/encoding.c:159-169):

from_ebcdic(buf, nbytes, nbuf);

if (looks_ascii(nbuf, nbytes, *ubuf, ulen)) {
        DPRINTF(("ebcdic %" SIZE_T_FORMAT "u\n", *ulen));
        *code = "EBCDIC";
        *code_mime = "ebcdic";
        ...
} else if (looks_latin1(nbuf, nbytes, *ubuf, ulen)) {
        DPRINTF(("ebcdic/international %" SIZE_T_FORMAT "u\n", *ulen));
        *code = "International EBCDIC";
        *code_mime = "ebcdic";

from_ebcdic (file/src/encoding.c:657) is a byte-for-byte table translation. Two tables back it:
ebcdic_to_ascii (file/src/encoding.c:599) and ebcdic_1047_to_8859 (file/src/encoding.c:633),
256 bytes each. The comment at encoding.c:583-597 notes that the five documented EBCDIC variants
agree on the characters that matter for this test, so one table is enough to identify EBCDIC text.

detect_text_encoding (polyfile/magic.py:3031) implements the ASCII, UTF-8, UCS and 8-bit
branches and stops there, returning None. MagicMatcher.match then takes the OctetStreamTest
branch (polyfile/magic.py:3453-3456).

Note the ordering constraint: EBCDIC is tried only after the other families fail, because most
EBCDIC buffers are also valid 8-bit buffers. Inserting the branch anywhere earlier would reclassify
ordinary Latin-1 text.

Impact

pocs-master/mocks/elf.bin in the corkami corpus, currently skipped by KNOWN_BAD_FILES in
tests/test_corkami.py. Its skip comment claimed PolyFile was the more accurate of the two, which
no longer holds after #3468.

Worth weighing before picking this up: the payload is two 256-byte tables plus a translation pass,
and EBCDIC input is rare outside mainframe data. It is a real fidelity gap against file rather
than an urgent one, and the corpus entry can stay skipped until someone wants the coverage.

Independent of #3488, which changes when the plain-text result is emitted rather than whether these
bytes count as text. Related to #3506, the missing NUL-trimming step in the same function.

Activity

  1. self-assigned this
    on Sep 11, 2026
  2. added this to the v0.6.0 milestone on Sep 11, 2026
  3. ESultanik commented on Sep 11, 2026

    @ESultanik
    CollaboratorAuthor

    Anchor correction

    Verified against master at 906d916.

    detect_text_encoding is now at polyfile/magic.py:4034-4057, not :3031. The EBCDIC branch
    goes at the tail of the dispatch chain, after _eight_bit_encoding at :4057, mirroring
    libmagic's order — EBCDIC is tried only once the ASCII, UTF-8, UCS and 8-bit families have failed,
    because most EBCDIC buffers are also valid 8-bit buffers.

    The OctetStreamTest fall-through is MagicMatcher.match at :4818-4822, not :3453-3456.

    Two more sites this needs beyond the function body:

    • a name in LIBMAGIC_ENCODING_NAMES at :4060-4069, for both EBCDIC and
      International EBCDIC
    • probably an entry in _EIGHT_BIT_ENCODINGS at :4076, and a translation helper alongside
      _eight_bit_encoding at :4025

    Sequencing

    Blocked by #3506, which rewrites the same ten-line body for trailing-NUL trimming. Branch from
    master after #3506 merges, not from #3506's branch.

    Verification

    Remove this issue's KNOWN_BAD_FILES entry from tests/test_corkami.py:25-28 (elf.bin, MD5
    d1531b1622de54fe3a0187c3344600e9) and show the Corkami suite green. That list is a plain
    skipTest with no unexpectedly-passing guard, so the removal is the proof.

    The two 256-byte tables are ebcdic_to_ascii (file/src/encoding.c:599) and
    ebcdic_1047_to_8859 (:633). The comment at encoding.c:583-597 explains why one table suffices.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions