A BOM-prefixed JSON document is reported as JSON, where file reports Unicode text #3500

Activity

1. ESultanik

   added

   bugSomething isn't workingSomething isn't working

   claude-code-assisted

   on Sep 10, 2026
2. ESultanik commented on Sep 10, 2026

   

   ESultanik

   on Sep 10, 2026

   CollaboratorAuthor

   More actions

   This is broader than a BOM. Python's json encoding detection also handles UTF-16 and UTF-32, so
   PolyFile reports JSON for inputs libmagic does not consider JSON at all.

   $ python3 -c "import pathlib; pathlib.Path('u16.json').write_bytes('{\"a\": 1}\n'.encode('utf-16-le'))"
   $ TZ=UTC ./file/src/file -b -k -r -m file/magic/magic.mgc u16.json
   Unicode text, UTF-16, little-endian text
   $ polyfile u16.json
   JSON text data

   libmagic's file_is_json walks raw bytes, so UTF-16-encoded JSON is simply not JSON to it — the
   same reason a UTF-8 BOM disqualifies a document there. parse_json decodes with
   json.detect_encoding first, which accepts all of UTF-8-with-BOM, UTF-16 and UTF-32.

   So the decision this issue asks for is one decision, not two: whether PolyFile's JSON test should
   work on bytes the way libmagic's does, or keep Python's encoding detection and diverge for every
   encoding it recognizes. Worth settling once rather than per-encoding.

   Confirmed pre-existing rather than introduced by recent work: master and the #3489 branch both
   report JSON text data for the file above, and #3489's precedes_soft_magic change deliberately
   keeps the JSON and CSV checks on the file's own bytes, so the decoded text buffer is not involved.
3. ESultanik

   marked this as blocked on Sep 11, 2026

   • detect_text_encoding has no EBCDIC branch, so EBCDIC text is reported as binary #3507
4. ESultanik

   marked this as blocking on Sep 11, 2026

   • A relative offset after a search resolves from the consumed extent, not the declared length #3503
5. ESultanik

   self-assigned this

   on Sep 11, 2026
6. ESultanik

   modified the milestones: v0.1.6, v0.6.0 on Sep 11, 2026
7. ESultanik

   marked this as blocking on Sep 11, 2026

   • Cut the v0.6.0 release #3533
8. ESultanik

   unmarked this as blocking on Sep 11, 2026

   • A relative offset after a search resolves from the consumed extent, not the declared length #3503
9. ESultanik

   unmarked this as blocked on Sep 11, 2026

   • detect_text_encoding has no EBCDIC branch, so EBCDIC text is reported as binary #3507
10. ESultanik commented on Sep 11, 2026

    

    ESultanik

    on Sep 11, 2026

    CollaboratorAuthor

    More actions

    Decision: report both

    This issue asked for the behavior to be decided before anything changed. Decided: emit the JSON
    match and let the text-encoding machinery also report Unicode text. Neither answer is lost.

    The reasoning is that a BOM-prefixed JSON file is JSON in practice — several editors write one — and
    PolyFile's model is to report every match rather than pick a winner. Suppressing the JSON match to
    mirror file would throw away the more useful of the two facts. RFC 8259 §8.1 permits either
    reading, so both tools stay in spec.

    This does diverge from file, which reports only Unicode text, UTF-8 (with BOM) text. That is a
    deliberate difference, not a defect, and the test should say so.

    Fix site

    parse_json at polyfile/magic.py:3700-3737. The single line is :3723:

    text = raw.decode(json.detect_encoding(raw), "surrogatepass")

    json.detect_encoding is what strips the BOM before the parser ever sees it. The docstring bullet
    list at :3707-3715 documents libmagic's rules and wants a third bullet recording this decision.

    JSONTest is at :3739-3784 (test at :3751-3763, precedes_soft_magic at :3768-3779);
    NDJSONTest at :3787-3790. Check whether precedes_soft_magic and appends_text_encoding need
    adjusting so the text-encoding description actually reaches the output alongside the JSON match —
    that is the part that makes "report both" real rather than nominal.

    Verification

    Add the test at around tests/test_magic.py:429, beside the existing JSON cases
    (test_newline_delimited_json :381, test_single_json_value :391,
    test_json_values_must_share_a_first_byte :409, test_bare_json_scalar_is_not_json :420).
    Pin all three of: the JSON match is present, the Unicode text description is present, and what
    file reports for the same bytes.

    Cover UTF-8, UTF-16 and UTF-32 BOMs — json.detect_encoding handles all of them.
11. ESultanik

    closed this as completedin #3544on Sep 11, 2026