Repository navigation
A search with no range sorts as though it declared a range of 100 #3580
Description
Summary
libmagic_str_range (polyfile/magic.py) gives a search that declared no range the
STRING_DEFAULT_RANGE of 100 when it builds the sort key. libmagic leaves that field zero, so the
two tools order such a test differently against its equal-strength neighbors.
string_range = getattr(data_type, "num_bytes", None) or 0
if isinstance(data_type, SearchType) and string_range == 0:
return STRING_DEFAULT_RANGE
return string_rangeWhy zero is what libmagic stores
parse_string_modifier runs only when a declaration carries a /, and it is the only place that
writes m->str_range (file/src/apprentice.c:1948). A bare search never reaches it, and
apprentice.c:2336 has already zeroed the field.
string_modifier_check does mention the 100 (file/src/apprentice.c:1769-1776), but it returns
early unless MAGIC_CHECK is set, and when MAGIC_CHECK is set it returns -1 after the warning,
which makes parse reject the whole definition line. So the value 100 never reaches a live entry:
without MAGIC_CHECK the range stays 0, and with it the test is dropped.
Reproducer
Two search tests of equal strength, the flagged one named so that ordering by description alone
would give the opposite answer:
$ printf '0\tsearch\tABCD\tAAA-bare\n0\tsearch/1\tABCD\tZZZ-1\n' > s.magic
$ printf 'ABCD\n' > in.bin
$ TZ=UTC ./file/src/file -b -k -m s.magic in.bin
ZZZ-1\012- AAA-bare, ASCII textfile puts ZZZ-1 first, so the bare search compares as 0 rather than as 100. PolyFile gives the
bare one a range of 100 and puts AAA-bare first. Compiling the definition with file -C first and
matching against the .mgc gives the same answer, so this is not an artifact of reading the source
form.
Exposure
Five level 0 search tests in polyfile/magic_defs/ declare no range: one each in ringdove,
sosi and bioinformatics, and two in javascript.
Removing the fallback moves 27 of the 3,877 level 0 tests in the default matcher. It changes no
file/tests corpus stem and no definition file's agreement with file -m <file> -l, so nothing in
the current test suite covers it either way.
Suggested fix
Drop the SearchType branch from libmagic_str_range so an undeclared range is zero for every
type, and pin it with a pair like the reproducer above in StringFlagSortKeyTest
(tests/test_magic.py).
SearchType.repetitions returning None for "search the whole buffer" is a separate matter and is
what libmagic does at match time; only the sort key is wrong.
Found while fixing #3568. It is in the same function but was not part of that issue's scope, since
changing it reorders shipped definitions.
Activity
ESultanik commentedon Sep 12, 2026
One correction to the reasoning, verified against the reference binary. The premise that the default is unreachable does not hold:
fileaccepts the line and matches at offset 2, which needs a non-zero range — sostring_modifier_check'sreturn -1does not reject the definition, and the 100 is applied.The divergence is real regardless, and is narrowly in the sort key.
filecompiles a baresearchandsearch/100to different entries — noDuplicate magic entrywarning, unlike thepstringcase — and orders them differently from PolyFile:PolyFile gives the bare declaration the same
str_rangeas the explicit one, so the pair ties and falls back to line order.So the fix stands, but it is not "remove an unreachable default" — it is "the compiled
str_rangefor a baresearchdiffers from what libmagic stores, even though both tools match the same bytes". Establish what libmagic actually stores before changing the value, rather than assuming zero.Added to milestone v0.6.0 as its own change, after #3568.
ESultanik commentedon Sep 12, 2026
Correcting my own correction above: it was wrong. The original issue body was right that libmagic stores
str_range = 0for a baresearch.My inference — that matching at offset 2 required a non-zero range — does not hold. The search loop is:
(
file/src/softmagic.c:2357). Zero means search the whole buffer, not search nothing. So a baresearchreaches further than an explicit one, which is the opposite of what I assumed:PR #3585 settled it by reading the compiled entry directly rather than inferring:
file -Cwrites astruct magicwhosestr_rangefield is 0 for a baresearchand 100 forsearch/100. That is the stored sort-key field itself.The observable divergence and the fix are unchanged; only my explanation was wrong. Apologies for the detour — anyone reading this issue should follow the PR's reasoning, not my earlier comment.