Repository navigation
The full word (f) flag anchors both sides, so no shipped f definition can match #3559
Description
Summary
The f (full word) string flag compiles to a word boundary on both sides of the value. libmagic
checks only the byte that follows the match, and it asks for whitespace or the end of the buffer
rather than a word boundary. PolyFile therefore misses matches that file reports, including one
from a shipped definition.
Reproducer
magic_defs/commands:118 declares 0 search/1/fwt #!\ /usr/bin/tclsh:
$ printf '#! /usr/bin/tclsh\n' > tcl.bin
$ TZ=UTC ./file/src/file -b -m ./file/magic/magic.mgc tcl.bin
Tcl/Tk script, ASCII text executablePolyFile on master at ed571ba:
from polyfile.magic import MagicMatcher
print({str(m) for m in MagicMatcher.DEFAULT_INSTANCE.match(b"#! /usr/bin/tclsh\n")}){'Tcl script, ASCII text executable', 'a /usr/bin/tclsh script, ASCII text executable'}
Tcl/Tk script text executable is absent. A smaller case, where the value is preceded by a word
character:
$ printf '0\tsearch/4/f\tABC\tfound\n' > fullword.magic
$ printf 'xxxABC' > in.bin
$ TZ=UTC ./file/src/file -b -m fullword.magic in.bin
found, ASCII text, with no line terminatorsPolyFile reports no match.
Cause
StringMatch.pattern_string wraps the pattern in \b on both sides:
if self.full_word_match:
pattern = rb"\b" + pattern + rb"\b"libmagic applies STRING_FULL_WORD once, after the comparison, and only to the trailing side
(file/src/softmagic.c:2127-2130):
if (len == 0 && v == 0 && (flags & STRING_FULL_WORD)) {
if (*b && !isspace(*b))
v = 1;
}That is two separate divergences:
- Nothing constrains what precedes the match. The leading
\brejectsxxxABC, and it rejects
magic_defs/commands:118outright, because\bat offset 0 needs a word character there and the
value starts with#. - The trailing condition is "whitespace or the end of the buffer", not "a word boundary". A value
ending in a word character followed by.satisfies Python's\bbut fails libmagic's check.
Impact
polyfile/magic_defs/commands carries 60 definitions with the flag, and every one of them declares a
value that starts with #, so the leading \b can never match and none of them can fire. That is
the whole of the file's interpreter-line table: file reports POSIX shell script for
#! /bin/sh, and PolyFile reports only a /bin/sh script from magic_defs/varied.script. Dropping
the flag from the declaration is enough to show the definition is otherwise fine:
>>> # 0 string/fwt #!\ /bin/sh POSIX shell script text executable
{'ASCII text'}
>>> # 0 string/wt #!\ /bin/sh POSIX shell script text executable
{'POSIX shell script, ASCII text executable'}The trailing side goes the other way and produces a false positive: with
0 search/4/f ABC found, file does not report ABC. and PolyFile does.
No corpus stem covers either side.
Found while fixing #3552.
Activity