Skip to content

The full word (f) flag anchors both sides, so no shipped f definition can match #3559

Closed

Description

@ESultanik

Summary

The f (full word) string flag compiles to a word boundary on both sides of the value. libmagic
checks only the byte that follows the match, and it asks for whitespace or the end of the buffer
rather than a word boundary. PolyFile therefore misses matches that file reports, including one
from a shipped definition.

Reproducer

magic_defs/commands:118 declares 0 search/1/fwt #!\ /usr/bin/tclsh:

$ printf '#! /usr/bin/tclsh\n' > tcl.bin
$ TZ=UTC ./file/src/file -b -m ./file/magic/magic.mgc tcl.bin
Tcl/Tk script, ASCII text executable

PolyFile on master at ed571ba:

from polyfile.magic import MagicMatcher

print({str(m) for m in MagicMatcher.DEFAULT_INSTANCE.match(b"#! /usr/bin/tclsh\n")})
{'Tcl script, ASCII text executable', 'a /usr/bin/tclsh script, ASCII text executable'}

Tcl/Tk script text executable is absent. A smaller case, where the value is preceded by a word
character:

$ printf '0\tsearch/4/f\tABC\tfound\n' > fullword.magic
$ printf 'xxxABC' > in.bin
$ TZ=UTC ./file/src/file -b -m fullword.magic in.bin
found, ASCII text, with no line terminators

PolyFile reports no match.

Cause

StringMatch.pattern_string wraps the pattern in \b on both sides:

if self.full_word_match:
    pattern = rb"\b" + pattern + rb"\b"

libmagic applies STRING_FULL_WORD once, after the comparison, and only to the trailing side
(file/src/softmagic.c:2127-2130):

if (len == 0 && v == 0 && (flags & STRING_FULL_WORD)) {
	if (*b && !isspace(*b))
		v = 1;
}

That is two separate divergences:

  1. Nothing constrains what precedes the match. The leading \b rejects xxxABC, and it rejects
    magic_defs/commands:118 outright, because \b at offset 0 needs a word character there and the
    value starts with #.
  2. The trailing condition is "whitespace or the end of the buffer", not "a word boundary". A value
    ending in a word character followed by . satisfies Python's \b but fails libmagic's check.

Impact

polyfile/magic_defs/commands carries 60 definitions with the flag, and every one of them declares a
value that starts with #, so the leading \b can never match and none of them can fire. That is
the whole of the file's interpreter-line table: file reports POSIX shell script for
#! /bin/sh, and PolyFile reports only a /bin/sh script from magic_defs/varied.script. Dropping
the flag from the declaration is enough to show the definition is otherwise fine:

>>> # 0  string/fwt  #!\ /bin/sh  POSIX shell script text executable
{'ASCII text'}
>>> # 0  string/wt   #!\ /bin/sh  POSIX shell script text executable
{'POSIX shell script, ASCII text executable'}

The trailing side goes the other way and produces a false positive: with
0 search/4/f ABC found, file does not report ABC. and PolyFile does.

No corpus stem covers either side.

Found while fixing #3552.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions