introduces a generic byte pattern matcher based on Ghidra

[ivg]

This PR implements the byte pattern matcher that could be fed with Ghidra patterns and used for function starts identification, instruction encoding (e.g., thumb vs arm), function names, code boundaries and many more. Even more than that, our implementation allows attaching arbitrary actions to the patterns. The semantics of actions is specified in Primus Lisp, with defmethod.

The PR is independent of Ghidra and the patterns themselves will be packed in the bap-signatures package in opam (with corresponding attributions). Once merged, and assuming that either bap-signatures or libghidra-data or both packages are installed, it will work out of the box and improve function start identification and, most importantly, significantly improve the disassembling of arm/thumb interworked binaries.

The patterns language is quite versatile and supports active patterns, i.e., other than matching with a sequence the pattern must satisfy with some non-trivial conditions. We implement actions in Primus Lisp to enable extensibility and fast prototyping. So far, we have (trivially) implemented only two actions, here is the current implementation, just to give you the flavor of actions,

(defmethod bap:patterns-action (action addr attrs)
  (when (is-in action 'funcstart 'possiblefuncstart)
    (promise-function-start addr)))

(defmethod bap:patterns-action (action addr attrs)
  (when (= action 'setcontext)
    (let ((name (patterns-attribute attrs :name))
          (mode (patterns-attribute attrs :value)))
      (when (and name mode (= name 'TMode))
        (let ((lang (if (= mode '1) :T32 :A32)))
          (arm-set-encoding addr lang))))))

So far it works perfectly even without complex constraints that some of the patterns impose, possible because bap is quite resilient to function start false positives, thanks to byteweight. We might later implement those constraints as necessary. In addition to the vast repository of patterns that are available from Ghidra (again, no need to install Ghidra we will have them in our sigs.tar.gz,
on the release page and installable via opam install bap-signatures), it is possible to define your own patterns and your own actions. The format is described in the bap --patterns-help man page, for convenience copied here,

patterns(3)                 BAP Programmer's Manual                patterns(3)



DESCRIPTION
       Applies semantic actions to the matching byte patterns. The patterns
       are specified in an XML format, described below, and the actions are
       implemented with Primus Lisp methods. Used to identify function starts,
       instruction encodings, function names, etc.

INPUT FORMAT
       The patterns are represented with XML files, each corresponding to a
       specific target. The patternconstraints.xml files are used as the table
       of contents and contain mapping between targets and files that provide
       patterns for that target.

       The file scheme itself is derived from the Ghidra bytesearch patterns,
       so that the patterns provided by Ghidra could be used as is.

PATTERNS SCHEME
       The file with patterns must have a single element  patternlist, which
       contains a list of pattern or patternpairs elements. The pattern
       element is composed of a list of data elements and a list of action
       elements. The data element descibes the ditted pattern, and an action
       is any element (other than data), which is translated to a Primus Lisp
       singal, bap:patterns-action name addr attrs, where name is the element
       name, addr is the address where the pattern matches, and attrs is the
       list of element attributes, accessible via patterns-attribute function.

       The patternspairs element is very similar to the patterns element
       except that the list of patterns is described by a cartesian product of
       two sets of patterns, prepatterns and postpatterns. Both prepatterns
       and postpatterns must contain a non-empty list of data elements and
       nothing more. The resulting patterns are made by concatenating every
       prepattern with every postpattern and leaving only those combinations
       that have the total number of non-masked bits equal to totalbits and
       the total number of non-masked bits in the postpattern part equal to
       postbits. When the resulting pattern matches with a sequence of bytes,
       the address of a byte that matches with the start of the postpattern is
       passed to the bap:patterns-action method.

PATTERNCONSTRAINTS SCHEME
       The patternconstraints.xml file is used as the table of contents and
       contain a mapping between targets (languages in Ghidra parlance) and
       paths to corresponding files, relative to the location of the
       patternconstraints.xml file.

       The file must have a single patternconstraints element that contains a
       list of language elements. The language element is required to have the
       id attribute which must be either the name of a BAP target (see bap
       list targets) or a Ghidra language specification, which is four-tuple
       of elements, separated with :. The first element is the architecture
       name, the second is the endiannes, the third is the bitness, and the
       last is the variant. Any field except the architecture, could use
       default or just * as the wildcard character that matches with anything.
       The endianness is specified as either LE for little or BE for big
       endianness. For instructions in little endian and data in big endian,
       use LEBE.

         The language element contains a list of patternfile or compiler
       elements. The patternfile element contains the path to the patterns
       file, and the compiler element contains the patternfile element with
       patterns specific to a compiler, which is specified in the required id
       attribute of the compiler element.

DITTED PATTERNS
       Each pattern is described as a ditted sequence of bits or nibbles. Each
       bit is represented with 0, 1, and . that match, correspondigly with,
       zero, one, and any bit. And a nibble is a sequence of hexadecimal
       digits, matching with their corresponding four-bit representation, and
       ., which matches with any four bits (i.e., with any binary number in
       the range from 0 to 0xF). The nibble sequence must start with 0x and
       continues until the next whitespace character. If the sequence doesn't
       start with 0x then it is assumed to be a sequence of bits. Sequences
       could be separated by the arbitrary number of whitespace characters.

BUILTIN ACTIONS
       All actions in the bap namespace, which is set as the default namespace
       when parsing the patterns file, are reserved to BAP. It is possible to
       add arbitrary actions, provided that they are not using the bap
       namespace. The following set of actions have predefined meaning.

       functionstart and possiblefuncstart mark the matching sequence as the
       function start. The attributes could be used to impose an extra
       constraint. The current implementation ignore them, but they will be
       implemented later.

       setcontext is used to control the disassembler context and currently
       the following two attributes are recongized, name and value. When the
       name is set to TMode then the matching sequence has the encoding T32 if
       the value is 1 and A32 otherwise.

 2.4.0-alpha+175870a                                               patterns(3)