overhauls the target/architecture abstraction (2/n) #1226

ivg · 2020-09-30T19:59:56Z

In the second patch of this series (#1225) we completely got rid of
Arch.t dependency in the disassembler engine that finally opens the
path for seamless integration of targets that are not representable
with Arch.t.

To achieve this, we introduced a proper dependency injection into the
disassembler driver so that it is no longer responsible for creating
the llvm MC disassembler. Instead a plugin that implements a target,
aka the target support package, has to create a disassembler and is
now in full control of all parameters and can choose backend, specify
the CPU and other details of encoding. The encoding is a new
abstraction in our knowledge base that breaks the tight connection
between the target and the way how the program for that target is
encoded. Unlike the target, which is a property of a unit of code, the
encoding is associated with a program itself, i.e., it is a property
of each instruction. That enables targets with context-dependent
encodings such ARM's thumb mode and MIPS16e for binary encodings as
well as paves the road for non-binary encodings for the same
architecture, e.g., text assembly (which also may have several
encodings on its own, cf. att vs intel syntax). We base this branch on
the enable-interworking (#1188) and this branch fully superseeds and
includes it, since encodings made it much more natural. It is still
highlty untested how it will work with real thumb binaries but we will
get back to it when we will merge #1178.

Another big update, is that the disassembler backend (which is
responsible for translating bits into machine instructions) is no
longer required to be implemented in C++ and it is now possible to
write your own backends/disassemblers in pure OCaml, e.g., to support
PIC microcontrollers. The Backend interface is pretty low-level and we
might provide higher-level interfaces later, see
Disasm_expert.Backend for the interface and detailed comments.

Finally, we rectify the interface introduced in the previous PR and
flatten the hierarchy of newly introduced to the Core Theory
abtractions, i.e., instead of Theory.Target.Endiannes we now have
Theory.Endianness and so on. We also made the Enum module public
which introduced enumerated types built on to of Knowledge.Values.

In the next episodes of this series we will gradually remove Arch.t
from other bap components and further clean up the newly introduced
interfaces.

What is interworking -------------------- Interworking is a feature of some architectures that enables mixing several instruction sets in the same compilation unit. Example, arm and thumb interworking that this branch is trying to add. What is done ------------- 1. We add the switch primitive to the basic interface that changes the dissassembler in the current disassembling state. It is a bold move and can have conseqeuences, should be carefully reviewed 2. Attributes each destination in the disassembler driver state with the architecture and calls switch every time we are going to disassemble the next chunk of memory. 3. The default rule that extends the unit architecture to all instructions in that unit is disabled for ARM/Thumb and is overriden in the arm plugin with the following behavior, if an arm unit has a file and that file has a symbol table then we provide information based on the last bit of that symbol table (todo: we should also check for abi), otherwise we propagate the unit arch to instructions. What is to be done ------------------ Next, the arm lifter shall provide a promise to compute destinations (which itself will require destinations, because we don't really want to compute them) and provide the destination architecture, based on the source encoding. We can safely examine any representation of the instruction since it is already will be lifted by that moment.

also makes Enum more strict by checking that the element is indeed a member of the set of elements and by preventing double declarations.

In the second patch of this series (BinaryAnalysisPlatform#1225) we completely got rid of Arch.t dependency in the disassembler engine that finally opens the path for seamless integration of targets that are not representable with Arch.t. To achieve this, we introduced a proper dependency injection into the disassembler driver so that it is no longer responsible for creating the llvm MC disassembler. Instead a plugin that implements a target, aka the target support package, has to create a disassembler and is now in full control of all parameters and can choose backend, specify the CPU and other details of encoding. The encoding is a new abstraction in our knowledge base that breaks the tight connection between the target and the way how the program for that target is encoded. Unlike the target, which is a property of a unit of code, the encoding is associated with a program itself, i.e., it is a property of each instruction. That enables targets with context-dependent encodings such ARM's thumb mode and MIPS16e for binary encodings as well as paves the road for non-binary encodings for the same architecture, e.g., text assembly (which also may have several encodings on its own, cf. att vs intel syntax). We base this branch on the enable-interworking (BinaryAnalysisPlatform#1188) and this branch fully superseeds and includes it, since encodings made it much more natural. It is still highlty untested how it will work with real thumb binaries but we will get back to it when we will merge BinaryAnalysisPlatform#1178. Another big update, is that the disassembler backend (which is responsible for translating bits into machine instructions) is no longer required to be implemented in C++ and it is now possible to write your own backends/disassemblers in pure OCaml, e.g., to support PIC microcontrollers. The Backend interface is pretty low-level and we might provide higher-level interfaces later, see `Disasm_expert.Backend` for the interface and detailed comments. Finally, we rectify the interface introduced in the previous PR and flatten the hierarchy of newly introduced to the Core Theory abtractions, i.e., instead of `Theory.Target.Endiannes` we now have `Theory.Endianness` and so on. We also made the `Enum` module public which introduced enumerated types built on to of `Knowledge.Value`s. In the next episodes of this series we will gradually remove Arch.t from other bap components and further clean up the newly introduced interfaces.

This PR is the continuation of the BinaryAnalysisPlatform#1225, BinaryAnalysisPlatform#1226, and BinaryAnalysisPlatform#1227 series of changes that were focused on substituting the old and inextensible `Arch.t` abstraction with the new `Theory.Target.t` representation. This episode is instigated by the upcoming implementation of the RISCV target. Since RISCV is the out target that is not supported with Arch.t it became a good test of the new Theory.Target.t abstraction. As the RISCV worked showed, we still have lots of code that depends on Arch.t, most importantly Primus, which was fully dependent on Arch.t. The main issue was that Theory.Target.t doesn't provide any means to encode register classes, which prevented us from using it everywhere in Primus, e.g., we need to know which register is the stack pointer in order to setup the stack. To implement this, we introduce a new abstraction called _role_. A _role_ could be generally applied to any entity but so far we are only talking about the roles of registers in various targets. The target definiton now acccepts the `regs` paramater that takes the register file specification with each register assigned one or more roles, e.g., here is the register file specification for 8086, ```ocaml Theory.Role.Register.[ [general; integer], main @< index @< segment; [stack_pointer], untyped [reg r16 "SP"]; [frame_pointer], untyped [reg r16 "BP"]; [Role.index], untyped index; [Role.segment], untyped segment; [status], untyped flags; [integer], untyped [ reg bool "CF"; reg bool "PF"; reg bool "AF"; reg bool "ZF"; reg bool "SF"; reg bool "OF"; ] ``` I.e., we assign a set of roles to a set of registers. We also now have two new functions `Theory.Target.regs` and `Theory.Target.reg` that enable querying the register file of the target for register that fulfill one or more roles. Whilst we publish a limited number of well-known (blessed) roles in the `Theory.Role.Register` module, more roles could be added as user need it. For example, in the code snippet above we have two non-standard roles that are specific to the x86 architectures, `Role.index` and `Role.segment`. With roles we can drop the dependency on Target in most of the places where it makes sense (I still left it in x86 and other target-specific plugins, which obviously are independent on the newly added architectures).

This PR is the continuation of the #1225, #1226, and #1227 series of changes that were focused on substituting the old and inextensible `Arch.t` abstraction with the new `Theory.Target.t` representation. This episode is instigated by the upcoming implementation of the RISCV target. Since RISCV is the out target that is not supported with Arch.t it became a good test of the new Theory.Target.t abstraction. As the RISCV worked showed, we still have lots of code that depends on Arch.t, most importantly Primus, which was fully dependent on Arch.t. The main issue was that Theory.Target.t doesn't provide any means to encode register classes, which prevented us from using it everywhere in Primus, e.g., we need to know which register is the stack pointer in order to setup the stack. To implement this, we introduce a new abstraction called _role_. A _role_ could be generally applied to any entity but so far we are only talking about the roles of registers in various targets. The target definiton now acccepts the `regs` paramater that takes the register file specification with each register assigned one or more roles, e.g., here is the register file specification for 8086, ```ocaml Theory.Role.Register.[ [general; integer], main @< index @< segment; [stack_pointer], untyped [reg r16 "SP"]; [frame_pointer], untyped [reg r16 "BP"]; [Role.index], untyped index; [Role.segment], untyped segment; [status], untyped flags; [integer], untyped [ reg bool "CF"; reg bool "PF"; reg bool "AF"; reg bool "ZF"; reg bool "SF"; reg bool "OF"; ] ``` I.e., we assign a set of roles to a set of registers. We also now have two new functions `Theory.Target.regs` and `Theory.Target.reg` that enable querying the register file of the target for register that fulfill one or more roles. Whilst we publish a limited number of well-known (blessed) roles in the `Theory.Role.Register` module, more roles could be added as user need it. For example, in the code snippet above we have two non-standard roles that are specific to the x86 architectures, `Role.index` and `Role.segment`. With roles we can drop the dependency on Target in most of the places where it makes sense (I still left it in x86 and other target-specific plugins, which obviously are independent on the newly added architectures).

resolves #19 The example was buggy as the size of the pointer was specified incorrectly. It was acceptable before but is not longer tolerated after we enabled [interworking][1] (several architectures in the same binary). [1]: BinaryAnalysisPlatform/bap#1226

ivg added 4 commits September 25, 2020 15:04

flattens the target interface, publishes the Enum module

c0a3d55

also makes Enum more strict by checking that the element is indeed a member of the set of elements and by preventing double declarations.

adds an llvm decode for x86

4808ca9

drops the dependency on arch from the disassembler driver

e6888e2

ivg force-pushed the overhauls-targets-part-2 branch 2 times, most recently from 6f2f7ad to 6355807 Compare October 1, 2020 12:58

ivg force-pushed the overhauls-targets-part-2 branch from 6355807 to d27fd0e Compare October 1, 2020 15:19

ivg merged commit aae5c10 into BinaryAnalysisPlatform:master Oct 1, 2020

ivg mentioned this pull request Feb 22, 2021

overhauls target/value abstraction and introduces roles (4/n) #1275

Merged

ivg mentioned this pull request Jul 27, 2021

fixes the lift_insn example BinaryAnalysisPlatform/bap-bindings#20

Merged

ivg deleted the overhauls-targets-part-2 branch December 1, 2021 19:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

overhauls the target/architecture abstraction (2/n) #1226

overhauls the target/architecture abstraction (2/n) #1226

ivg commented Sep 30, 2020

overhauls the target/architecture abstraction (2/n) #1226

overhauls the target/architecture abstraction (2/n) #1226

Conversation

ivg commented Sep 30, 2020