Rewrite x86 decoder to use Capstone for instruction decoding

[ceeac]

This PR completely replaces the old PentiumDecoder by a new CapstoneX86Decoder using Capstone for decoding x86 instructions. This has several benefits:

• No 50k source code lines file
• Faster compile times, especially with optimizations enabled
• Cleaner code
• Better unit test coverage
• Better regression test coverage
• Support for more x86 instructions (when finished)
• Better error messages when decoding instructions with unknown semantics
• Fixes crashes when decoding x86 instructions with swapped prefixes

This is a work in progress - do not merge yet.

[XVilka]

Why not to make it work on top of radare2 instead? See radareorg/radare2#10102

[ceeac]

Only using radare2 as an instruction decoder would ultimately limit flexibility: radare2 would then be a hard dependency. What I eventually want to do is being able to add decoders as plugins, so they can be changed at will. So using radare2 as an instruction decoder might be possible in the future, but as an optional dependency instead of as a hard dependendy.

[nemerle]

I agree with @ceeac, using capstone in a plugin is a better approach. And I'm afraid that integrating radare2 would lead the boomerang to becoming another radare2 frontend ( and there are a few of those already ? :) )

[nemerle]

Also, two things to consider:

• downloading + building a specific capstone version using CMake's ExternalProject support ( more stable environment )
• a way to integrate 16bit x86 ( maybe another plugin/separate issue altogether ? )

[ceeac]

Re. ExternalProject / 16bit x86 support:
I think these should be adressed after this PR is merged. I want to stabilize x86_32 support first; however adding 16 bit x86 support should not be too hard afterwards.

[XVilka]

Capstone is not very updated lately. You can try Zydis.

[ceeac]

There are a couple of things I want to say about this:

• I would not call a library that has had its last release three months ago "not updated" (although I agree that there was a large gap between the last release and the next-to-last one).
• Zydis only supports x86 and x86-64, while Capstone supports mote architectures. Using a single library for all instruction decoders reduces required maintenance, and why use another library when we most likely are going to use Capstone anyways in one way or another?
• The majority of the work left to do (for this PR) is in updating x86.ssl, which can be used regardless of which disassembler library is used. So imo it makes sense to keep Capstone because it works for the scope of this PR (The base IA-32 instruction set without the (now not-so-new) extensions like SSE, MMX etc.)
• After this PR is merged, it is not too hard to add a new decoder plugin that uses Zydis for disassembling instructions. Feel free to submit a PR for this @XVilka.

[ceeac]

To clarify what instructions still need semantics for this PR, these are:

• String instructions with and without prefixes (some are already implemented)
• Lots of FP instructions
• Some MUL/IMUL variants
• BSF/BSR instructions

In addition, all instructions still have to be checked for correctness before this PR can be merged.

[XVilka]

@ceeac Surely it is abandoned - author cannot support latest ARM instructions for example, PRs are rotting even without response: https://github.com/aquynh/capstone/pulls
This is why x96dbg for example switched to capstone https://x64dbg.com/blog/2017/10/18/goodbye-capstone-hello-zydis.html

I am not pushing, just as we (radare2 project) use capstone we met this inability to keep the library updated, thus rendering it is almost useless for any modern real world binaries.

[ceeac]

I have now tested the new decoder on a number of binaries and found it to work and perform much better than the old one. Some finishing touches will still be necessary, but I'll leave that for additional PRs.

Re. Zydis support: My current plan is to implement a Zydis decoder plugin after the release of 0.5.0 so users can choose between Capstone and Zydis.