BAP 2.0.0 release

[ivg]

This is a major update of the Binary Analysis Platform that brings lots of new features, libraries, and tools. We will properly advertise it later, using discuss and mailing list, but, so far, the most important and notable changes are:

• a new representation of program semantics (using tagless final embedding)
• a new backward chaining engine that runs co-dependent analyses

Despite the fact that we bump the major version there are no real breaking changes in the interfaces (some purist might say there are, depending on what you mean by a breaking change wrt to OCaml). We were able to implement the old interfaces using the new engine. The semantics of some functions slightly changed.

Features

BinaryAnalysisPlatform/bap#1016 adds unknown architecture
BinaryAnalysisPlatform/bap#1014 restores postinstall and man pages
BinaryAnalysisPlatform/bap#1013 tweaks the cache plugin
BinaryAnalysisPlatform/bap#1011 tweaks the subroutine ordering in the run plugin
BinaryAnalysisPlatform/bap#1006 extends the test coverage on Travis
BinaryAnalysisPlatform/bap#1005 introduces Bap_main the entry point to BAP
BinaryAnalysisPlatform/bap#1005 new command line interface and library
BinaryAnalysisPlatform/bap#1005 a new raw loader for opening unknown files and raw code
BinaryAnalysisPlatform/bap#1005 a new objdump command for linearly disassembling binaries
BinaryAnalysisPlatform/bap#1005 a new Byteweight threshold using the Bayes Factors procedure
BinaryAnalysisPlatform/bap#1005 adds more control over byteweight thresholding
BinaryAnalysisPlatform/bap#1005 implements custom thresholding procedures for Byteweight
BinaryAnalysisPlatform/bap#1005 extends the Trie module with iterators and printers
BinaryAnalysisPlatform/bap#1005 new toplevel based on utop
BinaryAnalysisPlatform/bap#1005 new bytecode frontend for debugging
BinaryAnalysisPlatform/bap#1005 adds a central location for regular readers and writers
BinaryAnalysisPlatform/bap#999 tweaks Graphlibs fixpoint to preserve previous solution
BinaryAnalysisPlatform/bap#998 represents rep prefix with while in x86
BinaryAnalysisPlatform/bap#993 prevents Primus from going to deep into PLT entries
BinaryAnalysisPlatform/bap#991 tweaks the print plugin
BinaryAnalysisPlatform/bap#990 represents ARM conditionals with ite expressions when possible
BinaryAnalysisPlatform/bap#998 implements more fine granular view on the image memory
BinaryAnalysisPlatform/bap#979 parametrize Lisp malloc with an initialization strategy
BinaryAnalysisPlatform/bap#960 new subroutine packing algorithm
BinaryAnalysisPlatform/bap#960 new incremental disassembler
BinaryAnalysisPlatform/bap#960 new knowledge representation library
BinaryAnalysisPlatform/bap#960 new representation of program semantics
BinaryAnalysisPlatform/bap#960 new bitvectors library
BinaryAnalysisPlatform/bap#948 disables Primus' taint GC by default

Bug fixes

BinaryAnalysisPlatform/bap#1013 resolves leaking files in the cache plugin
BinaryAnalysisPlatform/bap#1003 rectifies Primus semantics in case of exceptions
BinaryAnalysisPlatform/bap#1002 fixes bind operator in the Future library
BinaryAnalysisPlatform/bap#1000 fixes instruction properties computation for barriers
BinaryAnalysisPlatform/bap#985 fixes atexit Lisp stub
BinaryAnalysisPlatform/bap#980 fixes a bug in the configure script
BinaryAnalysisPlatform/bap#971 limits continuations at forks in the promiscuous mode
BinaryAnalysisPlatform/bap#970 fixes the argument evaluation order in call-return
BinaryAnalysisPlatform/bap#964 fixes Primus random generators
BinaryAnalysisPlatform/bap#962 fixes the semantics of signed division by zero in x86
BinaryAnalysisPlatform/bap#958 fixes Primus memory semantics with randomized memories
BinaryAnalysisPlatform/bap#955 improves stack handling in Primus for x86
BinaryAnalysisPlatform/bap#950 fixes the taint sanitization procedure

Changes that may affect/break your programs

1. You won't see anymore ret terms in the IR, as exits from subroutines are now represented as calls.
   Your plugins shall not perform any side-effects during evaluation except using the old Config.when_ready function or the new interface from the bap-main library.
2. If your application was a host program and was initializing the plugins manually, then you shall switch to the bap-main library. (It's easy, just do Bap_main.init instead of Plugins.run)
3. No more options for rooters, branchers, and reconstructors. They are subsumed with our new knowledge base and knowledge providers. The existing services and re-introduced via a backward compatibility layer. If you had a brancher of your own, then you will need to update it since we've changed their semantics, now a brancher shall not provide the fall-through edge.
   4.If you were using custom command-line parsing in your plugins this won't work anymore, you have to use the new bap-main library to specify your configuration options or the old Config module from Bap.Std, which still works fine (it is implemented in terms of bap-main).
4. New command-line interface that may break your bap orchestration utilities. We were trying to keep it as backward compatible as possible, e.g., bap /bin/ls --run will still work as before, but some of your scripts may stop working.

[ivg]

As always we're failing on conf-ida package, since obviously there is no ida available on the CI infrastructure, @kit-ty-kate we're requesting your permission to add the x-ci-accept-failures field to our conf-ida package for all distributions involved in the CI infrastructure, as well on a selected architectures for conf-llvm package (we will try to find an llvm version which is a the most common denominator for all available distributions, so far it is 6.0).

[camelus]

Commit: b5b0109

@ivg has posted 42 contributions.

☀️ All lint checks passed b5b0109

• These packages passed lint tests: bap-abi.2.0.0, bap-api.2.0.0, bap-arm.2.0.0, bap-beagle.2.0.0, bap-bil.2.0.0, bap-build.2.0.0, bap-bundle.2.0.0, bap-byteweight-frontend.2.0.0, bap-byteweight.2.0.0, bap-c.2.0.0, bap-cache.2.0.0, bap-callsites.2.0.0, bap-constant-tracker.2.0.0, bap-core-theory.2.0.0, bap-cxxfilt.2.0.0, bap-demangle.2.0.0, bap-disassemble.2.0.0, bap-dump-symbols.2.0.0, bap-dwarf.2.0.0, bap-elementary.2.0.0, bap-elf.2.0.0, bap-frontc.2.0.0, bap-frontend.2.0.0, bap-future.2.0.0, bap-ida-plugin.2.0.0, bap-ida-python.2.0.0, bap-ida.2.0.0, bap-knowledge.2.0.0, bap-llvm.2.0.0, bap-main.2.0.0, bap-mc.2.0.0, bap-microx.2.0.0, bap-mips.2.0.0, bap-objdump.2.0.0, bap-optimization.2.0.0, bap-phoenix.2.0.0, bap-piqi.2.0.0, bap-plugins.2.0.0, bap-powerpc.2.0.0, bap-primus-dictionary.2.0.0, bap-primus-lisp.2.0.0, bap-primus-powerpc.2.0.0, bap-primus-region.2.0.0, bap-primus-support.2.0.0, bap-primus-test.2.0.0, bap-primus-x86.2.0.0, bap-primus.2.0.0, bap-print.2.0.0, bap-raw.2.0.0, bap-recipe-command.2.0.0, bap-recipe.2.0.0, bap-relocatable.2.0.0, bap-report.2.0.0, bap-run.2.0.0, bap-signatures.2.0.0, bap-ssa.2.0.0, bap-std.2.0.0, bap-strings.2.0.0, bap-symbol-reader.2.0.0, bap-taint-propagator.2.0.0, bap-taint.2.0.0, bap-term-mapper.2.0.0, bap-trace.2.0.0, bap-traces.2.0.0, bap-trivial-condition-form.2.0.0, bap-veri.0.2.4, bap-warn-unused.2.0.0, bap-x86.2.0.0, bap.2.0.0, bare.2.0.0, bitvec-binprot.2.0.0, bitvec-order.2.0.0, bitvec-sexp.2.0.0, bitvec.2.0.0, conf-bap-llvm.1.5, graphlib.2.0.0, monads.2.0.0, ogre.2.0.0, regular.2.0.0, text-tags.2.0.0

---

☀️ Installability check (+80)

• new installable packages (80): bap.2.0.0 bap-abi.2.0.0 bap-api.2.0.0 bap-arm.2.0.0 bap-beagle.2.0.0 bap-bil.2.0.0 bap-build.2.0.0 bap-bundle.2.0.0 bap-byteweight.2.0.0 bap-byteweight-frontend.2.0.0 bap-c.2.0.0 bap-cache.2.0.0 bap-callsites.2.0.0 bap-constant-tracker.2.0.0 bap-core-theory.2.0.0 bap-cxxfilt.2.0.0 bap-demangle.2.0.0 bap-disassemble.2.0.0 bap-dump-symbols.2.0.0 bap-dwarf.2.0.0 bap-elementary.2.0.0 bap-elf.2.0.0 bap-frontc.2.0.0 bap-frontend.2.0.0 bap-future.2.0.0 bap-ida.2.0.0 bap-ida-plugin.2.0.0 bap-ida-python.2.0.0 bap-knowledge.2.0.0 bap-llvm.2.0.0 bap-main.2.0.0 bap-mc.2.0.0 bap-microx.2.0.0 bap-mips.2.0.0 bap-objdump.2.0.0 bap-optimization.2.0.0 bap-phoenix.2.0.0 bap-piqi.2.0.0 bap-plugins.2.0.0 bap-powerpc.2.0.0 bap-primus.2.0.0 bap-primus-dictionary.2.0.0 bap-primus-lisp.2.0.0 bap-primus-powerpc.2.0.0 bap-primus-region.2.0.0 bap-primus-support.2.0.0 bap-primus-test.2.0.0 bap-primus-x86.2.0.0 bap-print.2.0.0 bap-raw.2.0.0 bap-recipe.2.0.0 bap-recipe-command.2.0.0 bap-relocatable.2.0.0 bap-report.2.0.0 bap-run.2.0.0 bap-signatures.2.0.0 bap-ssa.2.0.0 bap-std.2.0.0 bap-strings.2.0.0 bap-symbol-reader.2.0.0 bap-taint.2.0.0 bap-taint-propagator.2.0.0 bap-term-mapper.2.0.0 bap-trace.2.0.0 bap-traces.2.0.0 bap-trivial-condition-form.2.0.0 bap-veri.0.2.4 bap-warn-unused.2.0.0 bap-x86.2.0.0 bare.2.0.0 bitvec.2.0.0 bitvec-binprot.2.0.0 bitvec-order.2.0.0 bitvec-sexp.2.0.0 conf-bap-llvm.1.5 graphlib.2.0.0 monads.2.0.0 ogre.2.0.0 regular.2.0.0 text-tags.2.0.0

---

🌤️ 4 ignored non-opam files:

• packages/bap-ida-python/bap-ida-python.2.0.0/files/bap.cfg.in
• packages/bap-llvm/bap-llvm.2.0.0/files/detect.travis
• packages/conf-bap-llvm/conf-bap-llvm.1.5/files/find-llvm.ml.in
• packages/conf-ida/conf-ida.0.2/files/find-ida.ml.in

[ivg]

OK, it looks like we're good to go. The only failures are either from the CI system itself or from conf-ida. Not sure whether x-ci-accept-failures works or not, at least we don't see any differences from our vantage point.

Could you please, guys merge it?

[mseri]

I was waiting for @kit-ty-kate to have a look and reply to your question before merging, but this could also be done later.

The name and version stanzas in the opam files is Redundant and overridden by opam with the ones deduced from the path. Is there any reason why you want to keep them? (This does not prevent merging)

The remove stanzas are no longer needed in opam2. Can you remove them?(I had missed the use of bapundle therej, sorry)

[ivg]

Thanks!