Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

remove symbolic information from the testsuite #1161

Closed
ivg opened this issue Jun 30, 2020 · 1 comment
Closed

remove symbolic information from the testsuite #1161

ivg opened this issue Jun 30, 2020 · 1 comment
Assignees
@gitoleg
Labels
CI testsuite

Comments

@ivg
Copy link
Member

ivg commented Jun 30, 2020

We are excluding a large and important portion of bap functionality from the tests by adding static symbolic information. I believe it is reminiscent of the bad old times, when we weren't able to setup binutils on Travis (not sure what stopped us then, though), but today we can have both radare and objdump on our runners so let's try to run as many tests as possible without relying on --read-symbols-from.
ivg added a commit to ivg/bap that referenced this issue Aug 3, 2020
@ivg
renovates the LLVM backend
62e5430 
Droping support of old LLVM and legacy backends
-----------------------------------------------

We drop a lot of old code (minus 3k lines of code) thus removing the support
burden and making it easier to maintain, fix, and upgrade the code.

Fixes BinaryAnalysisPlatform#1166

Simplifies the implementation
-----------------------------

The remaining code base is significanly simplified. We dropped the
separation between relocatable and non-relocatable files, removed any
transformations of addresses from the LLVM backend (we now emit
absolute virtual addresses). The whole logic of transforming from the
llvm view to the bap image view now fits into a hundred lines of
code (instead of hundreds lines spread across 16 files as it was
before).

Fixes BinaryAnalysisPlatform#1183
Fixes BinaryAnalysisPlatform#1189

Produces more information
-------------------------------

The relocation information is now emitted for all files (not only for
relocatable). Also, removes tons of checks that were preventing our
backends from emitting valuable symbolic information.

Paves the road to BinaryAnalysisPlatform#1135 and BinaryAnalysisPlatform#1161
@ivg ivg mentioned this issue Aug 3, 2020
Merged
ivg added a commit to ivg/bap that referenced this issue Aug 3, 2020
@ivg
renovates the LLVM backend
913c0ff 
Droping support of old LLVM and legacy backends
-----------------------------------------------

We drop a lot of old code (minus 3k lines of code) thus removing the support
burden and making it easier to maintain, fix, and upgrade the code.

Fixes BinaryAnalysisPlatform#1166

Simplifies the implementation
-----------------------------

The remaining code base is significanly simplified. We dropped the
separation between relocatable and non-relocatable files, removed any
transformations of addresses from the LLVM backend (we now emit
absolute virtual addresses). The whole logic of transforming from the
llvm view to the bap image view now fits into a hundred lines of
code (instead of hundreds lines spread across 16 files as it was
before).

Fixes BinaryAnalysisPlatform#1183
Fixes BinaryAnalysisPlatform#1189

Produces more information
-------------------------------

The relocation information is now emitted for all files (not only for
relocatable). Also, removes tons of checks that were preventing our
backends from emitting valuable symbolic information.

Paves the road to BinaryAnalysisPlatform#1135 and BinaryAnalysisPlatform#1161
ivg added a commit to ivg/bap that referenced this issue Aug 3, 2020
@ivg
calls radare2 directly
40cca78 
The ocaml-radare2 library is suffering from a few issues and it is
easier and much more efficient to call radare2 directly.

Here the list of issues that we have with ocaml-radare2
1) leaking file descriptors (they are never closed)
2) suboptimal inter-process communication (to get a 100 bytes
ocaml-radare makes a hundred system calls)
3) lingering processes (partially fixed)

In fact, we don't really need to keep the process open as we run
radare2 once and immediately close, so it is much easier for us just
to pass the command using the `-c` command line option.

@XVilka, I saw your fix and it is not yet working, the output channels
should be closed and exhausted for waitpid to return (or NOHANG)
should be passed. Also, without NOHANG, the code that sends signals is
actually unreachable. I've started fixing this and the other issues,
but later decided that for us it is much easier and much-much more
efficient to call radare2 directly.

@gitoleg, please update the corresponding opam package and drop the
dependency on ocaml-radare2. The new implementation also lacks the
version check as I hope that the new implementation will not hang even
with the older versions of radare2. If that is true, then it will help
us with BinaryAnalysisPlatform#1161. Also, please add the conf-radare2 package to the
upstream repo and add it as a depenedency to the bap-radare2. Please,
merge this PR after that.

Fixes BinaryAnalysisPlatform#1184.
@ivg ivg mentioned this issue Aug 3, 2020
Merged
ivg added a commit to ivg/bap that referenced this issue Aug 4, 2020
@ivg
renovates the LLVM backend
d1cc971 
Droping support of old LLVM and legacy backends
-----------------------------------------------

We drop a lot of old code (minus 3k lines of code) thus removing the support
burden and making it easier to maintain, fix, and upgrade the code.

Fixes BinaryAnalysisPlatform#1166

Simplifies the implementation
-----------------------------

The remaining code base is significanly simplified. We dropped the
separation between relocatable and non-relocatable files, removed any
transformations of addresses from the LLVM backend (we now emit
absolute virtual addresses). The whole logic of transforming from the
llvm view to the bap image view now fits into a hundred lines of
code (instead of hundreds lines spread across 16 files as it was
before).

Fixes BinaryAnalysisPlatform#1183
Fixes BinaryAnalysisPlatform#1189

Produces more information
-------------------------------

The relocation information is now emitted for all files (not only for
relocatable). Also, removes tons of checks that were preventing our
backends from emitting valuable symbolic information.

Paves the road to BinaryAnalysisPlatform#1135 and BinaryAnalysisPlatform#1161
ivg added a commit to ivg/bap that referenced this issue Aug 4, 2020
@ivg
renovates the LLVM backend
a3210a6 
Droping support of old LLVM and legacy backends
-----------------------------------------------

We drop a lot of old code (minus 3k lines of code) thus removing the support
burden and making it easier to maintain, fix, and upgrade the code.

Fixes BinaryAnalysisPlatform#1166

Simplifies the implementation
-----------------------------

The remaining code base is significanly simplified. We dropped the
separation between relocatable and non-relocatable files, removed any
transformations of addresses from the LLVM backend (we now emit
absolute virtual addresses). The whole logic of transforming from the
llvm view to the bap image view now fits into a hundred lines of
code (instead of hundreds lines spread across 16 files as it was
before).

Fixes BinaryAnalysisPlatform#1183
Fixes BinaryAnalysisPlatform#1189

Produces more information
-------------------------------

The relocation information is now emitted for all files (not only for
relocatable). Also, removes tons of checks that were preventing our
backends from emitting valuable symbolic information.

Paves the road to BinaryAnalysisPlatform#1135 and BinaryAnalysisPlatform#1161
ivg added a commit to ivg/bap that referenced this issue Aug 5, 2020
@ivg
renovates the LLVM backend
7110388 
Droping support of old LLVM and legacy backends
-----------------------------------------------

We drop a lot of old code (minus 3k lines of code) thus removing the support
burden and making it easier to maintain, fix, and upgrade the code.

Fixes BinaryAnalysisPlatform#1166

Simplifies the implementation
-----------------------------

The remaining code base is significanly simplified. We dropped the
separation between relocatable and non-relocatable files, removed any
transformations of addresses from the LLVM backend (we now emit
absolute virtual addresses). The whole logic of transforming from the
llvm view to the bap image view now fits into a hundred lines of
code (instead of hundreds lines spread across 16 files as it was
before).

Fixes BinaryAnalysisPlatform#1183
Fixes BinaryAnalysisPlatform#1189

Produces more information
-------------------------------

The relocation information is now emitted for all files (not only for
relocatable). Also, removes tons of checks that were preventing our
backends from emitting valuable symbolic information.

Paves the road to BinaryAnalysisPlatform#1135 and BinaryAnalysisPlatform#1161
ivg added a commit to ivg/bap that referenced this issue Aug 5, 2020
@ivg
renovates the LLVM backend
174b43f 
Droping support of old LLVM and legacy backends
-----------------------------------------------

We drop a lot of old code (minus 3k lines of code) thus removing the support
burden and making it easier to maintain, fix, and upgrade the code.

Fixes BinaryAnalysisPlatform#1166

Simplifies the implementation
-----------------------------

The remaining code base is significanly simplified. We dropped the
separation between relocatable and non-relocatable files, removed any
transformations of addresses from the LLVM backend (we now emit
absolute virtual addresses). The whole logic of transforming from the
llvm view to the bap image view now fits into a hundred lines of
code (instead of hundreds lines spread across 16 files as it was
before).

Fixes BinaryAnalysisPlatform#1183
Fixes BinaryAnalysisPlatform#1189

Produces more information
-------------------------------

The relocation information is now emitted for all files (not only for
relocatable). Also, removes tons of checks that were preventing our
backends from emitting valuable symbolic information.

Paves the road to BinaryAnalysisPlatform#1135 and BinaryAnalysisPlatform#1161
ivg added a commit that referenced this issue Aug 5, 2020
@ivg @gitoleg
renovates the LLVM backend (#1187)
27a7a5d 
* fixes the base calculation

1. For ELF files we compute base as the difference between the address of
any loadable code segment and its offset. If there are no loadable code
segments, then we find a section with minimal offset value and
substract its address from its offset.

2. For MachO, when the file is relocatable, i.e., it doesn't have addresses we
compute base as $vaddr - offset$, the same as we do in ELF. This
gives us results that match objdump (but do not match radare2, however
radare2 is not seeing any symbols, so it doesn't really matter)

3. For COFF nothing is done, and I am not sure that we need
to do anything.

4. Removed special computation of the base
address (Base.from_sections_offset) from ELF, MachO, and COFF.

It is not tested on LLVM versions below 6, but I believe it should
work up to 3.4.

resolves #1183

Co-authored-by: gitoleg <forown@yandex.ru>

* re-enables the failing test again

Hope we will pass it now.

* updates paths to artifacts

* renovates the LLVM backend

Droping support of old LLVM and legacy backends
-----------------------------------------------

We drop a lot of old code (minus 3k lines of code) thus removing the support
burden and making it easier to maintain, fix, and upgrade the code.

Fixes #1166

Simplifies the implementation
-----------------------------

The remaining code base is significanly simplified. We dropped the
separation between relocatable and non-relocatable files, removed any
transformations of addresses from the LLVM backend (we now emit
absolute virtual addresses). The whole logic of transforming from the
llvm view to the bap image view now fits into a hundred lines of
code (instead of hundreds lines spread across 16 files as it was
before).

Fixes #1183
Fixes #1189

Produces more information
-------------------------------

The relocation information is now emitted for all files (not only for
relocatable). Also, removes tons of checks that were preventing our
backends from emitting valuable symbolic information.

Paves the road to #1135 and #1161

Co-authored-by: gitoleg <forown@yandex.ru>
gitoleg added a commit that referenced this issue Aug 7, 2020
@ivg @gitoleg
calls radare2 directly (#1195)
c9980f4 
* calls radare2 directly

The ocaml-radare2 library is suffering from a few issues and it is
easier and much more efficient to call radare2 directly.

Here the list of issues that we have with ocaml-radare2
1) leaking file descriptors (they are never closed)
2) suboptimal inter-process communication (to get a 100 bytes
ocaml-radare makes a hundred system calls)
3) lingering processes (partially fixed)

In fact, we don't really need to keep the process open as we run
radare2 once and immediately close, so it is much easier for us just
to pass the command using the `-c` command line option.

@XVilka, I saw your fix and it is not yet working, the output channels
should be closed and exhausted for waitpid to return (or NOHANG)
should be passed. Also, without NOHANG, the code that sends signals is
actually unreachable. I've started fixing this and the other issues,
but later decided that for us it is much easier and much-much more
efficient to call radare2 directly.

@gitoleg, please update the corresponding opam package and drop the
dependency on ocaml-radare2. The new implementation also lacks the
version check as I hope that the new implementation will not hang even
with the older versions of radare2. If that is true, then it will help
us with #1161. Also, please add the conf-radare2 package to the
upstream repo and add it as a depenedency to the bap-radare2. Please,
merge this PR after that.

Fixes #1184.

* replaces the `radare2` dependency with the `conf-radare2`

Co-authored-by: gitoleg <forown@yandex.ru>
@ivg
Copy link
Member Author

ivg commented Aug 21, 2020

done in #1209

@ivg ivg closed this as completed Aug 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gitoleg gitoleg

Labels
CI testsuite
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ivg @gitoleg