[WIP] Convert Bash scripts to Python #147

[TheCodez]

I started the process of converting the Bash Scripts found in scripts to bash(#147)

All files compile without a problem, but there could be some runtime problems that I haven't found or tested yet.

The scripts should behave like their bash counterpart, as they are more or less a 1 to 1 translation that started off by using the Bash2Py tool and then fixing errors and applying some Python concepts

[TheCodez]

What's the right way to run python 3 code? Running the install-share script with python3 seems to only work on Linux.

[silverbacknet]

Both are python.exe on Windows, and with a default install, that will get you the last one installed. The standard windows way is "py -2" or "py -3" (a shortcut which resides in the windows folder) when people haven't manually renamed it.

[silverbacknet]

Still not quite working, but a lot closer. Fixed most of the bugs, now I need to dig in and see why the backend isn't working right for me. The .log is actually a .diff, but you can't upload those here. If I keep working on this I'll just clone and submit a pull request.

[silverbacknet]

Reuploading to remove a debugging line I left in.
retdec-py2.log
(reupload again with minor changes)

[TheCodez]

@silverbacknet Thanks for the help. Some of your fixes I had already fixed locally. But I will integrate the rest of your changes as soon as possible.

[silverbacknet]

That's always the tough part about having a public repo. But I'm stoked it's getting there, I'll be so happy when MSYS isn't needed anymore!

[TheCodez]

So for some reason the generated json config file is a lot shorter than the one generated by the bash script. As a result, the backend.ll file is a lot shorter too, which should be the reason why llvmir2hll doesn't do anything.
I really don't know whats the cause for this, because all command calls seem correct.

[Maroc-OS]

here you forgot about adding ".py" files.

[Maroc-OS]

here you forgot about adding ".py" files
if file_name.startswith('retdec-tests-') and not file.endswith('.sh') and not file.endswith('.py'):

[TheCodez]

Thanks for the help however I'm not quite sure if the code I wrote is even equivalent to what the original bash code intended to do:
find "$1" -name "retdec-tests-*" -type f $EXECUTABLE_FLAG | grep -v '\.sh$' | sort

Sadly there are also some other problems that prevent retdec_test_runner from working, because I'm currently running the executables incorrectly.

[Maroc-OS]

yes because you added python files to the folder we should skip them when searching for tests.

[Maroc-OS]

i have used this and seems okay to me, i did not compiled the tests but tested manually putting some test files on macOS and it runs.

[TheCodez]

Current status and things that need to be done:

• update CIs to use the scripts, needs Python upgrade to atleast 3.5

• retdec_archive_decompiler.py needs testing

• retdec_decompiler.py worked in all my test cases, might need more testing. In some cases better than the bash script? (Windown 10: retdec-decompiler.sh - strange behavior #350)

• retdec_unpacker.py finished. Is used successfully in decompiler script

• retdec_fileinfo.py works for simple files. Haven't tested with archives

• retdec_tests_runner.py doesn't really work, needs some work to be able to test in CI

• cmake/install-share.py is finished and working.

• yara_patterns/tools/compile-yara.py should work? Haven't tested.

• type_extractor/gen_cstdlib_and_linux_jsons.py is far from being finished.

• type_extractor/gen_windows_and_windrivers_jsons.py also needs help to finish.

[PeterMatula]

First of all, thanks, great work. 👍

I'm just looking into this and trying to run regression tests. The latest run (I hacked retdec-regression-tests-framework to use python implementation):
FAIL (failed 732 out of 5103 tests)
which is not too bad. However I don't think we test all the command line options and it will be very tricky to make sure every single option behave like it used to. But no matter, we can solve these problems later if somebody comes across them and creates an issue.

607 of those fails are related to retdec_fileinfo.py. The original script forwarded all the unknown arguments to retdec-fileinfo which it wraps. The new script does not. I'm currently rewriting it.

One question related to that: How/why did the shell=True get into this line: subprocess.call([config.FILEINFO, *fileinfo_params], shell=True)? It seems to me it is not working, and is not actually needed. Grep shows this happens on several places.

More important thing I'm thinking about is how to best collaborate on this. Other repositories (retdec-regression-tests-framework, retdec-idaplugin, ???) need to be modified. Should I (can I?) commit to this pull request, or should I merge this into a dedicated branch (e.g. bash-to-python) and continue the work on my own? I guess it depends if you (others) would like to continue working on this, or I should take over and finish it as soon as possible. It is hard to say how much work is left. It looks promising to me, but who knows what we come accross.

[TheCodez]

The original script forwarded all the unknown arguments to retdec-fileinfo which it wraps.

I noticed that in the beginning, but somehow lost track of it and implemented it without it.

How/why did the shell=True get into this line: subprocess.call([config.FILEINFO, *fileinfo_params], shell=True)

That's still a relict from using Bash2Py. It uses the parameter for every external programm call. So I left it like this, as it seemed to work for me. The decompiler script uses the CmdRunner class, so this problem only exists in some of the other scripts.

Should I (can I?) commit to this pull request, or should I merge this into a dedicated branch (e.g. bash-> to-python) and continue the work on my own? I guess it depends if you (others) would like to
continue working on this, or I should take over and finish it as soon as possible.

I'm fine with both. The latter however seems like the better option, as I don't have that much time the next few weeks. Also I think what's still left converting needs to be done by someone who has a better understanding of bash. If I (or others) would like to do some changes, we can still apply a pull request to the dedicated branch.

[PeterMatula]

If anyone would like to play with this, I did the following:

• Merged this into branch bash-to-python in the retdec repository.
• Created branch bash-to-python in the retdec-regression-tests-framework repository. Tests in this branch are using Python implementation - cca. 500 tests are failing at the moment.
• Created branch bash-to-python in the retdec-regression-tests repository. external.unit-tests.UnitTests needed to be changed to use Python script instead of Bash.
• Created branch bash-to-python in the retdec-idaplugin repository. This will also need to be updated to use Python implementation.

I will try to finish this as soon as possible.

[Maroc-OS]

Hello, i may help with this later am currently using it also as base because bash on macOS is not working.

retdec_decompiler.py does a great job but fail for selective decompilation. thats easy to be done

[TheCodez]

@Maroc-OS this can easily be fixed.
Replace the code starting at line 405:

if int(ranges[0]) > int(ranges[1]):

with:

start_range = int(ranges[0], 16 if ranges[0].startswith('0x') else 10)
end_range = int(ranges[1], 16 if ranges[1].startswith('0x') else 10)
if start_range > end_range:

This however only fixes the runtime error. The bash scripts produces different code, which means that the ranges are passed incorrectly to the underlying program.

[EDIT]
Fixed in #354