Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Convert Bash scripts to Python #147 #338

Merged
merged 48 commits into from
Jul 12, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
b237a43
Convert most Bash scripts to Python
TheCodez Jun 11, 2018
97ae641
Port cmake/install-share to python
TheCodez Jun 12, 2018
d922d13
Some cleanup
TheCodez Jun 12, 2018
fdb6b4a
Replaced install-share in cmake file with fully working python version
TheCodez Jun 12, 2018
8849cde
Attemp to fix compile error
TheCodez Jun 12, 2018
ed45adc
Fix compile for older python version
TheCodez Jun 12, 2018
97c456b
Try calling python 3 instead
TheCodez Jun 12, 2018
55c73dc
Make install-share.py executable
TheCodez Jun 14, 2018
2f51c69
Decompiler script now runs successfully until unpacking
TheCodez Jun 15, 2018
2d84524
Running until calling bin2llvmir now
TheCodez Jun 16, 2018
386556d
[skip ci] Integrate @silverbacknet changes + some fixes
TheCodez Jun 16, 2018
f10e7e3
[skip ci] Use CmdRunner.run_cmd everywhere in retdec_decompiler
TheCodez Jun 16, 2018
5248482
[skip ci] Latest fixes
TheCodez Jun 16, 2018
b6975d0
[skip ci] Check that options are correct + cleanup and fixes
TheCodez Jun 17, 2018
e29b06d
[skip ci] Fixed various errors
TheCodez Jun 17, 2018
4a91c1c
Try to fix running install-share script
TheCodez Jun 17, 2018
a53cc43
Should now work on every os
TheCodez Jun 17, 2018
a8de6fd
Fix compile error
TheCodez Jun 17, 2018
ea903dd
Convert compile-yara to python
TheCodez Jun 18, 2018
6d78c9b
[skip ci] Make test runner more portable
TheCodez Jun 18, 2018
c1635da
[skip ci] Use correct code style
TheCodez Jun 18, 2018
500e24e
[skip ci] Decompiler script now runs successfully
TheCodez Jun 18, 2018
0a2ab46
Now generates the same output as the bash script
TheCodez Jun 18, 2018
ef7b5ce
Try fixing Travis on macOS
TheCodez Jun 18, 2018
b6c52a6
Upgrade python instead
TheCodez Jun 18, 2018
4fa358a
Test scripts in travis
TheCodez Jun 18, 2018
f8d2ae8
Fix build
TheCodez Jun 18, 2018
90b9ba6
Fix path
TheCodez Jun 18, 2018
192c39e
Update build
TheCodez Jun 18, 2018
e21840f
Fix error in decompiler script
TheCodez Jun 18, 2018
576e605
Try to debug failure reason
TheCodez Jun 18, 2018
287f750
Use newer Python version and fix some errors
TheCodez Jun 18, 2018
df5058a
[skip ci] Little cleanup to make the code more clear
TheCodez Jun 19, 2018
8a89ad5
[skip ci] First version of reimplementing logging
TheCodez Jun 19, 2018
6a80440
[skip ci] Some fixes and cleanup
TheCodez Jun 19, 2018
e009193
[skip ci] Print memory usage, print output from unpacker, match code …
TheCodez Jun 19, 2018
7d73239
[skip ci] Fix crash when using cleanup option; fix crash when using c…
TheCodez Jun 19, 2018
f15127d
[skip ci] Fix --backend-aggressive-opts argument
TheCodez Jun 20, 2018
60e8080
[skip ci] Fix error when file arch is followed by a comment
TheCodez Jun 20, 2018
449c85f
[skip ci] Match Bash script more closely
TheCodez Jun 20, 2018
5415872
[skip ci] Fix a few comments
TheCodez Jun 21, 2018
9009c6c
[skip ci] Add some comments
TheCodez Jun 21, 2018
1a08e9f
[skip ci] Add early type_extractor/gen_cstdlib_and_linux_jsons.py and…
TheCodez Jun 21, 2018
6db0e24
Try Unit tests
TheCodez Jun 21, 2018
0654a51
Try to fix test
TheCodez Jun 21, 2018
e3b80d5
Use absolute path instead
TheCodez Jun 21, 2018
a04a067
[skip ci] Add check for python scripts
TheCodez Jul 3, 2018
c38bf5a
scripts/retdec_decompiler.py: use output if specified via -o option
PeterMatula Jul 11, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ matrix:
addons:
apt:
sources:
- deadsnakes
- ubuntu-toolchain-r-test
packages:
- build-essential
Expand All @@ -19,7 +20,7 @@ matrix:
- g++-4.9
- cmake
- perl
- python3
- python3.5
- flex
- bison
- autoconf
Expand Down Expand Up @@ -55,7 +56,9 @@ install:
- if [ "$TRAVIS_OS_NAME" = "osx" ]; then brew install gnu-getopt; fi
# bash 4
- if [ "$TRAVIS_OS_NAME" = "osx" ]; then brew install bash; fi

# python 3
- if [ "$TRAVIS_OS_NAME" = "osx" ]; then brew upgrade python; fi

before_script:
- eval "${MATRIX_EVAL}"
# We need to use newer versions of Flex and Bison on MacOS X (the ones from Homebrew).
Expand All @@ -79,18 +82,15 @@ script:
- cd ..
- rm -rf build
# Run unit tests.
- if [ "$TRAVIS_OS_NAME" = "osx" ]; then /usr/local/bin/bash retdec-install/bin/retdec-tests-runner.sh; fi
- if [ "$TRAVIS_OS_NAME" = "linux" ]; then ./retdec-install/bin/retdec-tests-runner.sh; fi
- python3 retdec-install/bin/retdec_tests_runner.py
# Run the decompilation script.
- if [ "$TRAVIS_OS_NAME" = "osx" ]; then /usr/local/bin/bash retdec-install/bin/retdec-decompiler.sh --help; fi
- if [ "$TRAVIS_OS_NAME" = "linux" ]; then ./retdec-install/bin/retdec-decompiler.sh --help; fi
- python3 retdec-install/bin/retdec_decompiler.py --help
# Run a simple decompilation.
- echo -e '#include <stdio.h>\n#include <stdlib.h>\nint main()\n{\n printf("hello world\\n");\n return 0;\n}\n' > hello-orig.c
- cat hello-orig.c
- gcc -m32 -o hello hello-orig.c
- ./hello
- if [ "$TRAVIS_OS_NAME" = "osx" ]; then /usr/local/bin/bash retdec-install/bin/retdec-decompiler.sh hello; fi
- if [ "$TRAVIS_OS_NAME" = "linux" ]; then ./retdec-install/bin/retdec-decompiler.sh hello; fi
- python3 retdec-install/bin/retdec_decompiler.py hello
- cat hello.c
- grep "int main(int argc, char \*\* argv)" hello.c

Expand Down
16 changes: 12 additions & 4 deletions cmake/install-external.cmake
Original file line number Diff line number Diff line change
@@ -1,8 +1,16 @@
install(CODE "
execute_process(
COMMAND sh \"${CMAKE_SOURCE_DIR}/cmake/install-share.sh\" \"${CMAKE_INSTALL_PREFIX}\"
RESULT_VARIABLE INSTALL_SHARE_RES
)
if (WIN32)
execute_process(
COMMAND py -3 \"${CMAKE_SOURCE_DIR}/cmake/install-share.py\" \"${CMAKE_INSTALL_PREFIX}\"
RESULT_VARIABLE INSTALL_SHARE_RES
)
else()
execute_process(
COMMAND python3 \"${CMAKE_SOURCE_DIR}/cmake/install-share.py\" \"${CMAKE_INSTALL_PREFIX}\"
RESULT_VARIABLE INSTALL_SHARE_RES
)
endif()

if(INSTALL_SHARE_RES)
message(FATAL_ERROR \"RetDec share directory installation FAILED\")
endif()
Expand Down
115 changes: 115 additions & 0 deletions cmake/install-share.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
#!/usr/bin/env python3

"""
Get RetDec share directory.
"""
import sys
import hashlib
import os
import shutil
import tarfile
import urllib.request

# Check arguments.
if len(sys.argv) != 2:
print('ERROR: Unexpected number of arguments.')
sys.exit(1)

###############################################################################

version_filename = 'version.txt'
arch_suffix = 'tar.xz'

sha256hash_ref = 'b54ba07e2f28143c9afe34a9d5b4114fb61f3c1175b9807caced471fec82001e'
version = '2018-02-08'

###############################################################################

arch_name = 'retdec-support' + '_' + version + '.' + arch_suffix

# Get install path from script options.
install_path = sys.argv[1]

share_dir = os.path.join(install_path, 'share')
share_retdec_dir = os.path.join(share_dir, 'retdec')
support_dir = os.path.join(share_retdec_dir, 'support')

arch_path = os.path.join(support_dir, arch_name)


###############################################################################

def cleanup():
if os.path.exists(support_dir):
for n in os.listdir(support_dir):
p = os.path.join(support_dir, n)
if os.path.isdir(p):
shutil.rmtree(p)
else:
os.unlink(p)


# Share directory exists.
if os.path.exists(support_dir):
# Version file exists.
if os.path.isfile(os.path.join(support_dir, version_filename)):
with open(os.path.join(support_dir, version_filename)) as version_file:
version_from_file = version_file.read().split('\n')[0]

if version == version_from_file:
print('%s already exists, version is ok' % support_dir)
sys.exit(0)
else:
print('versions is not as expected -> replace with expected version')

cleanup()

# Make sure destination directory exists.
os.makedirs(support_dir, exist_ok=True)

# Download archive
arch_url = 'https://github.com/avast-tl/retdec-support/releases/download/%s/%s' % (version, arch_name)
print('Downloading archive from %s ...' % arch_url)

try:
urllib.request.urlretrieve(arch_url, arch_path)
except (urllib.request.HTTPError, urllib.request.URLError):
print('ERROR: download failed')
cleanup()
sys.exit(1)

# Compute hash of the downloaded archive.
print('Verfifying archive\'s checksum ...')

sha256 = hashlib.sha256()
with open(arch_path, 'rb') as f:
try:
sha256.update(f.read())
except IOError:
print('ERROR: failed to compute the SHA-256 hash of the archive')
cleanup()
sys.exit(1)

sha256hash = sha256.hexdigest()

# Check that hash is ok.
if sha256hash != sha256hash_ref:
print('ERROR: downloaded archive is invalid (SHA-256 hash check failed)')
cleanup()
sys.exit(1)

# Unpack archive.
print('Unpacking archive ...')
with tarfile.open(arch_path) as tar:
try:
tar.extractall(support_dir)
except tarfile.ExtractError:
print('ERROR: failed to unpack the archive')
cleanup()
sys.exit(1)

# Remove archive.
os.remove(arch_path)

print('RetDec support directory downloaded OK')
sys.exit(0)
12 changes: 12 additions & 0 deletions scripts/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,15 @@ endif()
install(PROGRAMS "retdec-signature-from-library-creator.sh" DESTINATION bin)
install(PROGRAMS "retdec-unpacker.sh" DESTINATION bin)
install(PROGRAMS "retdec-utils.sh" DESTINATION bin)

# copy python scripts
install(PROGRAMS "retdec_config.py" DESTINATION bin)
install(PROGRAMS "retdec_archive_decompiler.py" DESTINATION bin)
install(PROGRAMS "retdec_decompiler.py" DESTINATION bin)
install(PROGRAMS "retdec_fileinfo.py" DESTINATION bin)
if(RETDEC_TESTS)
install(PROGRAMS "retdec_tests_runner.py" DESTINATION bin)
endif()
install(PROGRAMS "retdec_signature_from_library_creator.py" DESTINATION bin)
install(PROGRAMS "retdec_unpacker.py" DESTINATION bin)
install(PROGRAMS "retdec_utils.py" DESTINATION bin)
Loading