Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix integration test #31

Merged
merged 14 commits into from
Apr 29, 2024
Merged

Fix integration test #31

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
51f2de0
Decompress archive in obtain_input_file
Apr 16, 2024
3920ea2
Add tar.gz-archive to S3 properly
Apr 16, 2024
1c53828
Black
Apr 16, 2024
4871339
Update io_util.py
Veldhoen Apr 29, 2024
32be036
Update io_util.py
Veldhoen Apr 29, 2024
f97e8bd
removed spurious comment
Apr 29, 2024
809225c
apply_model is responsible for getting the approprate input from the …
Apr 29, 2024
0e07763
Removed unnecessary installation from _test
Apr 29, 2024
34e7adf
Temporarily commented out Flake8 because of weird version issues
Apr 29, 2024
0f72deb
Use test config for tests
Apr 29, 2024
6d0679b
Make data (instead of /data) the base mount in test
Apr 29, 2024
1fd870f
Updated test config
Apr 29, 2024
7b7469f
Apply black
Apr 29, 2024
d8faea0
Temporarily commented out mypy
Apr 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion io_util.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,9 @@ def obtain_input_file(s3_uri: str) -> ThisWorkerInput:
)
success = s3.download_file(bucket, object_name, output_folder)
if success:
# TODO uncompress the <input_base>.tar.gz
if input_file_path.find(".tar.gz") != -1:
# TODO: more elegant solution for hardcoded file name
input_file_path = untar_input_file(input_file_path) + f"/{source_id}.input"
Veldhoen marked this conversation as resolved.
Show resolved Hide resolved

provenance = Provenance(
activity_name="download",
Expand Down Expand Up @@ -333,3 +335,4 @@ def untar_input_file(tar_file_path: str):
path = str(Path(tar_file_path).parent)
with tarfile.open(tar_file_path) as tar:
tar.extractall(path=path, filter="data") # type: ignore
return path
Veldhoen marked this conversation as resolved.
Show resolved Hide resolved
5 changes: 3 additions & 2 deletions tests/integration/S3_integration_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,9 @@ def create_and_fill_buckets(aws, create_sample_input):
cfg.INPUT.S3_BUCKET_MODEL,
]:
client.create_bucket(Bucket=bucket)
client.put_object(
Body=fn_tar_in,

client.upload_file(
Filename=fn_tar_in,
Bucket=cfg.INPUT.S3_BUCKET,
Key=f"{cfg.INPUT.S3_FOLDER_IN_BUCKET}/{key_in}",
)
Expand Down
Loading