Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adjust ingest_2_parquet workflow. #248

Merged
merged 2 commits into from
Jun 13, 2024
Merged

Adjust ingest_2_parquet workflow. #248

merged 2 commits into from
Jun 13, 2024

Conversation

revit13
Copy link
Collaborator

@revit13 revit13 commented Jun 9, 2024 •
edited
Loading

This PR adjust ingest_2_parquet to the recent workflow changes.

This PR also uses a new input zip file as the current ones seems to be too large to upload to minio:

 mc cp --recursive ${ROOT_DIR}/../transforms/code/ingest_2_parquet/ray/test-data/input/ kfp/test/ingest_2_parquet/input/data-processing-lib.zip
mc: <ERROR> Failed to copy `/data/revital/prep/data-prep-lab/transforms/code/ingest_2_parquet/ray/test-data/input/https___github.com_00000o1_environments_archive_refs_heads_master.zip`. <html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx</center>
</body>
</html>
mc: <ERROR> Failed to copy `/data/revital/prep/data-prep-lab/transforms/code/ingest_2_parquet/ray/test-data/input/data-processing-lib.zip`. Object-prefix is already an object, please choose a different object-prefix name.
mc: <ERROR> Failed to copy `/data/revital/prep/data-prep-lab/transforms/code/ingest_2_parquet/ray/test-data/input/application-java.zip`. <html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx</center>
</body>
</html>

Alternatively, nginx.ingress.kubernetes.io/proxy-body-size: 64m can be added to the minio ingress to avoid the error.

@revit13 revit13 marked this pull request as draft June 9, 2024 10:16
@revit13 revit13 marked this pull request as ready for review June 10, 2024 08:12
@revit13 revit13 marked this pull request as draft June 10, 2024 09:02
@revit13 revit13 marked this pull request as ready for review June 10, 2024 10:44
blublinsky
blublinsky approved these changes Jun 10, 2024
View reviewed changes
Copy link
Collaborator

@blublinsky blublinsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

revit13 and others added 2 commits June 12, 2024 00:22
@revit13
Adjust ingest_2_parquet workflow.
5e12e17 
Signed-off-by: Revital Sur <eres@il.ibm.com>
Co-authored-by: Boris Lublinsky <blublinsky@ibm.com>
@revit13
Minor fix to ingest_2_parquet_wf.py
70df84d 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@revit13 revit13 force-pushed the ingestwf branch 2 times, most recently from 44ca4ee to 5e12e17 Compare June 13, 2024 08:28
roytman
roytman reviewed Jun 13, 2024
View reviewed changes
Copy link
Member

@roytman roytman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove Lines 221 and 222 from the pipeline file

@revit13 revit13 merged commit 1917eb5 into IBM:dev Jun 13, 2024
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@roytman roytman roytman left review comments

@blublinsky blublinsky blublinsky approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@revit13 @roytman @blublinsky