Add github action to push docker images with latest tag. #370

revit13 · 2024-07-02T10:59:21Z

Why are these changes needed?

This PR automates the process of pushing Docker images with latest tag upon each merge to the development branch. It accomplishes this by storing the images generated during the execution of the test-image target using the https://github.com/actions/upload-artifact action and subsequently downloading and pushing them to the repository following the completion of testing.
It also changes ray docker images to have a base image and thus avoids the need to call update_dockerfile.sh script to update the ray version in set-versions.

the pushed images rae:

quay.io/dataprep1/data-prep-kit/proglang_select-ray:latest
quay.io/dataprep1/data-prep-kit/proglang_select-python:latest
quay.io/dataprep1/data-prep-kit/malware-ray:latest
quay.io/dataprep1/data-prep-kit/malware-python:latest
quay.io/dataprep1/data-prep-kit/code_quality-ray:latest
quay.io/dataprep1/data-prep-kit/code_quality-python:latest
quay.io/dataprep1/data-prep-kit/code2parquet-ray:latest
quay.io/dataprep1/data-prep-kit/code2parquet-python:latest
quay.io/dataprep1/data-prep-kit/lang_id-ray::latest
quay.io/dataprep1/data-prep-kit/tokenization-ray:latest
quay.io/dataprep1/data-prep-kit/tokenization-python:latest
quay.io/dataprep1/data-prep-kit/resize-ray:latest
quay.io/dataprep1/data-prep-kit/resize-python:latest
quay.io/dataprep1/data-prep-kit/profiler-ray:latest
quay.io/dataprep1/data-prep-kit/noop-spark:latest
quay.io/dataprep1/data-prep-kit/noop-ray:latest
quay.io/dataprep1/data-prep-kit/noop-python:latest
quay.io/dataprep1/data-prep-kit/filter-spark:latest
quay.io/dataprep1/data-prep-kit/filter-ray:latest
quay.io/dataprep1/data-prep-kit/filter-python:latest
quay.io/dataprep1/data-prep-kit/fdedup-ray:latest
quay.io/dataprep1/data-prep-kit/ededup-ray:latest
quay.io/dataprep1/data-prep-kit/doc_id-spark:latest
quay.io/dataprep1/data-prep-kit/doc_id-ray:latest
quay.io/dataprep1/data-prep-kit/data-prep-kit-spark-3.5.1:latest
quay.io/dataprep1/data-prep-kit/lang_id-python:latest
quay.io/dataprep1/data-prep-kit/lang_id-ray:latest
quay.io/dataprep1/data-prep-kit/kfp-data-processing:latest

Related issue number (if any).

#176

Signed-off-by: Revital Sur <eres@il.ibm.com>

roytman · 2024-07-07T10:20:20Z

scripts/transforms/update_workflow_tags.sh

  cat << EOF
-	"Incorrect number of parameters provided. The required parameters are versions_file and pipeline_path. 
+	$1 $2 


should we add more explanation here? instead just $1 and $2

Thanks I removed the redundant printing...

roytman · 2024-07-07T10:22:19Z

tools/ingest2parquet/Dockerfile

-FROM docker.io/rayproject/ray:2.24.0-py310
+ARG BASE_IMAGE=docker.io/rayproject/ray:2.24.0-py310
+
+FROM ${BASE_IMAGE}


wow, that's very suitable.

.make.defaults

roytman

LGTM

daw3rd

@revit13 and I spoke and agreed we will try and simplify things by always publishing the "latest" docker images if a) it is a merge into dev and b) the make test-image passes.
Publishing will be done after make test-image only for dev merges and the tag in this case will be latest. The "releases" will be handled manually (for now?) and publishing will be done out of releases/vX.Y.Z branch. But perhaps this can somehow be automated as well? Maybe not using latest tag?

Signed-off-by: Revital Sur <eres@il.ibm.com>

revit13 · 2024-07-09T08:50:05Z

@revit13 and I spoke and agreed we will try and simplify things by always publishing the "latest" docker images if a) it is a merge into dev and b) the make test-image passes. Publishing will be done after make test-image only for dev merges and the tag in this case will be latest. The "releases" will be handled manually (for now?) and publishing will be done out of releases/vX.Y.Z branch. But perhaps this can somehow be automated as well? Maybe not using latest tag?

@daw3rd I changed the action to push as described above. Please let me know if thats OK. I also added a condition to push the images upon creating a new tag which should cover publishing for release. Thanks

Signed-off-by: Revital Sur <eres@il.ibm.com>

transforms/code/code2parquet/ray/Dockerfile

daw3rd · 2024-07-09T17:44:17Z

.make.defaults

+.PHONY: .defaults.docker-save-image
+.defaults.docker-save-image:
+	@# Help: Save docker image as tar file to ${ARTIFACTS_DIR} directory.
+	mkdir -p ${ARTIFACTS_DIR}/


Now that we're trying to add checks, can you add a check for DOCKER_IMAGE_NAME. Maybe this (and others) goes at the top of this file though instead of at each referencing target? I guess its the "global" ones like this that could go at the top of this file?

Checking DOCKER_IMAGE_NAME in the top of this file results in an error as its is set in the Makefiles that generate Docker images and include .make.defauls, for example in .make.transforms. So I added it to image and publish targets.

.make.versions

daw3rd

lgtm

revit13 added 5 commits July 2, 2024 01:56

Add github action to push docker images with latest tag.

73fb10a

Signed-off-by: Revital Sur <eres@il.ibm.com>

Merge branch 'dev' into github13

e375e43

Debug workflow.

fcf2975

Signed-off-by: Revital Sur <eres@il.ibm.com>

fix.

3a1a4b4

Signed-off-by: Revital Sur <eres@il.ibm.com>

Handle latest tag.

47d778e

Signed-off-by: Revital Sur <eres@il.ibm.com>

revit13 marked this pull request as draft July 2, 2024 10:59

revit13 force-pushed the github13 branch from 83aa9b4 to 73fb10a Compare July 2, 2024 11:11

revit13 added 6 commits July 2, 2024 15:47

Fixes after testing.

c3fbbee

Signed-off-by: Revital Sur <eres@il.ibm.com>

Merge branch 'dev' into github13

a9d7209

Fixes after testing.

421eaf3

Signed-off-by: Revital Sur <eres@il.ibm.com>

Add base image to ray images.

f466384

Signed-off-by: Revital Sur <eres@il.ibm.com>

Fix final stage.

4a5dc5e

Signed-off-by: Revital Sur <eres@il.ibm.com>

Minor change.

d1842a6

Signed-off-by: Revital Sur <eres@il.ibm.com>

revit13 force-pushed the github13 branch from aeeea03 to c3fbbee Compare July 3, 2024 10:20

revit13 added 6 commits July 3, 2024 14:38

Test publish phase.

58517f3

Signed-off-by: Revital Sur <eres@il.ibm.com>

Trgger push only on merge to dev.

64a806a

Signed-off-by: Revital Sur <eres@il.ibm.com>

Test push images.

9e69e11

Signed-off-by: Revital Sur <eres@il.ibm.com>

Minor change.

5defb5a

Signed-off-by: Revital Sur <eres@il.ibm.com>

Fixes after testsing.

a83fb7f

Signed-off-by: Revital Sur <eres@il.ibm.com>

Minor fix.

259bc4e

Signed-off-by: Revital Sur <eres@il.ibm.com>

revit13 force-pushed the github13 branch from bdd09e7 to f7de624 Compare July 4, 2024 10:40

Address review comments.

2631153

Signed-off-by: Revital Sur <eres@il.ibm.com>

revit13 force-pushed the github13 branch 8 times, most recently from 6fa57c5 to 4a5dc5e Compare July 4, 2024 21:27

revit13 marked this pull request as ready for review July 5, 2024 03:33

revit13 force-pushed the github13 branch 2 times, most recently from 122a094 to 9e69e11 Compare July 6, 2024 18:54

revit13 requested review from daw3rd, Mohammad-nassar10 and roytman July 7, 2024 10:07

roytman reviewed Jul 7, 2024

View reviewed changes

roytman approved these changes Jul 8, 2024

View reviewed changes

daw3rd requested changes Jul 8, 2024

View reviewed changes

revit13 added 2 commits July 9, 2024 08:32

Address review comments.

721b043

Signed-off-by: Revital Sur <eres@il.ibm.com>

Minor fix.

94fc34c

Signed-off-by: Revital Sur <eres@il.ibm.com>

revit13 force-pushed the github13 branch 2 times, most recently from a0601c0 to 721b043 Compare July 9, 2024 06:58

Minor fix.

37d9fb2

Signed-off-by: Revital Sur <eres@il.ibm.com>

Add a check for DOCKER_IMAGE_NAME

d2172f0

Signed-off-by: Revital Sur <eres@il.ibm.com>

daw3rd requested changes Jul 9, 2024

View reviewed changes

daw3rd approved these changes Jul 10, 2024

View reviewed changes

daw3rd merged commit 2042ced into IBM:dev Jul 10, 2024
21 checks passed

roytman mentioned this pull request Aug 20, 2024

[Bug] Updating transformer versions requires deployment relevant docker images #225

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add github action to push docker images with latest tag. #370

Add github action to push docker images with latest tag. #370

revit13 commented Jul 2, 2024 •

edited

Loading

roytman Jul 7, 2024

revit13 Jul 7, 2024

roytman Jul 7, 2024

roytman left a comment

daw3rd left a comment

revit13 commented Jul 9, 2024

daw3rd Jul 9, 2024

revit13 Jul 9, 2024

daw3rd left a comment

Add github action to push docker images with latest tag. #370

Add github action to push docker images with latest tag. #370

Conversation

revit13 commented Jul 2, 2024 • edited Loading

Why are these changes needed?

Related issue number (if any).

roytman Jul 7, 2024

Choose a reason for hiding this comment

revit13 Jul 7, 2024

Choose a reason for hiding this comment

roytman Jul 7, 2024

Choose a reason for hiding this comment

roytman left a comment

Choose a reason for hiding this comment

daw3rd left a comment

Choose a reason for hiding this comment

revit13 commented Jul 9, 2024

daw3rd Jul 9, 2024

Choose a reason for hiding this comment

revit13 Jul 9, 2024

Choose a reason for hiding this comment

daw3rd left a comment

Choose a reason for hiding this comment

revit13 commented Jul 2, 2024 •

edited

Loading