Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some new thoughts on cutting a release, especially scripts/release.sh #309

Merged
merged 37 commits into from
Jun 24, 2024
Merged

Some new thoughts on cutting a release, especially scripts/release.sh #309

merged 37 commits into from
Jun 24, 2024

Conversation

daw3rd
Copy link
Member

@daw3rd daw3rd commented Jun 19, 2024 •
edited
Loading

Why are these changes needed?

We need to automate the process of version management and release cutting. This is just a start with some initial thoughts based on recent discussion. Initial changes include

  1. Change .make.versions to define a common repo-wide version for all libraries and images.
  2. Add scripts/release.sh which tries to outline the steps needed to changes versions, create and tag a branch.
  3. Makefile new target to show-version based on .make.versions content
  4. Fixed a lot of set-versions problems
  5. Set all versions back to 0.2.0.dev6

Relies on make build publish being enough to actually publish the release to pypi and quay.io

Also see PR #297

Related issue number (if any).

#294

@daw3rd
Some new thoughts on cutting a release, especially scripts/release.sh
3dd1c9c 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd daw3rd marked this pull request as draft June 19, 2024 20:23
revit13
revit13 reviewed Jun 20, 2024
View reviewed changes
transforms/universal/profiler/kfp_ray/Makefile Outdated
@@ -29,8 +29,6 @@ image::

load-image::

set-versions: workflow-reconcile-requirements
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rebase, this should be fixed now in dev branch. thanks

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@roytman
Copy link
Member

roytman commented Jun 20, 2024

@daw3rd please do not do the release automation before we agree on the process. Some of the functionality can be done by GitHub actions

daw3rd added 18 commits June 20, 2024 11:17
@daw3rd
Minor updates to release.sh script
04fb1dc 
Signed-off-by: David Wood <dawood@us.ibm.com>

n#
@daw3rd
Update repo.md for 2 make users and add workflow test of global make …
98a41b5 
…rules

Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
Update release.sh script for testing
2d006a5 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
more testing or release.sh script
cc6b3aa 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
More release.sh
4fc4f9c 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
Revert to dev versions
8de2c49 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
release.sh
e49cbaa 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
release.sh
db921b9 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
release.sh
1451277 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
release.sh
c276b71 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
release.sh
a026584 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
release.sh
4b6f41b 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
release.sh
15317ef 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
updated repo.md on makefiles
7d5ca76 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
Merge branch 'dev' into releasing
679d637
@daw3rd
add publish target to superworkflows Makefile
c85c68a 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
More release work
2a7bcb9 
Added docs in .make.versions
Add RELEASE.md.
Fix checking for pypi credentials

Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
add publish target to kind/Makefile
12f315b 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
Copy link
Member Author

daw3rd commented Jun 20, 2024

@daw3rd please do not do the release automation before we agree on the process. Some of the functionality can be done by GitHub actions

I didn't think I was, except that I added the make tests for the test, clean, publish, build, set-versions top-level rules and some Makefiles needed to implement these

@daw3rd daw3rd mentioned this pull request Jun 20, 2024
Closed
@daw3rd
install ../python for spark transforms in .make.defaults
6c0d4ab 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
fix kfp set versions, and noop versions
f6bd444 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
Fix versioning in .make* and typo in noop/ray/pyproject.tmo
103f559 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
Merge branch 'dev' into releasing
7b3be84
@daw3rd
comments and dont created empty pyproject.toml in .make.defaults
7f7a264 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
update release.sh to apply final version changes
52e4eb7 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
Version updates, version setting fixes, noop spark Dockerfile,
6211b50 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
Merge branch 'dev' into releasing
af8ce1c
@daw3rd
set-versions inputcode2parquet'
51fe74c 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
restore INGEST_TO_PARQUET_VERSION removed in recent commit"
3db00f7 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
updated repo.md to point to include link to transform conventions
caa42b4 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
fix transform link added to repo.md in last commit
b43683f 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd daw3rd mentioned this pull request Jun 21, 2024
Closed
2 tasks
@daw3rd
Merge branch 'dev' into releasing
fcc6136
@daw3rd
fixed code2parquet versions from last merge and uninstall libs in bet…
67d3357 
…ter order in .make.defaults

Signed-off-by: David Wood <dawood@us.ibm.com>
revit13
revit13 reviewed Jun 24, 2024
View reviewed changes
transforms/universal/profiler/ray/Makefile Outdated
set-versions:
$(MAKE) TOML_VERSION=$(DOCKER_IMAGE_VERSION) .defaults.update-toml
$(MAKE) TRANSFORM_PYTHON_VERSION=$(DOCKER_IMAGE_VERSION) .transforms.set-versions
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think that's working , TRANSFORM_PYTHON_VERSION here is the value of PROFILER_RAY_VERSION while it should be PROFILER_PYTHON_VERSION. I tried to fix that in #323

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are correct. To be more clear i set to ...=not-used when there is a no python versions (profiler, e/fdedup, doc_id). code_quality, malware and proglang_select did need the change.

HOWEVER, in the end since all version numbers are now the same, it was working ok :)

@roytman
Copy link
Member

roytman commented Jun 24, 2024

We should deploy 2 kfp-data-processing images one for KFPv1 and another for KFPv2. Will try to update the script to proceed with that.

revit13
revit13 reviewed Jun 24, 2024
View reviewed changes
.make.versions Outdated
CODE_QUALITY_RAY_VERSION=0.4.0$(RELEASE_VERSION_SUFFIX)
CODE_QUALITY_RAY_VERSION=$(DPK_VERSION)
CODE_QUALITY_PYTHON_VERSION=$(DPK_VERSION)
CODE_QUALITY_RAY_VERSION=$(DPK_VERSION)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It appears twice.

@daw3rd daw3rd marked this pull request as ready for review June 24, 2024 19:20
@daw3rd
.make.versions changes - remove redancy and unused macros, and define…
87a7549 
… section for the repo's version dependencies.

Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd daw3rd merged commit 5aace3a into dev Jun 24, 2024
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@revit13 revit13 revit13 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@daw3rd @roytman @revit13