Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to exception processing doc #614

Closed
wants to merge 8 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 18 additions & 10 deletions RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
## Overview
Release are created from the main repository branch using the version
numbers, including an intermediate version suffix,
defined in `.make.versions`.
defined in `.make.versions`
The following points are important:

1. In general, common a version number is used for all published pypi wheels and docker images.
Expand All @@ -20,29 +20,35 @@ allows intermediate publishing from the main branch using version X.Y.Z.dev\<N\>
## Cutting the release
Creating the release involves

1. Edit the `release-notes.md` to list major/minor changes
1. Creating a release branch and tag and updating the main branch versions.
1. Creating a github release from the release branch and tag.
1. Building and publishing pypi library wheels and docker registry image.

Each is discussed below.

### Creating release branch and tag
### Editing release-notes.md
Make a dummy release on github (see below) to get a listing of all commits.
Use this to come up with the items.
Commit this to the main branch so it is ready for including in the release branch.

### Creating release branch
The `scripts/release-branch.sh` is currently run manually to create the branch and tags as follows:

1. Creates the `releases/vX.Y.Z` from the main branch where `X.Y.Z` are defined in .make.versions
1. Creates the `vX.Y.Z` branch for PR'ing back into the `releases/vX.Y.Z` branch.
1. In the new `vX.Y.Z` branch
1. Nulls out the version suffix in the new branch's `.make.version` file.
1. Applies the unsuffixed versions to the artifacts published from the repo using `make set-versions`..
1. Commits and pushes branch and tag
1. Commits and pushes branch
1. Creates the `pending-version-change/vX.Y.Z` branch for PR'ing back into the main branch.
1. In the `pending-version-change/vX.Y.Z` branch
1. Increments the minor version (i.e. Z+1) and resets the suffix to `dev0` in `.make.versions`.
1. Commits and pushes branch

To double-check the version that will be published from the release,
```
git checkout releasing/vX.Y.Z
git checkout vX.Y.Z
make show-version
```
This will print for example, 1.2.3.
Expand All @@ -60,13 +66,15 @@ After running the script, you should

### Github release
After running the `release-branch.sh` script, to create tag `vX.Y.Z` and branch `releases/vX.Y.Z`
and PRing/merging `vX.Y.Z` into `releases/vX.Y.Z`.
1. Go to the [releases page](https://github.com/IBM/data-prep-kit/releases).
2. Select `Draft a new release`
3. Select `Choose a tag -> vX.Y.Z`
4. Press `Generate release notes`
5. Add a title (e.g., Release X.Y.Z)
6. Add any additional relese notes.
7. Press `Publish release`
1. Select `Draft a new release`
1. Select target branch `releases/vX.Y.Z`
1. Select `Choose a tag`, type in vX.Y.Z, click `Create tag`
1. Press `Generate release notes`
1. Add a title (e.g., Release X.Y.Z)
1. Add any additional relese notes.
1. Press `Publish release`

### Publishing wheels and images
After creating the release branch and tag using the `scripts/release-branch.sh` script:
Expand Down
31 changes: 31 additions & 0 deletions release-notes.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,36 @@
# Data Prep Kit Release notes

## Release 0.2.1 - 9/24/2024

### General
1. Bug fixes across the repo
1. Added AI Alliance RAG demo, tutorials and notebooks and tips for running on google colab
1. Added new transforms and single package for transforms published to pypi
1. Improved CI/CD with targeted workflow triggered on specific changes to specific modules
1. New enhancements for cutting a release


### data-prep-toolkit libraries (python, ray, spark)

1. Restructure the repository to distinguish/separate runtime libraries
1. Split data-processing-lib/ray into python and ray
1. Spark runtime
1. Updated pyarrow version
1. Define required transform() method as abstract to AbstractTableTransform
1. Enables configuration of makefile to use src or pypi for data-prep-kit library dependencies


### KFP Workloads

1. Add a configurable timeout before destroying the deployed Ray cluster.

### Transforms

1. Added 7 new transdforms including: language identification, profiler, repo level ordering, doc quality, pdf2parquet, HTML2Parquet and PII Transform
1. Added ededup python implementation and incremental ededup
1. Added fuzzy floating point comparison


## Release 0.2.0 - 6/27/2024

### General
Expand Down