Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release v4.5.0 #6538

Merged
merged 10 commits into from
Jul 25, 2024
Merged

release v4.5.0 #6538

merged 10 commits into from
Jul 25, 2024

Conversation

jameslamb
Copy link
Collaborator

@jameslamb jameslamb commented Jul 12, 2024

Release checklist:

Copied from #6439 with a few changes.

deadline

August 3, 2024 (ref: #6522)

before merge

Items to be addressed before merging:

after merge

code to update tags (click me)
NEW_VERSION="4.4.0"
git checkout master
git pull upstream master
git fetch upstream --tags
git tag -d stable
git push upstream :refs/tags/stable
git tag stable
git tag "v${NEW_VERSION}"
git push upstream stable "v${NEW_VERSION}"

after the release is created by Azure DevOps job

  • Add release notes from release-drafter to the automatically-created GitHub release, convert it from Draft to published.
  • Upload release to CRAN
  • Upload release to test PyPI
  • Upload release to PyPI.
code to publish to PyPI (summary)
mkdir /tmp/lgb-release
cd /tmp/lgb-release

NEW_VERSION="4.4.0"

# NOTE: requires a GitHub personal access token with "repo" scope
gh release download \
    --repo microsoft/LightGBM \
    --dir ./artifacts \
    --pattern 'lightgbm*-py3-*.whl' \
    --pattern "lightgbm-${NEW_VERSION}.tar.gz" \
    "v${NEW_VERSION}"

# NOTE: manually downloaded arm64 wheels from CI

# config docs: https://packaging.python.org/en/latest/specifications/pypirc/
twine upload \
    -r testpypi \
    ./artifacts/*

(gh is the GitHub CLI, see https://cli.github.com/manual/gh_release_download)

Then confirmed that installing the latest wheel works.

pip install -i https://test.pypi.org/simple/ "lightgbm==${NEW_VERSION}"
python ./examples/python-guide/logistic_regression.py

Then pushed them to real PyPI.

twine upload \
    ./artifacts/*
code to open that dev version PR (click me)
RELEASE_PR_NUMBER="6439"
RELEASE_VERSION=$(cat ./VERSION.txt)
DEV_VERSION="${RELEASE_VERSION}.99"

git checkout -b ci/dev-version

echo "${DEV_VERSION}" > ./VERSION.txt
sed \
    -i .bak \
    "s|${RELEASE_VERSION}|${DEV_VERSION}|g" \
    .appveyor.yml

sed \
    -i .bak \
    "s|version = \"${RELEASE_VERSION}\"|version = \"${DEV_VERSION}\"|g" \
    ./python-package/pyproject.toml

sed \
    -i .bak \
    's|mode\: release|mode\: unreleased|g' \
    ./R-package/pkgdown/_pkgdown.yml

docker run \
    --rm \
    -v $(pwd):/opt/LightGBM \
    -w /opt/LightGBM \
    ubuntu:22.04 \
    ./R-package/recreate-configure.sh

git add \
    ./.appveyor.yml \
    ./R-package/configure \
    ./R-package/pkgdown/_pkgdown.yml \
    ./VERSION.txt \
    ./python-package/pyproject.toml

commit_msg="bump development version to ${DEV_VERSION}"
git commit -m "${commit_msg}"
git push upstream ci/dev-version

gh pr create \
    --repo microsoft/LightGBM \
    --base 'master' \
    --label 'maintenance' \
    --title "${commit_msg}" \
    --body "now that v${RELEASE_VERSION} has been released: #${RELEASE_PR_NUMBER}"

Notes for Reviewers

I believe this should be v4.5.0 instead of v4.4.1 because of some small breaking changes:

See https://github.com/microsoft/LightGBM/releases.

@jameslamb
Copy link
Collaborator Author

jameslamb commented Jul 13, 2024

/gha run r-valgrind

Workflow R valgrind tests has been triggered! 🚀
https://github.com/microsoft/LightGBM/actions/runs/9916742190

Status: success ✔️.

@@ -30,7 +30,7 @@ maintainers = [
name = "lightgbm"
readme = "README.rst"
requires-python = ">=3.7"
version = "4.4.0.99"
version = "4.4.0"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it was a typo.

Suggested change
version = "4.4.0"
version = "4.5.0"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah!! yes, thank you. Maybe in the future we should have a script that automates these changes, like my colleagues do in the RAPIDS repos: https://github.com/rapidsai/cuml/blob/branch-24.08/ci/release/update-version.sh

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great idea!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very sorry for resolving this without answering you @borchero , that was unintentional.

I'm -1 on using tools like setuptools-scm or versioneer here. In my experience, they introduce much more complexity than they reduce, at least for projects like LightGBM with a linear sequence of releases (no long-term-support branches, backporting, etc.).

For example, they can be the source of hard-to-debug differences between local and CI builds, unless you're diligent about running git fetch upstream --tags or similar regularly.

You might find some of the other discussion in https://discuss.python.org/t/please-make-package-version-go-away/58501 interesting (although I don't agree with the original post... I believe we should continue to support lightgbm.__version__).

@StrikerRUS
Copy link
Collaborator

I believe this should be v4.5.0 instead of v4.4.0 because of some small breaking changes:

I guess you meant v4.4.1. Agree with your proposal!

@jameslamb
Copy link
Collaborator Author

Yes I meant "instead of v4.4.1", just edited it. Sorry for all the typos 😅

@StrikerRUS
Copy link
Collaborator

StrikerRUS commented Jul 13, 2024

Seems that Azure Pipelines builds are constantly failing for the master branch.

image

For example, the latest one link: https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=16597&view=results

Failing jobs are:

  • macOS cpp_tests
[100%] Built target testlightgbm
/Applications/Xcode_13.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/c++/v1/locale:1417:26: runtime error: member call on address 0x612000003ad8 which does not point to an object of type 'std::streambuf'
0x612000003ad8: note: object is of type 'std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char> >'
 01 00 00 00  30 40 cf 0c 01 00 00 00  30 ba 41 5b f8 7f 00 00  00 00 00 00 00 00 00 00  00 00 00 00
              ^~~~~~~~~~~~~~~~~~~~~~~
              vptr for 'std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char> >'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /Applications/Xcode_13.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/c++/v1/locale:1417:26 in 
/Users/runner/work/1/s/.ci/test.sh: line 63:  8016 Abort trap: 6           ./testlightgbm

##[error]Bash exited with code '1'.
  • Windows bdist
2024-07-12T18:51:57.5021734Z Downloading pydistcheck-0.7.1-py3-none-any.whl (19 kB)
2024-07-12T18:51:58.6643657Z Installing collected packages: pydistcheck
2024-07-12T18:51:58.8103421Z Successfully installed pydistcheck-0.7.1
2024-07-12T18:51:59.8819463Z Traceback (most recent call last):
2024-07-12T18:51:59.8820899Z   File "<frozen runpy>", line 198, in _run_module_as_main
2024-07-12T18:51:59.8821855Z   File "<frozen runpy>", line 88, in _run_code
2024-07-12T18:51:59.8828650Z   File "C:\Miniconda\envs\test-env\Scripts\pydistcheck.exe\__main__.py", line 7, in <module>
2024-07-12T18:51:59.8829712Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\click\core.py", line 1157, in __call__
2024-07-12T18:51:59.8841284Z     return self.main(*args, **kwargs)
2024-07-12T18:51:59.8848274Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8848970Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\click\core.py", line 1078, in main
2024-07-12T18:51:59.8849427Z     rv = self.invoke(ctx)
2024-07-12T18:51:59.8850949Z          ^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8851311Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\click\core.py", line 1434, in invoke
2024-07-12T18:51:59.8857514Z     return ctx.invoke(self.callback, **ctx.params)
2024-07-12T18:51:59.8858902Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8861213Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\click\core.py", line 783, in invoke
2024-07-12T18:51:59.8863013Z     return __callback(*args, **kwargs)
2024-07-12T18:51:59.8863771Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8865099Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\pydistcheck\cli.py", line 256, in check
2024-07-12T18:51:59.8866530Z     errors += this_check(distro_summary=summary)
2024-07-12T18:51:59.8867326Z               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8868843Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\pydistcheck\checks.py", line 64, in __call__
2024-07-12T18:51:59.8870013Z     has_debug_symbols, cmd_str = _file_has_debug_symbols(
2024-07-12T18:51:59.8870823Z                                  ^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8871605Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\pydistcheck\shared_lib_utils.py", line 73, in _file_has_debug_symbols
2024-07-12T18:51:59.8877657Z     has_debug_symbols, cmd_str = _look_for_debug_symbols(lib_file=file_absolute_path)
2024-07-12T18:51:59.8878095Z                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8879249Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\pydistcheck\shared_lib_utils.py", line 48, in _look_for_debug_symbols
2024-07-12T18:51:59.8879935Z     stdout = _run_command(args=[*cmd_args, lib_file])
2024-07-12T18:51:59.8880333Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8880723Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\pydistcheck\shared_lib_utils.py", line 16, in _run_command
2024-07-12T18:51:59.8881941Z     stdout = subprocess.run(args, capture_output=True, check=True).stdout
2024-07-12T18:51:59.8882309Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8883036Z   File "C:\Miniconda\envs\test-env\Lib\subprocess.py", line 548, in run
2024-07-12T18:51:59.8896405Z     with Popen(*popenargs, **kwargs) as process:
2024-07-12T18:51:59.8897987Z          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8904669Z   File "C:\Miniconda\envs\test-env\Lib\subprocess.py", line 1026, in __init__
2024-07-12T18:51:59.8913959Z     self._execute_child(args, executable, preexec_fn, close_fds,
2024-07-12T18:51:59.8915143Z   File "C:\Miniconda\envs\test-env\Lib\subprocess.py", line 1538, in _execute_child
2024-07-12T18:51:59.8929852Z     hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
2024-07-12T18:51:59.8930741Z                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8933401Z ==================== running pydistcheck ====================
2024-07-12T18:51:59.8934510Z OSError: [WinError 122] The data area passed to a system call is too small

Should these fails be a blockers for this release?

@jameslamb
Copy link
Collaborator Author

Add release branch to RTD versions, trigger a new build, check docs

I'll check again once we're done merging changes here.

@jameslamb
Copy link
Collaborator Author

constantly failing on the master branch

Oh wow, yes I hadn't realized that most master builds for the last 2 weeks have failed with exactly those two issues! I just clicked through https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build?definitionId=1&_a=summary&repositoryFilter=1&branchFilter=2%2C2%2C2%2C2%2C2 to see that.

I do think we should try to figure this out for the release, but let's not block releasing the R package on them. If we get to, say, July 30 and have still not fixed it, I think we should at least do a release of the R package to CRAN, to avoid it being archived there.

Could you document these CI failures in a separate issue, and we could discuss further there?

I'm linking this relevant thing I found, that suggests that one issue could be a very long filepath on Windows: sarugaku/shellingham#8. Maybe that could explain the jobs only failing on master... maybe something about that changes the filepaths used by Azure DevOps or something? I'm not sure.

@StrikerRUS
Copy link
Collaborator

Could you document these CI failures in a separate issue, and we could discuss further there?

Sure, done!

If we get to, say, July 30 and have still not fixed it, I think we should at least do a release of the R package to CRAN, to avoid it being archived there.

Absolutely agree!

@jameslamb
Copy link
Collaborator Author

jameslamb commented Jul 22, 2024

/gha run r-valgrind

Workflow R valgrind tests has been triggered! 🚀
https://github.com/microsoft/LightGBM/actions/runs/10033742645

Status: success ✔️.

@jameslamb jameslamb changed the title WIP: release v4.5.0 release v4.5.0 Jul 22, 2024
@jameslamb jameslamb marked this pull request as ready for review July 22, 2024 02:43
@jameslamb
Copy link
Collaborator Author

I received the following email from CRAN (Prof Brian Ripley) 3 days ago.

subject: CRAN packages with g++ 14.x warnings

[lightgbm was] given a month's notice with a deadline of Aug 3. As the CRAN
summer shutdown starts on Aug 1, please submit a corrected update no
later than July 31.

I believe we should release the current state of this PR as v4.5.0 as soon as possible. I want to allow enough time for CRAN to reject the package and for us to put up another submission prior to their summer shutdown.

I do not think there are any other open PRs or issues that absolutely MUST be addressed in this release.

@guolinke @borchero @jmoralez @shiyu1994 @StrikerRUS could you please review, and approve if you agree?

I'd like to get at least 2 approvals (and at least 1 from either @shiyu1994 or @guolinke ) before releasing.

Copy link
Collaborator

@borchero borchero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go 🚀

@StrikerRUS
Copy link
Collaborator

StrikerRUS commented Jul 25, 2024

I think we should stop merging any non-trivial PRs before CRAN response. It should help to avoid situation similar to the following one: #4930 in case we will need 4.5.1.

@jameslamb
Copy link
Collaborator Author

Agreed!

I'll merge this so we can start the process of releasing. And I'll post follow-up comments here as I've done in previous releases.

Thanks for the reviews and help everyone!

@jameslamb jameslamb merged commit 3f7e608 into master Jul 25, 2024
45 checks passed
@jameslamb jameslamb deleted the release/v4.5.0 branch July 25, 2024 14:51
@jameslamb
Copy link
Collaborator Author

Ran the following to create the v4.5.0 tag and update the stable tag.

git checkout master
git pull upstream master
git fetch upstream --tags
git tag -d stable
git push upstream :refs/tags/stable
git tag stable
git tag v4.5.0
git push upstream stable v4.5.0

(NOTE: I alias this repo to upstream and my fork to origin in my git settings)

https://github.com/microsoft/LightGBM/tags

image

That triggered an Azure DevOps build which should create the release automatically: https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=16722&view=results

This takes around 90 minutes (because of the QEMU CI job).

I'll check back in a few hours and continue the release then.

@jameslamb
Copy link
Collaborator Author

One test job on the release failed, with what looked like a conda network error. Restarted it.

@jameslamb
Copy link
Collaborator Author

Remove the release branch release/v{n}.{n}.{n} from RTD versions

done

Add release notes from release-drafter to the automatically-created GitHub release, convert it from Draft to published.

done!

The release is now live: https://github.com/microsoft/LightGBM/releases/tag/v4.5.0

@jameslamb
Copy link
Collaborator Author

Submitted to CRAN with this additional note:

This release fixes the C++20 issues noted in the "gcc-12" checks at https://cran.r-project.org/web/checks/check_results_lightgbm.html. It is being submitted in response to a request from Prof. Brian Ripley to submit before July 31st.

image

@StrikerRUS
Copy link
Collaborator

Homebrew has been automatically updated (and bottles have been built): Homebrew/homebrew-core#178449.

https://formulae.brew.sh/formula/lightgbm#default:

image

@StrikerRUS
Copy link
Collaborator

Can't find ..macosx_..._arm64.whl macOS wheel for M chips in attached artifacts 🤔
Do we build them?
Seems that we did it for 4.4.0:
image

@StrikerRUS
Copy link
Collaborator

Previously failing CRAN gcc check is now OK! 🎉

https://cran.r-project.org/web/checks/check_results_lightgbm.html
image

@jameslamb
Copy link
Collaborator Author

Add new tag to RTD versions and trigger a new build.

This is done.

Screenshot 2024-07-26 at 8 40 10 AM

@jameslamb
Copy link
Collaborator Author

Can't find ..macosx_..._arm64.whl macOS wheel for M chips in attached artifacts 🤔

I haven't figured out a good way to make that automatic yet. They're built on GitHub Actions because (at least as of #6391), Azure DevOps didn't have an arm64 runner image.

So they have to be manually added to the release.

They can be found by clicking "🏠 Summary" on a run of the Python-Package workflow, and scrolling to the bottom. Here's the run for v4.5.0: https://github.com/microsoft/LightGBM/actions/runs/10096384964

Screenshot 2024-07-26 at 8 48 18 AM

I just manually downloaded that artifact, unzipped it, and uploaded the file to https://github.com/microsoft/LightGBM/releases/tag/v4.5.0.

@jameslamb
Copy link
Collaborator Author

Upload release to test PyPI

This is done: https://test.pypi.org/project/lightgbm/4.5.0/

And installing it (on my M2 Mac) worked fine.

pip install \
    --no-deps \
    -i https://test.pypi.org/simple/ \
    lightgbm==4.5.0

python -c "import lightgbm"
python examples/python-guide/sklearn_example.py

@StrikerRUS
Copy link
Collaborator

@jameslamb

They're built on GitHub Actions because (at least as of #6391), Azure DevOps didn't have an arm64 runner image.

Ah, I see! Thanks for the explanation!

* Just checked - still no arm64 macOS runner at Azure 😞

@jameslamb
Copy link
Collaborator Author

Upload release to PyPI.

this is done: https://pypi.org/project/lightgbm/4.5.0/#files

@jameslamb
Copy link
Collaborator Author

put up another PR incrementing development version here in GitHub

done: #6575

@StrikerRUS
Copy link
Collaborator

I think we should add one more release TODO item: "update conda-forge version" since @jameslamb is now official lightgbm feedstock's maintainer.

conda-forge/lightgbm-feedstock#60

@jameslamb
Copy link
Collaborator Author

Agreed! I just added it here so we'll remember it when this PR's description is copied into the next release PR.

@jameslamb
Copy link
Collaborator Author

Update cran-comments.md when new release is accepted on CRAN.

put up a PR: #6578

@jameslamb
Copy link
Collaborator Author

Upload release to NuGet

done: https://www.nuget.org/packages/LightGBM/4.5.0

That should be fully published some time in the next few hours.

image

@jameslamb
Copy link
Collaborator Author

With that, I believe this release is complete! Thanks for the help, everyone.

Really great having you back to help @StrikerRUS , the throughput in this project was great over the last month. This release covers just 6 weeks of changes, but included 49 commits from 16 different contributors (v4.4.0...v4.5.0). I'm very happy about that!

@StrikerRUS
Copy link
Collaborator

@jameslamb Thank you very much for your hard work on the release! 💪

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants