Release CmdStan 2.34

[WardBrian]

Feature freeze (January 8th, 2024):

• Ensure all expiring language deprecations have been removed or given new removal dates.
• Create and merge version updating pull requests in Math/Stan/Cmdstan. These should be the last PRs accepted before the freeze.
• Create Math/Stan RC releases.
• Create Stanc3 RC binary.
• Create a release candidate tarball for x86. Make sure RC tarballs include stanc3 binaries.
• Check external links in docs (i.e. TBB docs link) - use https://github.com/tcort/markdown-link-check
• Create a release candidate feature/bugfix list (major features/bugfixes that need testing, link to new docs in Github)
• Run CmdStanR tests with the RC tarball.
• Run CmdStanPy tests with the RC tarball.
• Make a Discourse RC post.
• Post a tweet with a link to the Discourse RC post.

Release (January 16th, 2024):

• Create the Math Release notes.
• Create the Stan Release notes.
• Create the Cmdstan Release notes.
• Create the Stanc3 Release notes, include new deprecations and removals.
• Rebuild and publish docs for the new version.
• Check that docs for the previous release links correctly to the newest docs.
• Create the Math release.
• Create the Stan release.
• Create the Stanc3 release.
• Create x86 CmdStan tarballs (check version, check that the extracted folder is in the cmdstan-version format).
• Create non-x86 CmdStan tarballs.
• Run CmdStanR tests with the release tarball.
• Make a Stan blog release announcement post (thank the sponsors and all contributors, mention new devs).
• Link to the blog post in a Discourse thread.
• Make a Twitter announcement.

[mitzimorris]

are we creating tarballs for Google Colab?

[WardBrian]

Yes, @serban-nicusor-toptal makes one each release (usually not for RCs, I don’t believe)

[serban-nicusor-toptal]

I build it always, for RC also. I have some credits and wrote a script so it's fairly easy to do.

[serban-nicusor-toptal]

Good morning @WardBrian, are we still waiting for stan-dev/stan#3249 ?

[WardBrian]

Let's give it a few hours to see if we can get #1234

[serban-nicusor-toptal]

Sounds good, we can also postpone for the following days if #1234 is important.

[WardBrian]

@serban-nicusor-toptal you should be good to run the RCs as soon as the currently-running jenkins pipelines finish. I'll get started on the release notes and a draft of the blog post.

[WardBrian]

Here's my working draft of the release notes: https://gist.github.com/WardBrian/e3a6b534f6220df5439df05f08445584

And the discourse announcement: https://gist.github.com/WardBrian/1ba59bfd0037d3a721efac8934e4ad36

@rok-cesnovar is helping me with the discourse post, it should be ready to go up as soon as the tarballs are ready

[serban-nicusor-toptal]

Jenkins finished all develop jobs, I'll start shortly.

[serban-nicusor-toptal]

Release candidates are up!
https://github.com/stan-dev/math/releases/tag/v4.8.0-rc1
https://github.com/stan-dev/stan/releases/tag/v2.34.0-rc1
https://github.com/stan-dev/cmdstan/releases/tag/v2.34.0-rc1
https://github.com/stan-dev/stanc3/releases/tag/v2.34.0-rc1

We were a bit unfortunate with the Jenkins maintenance timing so we'll have to wait until it's online for the non-x64 binaries builds. I'll watch it and do it later when it's back.
At the same time, for colab builds we were using transfer.sh to take our the builds from the colab machine which went offline recently :( I'll need to figure out a different way to take out the build.

[WardBrian]

Running cmdstanpy tests here: https://github.com/stan-dev/cmdstanpy/actions/runs/7453476403

Discourse post is up: https://discourse.mc-stan.org/t/cmdstan-stan-2-34-release-candidate/33756

[WardBrian]

Cmdstanpy tests have a failure because the unit_e metric is now producing an extra line of output in the CSV file compared to previous versions, due to stan-dev/stan#3230.

I think this is better classified as a cmdstanpy bug

[serban-nicusor-toptal]

stanc3 and cmdstan binaries are now ready!
I've refactored the colab script so we keep using it the same way as before but with file.io -> https://github.com/stan-dev/ci-scripts/blob/master/release-scripts/build-conda-tgz.sh

[WardBrian]

CmdStanPy tests passed: https://github.com/stan-dev/cmdstanpy/actions/runs/7464824088

@rok-cesnovar @andrjohns can one of you queue a run of the cmdstanr tests?

[WardBrian]

I ran the CmdStanR tests: https://github.com/stan-dev/cmdstanr/actions/runs/7477267203/job/20349538347

13 failures, but I'm having a hard time telling if they're relevant or not. @rok-cesnovar @andrjohns @jgabry can you weigh in on those tests?

[rok-cesnovar]

Looking at it quickly, they look fine, mostly around model code that errors now. Will eliminate them before we do the actual release.

[WardBrian]

Unless something comes in over the long weekend, we should be good to release Tuesday

[serban-nicusor-toptal]

I will create the release branches later today, add the release notes then create a merge to develop for all repos. This so the pipelines will finish until tomorrow morning.

[serban-nicusor-toptal]

You can find all the open PRs targeting develop (master for stanc3) that update the version and the release-notes below:

• Release/v4.8.0: updating version numbers math#3003
• Release/v2.34.0: updating version numbers stan#3252
• Release/v2.34.0: updating version numbers #1236
• Release/v2.34.0 stanc3#1394

It would be great if someone else could take a look then we're free to merge them in.

[serban-nicusor-toptal]

Good morning, the release is out now!

https://github.com/stan-dev/math/releases/tag/v4.8.0
https://github.com/stan-dev/stan/releases/tag/v2.34.0
https://github.com/stan-dev/cmdstan/releases/tag/v2.34.0
https://github.com/stan-dev/stanc3/releases/tag/v2.34.0

Please find and review the docs PRs:
stan-dev/docs#701
stan-dev/stan-dev.github.io#197

[WardBrian]

Thanks @serban-nicusor-toptal!

I'll work with @rok-cesnovar on getting a blog post up

[rok-cesnovar]

Blog, Twitter/X and Discourse posts are up. Thanks @WardBrian for writing the blog post and RC announcement. I think we can close this one. I will get the constant tests to turn green by the end of the week. The failures are all false positives, so we are all good.

[serban-nicusor-toptal]

Everything is almost ready! I'll get home in a few hours and will do the last thing, merge develop into master for math, stan, cmdstan.

[WardBrian]

The last-minute doc changes have been built and are now live on the website

[serban-nicusor-toptal]

Good morning, looks like we have some failures in CI
https://jenkins.flatironinstitute.org/blue/organizations/jenkins/Stan%2FCmdStan/detail/PR-1237/1/pipeline/
https://jenkins.flatironinstitute.org/blue/organizations/jenkins/Stan%2FStan/detail/PR-3254/1/pipeline/
https://jenkins.flatironinstitute.org/blue/organizations/jenkins/Stan%2FMath/detail/PR-3004/1/pipeline

Do we release a patch version for stan-dev/math#3002 ?

[WardBrian]

stan-dev/math#3002 isn't mission-critical enough for a patch release on its own in my opinion, but if we find reason to do one we'd want to make sure it was in there alongside whatever else

I think those test failures are due to issues in the PRs, starting with the conflicts @andrjohns noticed in the Math PR.

Is there a reason why we merge the release branch, rather than develop? I'm trying to understand why something seems to break in these merges every release - nothing else is ever happening in Master, so why can't we merge cleanly?

[serban-nicusor-toptal]

The only difference between develop and the release branch, is that the release branch has the submodule hardcoded to the current release version. (for stan and cmdstan)
There were just a few conflicts now, we could try with develop also.

[WardBrian]

The CI pipeline ignores the submodules anyway, I believe.

I think the correct way to do this is develop -> master in math, then develop -> master in Stan, then develop -> master here. I also think it would be reasonable if after each release we just did a git push develop:master rather than these PRs, since the merge commits they introduce seems to be what breaks things the next time around

[serban-nicusor-toptal]

That sounds good, I'll make a note for next time.
For now, I'll close the PRs and replace master with develop for math, stan, cmdstan.
As we used to do before:

git checkout master
git reset --hard develop
git push origin master

[WardBrian]

assuming that works, I think that's what we actually want to happen.
Because there is a history of merges between the two branches it might not be happy

[serban-nicusor-toptal]

That's now done! Let's hope everything goes smoothly!

[WardBrian]

#1238 is probably worthy of a patch. We should wait if anything else comes up too.

[WardBrian]

stan-dev/stan#3255 and stan-dev/stan#3251 would also go into 2.34.1

Nothing in Math at the moment

[WardBrian]

Tentative 2.34.1 release notes:

CmdStan

• Fixed an indexing error which could lead to crashes
• Fixed an issue where output was swallowed by the log_prob method if an error occurred

Stan

• Fixed a Windows-specific issue with std::regex in JSON file validation
• Added a missing include of cstdint

[WardBrian]

@serban-nicusor-toptal Does monday work for you to run the patch release?

Currently it would only be for the stan and cmdstan repos - math and stanc3 should be re-used from .0. If something comes up in Math we will need to do a cherry-pick because some non-bugfix PRs have already been merged there.

[serban-nicusor-toptal]

Hey Brian, Monday is fine with me!

[WardBrian]

We're also waiting to see if stan-dev/math#3002 is merged. If so, it would be nice to cherry pick a math patch release.

[WardBrian]

@serban-nicusor-toptal the pr to watch is now stan-dev/math#3007

This is targetting a branch called staging/v4.8.1. If it is merged before release, we can create release/v4.8.1, add the release notes, and use that as the submodule for the release/v2.34.1 branches elsewhere.

We can then merge the release branch back into develop of math.

[WardBrian]

Ok, I just put up #1243 which will hopefully be merged today, release tomorrow.

Here are the notes:

Math (releasing from branch staging/v4.8.1):

- The distributions defined over cholesky factors now check that their arguments are in support (#3007)
- Fixed an issue with the `inv_wishart_cholesky_rng` function (#3007)

Stan (releasing from develop):

- Fixed a Windows-specific issue with std::regex in JSON file validation (#3251)
- Added a missing include of `cstdint` (#3255)

CmdStan (releasing from develop):

- Fixed an indexing error which could lead to crashes (#1239)
- Fixed an issue where output was swallowed by the `log_prob` method if an error occurred (#1235)
- Fixed an issue where floating point arguments would be echoed back to the user with insufficient precision (#1243)

[serban-nicusor-toptal]

Good morning, patch release is ready!

https://github.com/stan-dev/math/releases/tag/v4.8.1
https://github.com/stan-dev/stan/releases/tag/v2.34.1
https://github.com/stan-dev/cmdstan/releases/tag/v2.34.1

With PRs targeting develop

stan-dev/math#3011
stan-dev/stan#3257
#1244

After PRs are approved & merged, I will align develop branch with master and that should be the last bit!

[WardBrian]

With 2.34.1 out I think we can close this issue. Thank you to everyone who helped with this release!

[WardBrian]

I have made https://github.com/stan-dev/math/tree/staging/v4.8.2, which currently contains cherry-picks of stan-dev/math#3012 and stan-dev/math#3014

I believe it may be worth doing a .2 for these fixes. We can discuss when it would happen