Release TestNg more frequently

[martinaldrin]

Hi,

I see a need of release TestNg more frequently, Today it seems to be 6-9 month between the releases. And the number of features included in TestNg is kind of big, In our project we do releases for every commit, which allow fast feed back.
And in our project we have restriction on using forks.

The risk I see now is if we find a bug in a release we need to wait very long time until next release, and then there is a risk that something else have changed.

Maybe a release every quarter wouldn't create to much extra job for your team.

What do you think?

[krmahadevan]

@martinaldrin - The frequency of a release is usually determined by what is getting shipped in a particular release. Today the rate at which issues are getting fixed is pretty slow, for lack of enough folks to help out. We would love to do more frequent releases, but we also need to ensure that we dont end up doing a release which contains a very small set of bug fixes/features.

What is the minimum number of issues which would warrant a release, this can be debated. But this primarily one of the main reasons behind elongated timelines between releases.

[askfor]

I think there could be some bug-fix releases between feature releases, 7.4.x for example.

[cbeust]

@krmahadevan I think it's perfectly fine to ship minor version releases more frequently even if they only contain a handful of bug fixes.

The way I set things up in the Maven repo and github should make it trivial for you and @juherr to deploy to Maven at any time, let's try to do that more frequently.

[martinaldrin]

That would be very helpful, and I think that will benefit both developers and users. You will get much faster feedback on the releases if a new issues is introduced.

It is always easier to correct bugs when the code is fresh, correcting bugs that was introduced 9 month ago can be much more tricky since the code might have changed and might be affected by other code commits.

I also believe that if you need to dig into code changes that was implemented long time ago will require more effort compare to something that was recently implemented.

[krmahadevan]

@krmahadevan I think it's perfectly fine to ship minor version releases more frequently even if they only contain a handful of bug fixes.

@cbeust Yes. I agree. Just that our few handle of bug fixes are also only trickling in, and not flowing in :) Hence the elongated time.

The way I set things up in the Maven repo and github should make it trivial for you and @juherr to deploy to Maven at any time, let's try to do that more frequently.

Yep. This would be the first time we are going to be doing a release after the overhaul of our gradle build. Will setup sometime for this two couple of issues which @martinaldrin is focussing on gets resolved so that he and the others can basically start using 7.5.0 (or) whatever that version is going to be.

[krmahadevan]

That would be very helpful, and I think that will benefit both developers and users. You will get much faster feedback on the releases if a new issues is introduced.

It is always easier to correct bugs when the code is fresh, correcting bugs that was introduced 9 month ago can be much more tricky since the code might have changed and might be affected by other code commits.

I also believe that if you need to dig into code changes that was implemented long time ago will require more effort compare to something that was recently implemented.

@martinaldrin - Trust me I would love to do more frequent releases (monthly or a quarterly). And I am not against that notion. But like I mentioned, we are short of hands on helping with bug fixes. Currently its mostly just me and @juherr and sometimes it gets difficult to time slice between OSS and your day job. That's the only reason why we have elongated releases. With support of folks like you in terms of helping out with bug fixes etc on TestNG, I am confident that we can get this sorted out pretty easily.

[martinaldrin]

@martinaldrin - Trust me I would love to do more frequent releases (monthly or a quarterly). And I am not against that notion. But like I mentioned, we are short of hands on helping with bug fixes. Currently its mostly just me and @juherr and sometimes it gets difficult to time slice between OSS and your day job. That's the only reason why we have elongated releases. With support of folks like you in terms of helping out with bug fixes etc on TestNG, I am confident that we can get this sorted out pretty easily.

I fully understand that, and we appreciate your excellent support

[juherr]

@krmahadevan Do you need any help with making a release? Feel free to ping me in case of a problem.

[krmahadevan]

@juherr - I will try to go ahead with doing a release once all the inflight defect fixes are merged. I will surely ping you incase I run into problems.

[krmahadevan]

We should now be able to do quarterly releases or more frequent bug fix releases depending upon the size of issues.

[martinaldrin]

Hi,
We have some bug corrections submitted on master, I wonder if it is time for a 7.5.1 release now?
It would be very valuable for us if we could get releases more often.

Thanks
/Martin

[cbeust]

I think it would be healthy to have a steady train of releases (e.g. monthly).

[krmahadevan]

@martinaldrin - TestNG 7.6.0 is now released. Please refer to the release notes here https://groups.google.com/g/testng-users/c/BAFB1vk-kok/m/uwQlDyqkAAAJ

[martinaldrin]

Great, many thanks for the fast release!

A question, the tag is named 7.6.0-rc1, is that correct and the release is 7.6.0?
https://github.com/cbeust/testng/tags

The reason why the tag is important for us is because our FOSS handling requires that we point out the source of the release. and now it is a missmatch between the release and the source

[krmahadevan]

@martinaldrin

A question, the tag is named 7.6.0-rc1, is that correct and the release is 7.6.0?

Yes the tag is correct.

[martinaldrin]

Hi @krmahadevan, then I don't understand the release procedure.
Shouldn't the tag match the actual release?

[krmahadevan]

@martinaldrin - The gradle plugin that we are using adds that as a tag because it attempts at doing release candidates.
You can refer to the documentation here

[martinaldrin]

Always when I see other projects releasing candidates there is a mapping between the version and the tag.
But I treat 7.6.0 as final release and 7.6.0-rc1 as a candidate of 7.6.0. So I do not understand this.
If you planned to release a candidate rc1, then I expect the version in maven central to be named rc1 as well and not 7.6.0

[Elisedlund-ericsson]

@martinaldrin

A question, the tag is named 7.6.0-rc1, is that correct and the release is 7.6.0?

Yes the tag is correct.

that does not make any sense at all,
So there is no official release, its just release candidate? or is the rc1 referring to a different commit than what 7.6.0 is based on?
which commit is 7.6.0 based on?

The gradle plugin that we are using adds that as a tag because it attempts at doing release candidates.

That is fine but a release candidate is not a release, its a candidate to become a release, you should probably still tag the release commit with the release number.
a release candidate and and a release can be the same commit which has two tags.
were the release tag is added first when the release candidate is proven to be stable and good enough to become a offical release.

[krmahadevan]

@Elisedlund-ericsson

So there is no official release, its just release candidate? or is the rc1 referring to a different commit than what 7.6.0 is based on?

7.6.0 is the official release. The commit represented by the tag 7.6.0-rc1 is what 7.6.0 was built on.

[martinaldrin]

I totally understand the procedure, but If I look at other projects that do release candidates. they do not tag the candidates, but they always tag the release. Here is an example from a different project

https://mvnrepository.com/artifact/org.aspectj/aspectjrt
https://github.com/eclipse/org.aspectj/releases

[Elisedlund-ericsson]

The commit represented by the tag 7.6.0-rc1 is what 7.6.0 was built on.

Why not tag it with 7.6.0 then like basically every other git project. (including every other TestNG release! even the ones that used release candidates)

[krmahadevan]

@Elisedlund-ericsson - Its not done yet. But would like to understand as to how would that be a blocker for you folks to consume the release. The release was done only in the morning. I am still going through the clean up activities associated with a release.

[Elisedlund-ericsson]

The release was done only in the morning. I am still going through the clean up activities associated with a release.

That is fine, and big thanks for the release. sorry if i have sounded pushy or negative.

But would like to understand as to how would that be a blocker for you folks to consume the release.

We can temporarily consume and test it.
But we cannot register it in any FOSS handling software's we have, (for handling licenses and vulnerabilities etc)
as it requires mapping between Artifact and commit/tag. that prevents us to use the release in an official manner

[martinaldrin]

Hi,

I have some bad news with last release, We tested a snapshot last week from master and that was working. But there seems to be a new commit in-between that affects us badly. non of the executions passed.

I expect it is related to this commit:
d190ade

Seems that some tmp path is added infront of the path

argument:
-xmlpathinjar suites/postrelease/emulator_wrat_postrelease.xml

Exception:
2022-05-18T08:44:26,315 ERROR
java.nio.file.NoSuchFileException: /tmp/testngXmlPathInJar-15086412835569336174/suites/old_action_test_suites/rnc_action_regression_with_configurationdata.xml

[krmahadevan]

@Elisedlund-ericsson

That is fine, and big thanks for the release. sorry if i have sounded pushy or negative.

Given the fact that I juggle around with a lot of things, apart from taking care of TestNG stuff, it would be good if you could please give it some time (at least 24 hours). I am still trying to come to terms with the Gradle plugin that we are using to do releases and understand what all it does internally. So I may be a bit rusty.

But we cannot register it in any FOSS handling software's we have, (for handling licenses and vulnerabilities etc)
as it requires mapping between Artifact and commit/tag. that prevents us to use the release in an official manner

Thanks for adding this context. I have created a new tag 7.6.0 which also tracks the same commit as 7.6.0-rc1 as can be seen here

[krmahadevan]

Hi,

I have some bad news with last release, We tested a snapshot last week from master and that was working. But there seems to be a new commit in-between that affects us badly. non of the executions passed.

I expect it is related to this commit: d190ade

Seems that some tmp path is added infront of the path

argument: -xmlpathinjar suites/postrelease/emulator_wrat_postrelease.xml

Exception: 2022-05-18T08:44:26,315 ERROR java.nio.file.NoSuchFileException: /tmp/testngXmlPathInJar-15086412835569336174/suites/old_action_test_suites/rnc_action_regression_with_configurationdata.xml

Can you please log this as a new issue ?

[martinaldrin]

yes, I will

[juherr]

@vlsi Any reason why the release step doesn't add a release tag?

[krmahadevan]

I think there should be bug fix releases, 7.6.1 for example. That means the branching model should change as well, not just releasing process.

We wouldn't be needing a branching model because we aren't running parallel releases. Its trunk based development model. So the pom file gets updated with whatever version.
For e.g.,

• if its a bugfix release only z gets changed in x.y.z
• if its a minor release we change y in x.y.z and
• if its a major release we change x

[askfor]

If use TBD model, maybe there should be some release branches, then it's possible to cherry pick bug fixes and release a bug fix version?
https://trunkbaseddevelopment.com/branch-for-release/

[krmahadevan]

We dont have release branches because at any point in time we would like to target for only 1 release.
What that release would be (Major/minor/bugfix) is something that we decide at that point in time.

I believe we are following https://trunkbaseddevelopment.com/release-from-trunk/
I have seen other projects such as Selenium project also follow the release-from-trunk approach and not create release branches.

We have just the master branch from which we release at any given point in time. Post that, the master branch gets tagged. I am personally not in favour of making the release process complicated with introducing release branches which gets cherry-picked etc., because its going to take time in getting it done.

The branch-for-release I think would work when you want to deploy an artifact into an environment which gets executed (for e.g., a microservice that gets deployed into an environment).

For libraries I think its fine to follow release-from-trunk

[Elisedlund-ericsson]

Considering there is a CVE fixed on master branch would it make sense to release a 7.6.2 (or 7.7.0)
https://nvd.nist.gov/vuln/detail/CVE-2022-4065

I surprised such a serious commit did not immediate new release.

What happened with more frequent releases?
Now there's huge amount of commits piled up and even if you do a release now, we risk that there is some other change that will prevent us from uplifting to the new release forcing us to either:

1. Live with or workaround the new issues
2. Keep using old TestNG version with the CVE
3. Having to create a fork with on the old release with the CVE patch released

[krmahadevan]

@Elisedlund-ericsson - Can you please help vet out this below release candidate and let me know if all is well at your end for version 7.7.0 ?

If you are using Maven then please add

<repository>
    <id>maven-central-staging</id>
    <url>https://oss.sonatype.org/content/repositories/orgtestng-1084</url>
</repository>

to your pom file.

If you are on Gradle please use

repositories {
    mavenCentral()

    maven {
        url 'https://oss.sonatype.org/content/repositories/orgtestng-1084' }
}

Once I hear back from you and if all is well then I will publish this release candidate.

[krmahadevan]

What happened with more frequent releases?

Because I was out of action for a couple of months owing to a medical emergency !

[krmahadevan]

Now there's huge amount of commits piled up and even if you do a release now, we risk that there is some other change that will prevent us from uplifting to the new release forcing us to either:

Do you have suggestions on how to make it better?

TestNG does not have a a lot of contributors. Majority of the time, it's only me who is fixing bugs etc., So that means that the number of fixes that lands in a version is very miniscule and it takes sometime before a sizeable number of issues are fixed which would warrant a release. That explains the lag in the release.

[Elisedlund-ericsson]

Can you please help vet out this below release candidate

Yes we can forsure test it, We even tested latest TestNG master mid November already. (which I guess went fine)

Because I was out of action for a couple of months owing to a medical emergency !

No worries, Prioritize health/family over work! Thanks for the great efforts,

Do you have suggestions on how to make it better?

TestNG does not have a a lot of contributors. Majority of the time, it's only me who is fixing bugs etc., So that means that the number of fixes that lands in a version is very miniscule and it takes sometime before a sizeable number of issues are fixed which would warrant a release. That explains the lag in the release.

How I usually do it and how I prefer is when every commit gets released. then automatically uplifted and used.
that way if there's a problem you run into it very quick and also it narrows it down to just a one or a few commits that could have caused the issue, and the person that did the change usually problem area still left in his recent memory.

basically test/use everything as early as possible and fail as fast as possible.

I'm fine with a release containing a major backwards compatibility issues sometimes so it can be identified as early as possible to then either be fixed/discussed or handled at the user side(migrated)

Doing it like above however means that the release process has to be fully automated. Even preferably with acceptance testing from end users included. Like us (or other big users like OpenJDK(i think?))

Maybe a more sane or reasonable first step is to make a new release like ever ~3-5 small commits or every time there is a big/important commit.
of course we already test a SNAPSHOT version this often, but we rather want use a release in our production this often

[krmahadevan]

Yes we can forsure test it, We even tested latest TestNG master mid November already. (which I guess went fine)

I will wait to hear back from you before I publish it to Maven central and wrap up the rest of the release formalities.

[krmahadevan]

How I usually do it and how I prefer is when every commit gets released. then automatically uplifted and used.
that way if there's a problem you run into it very quick and also it narrows it down to just a one or a few commits that could have caused the issue, and the person that did the change usually problem area still left in his recent memory.

Do you have an OSS project sample that does releases this way, which I can refer to ?

Doing it like above however means that the release process has to be fully automated. Even preferably with acceptance testing from end users included. Like us (or other big users like OpenJDK(i think?))

The release process is already automated. Just that the triggering part is manual. Do you have a bunch of acceptance tests that you can share, which we can try to incorporate and build some sort of an automated way of getting this done ?

Basically for every SNAPSHOT publishing that goes into Maven Central snapshot repository, this project's build job would run having it consume the SNAPSHOT. TestNG build snapshot would be the upstream job and the acceptance test would be the downstream job.

of course we already test a SNAPSHOT version this often, but we rather want use a release in our production this often

If you can share a bunch of acceptance tests, that can be used as a vetting out process, we can perhaps have it setup to be built after every merge to master, which publishes a snapshot version. That way, for every commit the acceptance tests are already checking if all is well.

[Elisedlund-ericsson]

Do you have an OSS project sample that does releases this way, which I can refer to ?

Everything I know of is closed source unfortunately. but basically its like your current process except that there is no manual triggering instead it runs the release flow on every commit, Then consumers(users) has CI to auto uplift to the new release and run tests, given any failures it would then report back to the origin project.

If you can share a bunch of acceptance tests, that can be used as a vetting out process, we can perhaps have it setup to be built after every merge to master, which publishes a snapshot version. That way, for every commit the acceptance tests are already checking if all is well.

Probably not that much that we can share (as in code) :(
and if we have something general enough it could as well just be contributed unit/integration tests merged to TestNG itself, stuff like: https://github.com/cbeust/testng/pull/2712/files#diff-72c2d0dc194a0c36f76620fbc687e4627fa52d80619f22ce400c6116ce809126 )

(Our testing is more in the nature of uplifting latest TestNG and running with all our know working "product" tests to see if something that worked previously instead now fails)

The final results of our integration/acceptance testing of TestNG can probably be shared however in some automated way.
Like posting an Issue in Github that SNAPSHOT X causes compatibility failures, then of course we would have to make an analysis of our internal closed logs / tc fails to pin point the exact issue and provide stacktraces.

[krmahadevan]

but basically its like your current process except that there is no manual triggering instead it runs the release flow on every commit, Then consumers(users) has CI to auto uplift to the new release and run tests, given any failures it would then report back to the origin project.

Are you suggesting that we release a version to Maven Central for every commit ? I don't think I quite understand this part, because our release process essentially releases the artifact into Maven Central (We first publish release candidates into staging area, which would wait there to be released or dropped). If we drop it from a staging area it means that we are discarding that release candidate, if we release it, then it means it gets published to Maven Central.
@juherr any idea on this ?

The straight forward way that I can think of is that, you folks have a dedicated job internally that consumes TestNG snapshot dependency and runs your evaluations on it and report back any breakages back to us on the snapshot so that it can be fixed. I dont see a different way of doing this, if there are no acceptance tests that can be shared.

[krmahadevan]

@Elisedlund-ericsson - Just to ensure that we aren't losing focus, please do ping back the results of your evaluation on the release candidate so that I can close the release, do the post release activities and send out release notes to the TestNG users community. I will wait to hear back from you on that.

[juherr]

I don't see any advantage to make a release on every commit. Here, we are not building a huge product that suffers from the synchronization of teams.

If you really want a release from every commit, you can use https://jitpack.io as a maven repository or you can make your own release process on a synchronized fork you manage.

[vlsi]

I guess the key issue for now (which is different from the initial comment in this "issue") is that it would be great to have TestNG releases shortly after CVE is fixed.

---

On top of that, it would probably make sense to do patch releases (security-only) for several past versions of TestNG since people will be under pressure of "fixing CVE", and they might not have enough time to upgrade to the latest TestNG as CVE arrives.

For pgjdbc, we settled on back-porting security fix to all past versions till 42.2.x (42.2.0 has been released ~5 years ago, and 42.2.x is the latest version that still supports Java 1.6 and 1.7). For instance, a recent GHSA-562r-vg33-8x8h has been fixed in four versions 42.2.27, 42.3.8, 42.4.3, 42.5.1, even though, the versions are most likely compatible. The users would most likely upgrade from 42.2 to 42.5 without any issues, however, we decouple "urgent upgrade needed for CVE resolution" from "upgrade for fixing bugs/adding features".

One of the reasons we do patch releases for past versions is that Spring Boot has a policy that they do not upgrade across "major" versions no matter what, so if a certain Spring Boot version uses pgjdbc 42.3.x, then that Spring Boot will use 42.3.x only. See pgjdbc/pgjdbc#2599

[bobshie]

Hello, I see there is a candidate version 7.7.0-rc2 7.7.0-rc1, where is the candidate release path, I can get the JAR and make some verification.

is /org/testng/testng/7.7.0-SNAPSHOT/testng-7.7.0-20221206.103750-3.jar the latest one?

[krmahadevan]

@bobshie - Did you read through the comments ?
Please see here #2665 (comment)

[bobshie]

@krmahadevan , sorry, I didn't go though all comments here. OK, I'll do some verification in this week on the version:
https://oss.sonatype.org/content/repositories/orgtestng-1084/org/testng/testng/7.7.0/

[bradh]

I added staging and bumped to 7.7.0 (we were previously on 7.6.1), and did a full rebuild.

No problems noted.

Our use of testng is pretty basic, and certainly wouldn't have tested every capability. It is probably reasonably representative of a lot of other projects though.

[krmahadevan]

@Elisedlund-ericsson - Is there an ETA by when you would be able to tell me the result?

[krmahadevan]

I guess the key issue for now (which is different from the initial comment in this "issue") is that it would be great to have TestNG releases shortly after CVE is fixed.

On top of that, it would probably make sense to do patch releases (security-only) for several past versions of TestNG since people will be under pressure of "fixing CVE", and they might not have enough time to upgrade to the latest TestNG as CVE arrives.

For pgjdbc, we settled on back-porting security fix to all past versions till 42.2.x (42.2.0 has been released ~5 years ago, and 42.2.x is the latest version that still supports Java 1.6 and 1.7). For instance, a recent GHSA-562r-vg33-8x8h has been fixed in four versions 42.2.27, 42.3.8, 42.4.3, 42.5.1, even though, the versions are most likely compatible. The users would most likely upgrade from 42.2 to 42.5 without any issues, however, we decouple "urgent upgrade needed for CVE resolution" from "upgrade for fixing bugs/adding features".

One of the reasons we do patch releases for past versions is that Spring Boot has a policy that they do not upgrade across "major" versions no matter what, so if a certain Spring Boot version uses pgjdbc 42.3.x, then that Spring Boot will use 42.3.x only. See pgjdbc/pgjdbc#2599

@vlsi - Thanks for sharing those additional details in terms of back patching. I will keep this part in mind going forward.
There was a PR that someone had raised to fix this issue #2806

I read an interesting comment in the same PR that basically questions the seriousness of this issue since the code path in question is related to a test jar mode of execution of TestNG. Details here

[Elisedlund-ericsson]

@Elisedlund-ericsson - Is there an ETA by when you would be able to tell me the result?

All test so far have been fine, like we tested on roughly 300+ repos(typical small OSS libraries) that does just typical unit testing.

However the more heavy stuff like function/system/integration testing we have however not finished yet which uses much more of TestNG like: every type of Listener and every annotation and many of the application parameters including -xmlPathInJar

I read an interesting comment in the same PR that basically questions the seriousness of this issue since the code path in question is related to a test jar mode of execution of TestNG. Details here

I also don't fully agree with the seriousness of the issue either, Have a very hard time seeing any real world scenarios were it can be abused.
in this case its more about company policies of using software with CVE and "management" not liking to see CVE: 7.8 HIGH as it looks bad.

[krmahadevan]

@Elisedlund-ericsson -

However the more heavy stuff like function/system/integration testing we have however not finished yet which uses much more of TestNG like: every type of Listener and every annotation and many of the application parameters including -xmlPathInJar

Thanks for the inputs. Can you please tell me by when can we expect to hear back from you on the completeness of those tests as well? That way I would be able to wrap up the release part in time. Please let me know. Given the fact that there are now queries and apprehensions on the CVE, I don't want to stall the release more because now I will have to keep answering all those queries :)

[Elisedlund-ericsson]

Thanks for the inputs. Can you please tell me by when can we expect to hear back from you on the completeness of those tests as well? That way I would be able to wrap up the release part in time. Please let me know. Given the fact that there are now queries and apprehensions on the CVE, I don't want to stall the release more because now I will have to keep answering all those queries :)

I actually don't know why it taking this long either, (and I'm not running it myself personally) I asked for ETA

From my perspective it ok if you go ahead and release right now.
We did a complete test during mid November so the majority of the new commits(since 7.6.1) does not cause us any issues for us atleast.
From the newer commits I cannot see anything I foresee as risky except possibly the "dependsOnMethods" commits which we rely on and that has caused us issues before.
on the ~smaller chance of issues/failure we could instead look into a 7.7.1 so other users do not have to wait for a 7.7.0

[krmahadevan]

@Elisedlund-ericsson - Thank you so much. I will proceed with the release.

[krmahadevan]

@Elisedlund-ericsson TestNG 7.7.0 is released.

[Elisedlund-ericsson]

TestNG 7.7.0 is released.

Great, All of our tests came back successful now also :)

Thanks for all the help 🥇