Add task for closing a staging repository

[marcphilipp]

• Allow close task to be run in a separate Gradle invocation. That requires that initialize task to write the necessary information (staging id?) to a file so the close task can read it from there
• Add mustRunAfter dependencies to all publish tasks (across projects) to support parallel multi-project builds

[vlsi]

Note: "staging repository id" should be visible to the end-user anyway.
There should be a way to use the ide for other tasks.

Use case: generate draft mail on "Nexus repository preview"

[szpak]

Yes, we plan to expose it as an extension property as currently in GNSP.

[vlsi]

Can it be a task property rather than an extension property?

[szpak]

As I will be set by one task (initialize*) and consumed by others (close* and release*) for me it's better to keep it in a "global" space - the extension. However, having good argument against that could be a good input to rethink the idea.
For example, a potential issue I see is the fact that the plugin will support multiple initialize* tasks for different Nexus servers which would complicate the extension. However, maybe we will have to have there some mapping server -> server properties anyone. Would it be enough for you? Is there any other reason that you would prefer to keep it in the task (and which task - init/close/release/drop/all of them?) instead?

[vlsi]

Is there any other reason that you would prefer to keep it in the task

The value is not usable before the task is executed, so it looks pretty natural to add dependsOn(initialize) + treat the id as the task output.

[szpak]

The value is not usable before the task is executed

Not always. You may want to close/release a staging repository created with some older Gradle execution (on demand or dealing with some errors).
It would be especially usable with drop* task (once/if implemented).

[vlsi]

Do you mean the property would be "read only"?
That is user would never need to use .set(..)

If that is the case, it might be ok. However, having a single property would indeed impact the ability to have multiple repositories.

Then, Gradle "forbids" cross-project access. In other words, the order of project evaluation is not really consistent, so it might be the case that subproject is evaluated before the root project, thus the extension might be not there (or be empty).

AFAIK "it is a code smell every time cross-project APIs are invoked".
I guess this might be relevant: https://docs.gradle.org/current/userguide/troubleshooting_dependency_resolution.html#sub:configuration_resolution_constraints

[marcphilipp]

I think it should be a property of NexusRepository which is part of the extension which is present in each project that applies the plugin.

[marcphilipp]

If possible, we should probably only expose a Provider, not a Property so users can only read but not write.

[vlsi]

That requires that initialize task to write the necessary information (staging id?) to a file so the close task can read it from there

I gave it a bit of though, and now I think the "minimal" implementation should allow to store the "necessary information" at the Nexus (server) side (e.g. in staging repository comment or something like that).

In other words, I would like to have a task (or API) like findStagingRepositoryByComment(...) or even getAllStagingRepositories.
Then I would no longer need local files to keep staging ids.

It would enable end-users to clean the workspace and still associate staging repository with the intended release step.

Here's a similar case: vlsi/vlsi-release-plugins#26

Apache releases are typically "staged" to SVN folders like https://dist.apache.org/repos/dist/dev/jmeter/apache-jmeter-5.2.1-rc1/ , and for release the contents is moved to folders under https://dist.apache.org/repos/dist/release/jmeter/

There we have the same question: "we need to track the set of the file names to be released".

My initial (current) implementation was to use build artifacts to get the set of release files, and that turned out to be bad, because "publish release" task starts depending on the full build procedure, and it becomes fragile (you don't want to pray in a hope that all the tasks would be UP-TO-DATE).

---

As of now, I'm inclined that external service should be treated as a primary store. That is developer should be able to "stage the artifacts", perform "./gradlew clean", and then they should be able to release the repository.

I propose the following workflow:

Stage release artifacts

1. As release manager starts the release, they decide release version. For instance, it could be v1.0-rc1. The version could be configured in the sources or passed via parameters (it does not matter)
2. initializeStagingRepository takes that version number, and puts it to the staging repository metadata. I'm not sure what are the possibilities, however, I'm sure the version can be put to the "comment" field
3. initializeStagingRepository produces an @Output with stagingRepositoryId. It could save the id to file (files are commonly expected task inputs and outputs). It could provide the id via Property<String>.

Verify release

4. Everybody might want to verify the artifacts. They know "release candidate index", and they can pass it to the build script, so it locates the relevant staging repository, dowloads the artifacts, and validates them.

Publish release

The release manager remembers "release candidate version" to be released, and it passes that parameter to the build script.
4) findStagingRepository(version=...) task queries Nexus, and it detects the staging repository id. Then regular closeRepositoryOrWhatever task could be used to close&promote the repository.

^^^ Note: the above is not something that must be used by everybody. It is just a sample.
Of course, one could combine all the tasks into a single Gradle execution.

In other words, I suggest using Nexus Server for storage of "repository id" rather than "repository extension".

@marcphilipp , @szpak , what do you think?

[marcphilipp]

Would it suffice if the close task would check for open staging repos and fail if there's more than one, but print as much info (URL, comment) of each as possible?

Users could then re-run the goal and pass the one they meant explicitly.

[ihostage]

Would it suffice if the close task would check for open staging repos and fail if there's more than one, but print as much info (URL, comment) of each as possible?

Hmmm 🤔If I use one credential for Sonatype I can't release some modules parallel? Moreover, we have a case with release one module for different Scala targets (2.11, 2.12, 2.13) from parallel Travis jobs, that open many staging repos at the same time.

[marcphilipp]

Sure, you can. My proposal was only when the close task was executed by itself with a clean workspace and no parameters. If you close from the build that published, we would know which staging repo to close.

[vlsi]

If you close from the build that published, we would know which staging repo to close.

I don't like how the whole thing becomes fragile if clean is executed :(

My proposal was only when the close task was executed by itself with a clean workspace and no parameters

In 99.42% of the cases, users want reasonable comments for their staging repositories.
For instance, I have the following:

It corresponds to ~5 or so "release candidates".
The version tree between them looks as follows:

I have multiple "closed" repositories, and I can easily test each version individually, and later I can decide which one is better.

What if I want to promote v3.0-rc5 as the final release?
There's no clear way to tell which staging repository belongs to each RC number :(

---

On the other hand, if staging repositories did include user-provided reference (e.g. version number, git sha, or whatever else), then end user would be able to promote the relevant repository.

In other words, it would be possible to promote the release candidate by specifying a single rc index: ./gradlew publishDist -Prc=5

Note: as you might guess, I have built a lot of different release candidates, and I probably did ./gradlew clean in between. So I keeping rc index <-> staging repo id link in the build folder is not really an option.

[szpak]

Closing this task as closing a staging repository is implemented.

@vlsi I extracted your enhancement proposal to a separate issue - #19.