Generate plugins lock file from list of plugins

[simonvanderveldt]

What feature do you want to see added?

Like most tools that support installing packages with dependencies it would be nice if it would be possible to generate a list of all plugins, their (transitive) dependencies and have all their versions pinned. This to enable a reproducible build (in so far as that it would have the same versions of all components/plugins).

Until the removal of install-plugins.sh in the Jenkins Docker image we were using a plugins.txt which contains a (unversioned) list of the plugins that we want to use and from that we generated a plugins.lock.txt which contained all the plugins, including (transitive) dependencies, with their versions pinned. I've not been able to find a simple/direct way to do this with jenkins-plugin-cli.

$ jenkins-plugin-cli -f plugins.txt --list --output txt

gets me somewhat close, but it outputs irrelevant stuff ("Installed plugins", "Bundled plugins", etc sections).

Also it would be nice if one could pass a filename to write this list to so that the tool handles everything in one go. Plus that wayone can use --verbose at the same time as well.
(#448 might be a related change, but the separation to stderr/stdout is based on a long running misunderstanding, stderr is for errors not for regular output).

Upstream changes

No response

[lemeurherve]

$ jenkins-plugin-cli -f plugins.txt --list --output txt

gets me somewhat close, but it outputs irrelevant stuff ("Installed plugins", "Bundled plugins", etc sections).

You can pass --available-updates argument to the CLI to retrieve only the new versions of your plugins.

Also it would be nice if one could pass a filename to write this list to so that the tool handles everything in one go

Just pipe it to a file?

For the record, we're still using such plugins.txt on Jenkins Infrastructure instances with all our plugins pinned and updated with (for example) https://github.com/jenkins-infra/docker-jenkins-weekly/blob/main/bin/update-plugins.sh

To get the initial complete list of plugins including dependencies, we're using this script in the Jenkins controller script console, courtesy of @MarkEWaite:

Jenkins.instance.pluginManager.plugins
    .collect()
    .sort { it.getShortName() }
    .each {
        plugin -> println("${plugin.getShortName()}:${plugin.getVersion()}")
    }
return

[simonvanderveldt]

Thanks for the reply/info :)

You can pass --available-updates argument to the CLI to retrieve only the new versions of your plugins.

That's not the same as generating a complete list including transitive dependencies from a list of plugins, right?

Just pipe it to a file?

But then you can't see progress/there's no feedback what's happening

For the record, we're still using such plugins.txt on Jenkins Infrastructure instances with all our plugins pinned and updated with (for example) https://github.com/jenkins-infra/docker-jenkins-weekly/blob/main/bin/update-plugins.sh

To get the initial complete list of plugins including dependencies, we're using this script in the Jenkins controller script console, courtesy of @MarkEWaite:

That is exactly the output I'd like to generate, but I'd like to be able to do so directly from a list of plugins. It doesn't really make sense to have to run Jenkins first, then run a script inside Jenkins to do this.

[lemeurherve]

You can pass --available-updates argument to the CLI to retrieve only the new versions of your plugins.

That's not the same as generating a complete list including transitive dependencies from a list of plugins, right?

It isn't but it's also not why I specified this arg, I was responding to "but it outputs irrelevant stuff ("Installed plugins", "Bundled plugins", etc sections)"

That is exactly the output I'd like to generate, but I'd like to be able to do so directly from a list of plugins. It doesn't really make sense to have to run Jenkins first, then run a script inside Jenkins to do this.

You only have to run this script in Jenkins when you're adding new plugins (to get their dependencies), or periodically to check if existing plugins didn't integrated new dependencies since the last time you've run the script. (I have no idea if it's a frequent occurrence).
Every other times it can be run independently to retrieve last plugins versions.

I don't know if your feature request represents a huge coding effort or not (I think it could be a nice addition), but at least you've got here a workaround. Hope it can alleviate some of your issues.

[jiakuanghe]

Hi there, I am a new contributor and I am interested in this issue. But I don't have any idea what is the purpose of the plugin's lock file.

If you want to download the specific version, you can use the option --plugins, and specify the plugins you want to install and notate their version.

If you want to get the available updates in a file, you can use the option --available-updates --output txt, and you will get the following output:

mailer:1.32

If the above solution cannot satisfy your question. My understanding is you would like to create a file called plugins.lock.txt, and then after you installed the plugins, the installed plugins and their version will be recorded in this file, just similar to a node.js project. Then, you may use this created file with the --plugins option to install plugins. Is it correct?

[MarkEWaite]

Hi there, I am a new contributor and I am interested in this issue. But I don't have any idea what is the purpose of the plugin's lock file.

In the context of plugins.txt, I think that the plugins.txt file would be the list of plugins declared by the user (like "blueocean:1.27.3 docker-workflow:563.vd5d2e5c4007f") while the plugins-lock.txt file would be the full list of plugins generated at a point in time with a specific Jenkins version and would include the plugins declared by the user and all the dependencies of those plugins.

When the plugins.txt file is read, it is ambiguous because it depends on the current state of the Jenkins update center and the Jenkins version that was specified. If I requested "blueocean:1.27.3 docker-workflow:563.vd5d2e5c4007f" 10 days ago with Jenkins 2.361.4, I would get a different final set of plugins installed than if I requested "blueocean:1.27.3 docker-workflow:563.vd5d2e5c4007f" today with Jenkins 2.387.1.

The plugins-lock.txt file would be unambiguous, because it would describe all the plugins and versions that were read from the Jenkins update center at the time the command was executed.

[simonvanderveldt]

What @MarkEWaite describes is indeed correct. The underlying reason being that you want your build to be deterministic/you want to be able to reproduce the same state at any point in time, so you can be sure that your Jenkins including all plugins is the same as the setup you validated.

[Louai-Abdelsalam]

@simonvanderveldt Assuming I understood your requirement properly, you actually wanna redirect the stdout only onto a file, and that file becomes your lock file.

I dunno why it's set up like that, but the stuff you called:

irrelevant stuff ("Installed plugins", "Bundled plugins", etc sections)

is actually in the stderr, with only the part of interest (i.e the plugins and their versions) neatly being in the stdout.
I dunno why non-error stuff is in the stderr, and I dunno why the stderr is printed even when the command exits with a 0 status code. For a while a redirect didn't even cross my mind as a solution because I thought everything I was seeing (i.e the useful stuff and the irrelevant stuff) were all stdout.

So anyway what you wanna do is jenkins-plugin-cli --list --output txt 1> plugins.txt. This works with --output yaml too.
It was a lucky find though honestly; I'm not sure why this isn't documented.

For reference I was using version 2.12.13.