featureRequest: preExtract and preUpgradeTasks configuration option.

[r-plus]

What would you like Renovate to be able to do?

We try to create PR for CocoaPods managed dependency, but renovate display error like this.

{
  "artifactErrors": [
    {
      "lockFile": "Podfile.lock",
      "stderr": "Analyzing dependencies\n[!] Unable to find the Xcode project `/mnt/renovate/gh/username/repo/repo.xcodeproj` for the target `Pods-repo`.\n"
    }
  ]
}

yes, we are not putting xcodeproj file in git repository.
instead, using xcodegen https://github.com/yonaskolb/XcodeGen to create xcodeproj file from cli before we build the project.

Did you already have any implementation ideas?

like the postUpgradeTasks https://docs.renovatebot.com/configuration-options/#postupgradetasks configuration, we want to execute arbitrary commands by preUpgradeTasks to prepare for renovate.

[rarkins]

We don't have any timeline for supporting custom tasks in the hosted app, so you'd need to self-host once such a feature existed.

Regarding your specific need, do you know if this can even be run on Linux? From the project's install instructions:

[github-actions]

This issue has been labeled with status:requirements. This indicates that we don't know enough to start work and further requirements or a reproduction are needed first.

This label will be replaced with status:ready once all requirements and reproductions necessary to start work have been met.

If it's not clear what is missing to move this issue forward, ask for clarification in a new comment. If you think we already have what we need to move forward, mention this in a new comment.

[r-plus]

thanks for your reply.

I didn't test it on Linux.
But project unit testing will execute on ubuntu-latest when every commit by github CI.
https://github.com/yonaskolb/XcodeGen/runs/1906702739

So maybe.

[rarkins]

Back to the generic concept of preUpgradeTasks:

Ideally they are configurable per-upgrade.

We also need to consider how, where and when they are run:

• Do they need to be run before we edit the package file, or just before artifacts updating?
• Do they need to be run within the same container as the artifacts updating (e.g. to install a necessary tool inside the container) or run independently - meaning that the result of such a task needs to be visible in the repo

[Leksat]

A use case from #8984: Drupal project updates.

Drupal configuration is stored in Git, but updates from Drupal core or modules can modify it. To achieve completely automated updates we need:

• Pre: Install Drupal using existing configuration.
• Update dependencies.
• Post: Run Drupal updates and export the updated config.

[Leksat]

For the above Drupal config use case:

Do they need to be run before we edit the package file, or just before artifacts updating?

Ideally before any changes are made.

And a counter question:
If we install dependencies in the pre-upgrade task (and also in the post-upgrade), can this be a problem for Renovate?

Do they need to be run within the same container as the artifacts updating (e.g. to install a necessary tool inside the container) or run independently - meaning that the result of such a task needs to be visible in the repo

In the same container. In the post-upgrade task we need to have the database (sqlite) generated in the pre-upgrade.

[rarkins]

Please explain why Drupal needs to be installed prior to updating dependencies. Keeping in mind that Renovate updates dependencies directly. I normally would not see any reason why a command needs to be run prior to us editing the e.g. requirements.txt ourselves.

[Leksat]

That's the way it works 🤷

Drupal, as a usual CMS, consists of two parts: code and database. But there is also a third one: configuration. The active configuration lives in the database, but it can be exported to (or imported from) yaml files. The best practice is to not change any configuration on prod, but rather update it in dev/local environment, export to Git and then deploy to prod where it will be imported.

(So code and config are stored in Git. Database is not. To run tests in CI, we install Drupal importing the configuration from Git. Same as for local development.)

Drupal core and modules sometimes provide their own config, which is still can be edited by users. Modules can provide updates which can modify the active configuration. They never touch the exported one (because, even if it's the best practice, the exported config may not exist). So module updates need to be executed within an active Drupal installation.

Also, losing the config updates is not that critical because they come rarely. I guess most of people don't even notice the loss 😅

[rarkins]

module updates need to be executed within an active Drupal installation.

Define "module updates"?

We are literally editing a text file. I think you're referring to some command someone would normally run?

[Leksat]

Oh, yes, sorry. So a module release can provide one or more update functions which need to be executed after the code update. I think the most common word for this would be "migrations". But in Drupal they are called updates.

[buffcode]

We have a similar problem. We use link:<path> in our package.json where <path> references a package installed via composer. As renovate runs both managers independently yarn upgrades will remove linked dependencies from yarn.lock.

A composer install as preUpgradeTask is required to fix this problem.

[buffcode]

A use case from #8984: Drupal project updates.

Drupal configuration is stored in Git, but updates from Drupal core or modules can modify it. To achieve completely automated updates we need:

• Pre: Install Drupal using existing configuration.
• Update dependencies.
• Post: Run Drupal updates and export the updated config.

IMHO this it out-of-scope of renovate and better suited for a CI task.
For example:

• Renovate updates dependencies and creates PR
• CI pipelines detects changes via Renovate
  • installs drupal
  • runs module updates
  • exports new config as artifact or even commits back these changes to the branch

I argue that moving (complex?) tasks from a CI pipeline into Renovate pre/postUpgradeTasks is an anti-pattern.

[rarkins]

FWIW I think we need to consider semantics similar to beforeAll and beforeEach, to differentiate commands needing to be run once even if there are multiple updates in the PR, versus ones to be run once per-upgrade

[TheKevJames]

Another potentially-related use-case here would be for Phoenix projects.

The default setup for Phoenix does not allow for Renovate to update your node_modules since some of your dependencies are generated files; you need the Phoenix buildchain to generate them before you can run npm install. See an example Renovate failure here.

Basically, that's because Phoenix always adds the following to your package.json:

  "dependencies": {
    "phoenix": "file:../deps/phoenix",
    "phoenix_html": "file:../deps/phoenix_html"
  },

and those phoenix and phoenix_html files are generated via mix commands.

In this case, we don't actually need Renovate to manage those dependencies, so being able to either remove these dependencies before renovate runs OR run the mix compilation would do the trick. To put that into psuedo-code, you could imagine making this work with one of the following two options:

• {"beforeAll": "npm uninstall --save phoenix phoenix_html", "afterAll": "npm install --save ../deps/phoenix ../deps/phoenix_html"} (actually, would this even work? Re-introducing the deps might not work without them existing... I suppose we could add a touch ../deps/phoenix ../deps/pheonix_html in there...)
• {"beforeAll": "mix local.hex; mix local.rebar; mix deps.get,deps.compile,compile"}

[Himani-relan]

We have another use-case in our project where some of the files are generated by openapigenerator( these are the files which aren't committed in git). renovate scans the repo and remove the dependencies from go.mod as it doesn't see any file which is using the dependency.
So we would need an option wherein openapigenerator could generate the files in project and then renovate can update the dependencies.

Is there any way we can achieve this now with current self-hosted configurations ?

[rarkins]

I can't think of any way to achieve it now

[Himani-relan]

I can't think of any way to achieve it now

So i tried with below configuration:
"allowedPostUpgradeCommands": ["^openapi-generator-cli .","go .","^test ."],
"postUpgradeTasks": {
"commands": ["test -f swagger-ui/openapi.json && openapi-generator-cli generate -i swagger-ui/openapi.json -g go-server -o src/ --package-name api ", "go mod tidy"],
"fileFilters": ["go.mod"],
"executionMode": "branch"
},

But still go.mod removes packages which are part of generated files.
Ideally with above configuration it should use updated go.mod and commit the same in PR.

Am i missing something here?

[anopheles]

@Himani-relan
postUpgradeTasks is executed after (post) the upgrade, i.e. the generated files are not considered during renovate's update operation.

I haven't found a way yet to solve the generation problem.

[PlusMinus0]

Practically the same usecase as @Himani-relan, we use ent as our entity relation framework and for go get not to fail, a call to go generate ./ent would be necessary.

[n0rad]

Same here, some assets files are embedded in the go binary by generating go source files with https://github.com/go-bindata/go-bindata.
go-bindata binary must be run prior resolving / fetching dependencies.

[knutwannheden]

We have another use-case for which we would need pre-upgrade tasks: We would like to be able to provide "migration scripts" based on OpenRewrite which modify the source code via AST. The migrations would be selected based on the source and target versions as determined by Renovate. These migrations need to be executed before Renovate updates the dependency, because otherwise the source may contain unresolved references in the code, which cannot be migrated.

[rarkins]

I'd like to hear more about this, because I'd always assumed migrations would occur after the dependency upgrade. For example let's say the new version renames an API function. Would you migrate the code to the new API function name before updating the dependency?

[knutwannheden]

Yes. The AST-based tools typically require a fully resolved AST to operate on, so that a referenced symbol can correctly be resolved while respecting overloading concepts the language may have. After the migration the references would resolve against some stubs.

As a workaround I will try to use the Git stash. Will that work?

[rarkins]

It might, but also don't forget you may need to run an install after stash and before migration?

[knutwannheden]

Not sure what the "install" is you are referring to. Naively I was expecting that I would only have to restore the changes made by Renovate using git stash pop at the end of my post-upgrade task.

[rarkins]

I assume you might be doing Java? But if it were JS for example then I was going to guess you'd need your node modules populated with the "old" dependencies before you ran the migration.

BTW I'm interested in incorporating OpenRewrite into Renovate natively so feel free to reach out to me separately on the topic so we don't flood this thread.

[r65535]

I'll copy in my use case, as I duplicated this issue! My team is running renovate against some golang repositories. We have set postUpdateOptions in renovate.json to run gomodtidy. Before this can be run successfully, we need to run some initialization commands to the code base.

Example:
We use swaggo/swag to generate a docs/ module which we don't commit to git. Renovate needs to run swag init ./... before updating dependencies.

[rarkins]

So would those commands be executed before we run any go commands to update the go.sum file

[r65535]

So would those commands be executed before we run any go commands to update the go.sum file

Yeah exactly - the swag init ./... command will generate a docs/ module that go will ensure exists before making any changes

[rarkins]

So it would be something like this?

swag init
go get
go mod tidy

[r65535]

Exactly :D

[RemiBou]

Found a workaround : manually git clone in the cache dir, then execute your command then renovate

current=$(pwd)
rm -rf ./baseDir
mkdir -p  ./baseDir/repos/bitbucket-server/myproject/myrepo
git clone ssh://mybitbucket.com/myproject/myrepo.git ./baseDir/repos/bitbucket-server/myproject/myrepo
cd ./baseDir/repos/bitbucket-server/myproject/myrepo
go generate ./...
cd $current
docker run --rm -v "$(pwd)/baseDir:/tmp/renovate" -v "$(pwd)/conf.js:/usr/src/app/config.js" renovate/renovate

And your configuration must have persistRepoData: true,

[codersasha]

We have another use case for this -- we don't check-in generated go protogen files, but they are needed to run go get -d -t ./.... They are not needed for building as we use bazel for building.

[blizzy78]

We have another use case for this -- we don't check-in generated go protogen files

Same when using sqlc.

[colinodell]

We have another use case for this -- we don't check-in generated go protogen files

Same when using sqlc.

Same when using wire which generates dependency injection code.

In my case, we have some dependencies listed in go.mod that are only referenced in wire-generated files. Having Renovate perform gomodtidy causes those (seemingly-unused) dependencies to get erased from go.mod. Being able to run a go generate command beforehand would avoid this problem.

[rarkins]

Let's expand this to be more generic and allow arbitrary tasks to be executed at any state where it's useful, e.g. postClone/preExtract, preUpgrade, postUpgrade, postPrCreation, etc.

[bobvandevijver]

Just to chime in: having a preUpgrade hook would most probably solve Yarn package updates when combined with Symfony UX packages. Currently, a package upgrade would fail due to a package not being installed as it comes from a composer dependency. Being able to run composer install before the upgrade would most probably solve this.

error Package "" refers to a non-existing file '"/cache/renovate/company/repos/gitlab/group/project/vendor/symfony/stimulus-bundle/assets"'.

[D1X7R4]

Adding postCommit/prePrCreation as well might be a +1. It will allow tools to modify CHANGELOG.md file based on commits.

[mlahargou]

I opened a discussion about this same problem before seeing this issue, sorry! I've since closed that discussion.

We use syncpack to enforce certain rules about dependencies in our monorepo. That has to run before calling pnpm install. A preUpgradeTasks hook would solve our issue as well.

[rarkins]

This feature has label status:requirements because it still needs a "design" of how the config would be structured, etc.