Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(datasource/custom): add ability for local registry (file://) #25734

Merged
merged 19 commits into from
Nov 24, 2023

Conversation

nSimonFR
Copy link
Contributor

@nSimonFR nSimonFR commented Nov 13, 2023

Context

Renovate already allows to use your own dependencies from any HTTP server => https://docs.renovatebot.com/modules/datasource/custom/#custom-offline-dependencies

But sometimes, you may not want to expose your dependencies, or those may be local from your current repository, thus, the need to use local files as registry.

This allows a setup as such:

  • renovate can manage dependencies from "remote" providers
  • and use a "local" provider - for example to enforce the alignment of dependencies between two similar files and a regex matcher / custom local datasource

My organization especially needs this to "buffer" dependencies between two different environments (Our usecase is kubernetes manifests for [pre-]prod on the same main branch), this will allow us to use renovate as an automated way to manage our envs - without exposing our services versions through the internet.

This can be seen as an alternative to #23410 - a simpler version without the limitation for self-hosting renovate - especially useful for usecases such as #23410 (comment).

Changes

  • Refactor json format and move it to format folder
  • Add a read method to all format files that parses through filesystem
  • Extend CustomDatasourceConfig use file://
  • Extend datasource tests
  • Extend custom datasource docs

Documentation (please check one with an [x])

  • I have updated the documentation, or
  • No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

  • Code inspection only, or
  • Newly added/modified unit tests, or
  • No unit tests but ran on a real repository, or
  • Both unit tests + ran on a real repository

@nSimonFR nSimonFR changed the title feat(datasource/custom): add defaultRegistryPathTemplate feat(datasource/custom): add isLocalRegistry Nov 13, 2023
@nSimonFR nSimonFR changed the title feat(datasource/custom): add isLocalRegistry feat(datasource/custom): add local registry Nov 13, 2023
@nSimonFR nSimonFR marked this pull request as ready for review November 13, 2023 20:21
lib/modules/datasource/custom/formats/json.ts Outdated Show resolved Hide resolved
lib/modules/datasource/custom/formats/plain.ts Outdated Show resolved Hide resolved
lib/modules/datasource/custom/formats/json.ts Outdated Show resolved Hide resolved
lib/config/types.ts Outdated Show resolved Hide resolved
@secustor
Copy link
Collaborator

Please do not force push

@nSimonFR nSimonFR changed the title feat(datasource/custom): add local registry feat(datasource/custom): add ability for local registry (file://) Nov 15, 2023
@viceice
Copy link
Member

viceice commented Nov 15, 2023

@rarkins what's your opinion about this idea?

@daveconde
Copy link

This would be very useful for the case of GitOps promotions where we'd like to be able to force specific versions are picked up.

@daveconde
Copy link

Is there anything I can do to help get this merged?

@rarkins
Copy link
Collaborator

rarkins commented Nov 22, 2023

Does this support reading both local files (within the repo) as well as absolute files off the operating system, or just within the repo?

If it's within the repo, what's the use case here? e.g. first PR updates some file1 and is merged, then a second PR updating file2 based off file1's contents is made using this capability? And in the meantime file1 and file2 are "out of sync" on the main branch?

@nSimonFR
Copy link
Contributor Author

Does this support reading both local files (within the repo) as well as absolute files off the operating system, or just within the repo?

I've never used renovate self-hosted, but the implementation uses the readLocalFile from lib and takes any path, so I would assume, both ?

If it's within the repo, what's the use case here? e.g. first PR updates some file1 and is merged, then a second PR updating file2 based off file1's contents is made using this capability? And in the meantime file1 and file2 are "out of sync" on the main branch?

Yes, this is indeed the exact usecase I am trying to emulate, which would allow renovate to manage (and automate) our "pre-production" and "production" gitops manifests in the same branches.

I thought about using a custom GHA for this but it seemed like extending renovate custom datasource was appropriate too.
If there are other ways to accomplish such a thing with renovate, I'd gladly take it too !

@rarkins
Copy link
Collaborator

rarkins commented Nov 23, 2023

I'm ok with this being for local files in the repo

@viceice
Copy link
Member

viceice commented Nov 23, 2023

readLocalFile only allows files from repo, it throws errors when trying to read outside of the repo.

Copy link
Collaborator

@secustor secustor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will look into moving this to a class based approach after the merge of this PR

@rarkins rarkins added this pull request to the merge queue Nov 24, 2023
Merged via the queue into renovatebot:main with commit f7b74c3 Nov 24, 2023
36 checks passed
@renovate-release
Copy link
Collaborator

🎉 This PR is included in version 37.67.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 25, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants