feat(datasource/custom): add ability for local registry (file://)

[nSimonFR]

Context

Renovate already allows to use your own dependencies from any HTTP server => https://docs.renovatebot.com/modules/datasource/custom/#custom-offline-dependencies

But sometimes, you may not want to expose your dependencies, or those may be local from your current repository, thus, the need to use local files as registry.

This allows a setup as such:

• renovate can manage dependencies from "remote" providers
• and use a "local" provider - for example to enforce the alignment of dependencies between two similar files and a regex matcher / custom local datasource

My organization especially needs this to "buffer" dependencies between two different environments (Our usecase is kubernetes manifests for [pre-]prod on the same main branch), this will allow us to use renovate as an automated way to manage our envs - without exposing our services versions through the internet.

This can be seen as an alternative to #23410 - a simpler version without the limitation for self-hosting renovate - especially useful for usecases such as #23410 (comment).

Changes

• Refactor json format and move it to format folder
• Add a read method to all format files that parses through filesystem
• Extend CustomDatasourceConfig use file://
• Extend datasource tests
• Extend custom datasource docs

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests but ran on a real repository, or
• Both unit tests + ran on a real repository

[secustor]

Please do not force push

[viceice]

@rarkins what's your opinion about this idea?

[daveconde]

This would be very useful for the case of GitOps promotions where we'd like to be able to force specific versions are picked up.

[daveconde]

Is there anything I can do to help get this merged?

[rarkins]

Does this support reading both local files (within the repo) as well as absolute files off the operating system, or just within the repo?

If it's within the repo, what's the use case here? e.g. first PR updates some file1 and is merged, then a second PR updating file2 based off file1's contents is made using this capability? And in the meantime file1 and file2 are "out of sync" on the main branch?

[nSimonFR]

Does this support reading both local files (within the repo) as well as absolute files off the operating system, or just within the repo?

I've never used renovate self-hosted, but the implementation uses the readLocalFile from lib and takes any path, so I would assume, both ?

If it's within the repo, what's the use case here? e.g. first PR updates some file1 and is merged, then a second PR updating file2 based off file1's contents is made using this capability? And in the meantime file1 and file2 are "out of sync" on the main branch?

Yes, this is indeed the exact usecase I am trying to emulate, which would allow renovate to manage (and automate) our "pre-production" and "production" gitops manifests in the same branches.

I thought about using a custom GHA for this but it seemed like extending renovate custom datasource was appropriate too.
If there are other ways to accomplish such a thing with renovate, I'd gladly take it too !

[rarkins]

I'm ok with this being for local files in the repo

[viceice]

readLocalFile only allows files from repo, it throws errors when trying to read outside of the repo.

[secustor]

I will look into moving this to a class based approach after the merge of this PR

resolved

[renovate-release]

🎉 This PR is included in version 37.67.0 🎉

The release is available on:

• GitHub release
• 37.67.0

Your semantic-release bot 📦🚀