feat(datasource): add debian datasource

[oxdev03]

Changes

• Built upon @Ka0o0 implementation of the Debian datasource.
• Includes minor improvements.
• Addresses coverage issues identified in the previous review of the Debian datasource.
• Fixes earlier review findings (except the caching issue).

Context

Resolves: #7041, #24906

Concept

Based on the discussion and the reused implementation (see the related issue for more details):

1. Evaluation of Provided or Default Registry URL

   • Example: The URL https://ftp.debian.org/debian?release=bullseye&components=main,contrib&binaryArch=amd64 is evaluated to:
     • .../debian/dists/bullseye/main/binary-amd64
     • .../debian/dists/bullseye/contrib/binary-amd64
   • Supports multiple component URLs.

2. Check for Existing Compressed Package

   • If the compressed Packages.gz file has already been downloaded and extracted:
     • Send a HEAD request to check if the upstream file has been updated.
   • If not already downloaded:
     • Download the compressed file.
     • Extract the contents.
     • Delete the compressed file.
   • Currently, only gzip compression is supported.

3. Iterate through Package Index

   • Gather relevant information for the requested package.
   • Stop iteration, if found

4. Return relevant release information

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

included documentation for new datasource

How I've tested my work (please select one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests but ran on a real repository, or
• Both unit tests + ran on a real repository

Real Test

Open Points

• Does the current caching implementation for the datasource need to be reworked? Can the Renovate team provide a concept/details for this? (e.g., invalidation of the compressed file based on its timestamp)

• Is the current approach generic enough? Are there any points that were not considered?

• If significant rework is required, please let me know in advance.

[CLAassistant]

All committers have signed the CLA.

[secustor]

Thanks for your contribution, and the well documented functionality!

This is my first pass as there is a lot going on here and we have to be sure that the caching is setup efficiently.

Would this work for every APT repository?

[secustor]

Does the current caching implementation for the datasource need to be reworked? Can the Renovate team provide a concept/details for this? (e.g., invalidation of the compressed file based on its timestamp)

We should start simple and then start to optimize.
The Package.gz will be cached non transparently by the http layer, so the file will be downloaded only when necessary.
I suggest to simply cache the extracted file on a time base ( extract it to a different function and use the annotation ).

[oxdev03]

This is my first pass as there is a lot going on here and we have to be sure that the caching is setup efficiently.

Thx for the initial review, I will fix the findings soon.

Would this work for every APT repository?

should work with every apt repository, which follows the https://wiki.debian.org/DebianRepository standard.

[oxdev03]

This will merge results from different registries. Is it really what you want here? This is only used if there is a a second registry which has more metadata.

The registryStrategy=merge is required for the following use case: when there are multiple apt packages, some of which are only included in the remote repository and others in the internal Artifactory. With the current registryStrategy (first), it would require two separate configurations and additional labeling of internal dependencies

[oxdev03]

This will merge results from different registries. Is it really what you want here? This is only used if there is a a second registry which has more metadata.

The registryStrategy=merge is required for the following use case: when there are multiple apt packages, some of which are only included in the remote repository and others in the internal Artifactory. With the current registryStrategy (first), it would require two separate configurations and additional labeling of internal dependencies

Can we add this though or something speaking against it?

[secustor]

This will merge results from different registries. Is it really what you want here? This is only used if there is a a second registry which has more metadata.

The registryStrategy=merge is required for the following use case: when there are multiple apt packages, some of which are only included in the remote repository and others in the internal Artifactory. With the current registryStrategy (first), it would require two separate configurations and additional labeling of internal dependencies

Can we add this though or something speaking against

Add it, the only downside is for scenarios in which the metadata conflicts.
Though in that case we are not extracting much of them.

[viceice]

@zharinov should we enable the cachable property to true? so this call is also cached?

[oxdev03]

it is already cached in getPackageIndex, so it would be redundant

[zharinov]

The @cache decorator is the more functional alternative, e.g. you can set the TTL

[secustor]

We will get cache bust for each configuration option and their order in the registryUrl though.
The current implementation checks cache after breaking down this options.

[oxdev03]

I meant its already done here.

  @cache({
    namespace: `datasource-${DebDatasource.id}`,
    key: (componentUrl: string) => componentUrl,
  })
  async getPackageIndex(
    componentUrl: string,
  ): Promise<Record<string, PackageDescription[]>> {
    const { extractedFile, lastTimestamp } =
      await this.downloadAndExtractPackage(componentUrl);
    return await this.parseExtractedPackageIndex(extractedFile, lastTimestamp);
  }

No async operations happen in getReleases, if the cache of getPackageIndex is not stale.

but I still added this.

[oxdev03]

let me revert the change again, github seems bugging and not loading latest the conversations 😅.

[viceice]

Pleas add cache deocator to this function, so the aggregated result is cached too. (perf)

[viceice]

Pleas add cache deocator to this function, so the aggregated result is cached too. (perf)