feat(datasource/custom): Support digest in custom datasource #26299

fstehle · 2023-12-14T16:24:24Z

Changes

To support digests a datasource has to implement the getDigest method.
This PR implements the method returning a null promise.
The value itself can be provided in the getReleases method.

Context

Documentation (please check one with an [x])

I have updated the documentation, or
No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

Code inspection only, or
Newly added/modified unit tests, or
No unit tests but ran on a real repository, or
Both unit tests + ran on a real repository

To support digests a datasource has to implement the getDigest method. The value itself can be provided in the getReleases method.

secustor

Not fond of this solution, tough a better fix would need bigger changes.

lib/workers/repository/process/lookup/index.spec.ts

Co-authored-by: Sebastian Poxhofer <secustor@users.noreply.github.com>

viceice · 2023-12-15T08:21:00Z

why do we need this? that function shouldn't be called when the digest is already returned in releases.

fstehle · 2023-12-15T08:21:41Z

Not fond of this solution, tough a better fix would need bigger changes.

Yes, absolutely. Would you suggest to modify supportsDigests in

renovate/lib/modules/datasource/index.ts

Line 392 in 6884d83

export function supportsDigests(datasource: string | undefined): boolean {

instead?

fstehle · 2023-12-15T08:23:37Z

why do we need this? that function shouldn't be called when the digest is already returned in releases.

That is not what I observed, supportsDigests returns false and no digest update is made.

See the following lines for that:

renovate/lib/modules/datasource/index.ts

Line 392 in 6884d83

export function supportsDigests(datasource: string | undefined): boolean {

)

renovate/lib/workers/repository/process/lookup/index.ts

Lines 417 to 545 in 6884d83

    
             if (supportsDigests(config.datasource)) { 
        
               if (config.currentDigest) { 
        
                 if (!config.digestOneAndOnly || !res.updates.length) { 
        
                   // digest update 
        
                   res.updates.push({ 
        
                     updateType: 'digest', 
        
                     newValue: compareValue, 
        
                   }); 
        
                 } 
        
               } else if (config.pinDigests) { 
        
                 // Create a pin only if one doesn't already exists 
        
                 if (!res.updates.some((update) => update.updateType === 'pin')) { 
        
                   // pin digest 
        
                   res.updates.push({ 
        
                     isPinDigest: true, 
        
                     updateType: 'pinDigest', 
        
                     // TODO #22198 
        
                     newValue: config.currentValue!, 
        
                   }); 
        
                 } 
        
               } 
        
               if (versioning.valueToVersion) { 
        
                 // TODO #22198 
        
                 res.currentVersion = versioning.valueToVersion(res.currentVersion!); 
        
                 for (const update of res.updates || /* istanbul ignore next*/ []) { 
        
                   // TODO #22198 
        
                   update.newVersion = versioning.valueToVersion(update.newVersion!); 
        
                 } 
        
               } 
        
               if (res.registryUrl) { 
        
                 config.registryUrls = [res.registryUrl]; 
        
               } 
        
               // massage versionCompatibility 
        
               if ( 
        
                 is.string(config.currentValue) && 
        
                 is.string(compareValue) && 
        
                 is.string(config.versionCompatibility) 
        
               ) { 
        
                 for (const update of res.updates) { 
        
                   logger.debug({ update }); 
        
                   if (is.string(config.currentValue) && is.string(update.newValue)) { 
        
                     update.newValue = config.currentValue.replace( 
        
                       compareValue, 
        
                       update.newValue, 
        
                     ); 
        
                   } 
        
                 } 
        
               } 
        
               // update digest for all 
        
               for (const update of res.updates) { 
        
                 if (config.pinDigests === true || config.currentDigest) { 
        
                   // TODO #22198 
        
                   update.newDigest ??= 
        
                     dependency?.releases.find((r) => r.version === update.newValue) 
        
                       ?.newDigest ?? (await getDigest(config, update.newValue))!; 
        
                   // If the digest could not be determined, report this as otherwise the 
        
                   // update will be omitted later on without notice. 
        
                   if (update.newDigest === null) { 
        
                     logger.debug( 
        
                       { 
        
                         packageName: config.packageName, 
        
                         currentValue: config.currentValue, 
        
                         datasource: config.datasource, 
        
                         newValue: update.newValue, 
        
                         bucket: update.bucket, 
        
                       }, 
        
                       'Could not determine new digest for update.', 
        
                     ); 
        
                     // Only report a warning if there is a current digest. 
        
                     // Context: https://github.com/renovatebot/renovate/pull/20175#discussion_r1102615059. 
        
                     if (config.currentDigest) { 
        
                       res.warnings.push({ 
        
                         message: `Could not determine new digest for update (${config.datasource} package ${config.packageName})`, 
        
                         topic: config.packageName, 
        
                       }); 
        
                     } 
        
                   } 
        
                 } else { 
        
                   delete update.newDigest; 
        
                 } 
        
                 if (update.newVersion) { 
        
                   const registryUrl = dependency?.releases?.find( 
        
                     (release) => release.version === update.newVersion, 
        
                   )?.registryUrl; 
        
                   if (registryUrl && registryUrl !== res.registryUrl) { 
        
                     update.registryUrl = registryUrl; 
        
                   } 
        
                 } 
        
               } 
        
             } 
        
             if (res.updates.length) { 
        
               delete res.skipReason; 
        
             } 
        
             // Strip out any non-changed ones 
        
             res.updates = res.updates 
        
               .filter( 
        
                 (update) => update.newValue !== null || config.currentValue === null, 
        
               ) 
        
               .filter((update) => update.newDigest !== null) 
        
               .filter( 
        
                 (update) => 
        
                   (is.string(update.newName) && 
        
                     update.newName !== config.packageName) || 
        
                   update.isReplacement === true || 
        
                   update.newValue !== config.currentValue || 
        
                   update.isLockfileUpdate === true || 
        
                   // TODO #22198 
        
                   (update.newDigest && 
        
                     !update.newDigest.startsWith(config.currentDigest!)), 
        
               ); 
        
             // If range strategy specified in config is 'in-range-only', also strip out updates where currentValue !== newValue 
        
             if (config.rangeStrategy === 'in-range-only') { 
        
               res.updates = res.updates.filter( 
        
                 (update) => update.newValue === config.currentValue, 
        
               ); 
        
             } 
        
             // Handle a weird edge case involving followTag and fallbacks 
        
             if (config.rollbackPrs && config.followTag) { 
        
               res.updates = res.updates.filter( 
        
                 (update) => 
        
                   res.updates.length === 1 || 
        
                   /* istanbul ignore next */ update.updateType !== 'rollback', 
        
               ); 
        
             } 
        
           } catch (err) /* istanbul ignore next */ {

lib/modules/datasource/custom/index.ts

secustor · 2023-12-17T23:24:40Z

lib/workers/repository/process/lookup/__snapshots__/index.spec.ts.snap

The snapshot is with the latest change obsolete.
You can remove it with pnpm run jest -u

secustor · 2023-12-17T23:32:18Z

Not fond of this solution, tough a better fix would need bigger changes.

Yes, absolutely. Would you suggest to modify supportsDigests in

renovate/lib/modules/datasource/index.ts

Line 392 in 6884d83

export function supportsDigests(datasource: string | undefined): boolean {

instead?

The logic in lookup/index.ts would need to modified, which needs extensive testing on real repos.

Support digets for custom datasource. The getDigest method needs to exist on the custom datasource for renovate to use the digest provided from the main versions/releases endpoint. If the versions response always includes the digest, the getDigest method doesn't actually need to be implemented. https://github.com/renovatebot/renovate/blob/14b67888307e2a83219e4bb90f7f0f71f5e608bd/lib/workers/repository/process/lookup/index.ts#L446 See renovatebot#26299

rarkins

Needs deconflicting and continuing

Support digets for custom datasource. The getDigest method needs to exist on the custom datasource for renovate to use the digest provided from the main versions/releases endpoint. If the versions response always includes the digest, the getDigest method doesn't actually need to be implemented. https://github.com/renovatebot/renovate/blob/14b67888307e2a83219e4bb90f7f0f71f5e608bd/lib/workers/repository/process/lookup/index.ts#L446 See renovatebot#26299

acortelyou · 2024-04-29T21:21:51Z

This PR would resolve request for help discussion #28343.

I hope we can find a way to get this code moving again.

It looks like the suggestions from the review were just nits: if @fstehle doesn't have bandwidth to revisit, perhaps the maintainers can update the PR with their fixes or an interested party (myself) could fork the PR.

rarkins · 2024-04-30T04:06:18Z

The latter

fstehle · 2024-04-30T09:51:17Z

Currently quite busy, would be great if you could create a fork PR

Co-authored-by: Sebastian Poxhofer <secustor@users.noreply.github.com>

# Conflicts: # lib/workers/repository/process/lookup/index.spec.ts

rarkins

Fixed conflicts, but now I defer to @secustor

viceice · 2024-04-30T13:30:46Z

lib/modules/datasource/custom/index.ts

+    newValue?: string,
+  ): Promise<string | null> {
+    // Return null here to support setting a digest: value can be provided digest in getReleases
+    // istanbul ignore next


should be removed, it seems there's a test for it

secustor

Needs test fix and removal of ignore statement, else LGTM

acortelyou · 2024-04-30T23:18:18Z

See linked PR for commits to address outstanding items.

Either cherry-pick the commits on to here (i don't have permissions) or publish that one and merge from there.

Thanks!

acortelyou · 2024-04-30T16:59:47Z

lib/workers/repository/process/lookup/index.spec.ts

+      const res = await lookup.lookupUpdates(config);
+      expect(res).toMatchObject({
+        updates: [
+          {
+            newDigest: '0123456789abcdef',
+            newValue: '1.0.0',
+            updateType: 'digest',
+          },
+        ],
+      });


Suggested change

const res = await lookup.lookupUpdates(config);

expect(res).toMatchObject({

updates: [

{

newDigest: '0123456789abcdef',

newValue: '1.0.0',

updateType: 'digest',

},

],

});

const { updates } = await Result.wrap(

lookup.lookupUpdates(config),

).unwrapOrThrow();

expect(updates).toEqual([

{

newDigest: '0123456789abcdef',

newValue: '1.0.0',

updateType: 'digest',

},

]);

lib/modules/datasource/custom/index.ts

+    newValue?: string,
+  ): Promise<string | null> {
+    // Return null here to support setting a digest: value can be provided digest in getReleases
+    // istanbul ignore next


acortelyou · 2024-04-30T23:19:24Z

lib/modules/datasource/custom/index.ts

+    newValue?: string,
+  ): Promise<string | null> {
+    // Return null here to support setting a digest: value can be provided digest in getReleases
+    // istanbul ignore next


Suggested change

// istanbul ignore next

ivankatliarchuk · 2024-05-01T07:45:59Z

Does anyone know whether or not these changes will support Gitlab MD5 https://gitlab-docker-machine-downloads.s3.amazonaws.com/v0.16.2-gitlab.25/index.html?

secustor · 2024-05-01T08:02:56Z

Does anyone know whether or not these changes will support Gitlab MD5 https://gitlab-docker-machine-downloads.s3.amazonaws.com/v0.16.2-gitlab.25/index.html?

No.

Closing as this has been implemented in the linked PR.

fstehle changed the title ~~Support digest in custom datasource~~ fix(datasource/custom): Support digest in custom datasource Dec 14, 2023

fstehle force-pushed the custom_datasource_digest_support branch from c470002 to dfc34f7 Compare December 14, 2023 16:34

Support digest in custom datasource

08edb90

To support digests a datasource has to implement the getDigest method. The value itself can be provided in the getReleases method.

fstehle force-pushed the custom_datasource_digest_support branch from dfc34f7 to 08edb90 Compare December 14, 2023 16:34

fstehle marked this pull request as ready for review December 14, 2023 16:34

rarkins requested a review from secustor December 14, 2023 16:40

Add test for getDigest in custom datasource

e04f88f

secustor requested changes Dec 15, 2023

View reviewed changes

lib/workers/repository/process/lookup/index.spec.ts Outdated Show resolved Hide resolved

Use toMatchObject

3f379ba

Co-authored-by: Sebastian Poxhofer <secustor@users.noreply.github.com>

fstehle requested a review from secustor December 15, 2023 08:23

secustor requested changes Dec 17, 2023

View reviewed changes

rarkins requested changes Mar 18, 2024

View reviewed changes

rarkins and others added 3 commits April 30, 2024 12:30

Update lib/modules/datasource/custom/index.ts

0eabe34

Co-authored-by: Sebastian Poxhofer <secustor@users.noreply.github.com>

remove snapshot

a318a24

Merge branch 'main' into custom_datasource_digest_support

3e55527

# Conflicts: # lib/workers/repository/process/lookup/index.spec.ts

rarkins requested a review from secustor April 30, 2024 10:35

rarkins reviewed Apr 30, 2024

View reviewed changes

prettier fix

981f0af

viceice reviewed Apr 30, 2024

View reviewed changes

viceice changed the title ~~fix(datasource/custom): Support digest in custom datasource~~ feat(datasource/custom): Support digest in custom datasource Apr 30, 2024

secustor reviewed Apr 30, 2024

View reviewed changes

acortelyou mentioned this pull request Apr 30, 2024

feat(datasource/custom): Support digest in custom datasource #28760

Merged

acortelyou approved these changes Apr 30, 2024

View reviewed changes

secustor closed this May 1, 2024

github-actions bot locked as resolved and limited conversation to collaborators Jun 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(datasource/custom): Support digest in custom datasource #26299

feat(datasource/custom): Support digest in custom datasource #26299

fstehle commented Dec 14, 2023 •

edited

Loading

secustor left a comment

viceice commented Dec 15, 2023

fstehle commented Dec 15, 2023

fstehle commented Dec 15, 2023

secustor Dec 17, 2023

secustor commented Dec 17, 2023

rarkins left a comment

acortelyou commented Apr 29, 2024

rarkins commented Apr 30, 2024

fstehle commented Apr 30, 2024

rarkins left a comment

viceice Apr 30, 2024

This comment was marked as outdated.

secustor left a comment

acortelyou commented Apr 30, 2024

acortelyou Apr 30, 2024

This comment was marked as outdated.

acortelyou Apr 30, 2024

ivankatliarchuk commented May 1, 2024

secustor commented May 1, 2024

feat(datasource/custom): Support digest in custom datasource #26299

feat(datasource/custom): Support digest in custom datasource #26299

Conversation

fstehle commented Dec 14, 2023 • edited Loading

Changes

Context

Documentation (please check one with an [x])

How I've tested my work (please select one)

secustor left a comment

Choose a reason for hiding this comment

viceice commented Dec 15, 2023

fstehle commented Dec 15, 2023

fstehle commented Dec 15, 2023

secustor Dec 17, 2023

Choose a reason for hiding this comment

secustor commented Dec 17, 2023

rarkins left a comment

Choose a reason for hiding this comment

acortelyou commented Apr 29, 2024

rarkins commented Apr 30, 2024

fstehle commented Apr 30, 2024

rarkins left a comment

Choose a reason for hiding this comment

viceice Apr 30, 2024

Choose a reason for hiding this comment

This comment was marked as outdated.

secustor left a comment

Choose a reason for hiding this comment

acortelyou commented Apr 30, 2024

acortelyou Apr 30, 2024

Choose a reason for hiding this comment

This comment was marked as outdated.

acortelyou Apr 30, 2024

Choose a reason for hiding this comment

ivankatliarchuk commented May 1, 2024

secustor commented May 1, 2024

fstehle commented Dec 14, 2023 •

edited

Loading