Configure Maven Private Artifactory by default using config.js on self hosted

[erzz]

I am really struggling to get this work having tried dozens of permutations!

Simple requirements:

• If the package begins com.myorg then use our artifactory maven repo https://artifactory.myorg.com:443/artifactory/projectname-maven-virtual
• Else use default maven.org etc

We are given url / credentials etc:

repo: https://artifactory.myorg.com:443/artifactory/projectname-virtual
username: xyz
password: a.JWT.token (an access token in Artifactory parlance I believe)

Testing with PAW / Curl

• Using curl/PAW I can get a 200 response using Authorization: Bearer a.JWT.token header
• Using curl/PAW I can get a 200 response using basic auth -u username:a.JWT.token

Configuring the following options in the config.js I get only 401's when renovate-runner tries to get the dependencies in artifactory (and then it falls back to maven.org as expected - but misses entirely our private dependencies of course)

config.js using basic auth

module.exports = {
  trustLevel: "high",
  hostRules: [
    {
      hostType: "maven",
      domainName: "artifactory.myorg.com",
      authType: "Basic",
      username: process.env.ARTIFACTORY_USER,
      password: process.env.ARTIFACTORY_JWT
    },
  ]
};

config.js using token

module.exports = {
  trustLevel: "high",
  hostRules: [
    {
      hostType: "maven",
      domainName: "artifactory.myorg.com",
      authType: "Bearer",
      token: process.env.ARTIFACTORY_JWT
    },
  ]
};

I have also experimented with and without package rules such as

packageRules: [
           {
             managers: ["maven"],
             registryUrls: [
               "https://artifactory.myorg.com:443/artifactory/projectname-virtual",
             ],
           },
         ],

ARTIFACTORY_USER and ARTIFACTORY_JWT are CICD variables that I can see stamped out in the debug log so I know that they are accessible and expanded

e.g.

DEBUG: Checking for config file in /builds/retail-pim/supporting-infrastructure/renovate-bot/module.exports = {
         hostRules: [
           {
             hostType: "maven",
             hostName: "artifactory.myorg.com",
             authType: "Bearer",
             token:
               "[MASKED]",
           },
         ],
         packageRules: [
           {
             managers: ["maven"],
             registryUrls: [
               "https://artifactory.myorg.com:443/artifactory/projectname-virtual",
             ],
           },
         ],
       };

And a few dozen variations :)

Can anyone help me out? Am I doing it totally wrong? Is it even possible to do it this way via config.js for all projects?

[rarkins]

• Using curl/PAW I can get a 200 response using Authorization: Bearer a.JWT.token header

If this is the case then the simplest hostRules would be:

    {
      hostName: "artifactory.myorg.com",
      token: process.env.ARTIFACTORY_JWT
    }

Please note that artifactory.myorg.com is not a domain name (it's a subdomain/hostname), so that could be one reason of why things weren't working (there may be others). I've changed it to hostName in the above

[erzz]

Thanks for the answer - still no luck though I am afraid. If I take the debug logs for a single specific package

DEBUG: Looking up com.github.spotbugs:spotbugs-annotations in repository https://repo.maven.apache.org/maven2/ (repository=groupname/projectname)

DEBUG: Found 42 new releases for com.github.spotbugs:spotbugs-annotations in repository https://repo.maven.apache.org/maven2/ (repository=groupname/projectname)

DEBUG: Looking up com.github.spotbugs:spotbugs-annotations in repository https://artifactory.myorg.com:443/artifactory/projectname/ (repository=groupname/projectname)

DEBUG: Looking up com.github.spotbugs:spotbugs-annotations in repository https://repo.spring.io/milestone/ (repository=groupname/projectname)

DEBUG: Dependency lookup unauthorized. Please add authentication with a hostRule (repository=groupname/projectname)
       "failedUrl": "https://artifactory.myorg.com/artifactory/projectname-virtual/com/github/spotbugs/spotbugs-annotations/maven-metadata.xml"

DEBUG: Content is not found for Maven url: https://artifactory.myorg.com/artifactory/rpim-maven-virtual/com/github/spotbugs/spotbugs-annotations/maven-metadata.xml (repository=groupname/projectname)

DEBUG: Looking up com.github.spotbugs:spotbugs-annotations in repository https://repo.spring.io/milestone/ (repository=groupname/projectname)

DEBUG: Content is not found for Maven url: https://repo.spring.io/milestone/com/github/spotbugs/spotbugs-annotations/maven-metadata.xml (repository=groupname/projectname)

So it is attempting at least - and attempting the correct path - but still a 401

{
                 "datasource": "maven",
                 "depName": "com.github.spotbugs:spotbugs-annotations",
                 "currentValue": "4.2.2",
                 "fileReplacePosition": 5682,
                 "registryUrls": [
                   "https://repo.maven.apache.org/maven2",
                   "https://artifactory.myorg.com:443/artifactory/projectname-virtual",
                   "https://repo.spring.io/milestone"
                 ],
                 "depIndex": 8,
                 "updates": [],
                 "warnings": [],
                 "sourceUrl": "https://github.com/spotbugs/spotbugs",
                 "homepage": "https://spotbugs.github.io/",
                 "fixedVersion": "4.2.2"
               },

Current config is simply:

module.exports = {
  trustLevel: "high",
  hostRules: [
    {
      hostName: "artifactory.myorg.com",
      token: process.env.ARTIFACTORY_JWT
    },
  ]
};

[erzz]

I don't know if its possibly a reason for it - but when running in PAW for example the initial response is in fact a 302 and I need to specifically enable to follow links and Redirect Authorization

Not sure if that is handled in Renovate? If I disable the redirction of auth then I can recreate the 401

[rarkins]

yes, maybe got does see a new host because of the port change `` -> 443 and removed the auth header

Possibly when following a redirect like this:

• got removes the existing authorization header
• Our auth logic is not reapplied because got handles it internally

We could work around it by handling redirects ourselves in the datasource layer but it could be a lot of work.

[viceice]

I think we should start to reproduce in our test cases before doing any bigger refactoring

[viceice]

ok, added simple test case, but cant reproduce:

renovate/lib/datasource/maven/index.spec.ts

Lines 336 to 392 in e4e9332

	it('should redirect with authentication header', async () => {
	nock.cleanAll();
	hostRules.add({
	baseUrl:
	'https://artifactory.myorg.com/artifactory/projectname-virtual',
	token: 'ABC',
	});
	httpMock
	.scope(
	'https://artifactory.myorg.com:443/artifactory/projectname-virtual',
	{
	reqheaders: { authorization: 'Bearer ABC' },
	}
	)
	.get('/com/github/spotbugs/spotbugs-annotations/maven-metadata.xml')
	.reply(
	200,
	`<?xml version="1.0" encoding="UTF-8"?><metadata>
	<groupId>com.github.spotbugs</groupId>
	<artifactId>spotbugs-annotations</artifactId>
	<version>4.2.2</version>
	<versioning>
	<versions>
	<version>4.2.2</version>
	<version>4.2.1</version>
	</versions>
	<lastUpdated>20130301200000</lastUpdated>
	</versioning>
	</metadata>`
	)
	.head(
	'/com/github/spotbugs/spotbugs-annotations/4.2.2/spotbugs-annotations-4.2.2.pom'
	)
	.reply(200)
	.get(
	'/com/github/spotbugs/spotbugs-annotations/4.2.2/spotbugs-annotations-4.2.2.pom'
	)
	.reply(200)
	.head(
	'/com/github/spotbugs/spotbugs-annotations/4.2.1/spotbugs-annotations-4.2.1.pom'
	)
	.reply(404);
	const releases = await getPkgReleases({
	...config,
	depName: 'com.github.spotbugs:spotbugs-annotations',
	registryUrls: [
	'https://artifactory.myorg.com:443/artifactory/projectname-virtual',
	],
	});
	expect(httpMock.getTrace()).toMatchSnapshot();
	expect(releases).not.toBeNull();
	expect(releases.releases).toEqual(generateReleases(['4.2.2']));
	});
	});
	});

So renovate is doing the following requests:

renovate/lib/datasource/maven/__snapshots__/index.spec.ts.snap

Lines 3 to 46 in e4e9332

	exports[`datasource/maven getReleases should redirect with authentication header 1`] = `
	Array [
	Object {
	"headers": Object {
	"accept-encoding": "gzip, deflate",
	"authorization": "Bearer ABC",
	"host": "artifactory.myorg.com",
	"user-agent": "https://github.com/renovatebot/renovate",
	},
	"method": "GET",
	"url": "https://artifactory.myorg.com/artifactory/projectname-virtual/com/github/spotbugs/spotbugs-annotations/maven-metadata.xml",
	},
	Object {
	"headers": Object {
	"accept-encoding": "gzip, deflate",
	"authorization": "Bearer ABC",
	"host": "artifactory.myorg.com",
	"user-agent": "https://github.com/renovatebot/renovate",
	},
	"method": "HEAD",
	"url": "https://artifactory.myorg.com/artifactory/projectname-virtual/com/github/spotbugs/spotbugs-annotations/4.2.2/spotbugs-annotations-4.2.2.pom",
	},
	Object {
	"headers": Object {
	"accept-encoding": "gzip, deflate",
	"authorization": "Bearer ABC",
	"host": "artifactory.myorg.com",
	"user-agent": "https://github.com/renovatebot/renovate",
	},
	"method": "HEAD",
	"url": "https://artifactory.myorg.com/artifactory/projectname-virtual/com/github/spotbugs/spotbugs-annotations/4.2.1/spotbugs-annotations-4.2.1.pom",
	},
	Object {
	"headers": Object {
	"accept-encoding": "gzip, deflate",
	"authorization": "Bearer ABC",
	"host": "artifactory.myorg.com",
	"user-agent": "https://github.com/renovatebot/renovate",
	},
	"method": "GET",
	"url": "https://artifactory.myorg.com/artifactory/projectname-virtual/com/github/spotbugs/spotbugs-annotations/4.2.2/spotbugs-annotations-4.2.2.pom",
	},
	]
	`;

@erzz Can you curl those urls to check if there is any redirect on them?

[erzz]

To save the sanitisation and possibly omitting too much - I emailed some curls directly to you @rarkins

Summary being the 302 is only for the repo root https://artifactory.myorg.com/artifactory/projectname-virtual and everything else is a direct response

[viceice]

As I know renovate does not call the virtual repo root, only matadata.xml and pom.

[rarkins]

Quick question: which type and version of Renovate are you running? e.g. npm CLI, Docker CLI, or WhiteSource Renovate On-Premises? I want to check that you've got a very recent version in use

[erzz]

Morning!

We are using the gitlab-runner.

Current .gitlab-ci.yml

include:
  - project: 'renovate-bot/renovate-runner'
    file: '/templates/renovate.gitlab-ci.yml'

renovate:
  variables:
    RENOVATE_EXTRA_FLAGS: <list of projects>
    RENOVATE_ONBOARDING: 'true'
    RENOVATE_ONBOARDING_CONFIG: '{"$$schema": "https://docs.renovatebot.com/renovate-schema.json", "extends": ["config:base", ":semanticCommits"], "branchPrefix": "rv8/", "prConcurrentLimit": 3, "branchConcurrentLimit": 3, "hashedBranchLength": 20 }'
    RENOVATE_CONFIG_FILE: $CI_PROJECT_DIR/config.js
    LOG_LEVEL: debug

[viceice]

Why you don't use the dind version? it should be faster, because it doesn't have to download the big fat renovate full image. It's instead a smaller slim image and docker sidecar container for other tools, which are downloaded on demand.

[erzz]

As we run our own gitlab runners in our own GKE cluster, we avoid DinD and privileged pods!

[erzz]

But Good News!!! I had sent a mail to artifactory team and explained the issue - it seems like they have made a tweak to the HA config that has perhaps resolved the problem!

I am still testing but first things look good - I have already asked them for the specifics of what they changed so that I can report back here