Support private repository requiring authentication with Authorization http header in a more secure manner #80

guw · 2019-03-22T14:21:52Z

The current support for HTTP basic authentication relies on username:password@ URL schema.

There are few flaws with this approach:

Not ever repository server supports the username:password@ URL scheme. Some admins disable it intentionally for security reasons.
The username:password@ URL scheme requires persisting username and passwords in clear text in Bazel files, which are committed to SCM. This is almost impossible to deploy in any corporate environment.
- Credentials should be supplied using an external configuration file and/or other form of parameters (eg., secure store, environment variables, etc.).
- User need to be able to provide their own credentials at build time.

Bonus points:

Allow to provide URL overrides at build time, eg. use a local proxy/mirror instead of what is configured in the Bazel files.
Repository credentials are read from ~/.m2/settings.xml
Repository mirror urls are read from ~/.m2/settings.xml

The text was updated successfully, but these errors were encountered:

jin · 2019-03-22T21:59:08Z

Thanks for reporting this issue -- this is one of the areas that we want to tackle as soon as possible.

@dkelmer is looking into a better HTTP downloader mechanism in Bazel that works with more auth approaches, including the ones you've mentioned.

Currently, we rely on Coursier CLI's API, which only supports the URL scheme approach. To work around directly checking in usernames and passwords in your WORKSPACE, I thought of a few approaches:

Load strings from a .bzl file that's not checked in.

load("//secrets:maven.bzl", "USERNAME", "PASSWORD")
maven_install(
    # ...
    repositories = [
        "http://%s:%s@localhost:8081/artifactory/my-repository" % (USERNAME, PASSWORD),
    ],
)

Load strings from a remote Bazel WORKSPACE using http_archive/git_repository:

git_repository(
    name = "secrets_repository",
    remote = "git@githost.com:secrets/secrets.git",
    tag = "...",
)
load("@secrets_repository//secrets:maven.bzl", "USERNAME", "PASSWORD")
maven_install(
    # ...
    repositories = [
        "http://%s:%s@localhost:8081/artifactory/my-repository" % (USERNAME, PASSWORD),
    ],
)

I don't think http_archive supports authentication yet, but the git_repository rule uses system git, so it works with your regular ssh keys.

Load strings from another local (or mounted / authenticated) directory using local_repository. This is similar to the previous approach.

local_repository(
    name = "secrets_repository",
    path = "/path/to/secrets/",
)
load("@secrets_repository//secrets:maven.bzl", "USERNAME", "PASSWORD")
maven_install(
    # ...
    repositories = [
        "http://%s:%s@localhost:8081/artifactory/my-repository" % (USERNAME, PASSWORD),
    ],
)

Do any of these approaches work for you?

guw · 2019-03-22T22:08:20Z

Thanks for the feedback. I already thought of something like approach 1 as a workaround, i.e. have a Bazel file in .gitignore that will be included in the build. That should work.

Can you think of a solution that would work with .bazelrc-local?

Thanks for the update on the HTTP downloader feature request. Really appreciate it. I believe this will be the key because we really need http header auth for our repo servers.

jin · 2019-03-22T22:27:49Z

Can you think of a solution that would work with .bazelrc-local?

We can do something with environment variables.

$ MAVEN_USERNAME=foo MAVEN_PASSWORD=bar bazel build //...

and in the WORKSPACE file:

maven_install(
    # ...
    repositories = [
        "http://{MAVEN_USERNAME}:{MAVEN_PASSWORD}@localhost:8081/artifactory/my-repository",
    ],
)

Just before we invoke Coursier, we call .format() on the URLs.

This approach also works with .bazelrc / command line flags:

$ bazel build //... --action_env=MAVEN_USERNAME=foo --action_env=MAVEN_PASSWORD=bar

# Assuming MAVEN_USERNAME and MAVEN_PASSWORD are set
build --action_env=MAVEN_USERNAME
build --action_env=MAVEN_PASSWORD

Note that the --action_env approach will apply the env var to all your build actions.

What do you think about this approach?

jin · 2019-03-29T18:47:34Z

Looks like Coursier is reworking its credential handling upstream, with support for environment variables and reading from a properties file: coursier/coursier#1102

jin · 2019-04-02T18:27:50Z

Documentation: https://github.com/coursier/coursier/blob/0bccb51d3572f36fa664e825cb48e749c7ba1845/doc/docs/other-credentials.md

guw · 2019-04-15T06:41:14Z

Would either or both be a required dependency for fixing this one?

Support Authentication in jvm_maven_import_external/java_import_external bazelbuild/bazel#7443 (authentication in jvm_maven_import_external/java_import_external)
Support authentication for repository_ctx.download bazelbuild/bazel#7469 (authentication for repository_ctx.download)

jin · 2019-04-15T15:28:44Z

cc @dkelmer

Both jvm_maven_import_external and java_import_external delegates downloading to repository_ctx.download. Currently rules_jvm_external doesn't use repository_ctx.download, instead it delegates downloading to Coursier. To fix this issue, we will have to upstream more authentication avenues to Coursier, or implement bazelbuild/bazel#7469 and switch rules_jvm_external to use that.

guw · 2019-06-11T23:46:28Z

FWIW, I've created a local little proxy to deal with authenticated Maven repositories. It also supports streaming from the local .m2 Maven repo when an artifact is available, which will avoid server round trips completely.

https://github.com/guw/bazel-maven-proxy

KaoruDev · 2019-09-13T17:10:16Z

I have a Coursier properties file in: /Library/Preferences/Coursier/credentials.properties but still can't properly authenticate to our private repo. I'm confused, is Coursier looking elsewhere for this file?

jin · 2019-09-13T17:13:32Z

@KaoruDev I think that may be due to coursier/coursier#1319

puffnfresh · 2019-09-18T08:55:03Z

I had to add this:

myrepo.auto=true

Now my build seems to be getting closer to working.

KaoruDev · 2019-09-20T01:13:59Z

Seems coursier/coursier#1350 fixed the issue, looks like bumping to version RC-3-4 would pull in the fix.

puffnfresh · 2019-09-20T02:39:54Z

I got them to fully publish that version yesterday, it should be possible to bump it now.

KaoruDev · 2019-09-27T01:19:29Z

😭 Sorry doesn't seem to have fix the issue. I've tried:

via property
via ENV

puffnfresh · 2019-09-27T05:18:01Z

Mine is being found via ~/.config/coursier/credentials.properties on Linux.

KaoruDev · 2019-09-27T12:52:02Z

Maybe this is a MacOSx problem, I have a config file in ~/Library/Preferences/Coursier/credentials.properties and in ~/.config/coursier/credentials.properties (just in case)

jin · 2019-09-27T13:41:27Z

Reopening as issue is still reported on macOS.

whs · 2019-10-11T09:16:37Z

I've tried on macOS with credentials in ~/Library/Preferences/Coursier/credentials.properties with myrepo.auto=true as mentioned in above comment, I got past fetching to

ERROR: An error occurred during the fetch of repository 'maven':
local variable 'primary_artifact_path' is referenced before assignment.

jin · 2019-10-22T20:34:04Z

@whs, @puffnfresh provided a fix in #248. Could you please try pulling rules_jvm_external from the latest commit and give it a shot?

whs · 2019-10-30T12:43:05Z

Seems to be working for me. (with auto=true, I didn't test without auto=true) Thanks!

KaoruDev · 2019-10-30T20:49:38Z

sorry, I'm missing where to put auto=true?

whs · 2019-10-31T04:30:03Z

In credentials.properties, #80 (comment)

KaoruDev · 2019-10-31T14:09:17Z

Ah ok thanks @whs I thought this was a bazel config which confused me. As a note to my future self, I replaced myrepo in <myrepo>.auto=true to assign the value to the group. @jin feel free to close this out, thanks!

jin · 2019-10-31T18:53:32Z

Thanks for verifying, folks.

KaoruDev · 2019-12-02T23:47:52Z

@jin sorry I was too quick to declare this fixed and I now believe this is a rules_jvm_external issue.

What changed?

When I confirmed things were working, I just ran bazel run @unpinned_repo//:pin and did not see any errors. I didn't actually declare it a dependency in one of my targets.

Now that I'm bringing in a project that is using a private version, I encountered an Authentication issue when running sync via Intellij. I verified pulling down the artifact using the Coursier CLI works.

KaoruDev · 2019-12-03T11:13:53Z

Ok I was wrong ^ the issue I'm facing is not related to auth but rather the fact that the JAR i'm trying to pull down has additional letters to it. As an example: org.postgresql:postgresql:9.4-1201-jdbc41 whatever mechanism is used to copy over the jar that Coursier pulled down to be used as Bazel's cache is not working if the jar's version has extra data, will open new issue

ablyes · 2020-09-22T14:38:45Z

An error occurred during the fetch of reposit

I have a similar error, take a look please:

If the definition of 'toto' was updated, verify that the hashes were also updated.
ERROR: An error occurred during the fetch of repository 'toto':
   Traceback (most recent call last):
        File "/private/var/tmp/_bazel_lyes.abdelfettah/bdbe9cb9aff117eb863b0a1426519f31/external/rules_jvm_external/coursier.bzl", line 783, column 34, in _coursier_fetch_impl
                mirror_urls = [url + "/" + primary_artifact_path for url in repository_urls]
Error: unsupported binary operation: string + NoneType

I'm fighting with this.
There is no documentation at all !
I had first:

to export this var COURSIER_CREDENTIALS=/Users//Library/ApplicationSupport/Coursier/credentials.properties
to add the auto=true described

-> But i still get the above error.
What can I do ?

plaird · 2020-09-22T15:07:22Z

@ablyes does your user token or password contain a / character or @ character? I reported this in #414 and there is also #418 and #368. They are all the same issue, I believe.

ablyes · 2020-09-22T15:11:11Z

@ablyes does your user token or password contain a / character or @ character? I reported this in #414 and there is also #418 and #368. They are all the same issue, I believe.

The username contains @ :)
I will read the 3 topics, thank you.
Is there a solution right now ?

plaird · 2020-09-22T15:22:31Z

I describe a path to a workaround using a private fork in #414 but my fix was for / not @ case so my private fork workaround wouldn't work for you.

ablyes · 2020-09-22T19:14:20Z

Good job !
I pushed a pull request for my problem (@ in login).
But I forked and tagged a version if somebody else want to use my temporary fix.

guw mentioned this issue Mar 23, 2019

incompatible_remove_native_maven_jar bazelbuild/bazel#6799

Closed

jin mentioned this issue Mar 25, 2019

using maven_jar extension in windows and having ubuntu shell installed seems to be broken bazelbuild/bazel#7781

Closed

guw mentioned this issue Apr 15, 2019

cannot use bazel with protected nexus maven repository to fetch dependencies bazelbuild/bazel#5689

Closed

jin added the type: feature request label Apr 22, 2019

jin mentioned this issue Jul 18, 2019

Support auth from settings.xml bazelbuild/bazel#498

Closed

jin mentioned this issue Sep 20, 2019

Bump Coursier to 2.0.0-RC3-4 #249

Merged

jin closed this as completed in #249 Sep 20, 2019

jin reopened this Sep 27, 2019

jin closed this as completed Oct 31, 2019

waffel mentioned this issue Nov 12, 2019

maven repository credential settings does not work (or missing correct example) #289

Open

KaoruDev mentioned this issue Dec 3, 2019

Bazel using JAR Coursier downloads when jar version has extra metadata #302

Closed

gonzojive mentioned this issue May 30, 2020

Document how to work with private git repositories bazelbuild/bazel#11523

Open

thundergolfer mentioned this issue Apr 19, 2021

Pass --isolated to pip bazelbuild/rules_python#232

Merged

magiccrafter mentioned this issue Nov 5, 2021

Add http headers authentication support for private repositories #616

Open

dr-dime mentioned this issue Apr 13, 2022

java_export should not generate publisher script with credentials in it #679

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support private repository requiring authentication with Authorization http header in a more secure manner #80

Support private repository requiring authentication with Authorization http header in a more secure manner #80

guw commented Mar 22, 2019 •

edited

Loading

jin commented Mar 22, 2019 •

edited

Loading

guw commented Mar 22, 2019

jin commented Mar 22, 2019 •

edited

Loading

jin commented Mar 29, 2019

jin commented Apr 2, 2019

guw commented Apr 15, 2019

jin commented Apr 15, 2019 •

edited

Loading

guw commented Jun 11, 2019

KaoruDev commented Sep 13, 2019

jin commented Sep 13, 2019

puffnfresh commented Sep 18, 2019

KaoruDev commented Sep 20, 2019

puffnfresh commented Sep 20, 2019

KaoruDev commented Sep 27, 2019

puffnfresh commented Sep 27, 2019

KaoruDev commented Sep 27, 2019

jin commented Sep 27, 2019

whs commented Oct 11, 2019 •

edited

Loading

jin commented Oct 22, 2019 •

edited

Loading

whs commented Oct 30, 2019

KaoruDev commented Oct 30, 2019

whs commented Oct 31, 2019

KaoruDev commented Oct 31, 2019 •

edited

Loading

jin commented Oct 31, 2019

KaoruDev commented Dec 2, 2019

KaoruDev commented Dec 3, 2019

ablyes commented Sep 22, 2020 •

edited

Loading

plaird commented Sep 22, 2020

ablyes commented Sep 22, 2020

plaird commented Sep 22, 2020

ablyes commented Sep 22, 2020

Support private repository requiring authentication with Authorization http header in a more secure manner #80

Support private repository requiring authentication with Authorization http header in a more secure manner #80

Comments

guw commented Mar 22, 2019 • edited Loading

jin commented Mar 22, 2019 • edited Loading

guw commented Mar 22, 2019

jin commented Mar 22, 2019 • edited Loading

jin commented Mar 29, 2019

jin commented Apr 2, 2019

guw commented Apr 15, 2019

jin commented Apr 15, 2019 • edited Loading

guw commented Jun 11, 2019

KaoruDev commented Sep 13, 2019

jin commented Sep 13, 2019

puffnfresh commented Sep 18, 2019

KaoruDev commented Sep 20, 2019

puffnfresh commented Sep 20, 2019

KaoruDev commented Sep 27, 2019

puffnfresh commented Sep 27, 2019

KaoruDev commented Sep 27, 2019

jin commented Sep 27, 2019

whs commented Oct 11, 2019 • edited Loading

jin commented Oct 22, 2019 • edited Loading

whs commented Oct 30, 2019

KaoruDev commented Oct 30, 2019

whs commented Oct 31, 2019

KaoruDev commented Oct 31, 2019 • edited Loading

jin commented Oct 31, 2019

KaoruDev commented Dec 2, 2019

What changed?

KaoruDev commented Dec 3, 2019

ablyes commented Sep 22, 2020 • edited Loading

plaird commented Sep 22, 2020

ablyes commented Sep 22, 2020

plaird commented Sep 22, 2020

ablyes commented Sep 22, 2020

guw commented Mar 22, 2019 •

edited

Loading

jin commented Mar 22, 2019 •

edited

Loading

jin commented Mar 22, 2019 •

edited

Loading

jin commented Apr 15, 2019 •

edited

Loading

whs commented Oct 11, 2019 •

edited

Loading

jin commented Oct 22, 2019 •

edited

Loading

KaoruDev commented Oct 31, 2019 •

edited

Loading

ablyes commented Sep 22, 2020 •

edited

Loading