Artifact download with basic authentication #8853
Comments
|
Hi @orcuny! What you've got here should be working for sure. Can you confirm that you're URL encoding the auth string (see https://github.com/hashicorp/go-getter#basic-authentication) |
|
Hi @tgross. Actually, my example includes the values I used for my tests. Username is downloader and password is download1. So, I suppose it should be about something different. A couple of minutes ago, I came across a similar case with ansible get_url module. Ansible get_url also got exception with the url and the same username as url_username and the same password as url_password. Then, I noticed the force_basic_auth parameter which resolved the issue with get_url. Description of that parameter is: Force the sending of the Basic authentication header upon initial request. httplib2, the library used by the uri module only sends authentication information when a webservice responds to an initial request with a 401 status. Since some basic auth services do not properly send a 401, logins will fail. I don't know if it helps but can it be some kind of hint or sth? |
|
Ok, thanks for that extra info. Let me see if I can make a minimal reproduction for investigation. I'm hoping it's not a bug in go-getter itself but if so, we'll just have to fix that in our upstream as well. |
|
Hi @orcuny I tried to make a minimal reproduction and it seems to work as I'd expect. So there might be some element we're missing here. Is there a particular artifact server implementation (ex. is this Nginx? Artifactory?) you've got that I might be able to try? bottle.py artifact server with HTTP Basic Authfrom bottle import auth_basic, route, run, static_file
def check(user, pw):
if user != "user":
return False
if pw != "password1":
return False
return True
@route('/')
@auth_basic(check)
def index():
return static_file("text.txt", root="./")
run(host='localhost', port=8080)jobspecJobfile: job "example" {
datacenters = ["dc1"]
group "box" {
task "box" {
driver = "docker"
config {
image = "busybox:1"
command = "/bin/sh"
args = ["-c", "cat /local/result.txt; sleep 300"]
}
artifact {
source = "http://user:password1@localhost:8080"
destination = "local/result.txt"
}
resources {
cpu = 256
memory = 128
}
}
}
}Run Artifact server logs: |
|
Hi @tgross. Environment I'm working on is Linux amd with sonatype/nexus3:3.24.0 docker image as file server and Fabio LB. In the mean time, I tried with Nomad 0.10.5 and nexus3:3.26.1. Sadly, the problem persists. On the other hand, you might be on the correct path. There seems to be a 302 response followed by a redirection to the artifact. |
|
@orcuny I'm not at all familiar with sonatype and in trying to reproduce this I just crashed Sonatype a lot (?). Do you have a working Nomad job that can run sonatype (ideally with your fabio setup as well)? |
|
@tgross I’ll try to make you reproduce the issue. First, create the directory structure /opt/nexus/data/etc/ and place attached nexus.properties under /opt/nexus/data/etc/ with nexus host ip address corrected. Then, change ownership of /opt/nexus for 200:200 recursively. Volume definition in my client config is as follows: Execute attached nexus.nomad and it should run. You can find initial admin password in /opt/nexus/data/admin.password after some time. Login to admin console and disable anonymous access when prompted. Upload a jar file to maven-releases repo. I also attached an example maven project. Coordinates requested should match pom.xml definitions. You can upload the jar with spring-boot as classifier. So, classifier is spring-boot and extension is jar. Finally, execute test.nomad and it will fail with 401. My fabio config is a little complicated with TLS. I attached the skeleton parts, but I suppose any fabio config will work. Thanks in advance. nexus.properties nexus.nomad test.nomad fabio.nomad pom.xml SleeperApplication.java curl |
|
Hello again @orcuny. It took me a bit to get the sonatype container working because it has some weird requirements around volume permissions. But I wanted to make sure we were solving the right problem here, so I modified your setup to remove Fabio from the mix. I had to make a few modifications to get this all working, but I was finally able to reproduce the error. sonatype jobspecjob "nexus3" {
datacenters = ["dc1"]
type = "service"
group "nexus" {
network {
port "http" {
static = 8081
to = 8081
}
}
task "nexus" {
driver = "docker"
config {
image = "sonatype/nexus3:3.24.0"
ports = ["http"]
# note: we have to set permissions manually on the host because
# sonatype doesn't understand how users work in Docker
# chown -R 200 /srv/nexus
volumes = [
"/srv/nexus:/nexus-data"
]
}
resources {
cpu = "1048"
memory = "2048"
}
}
}
}java job that needs the artifactjob "example" {
datacenters = ["dc1"]
type = "batch"
group "test" {
task "test" {
driver = "java"
config {
jar_path = "/local/helloworld.jar"
jvm_options = ["-Xmx256m"]
}
# derived from:
# curl -L -u downloader:download1 \
# -o helloworld.jar \
# "http://10.0.2.15:8081/service/rest/v1/search/assets/download?group=org.example&name=helloworld&maven.baseVersion=1.0&sort=version"
artifact {
source = "http://downloader:download1@10.0.2.15:8081/service/rest/v1/search/assets/download"
options {
group = "org.example"
name = "helloworld"
maven.baseVersion = "1.0"
sort = "version"
}
}
resources {
cpu = 256
memory = 300
}
}
}
}I packet-captured the communication between Nomad and the Sonatype container (with This doesn't look like it's some hidden default behavior of golang. The following program returns the length in bytes of my artifact, as I'd expect. package main
import (
"fmt"
"io/ioutil"
"net/http"
)
func main() {
client := &http.Client{}
req, err := http.NewRequest("GET", "http://10.0.2.15:8081/service/rest/v1/search/assets/download?group=org.example&maven.baseVersion=1.0&name=helloworld&sort=version", nil)
req.SetBasicAuth("downloader", "download1")
resp, err := client.Do(req)
if err != nil {
panic(err)
}
body, err := ioutil.ReadAll(resp.Body)
defer resp.Body.Close()
fmt.Println(len(body))
}But using go-getter as vendored in Nomad results in the 401: package main
import (
"github.com/hashicorp/go-getter"
)
func main() {
src := "http://downloader:download1@10.0.2.15:8081/service/rest/v1/search/assets/download?group=org.example&maven.baseVersion=1.0&name=helloworld&sort=version"
dst := "./hello.jar"
client := &getter.Client{
Src: src,
Dst: dst,
}
err := client.Get()
if err != nil {
panic(err)
}
}A more recent version of go-getter fails in the same way. So as far as I can tell, the problem is in go-getter: we follow the redirect but for some reason don't pass along the authorization headers we'd expect to see there. I'll open an issue in the upstream repository and see what we can do to get that fixed soon. In the meantime, if it's at all possible I would recommend not using the Sonatype search interface, but get the well-known artifact path out of Sonatype and use that directly. |
tgross
added
stage/accepted
|
Hi @tgross Glad to see you reproduce the issue. Last comment is unfortunately not possible with Sonatype as it appends suffixes to snapshot artifacts in order not to override previous ones. Because of those suffixes, one cannot know the exact artifact paths. Search api is Sonatype's only way to find and return the most recent snapshot to client with that sort=version parameter. Looking forward to fix :-) |
|
Ok, after a bit more investigation and comparing the exact responses I captured, I've discovered this is actually an issue between Sonatype's redirect and golang's redirect handling behavior. When Nexus returns the redirect, it's returning the full URL, but with the authentication string stripped off. Which means that when the HTTP client follows the redirect, it no longer has the auth information. This is all happening "below" the go-getter library... and as far as I can tell this is the correct behavior in
(The relevant bit of the go stdlib: https://golang.org/src/net/http/client.go?s=1950:3998#L981) @orcuny this might provide a workaround option for you as well if you can configure the Nexus to provide a relative URL rather than a full URL in its redirects. Or if you can verify that the Nexus is providing the redirect to its DNS name and not the IP address. The golang program below exercises this in detail: package main
import (
"fmt"
"net/http"
getter "github.com/hashicorp/go-getter"
)
var redirect = ""
func search() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
username, password, ok := r.BasicAuth()
if username != "test" || password != "test" || !ok {
w.WriteHeader(http.StatusUnauthorized)
return
}
http.Redirect(w, r, redirect, http.StatusFound)
return
})
}
func artifact() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
username, password, ok := r.BasicAuth()
if username != "test" || password != "test" || !ok {
w.WriteHeader(http.StatusUnauthorized)
return
}
fmt.Fprintln(w, "Hello, World!")
})
}
func runServer() *http.Server {
router := http.NewServeMux()
router.Handle("/search", search())
router.Handle("/artifact", artifact())
srv := &http.Server{
Addr: ":8000",
Handler: router,
}
go srv.ListenAndServe()
return srv
}
func runTest() {
fmt.Printf("test with redirect: %q\n", redirect)
src := "http://test:test@localhost:8000/search"
client := &http.Client{}
req, err := http.NewRequest("GET", src, nil)
resp, err := client.Do(req)
if err != nil {
panic(err)
}
fmt.Printf(" golang HTTP client got HTTP status code: %v\n", resp.StatusCode)
getter := &getter.Client{
Src: src,
Dst: "./hello.jar",
}
err = getter.Get()
if err != nil {
fmt.Printf(" go-getter client got: %v\n", err)
} else {
fmt.Println(" go-getter worked!")
}
}
func main() {
srv := runServer()
defer srv.Close()
redirect = "http://localhost:8000/artifact"
runTest()
redirect = "/artifact"
runTest()
} |
tgross
added
stage/not-a-bug
and removed
stage/accepted
|
@orcuny at this point I'm going to close this issue, because the problem is with the Nexus configuration and its redirects. If you provide a packet capture of the unencrypted HTTP traffic between Fabio and Nexus I might be able to help you poke at the specific configuration problem, but this isn't really a Nomad issue at this point. |
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Hi,
I suppose root cause is something related to go-getter library, but in my case, as I can't make Nomad download artifacts, I'm posting the issue in Nomad list.
Summary: When remote file server needs basic authentication, artifact source as user:pass@url does not work as expected.
Nomad version:
0.11.4
Operating system and Environment details
Environment: Linux amd
File server: Nexus3 3.24.0
Issue
Without basic authentication, artifacts can be downloaded successfully.
When basic authentication is taken into action, artifact download fails with 401 Unauthorized.
I tested Nexus api with curl and when user:pass is given with --user option, it starts to download with no problem.
Reproduction steps
Run a file server with no anonymous user support.
Run nomad with go-getter's user:pass@url format.
Job file (if appropriate)
Nomad Client logs (if appropriate)
2020-09-09T11:14:58.369+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.artifacts: downloading artifact: alloc_id=1298bb82-2d4a-6af8-36c3-26d2e175a5b1 task=task artifact=https://downloader:download1@fabio.service.dc1.consul/nexus/service/rest/v1/search/assets/download
2020-09-09T11:14:58.550+0300 [DEBUG] client: updated allocations: index=21985 total=7 pulled=5 filtered=2
2020-09-09T11:14:58.550+0300 [DEBUG] client: allocation updates: added=0 removed=0 updated=5 ignored=2
2020-09-09T11:14:58.558+0300 [ERROR] client.alloc_runner.task_runner: prestart failed: alloc_id=1298bb82-2d4a-6af8-36c3-26d2e175a5b1 task=task error="prestart hook "artifacts" failed: failed to download artifact "https://downloader:download1@fabio.service.dc1.consul/nexus/service/rest/v1/search/assets/download": bad response code: 401"
Nomad Server logs (if appropriate)
The text was updated successfully, but these errors were encountered: