Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Client panic when submitting a job with exposed service check and no sidecar_service stanza #10115

Closed
krishicks opened this issue Mar 4, 2021 · 2 comments · Fixed by #10154
Closed

Client panic when submitting a job with exposed service check and no sidecar_service stanza #10115

krishicks opened this issue Mar 4, 2021 · 2 comments · Fixed by #10154
Assignees
@notnoop
Labels
theme/crash type/bug

Comments

@krishicks
Copy link
Contributor

krishicks commented Mar 4, 2021 •
edited

Nomad version

Nomad v1.0.4 (9294f35)

Operating system and Environment details

vagrant-nomad with v1.0.4

Issue

When submitting a job with a check that has expose = true, but no sidecar_service stanza, the client panics.

Reproduction steps

Submit an otherwise complete countdash.nomad, without the sidecar_service stanza on the count-api service.

Expected Result

A validation error of some kind is returned, because the service's check has expose = true, and that's invalid without a minimal connect stanza, meaning one that also has an empty sidecar_service stanza.

Actual Result

The CLI returns an EOF:

nomad job run countdash.nomad
Error submitting job: Put "http://10.199.0.10:4646/v1/jobs": EOF

If we submit an equivalent job via the HTTP API, we confirm the empty response:

curl --data @countdash.json http://10.199.0.10:4646/v1/jobs
curl: (52) Empty reply from server

Job file (if appropriate)

job "countdash" {
  datacenters = ["dc1"]

  group "api" {
    network {
      mode = "bridge"
    }

    service {
      name = "count-api"
      port = "9001"

      connect {
      #   sidecar_service {}
      }

      check {
        expose   = true
        type     = "http"
        name     = "api-health"
        path     = "/health"
        interval = "10s"
        timeout  = "3s"
      }
    }

    task "web" {
      driver = "docker"

      config {
        image = "hashicorpnomad/counter-api:v3"
      }
    }
  }

  group "dashboard" {
    network {
      mode = "bridge"

      port "http" {
        static = 9002
        to     = 9002
      }
    }

    service {
      name = "count-dashboard"
      port = "9002"

      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "count-api"
              local_bind_port  = 8080
            }
          }
        }
      }
    }

    task "dashboard" {
      driver = "docker"

      env {
        COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
      }

      config {
        image = "hashicorpnomad/counter-dashboard:v3"
      }
    }
  }
}

Nomad Server logs (if appropriate)

N/A

Nomad Client logs (if appropriate)

2021-03-04T03:31:06.128Z [DEBUG] http: request complete: method=PUT path=/v1/jobs duration=584.231µs
2021-03-04T03:31:06.128Z [ERROR] http: http: panic serving 10.199.0.1:39850: runtime error: invalid memory address or nil pointer dereference
goroutine 421158 [running]:
net/http.(*conn).serve.func1(0xc0009580a0)
	net/http/server.go:1801 +0x147
panic(0x2e148a0, 0x5116340)
	runtime/panic.go:975 +0x47a
github.com/hashicorp/nomad/nomad.serviceExposeConfig(...)
	github.com/hashicorp/nomad/nomad/job_endpoint_hook_expose_check.go:79
github.com/hashicorp/nomad/nomad.jobExposeCheckHook.Mutate(0xc00118a240, 0x0, 0x1000100, 0x0, 0xffffffffffffffff, 0xc00044c410, 0x35f3650)
	github.com/hashicorp/nomad/nomad/job_endpoint_hook_expose_check.go:30 +0x11c
github.com/hashicorp/nomad/nomad.(*Job).admissionMutators(0xc0003d81e0, 0xc00118a240, 0x0, 0xc00006e800, 0x0, 0x47251b, 0x269bb6bccfa6, 0xc007a94721)
	github.com/hashicorp/nomad/nomad/job_endpoint_hooks.go:61 +0xd1
github.com/hashicorp/nomad/nomad.(*Job).admissionControllers(0xc0003d81e0, 0xc00118a240, 0x7, 0x3a54301, 0xc000707620, 0x32be5c0, 0xc000707620, 0x2f9aae0)
	github.com/hashicorp/nomad/nomad/job_endpoint_hooks.go:43 +0x3f
github.com/hashicorp/nomad/nomad.(*Job).Register(0xc0003d81e0, 0xc000707620, 0xc001723270, 0x0, 0x0)
	github.com/hashicorp/nomad/nomad/job_endpoint.go:96 +0x3bd
reflect.Value.call(0xc000370e40, 0xc00012f558, 0x13, 0x34e043f, 0x4, 0xc000a1b5a8, 0x3, 0x3, 0xc001723270, 0xc000a1b600, ...)
	reflect/value.go:476 +0x8c7
reflect.Value.Call(0xc000370e40, 0xc00012f558, 0x13, 0xc000a1b5a8, 0x3, 0x3, 0x0, 0x16, 0x16)
	reflect/value.go:337 +0xb9
net/rpc.(*service).call(0xc0003c5080, 0xc0003d8000, 0xc000902bf0, 0x0, 0xc0001aad00, 0xc00026f640, 0x32be5c0, 0xc000707620, 0x16, 0x2f9aae0, ...)
	net/rpc/server.go:377 +0x189
net/rpc.(*Server).ServeRequest(0xc0003d8000, 0x3a32cc0, 0xc000e0d7c0, 0xc001723090, 0x34e3ac7)
	net/rpc/server.go:498 +0x23c
github.com/hashicorp/nomad/nomad.(*Server).RPC(0xc00020d200, 0x34f1a12, 0xc, 0x32be5c0, 0xc0007075c0, 0x2f9aae0, 0xc001723090, 0xc000e0d601, 0xc001723090)
	github.com/hashicorp/nomad/nomad/server.go:1568 +0xb9
github.com/hashicorp/nomad/command/agent.(*Agent).RPC(0xc0002621e0, 0x34f1a12, 0xc, 0x32be5c0, 0xc0007075c0, 0x2f9aae0, 0xc001723090, 0x0, 0x0)
	github.com/hashicorp/nomad/command/agent/agent.go:989 +0x7d
github.com/hashicorp/nomad/command/agent.(*HTTPServer).jobUpdate(0xc0005ae550, 0x3a16500, 0xc000707560, 0xc0000a0200, 0x0, 0x0, 0xc000517580, 0x0, 0x0, 0xc00072a940)
	github.com/hashicorp/nomad/command/agent/job_endpoint.go:405 +0x3a5
github.com/hashicorp/nomad/command/agent.(*HTTPServer).JobsRequest(0xc0005ae550, 0x3a16500, 0xc000707560, 0xc0000a0200, 0x4e2ee6, 0x6040547a, 0x7a248ff, 0x269bb6b5d171)
	github.com/hashicorp/nomad/command/agent/job_endpoint.go:23 +0x85
github.com/hashicorp/nomad/command/agent.(*HTTPServer).wrap.func1(0x3a16500, 0xc000707560, 0xc0000a0200)
	github.com/hashicorp/nomad/command/agent/http.go:450 +0x179
net/http.HandlerFunc.ServeHTTP(0xc0005d8900, 0x3a16500, 0xc000707560, 0xc0000a0200)
	net/http/server.go:2042 +0x44
net/http.(*ServeMux).ServeHTTP(0xc000517580, 0x3a16500, 0xc000707560, 0xc0000a0200)
	net/http/server.go:2417 +0x1ad
github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1(0x3a1ee40, 0xc00142a0e0, 0xc0000a0200)
	github.com/NYTimes/gziphandler@v1.0.1/gzip.go:277 +0x1e6
net/http.HandlerFunc.ServeHTTP(0xc000658000, 0x3a1ee40, 0xc00142a0e0, 0xc0000a0200)
	net/http/server.go:2042 +0x44
net/http.serverHandler.ServeHTTP(0xc00027dce0, 0x3a1ee40, 0xc00142a0e0, 0xc0000a0200)
	net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0009580a0, 0x3a31dc0, 0xc000e0d4c0)
	net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
	net/http/server.go:2969 +0x36c
@tgross tgross added this to Needs Triage in Nomad - Community Issues Triage via automation Mar 4, 2021
@notnoop
Copy link
Contributor

notnoop commented Mar 9, 2021

The panic here affect the server RPC handler, affecting the goroutine processing the request. The server continues to operate undisturbed without a full process crash. The API consumers do get indecipherable EOF error message though.

The bug sadly dates back to 0.10.0! The stack traces are a bit different. Nomad 0.11.0 fails with the same stack trace as 1.0.4 and included above in the original report.

Server 0.11.0

    2021/03/09 17:13:30.869039 http: panic serving 127.0.0.1:58525: runtime error: invalid memory address or nil pointer dereference
goroutine 433 [running]:
net/http.(*conn).serve.func1(0xc0000de780)
	net/http/server.go:1772 +0x139
panic(0x6497780, 0x8e7af10)
	runtime/panic.go:973 +0x3e3
github.com/hashicorp/nomad/nomad.serviceExposeConfig(...)
	github.com/hashicorp/nomad/nomad/job_endpoint_hook_expose_check.go:77
github.com/hashicorp/nomad/nomad.jobExposeCheckHook.Mutate(0xc000194960, 0xc000b24b08, 0x400f4c8, 0x60, 0x62dda60, 0x0, 0x0)
	github.com/hashicorp/nomad/nomad/job_endpoint_hook_expose_check.go:28 +0x11d
github.com/hashicorp/nomad/nomad.(*Job).admissionMutators(0xc000632410, 0xc000194960, 0xc000b24c30, 0x4898ed0b, 0xc000880000, 0xb46a9b8, 0x0, 0x4059b39)
	github.com/hashicorp/nomad/nomad/job_endpoint_hooks.go:61 +0xd3
github.com/hashicorp/nomad/nomad.(*Job).admissionControllers(0xc000632410, 0xc000194960, 0x7, 0x7208001, 0xc00071f680, 0x688a0e0, 0xc00071f680, 0x65fa220)
	github.com/hashicorp/nomad/nomad/job_endpoint_hooks.go:43 +0x4d
github.com/hashicorp/nomad/nomad.(*Job).Register(0xc000632410, 0xc00071f680, 0xc000639400, 0x0, 0x0)
	github.com/hashicorp/nomad/nomad/job_endpoint.go:93 +0x3b8
reflect.Value.call(0xc0006a1a40, 0xc000198f00, 0x13, 0x6abf76e, 0x4, 0xc000b255d0, 0x3, 0x3, 0xc000639400, 0xc000b25628, ...)
	reflect/value.go:460 +0x8ab
reflect.Value.Call(0xc0006a1a40, 0xc000198f00, 0x13, 0xc000b255d0, 0x3, 0x3, 0x8, 0x16, 0x16)
	reflect/value.go:321 +0xb4
net/rpc.(*service).call(0xc0005f4f00, 0xc000632280, 0xc00098d0e8, 0x0, 0xc0006b4000, 0xc00072c040, 0x688a0e0, 0xc00071f680, 0x16, 0x65fa220, ...)
	net/rpc/server.go:377 +0x17f
net/rpc.(*Server).ServeRequest(0xc000632280, 0x72017a0, 0xc000adc480, 0xc0007b7260, 0xc000880000)
	net/rpc/server.go:498 +0x23e
github.com/hashicorp/nomad/nomad.(*Server).RPC(0xc000470c00, 0x6acdf07, 0xc, 0x688a0e0, 0xc00071f620, 0x65fa220, 0xc0006393b0, 0xc0004ca001, 0xc0006393b0)
	github.com/hashicorp/nomad/nomad/server.go:1517 +0xb9
github.com/hashicorp/nomad/command/agent.(*Agent).RPC(0xc0000ed520, 0x6acdf07, 0xc, 0x688a0e0, 0xc00071f620, 0x65fa220, 0xc0006393b0, 0x0, 0x8)
	github.com/hashicorp/nomad/command/agent/agent.go:924 +0x7d
github.com/hashicorp/nomad/command/agent.(*HTTPServer).jobUpdate(0xc00043bc20, 0x71e72e0, 0xc00071f5c0, 0xc0004ca000, 0x0, 0x0, 0x485e41f8, 0x13464485e41f8, 0x100000001, 0xc000b8f930)
	github.com/hashicorp/nomad/command/agent/job_endpoint.go:426 +0x383
github.com/hashicorp/nomad/command/agent.(*HTTPServer).JobsRequest(0xc00043bc20, 0x71e72e0, 0xc00071f5c0, 0xc0004ca000, 0x40d06e6, 0x6047f30a, 0x33790850, 0x13464485e41f8)
	github.com/hashicorp/nomad/command/agent/job_endpoint.go:22 +0x84
github.com/hashicorp/nomad/command/agent.(*HTTPServer).wrap.func1(0x71e72e0, 0xc00071f5c0, 0xc0004ca000)
	github.com/hashicorp/nomad/command/agent/http.go:444 +0x176
net/http.HandlerFunc.ServeHTTP(0xc000824240, 0x71e72e0, 0xc00071f5c0, 0xc0004ca000)
	net/http/server.go:2012 +0x44
net/http.(*ServeMux).ServeHTTP(0xc0008a8440, 0x71e72e0, 0xc00071f5c0, 0xc0004ca000)
	net/http/server.go:2387 +0x1a5
github.com/hashicorp/nomad/vendor/github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1(0x71eace0, 0xc000270460, 0xc0004ca000)
	github.com/hashicorp/nomad/vendor/github.com/NYTimes/gziphandler/gzip.go:269 +0x1e6
net/http.HandlerFunc.ServeHTTP(0xc0008b6ab0, 0x71eace0, 0xc000270460, 0xc0004ca000)
	net/http/server.go:2012 +0x44
net/http.serverHandler.ServeHTTP(0xc000270000, 0x71eace0, 0xc000270460, 0xc0004ca000)
	net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc0000de780, 0x72014e0, 0xc000721dc0)
	net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
	net/http/server.go:2933 +0x35c

Server 0.10.0

    2021/03/09 17:12:14.740069 http: panic serving 127.0.0.1:58423: runtime error: invalid memory address or nil pointer dereference
goroutine 572 [running]:
net/http.(*conn).serve.func1(0xc00028eb40)
	/tmp/nomad-workspace/2019-10-22-1571747903/go/src/net/http/server.go:1769 +0x139
panic(0x5cb1d60, 0x7ba30a0)
	/tmp/nomad-workspace/2019-10-22-1571747903/go/src/runtime/panic.go:522 +0x1b5
github.com/hashicorp/nomad/nomad/structs.(*ConsulSidecarService).Copy(...)
	/tmp/nomad-workspace/2019-10-22-1571747903/gopath/src/github.com/hashicorp/nomad/nomad/structs/services.go:617
github.com/hashicorp/nomad/nomad/structs.(*ConsulConnect).Copy(0xc000b0a9e0, 0xc000bb0100)
	/tmp/nomad-workspace/2019-10-22-1571747903/gopath/src/github.com/hashicorp/nomad/nomad/structs/services.go:557 +0x39
github.com/hashicorp/nomad/nomad/structs.(*Service).Copy(0xc0001dd4d0, 0x1)
	/tmp/nomad-workspace/2019-10-22-1571747903/gopath/src/github.com/hashicorp/nomad/nomad/structs/services.go:354 +0x200
github.com/hashicorp/nomad/nomad/structs.(*TaskGroup).Copy(0xc0002b0700, 0x1)
	/tmp/nomad-workspace/2019-10-22-1571747903/gopath/src/github.com/hashicorp/nomad/nomad/structs/structs.go:4776 +0x352
github.com/hashicorp/nomad/nomad/structs.(*Job).Copy(0xc0000ce960, 0x0)
	/tmp/nomad-workspace/2019-10-22-1571747903/gopath/src/github.com/hashicorp/nomad/nomad/structs/structs.go:3452 +0x221
github.com/hashicorp/nomad/nomad.(*Job).admissionValidators(0xc0003027d0, 0xc0000ce960, 0xc0000ce960, 0x0, 0x0, 0x0, 0x0)
	/tmp/nomad-workspace/2019-10-22-1571747903/gopath/src/github.com/hashicorp/nomad/nomad/job_endpoint_hooks.go:58 +0x40
github.com/hashicorp/nomad/nomad.(*Job).admissionControllers(0xc0003027d0, 0xc0000ce960, 0x7, 0x1, 0x3, 0xc00a1a8fac18ecd8, 0x21fb9762b, 0x7be45c0)
	/tmp/nomad-workspace/2019-10-22-1571747903/gopath/src/github.com/hashicorp/nomad/nomad/job_endpoint_hooks.go:31 +0xa9
github.com/hashicorp/nomad/nomad.(*Job).Register(0xc0003027d0, 0xc00081c7e0, 0xc000577180, 0x0, 0x0)
	/tmp/nomad-workspace/2019-10-22-1571747903/gopath/src/github.com/hashicorp/nomad/nomad/job_endpoint.go:97 +0x3d4
reflect.Value.call(0xc0003bbf80, 0xc0005c6640, 0x13, 0x6099ca2, 0x4, 0xc000bc9728, 0x3, 0x3, 0xc000577180, 0x16, ...)
	/tmp/nomad-workspace/2019-10-22-1571747903/go/src/reflect/value.go:447 +0x461
reflect.Value.Call(0xc0003bbf80, 0xc0005c6640, 0x13, 0xc000bc9728, 0x3, 0x3, 0x16, 0x0, 0x5ed5bc0)
	/tmp/nomad-workspace/2019-10-22-1571747903/go/src/reflect/value.go:308 +0xa4
net/rpc.(*service).call(0xc00002c9c0, 0xc000302690, 0xc000b14d38, 0x0, 0xc000526580, 0xc000304000, 0x5ed5bc0, 0xc00081c7e0, 0x16, 0x5d618e0, ...)
	/tmp/nomad-workspace/2019-10-22-1571747903/go/src/net/rpc/server.go:384 +0x14e
net/rpc.(*Server).ServeRequest(0xc000302690, 0x65051c0, 0xc000b7e9c0, 0xc000b073e0, 0xc00058a800)
	/tmp/nomad-workspace/2019-10-22-1571747903/go/src/net/rpc/server.go:505 +0x23e
github.com/hashicorp/nomad/nomad.(*Server).RPC(0xc000401600, 0x60a7299, 0xc, 0x5ed5bc0, 0xc00081c780, 0x5d618e0, 0xc000577130, 0xc000b76401, 0xc000577130)
	/tmp/nomad-workspace/2019-10-22-1571747903/gopath/src/github.com/hashicorp/nomad/nomad/server.go:1441 +0xb9
github.com/hashicorp/nomad/command/agent.(*Agent).RPC(0xc000372160, 0x60a7299, 0xc, 0x5ed5bc0, 0xc00081c780, 0x5d618e0, 0xc000577130, 0x17, 0x8)
	/tmp/nomad-workspace/2019-10-22-1571747903/gopath/src/github.com/hashicorp/nomad/command/agent/agent.go:919 +0x7a
github.com/hashicorp/nomad/command/agent.(*HTTPServer).jobUpdate(0xc0000e10e0, 0x64f1a80, 0xc00081c720, 0xc000b76400, 0x0, 0x0, 0xc08f0b5304, 0x134528f0b5304, 0x405bed0, 0xc000000038)
	/tmp/nomad-workspace/2019-10-22-1571747903/gopath/src/github.com/hashicorp/nomad/command/agent/job_endpoint.go:422 +0x386
github.com/hashicorp/nomad/command/agent.(*HTTPServer).JobsRequest(0xc0000e10e0, 0x64f1a80, 0xc00081c720, 0xc000b76400, 0xc0004bb760, 0xc000252ac0, 0x38, 0xc000252ab8)
	/tmp/nomad-workspace/2019-10-22-1571747903/gopath/src/github.com/hashicorp/nomad/command/agent/job_endpoint.go:21 +0x8a
github.com/hashicorp/nomad/command/agent.(*HTTPServer).wrap.func1(0x64f1a80, 0xc00081c720, 0xc000b76400)
	/tmp/nomad-workspace/2019-10-22-1571747903/gopath/src/github.com/hashicorp/nomad/command/agent/http.go:294 +0x149
net/http.HandlerFunc.ServeHTTP(0xc000472040, 0x64f1a80, 0xc00081c720, 0xc000b76400)
	/tmp/nomad-workspace/2019-10-22-1571747903/go/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xc0003c0100, 0x64f1a80, 0xc00081c720, 0xc000b76400)
	/tmp/nomad-workspace/2019-10-22-1571747903/go/src/net/http/server.go:2375 +0x1d6
github.com/hashicorp/nomad/vendor/github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1(0x64f3fc0, 0xc0002b0380, 0xc000b76400)
	/tmp/nomad-workspace/2019-10-22-1571747903/gopath/src/github.com/hashicorp/nomad/vendor/github.com/NYTimes/gziphandler/gzip.go:269 +0x1db
net/http.HandlerFunc.ServeHTTP(0xc000368180, 0x64f3fc0, 0xc0002b0380, 0xc000b76400)
	/tmp/nomad-workspace/2019-10-22-1571747903/go/src/net/http/server.go:1995 +0x44
net/http.serverHandler.ServeHTTP(0xc000342dd0, 0x64f3fc0, 0xc0002b0380, 0xc000b76400)
	/tmp/nomad-workspace/2019-10-22-1571747903/go/src/net/http/server.go:2774 +0xa8
net/http.(*conn).serve(0xc00028eb40, 0x6504f00, 0xc000b72f80)
	/tmp/nomad-workspace/2019-10-22-1571747903/go/src/net/http/server.go:1878 +0x851
created by net/http.(*Server).Serve
	/tmp/nomad-workspace/2019-10-22-1571747903/go/src/net/http/server.go:2884 +0x2f4

@notnoop notnoop mentioned this issue Mar 9, 2021
Merged
Nomad - Community Issues Triage automation moved this from Needs Triage to Done Mar 10, 2021
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 21, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@notnoop notnoop

Labels
theme/crash type/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

server: handle invalid jobs in expose handler hook hashicorp/nomad
3 participants
@krishicks @notnoop @tgross