Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nomad: vault integration > nomad doesnt set user-agent header when making requests to vault #15629

Closed
noahehall opened this issue Dec 31, 2022 · 5 comments
Closed

nomad: vault integration > nomad doesnt set user-agent header when making requests to vault #15629

noahehall opened this issue Dec 31, 2022 · 5 comments
Assignees
@shoenig
Labels
theme/vault type/bug

Comments

@noahehall
Copy link

noahehall commented Dec 31, 2022 •
edited
Loading

Nomad version

Output from nomad version

08:29 AM (nomad *$|u=) 
$ nomad -version
Nomad v1.4.3 (f464aca721d222ae9c1f3df643b3c3aaa20e2da7)

Operating system and Environment details

08:29 AM (nomad *$|u=) 
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy

Issue

  • following the vault integration docs (really good by the way)
  • nomad doesnt set user-agent header when making requests to vault over http
  • haproxy ACL documentation recommends blocking requests lacking proper user agent header
Length of the user-agent string

Some attackers will attempt to bypass normal 
user agent strings by using a random md5sum, 
which can be identified by length and immediately blocked:

Attackers can vary more with their attacks, 
so you can rely on the fact that legitimate 
user agents are longer while also being set 
to a minimum length:

http-request deny if { req.hdr(user-agent) -m len le 32 }

This will then block any requests which 
have a user-agent header shorter than 32 characters.

Reproduction steps

Expected Result

  • nomad correclty implements http rfc 2048 and sets a user agent string
  • vault (IMO) should exceed the standard set by github and not accept requests over http lacking user agent header
  • nomad doesnt require me to modify this line
http-request deny if { req.hdr(user-agent) -m len le 32 }

Actual Result

  • haproxy returns 403 because nomad request doesnt include user agent string

Job file (if appropriate)

Nomad Server logs (if appropriate)

---
> issue 1: 403 Request forbidden by administrative rules.
- its an edge issue: if we curl  8300 it works, check the deny rules
- its not an edge issue, its a security issue, nomad doesnt set user agent when making request
  - `http-request deny if { req.hdr(user-agent) -m len le 32 }`
```sh
$     2022-12-31T06:53:10.877-0700 [WARN]  nomad.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
    2022-12-31T06:53:38.937-0700 [WARN]  nomad.vault: failed to contact Vault API: retry=30s
  error=
  | Error making API request.
  | 
  | URL: GET https://dev.nirv.ai:8200/v1/sys/health?drsecondarycode=299&performancestandbycode=299&sealedcode=299&standbycode=299&uninitcode=299
  | Code: 403. Raw Message:
  | 
  | <html><body><h1>403 Forbidden</h1>
  | Request forbidden by administrative rules.
  | </body></html>
  
### Nomad Client logs (if appropriate)
@noahehall noahehall changed the title nomad: vaullt integration > nomad doesnt set user-agent header when making requests to vault limiting edge router security posture nomad: vault integration > nomad doesnt set user-agent header when making requests to vault limiting edge router security posture Dec 31, 2022
@noahehall noahehall changed the title nomad: vault integration > nomad doesnt set user-agent header when making requests to vault limiting edge router security posture nomad: vault integration > nomad doesnt set user-agent header when making requests to vault limiting reverse proxy security posture Dec 31, 2022
This was referenced Dec 31, 2022
Open
Closed
@noahehall noahehall changed the title nomad: vault integration > nomad doesnt set user-agent header when making requests to vault limiting reverse proxy security posture nomad: vault integration > nomad doesnt set user-agent header when making requests to vault Jan 2, 2023
@tgross
Copy link
Member

tgross commented Jan 3, 2023

Adding a note here that we should probably verify whether this bug exists for consul-template as well, as we use that for template rendering. If we fix it in Nomad and not in consul-template, we'll be able to mint Vault tokens but not render secrets.

@shoenig
Copy link
Contributor

shoenig commented Jan 3, 2023

We have https://github.com/hashicorp/nomad/blob/v1.4.3/helper/useragent/useragent.go but it doesn't get used in a lot places it probably should. Sometimes I wonder if we shouldn't substitute go-cleanhttp with our own Nomad specific http client factory.

@shoenig
Copy link
Contributor

shoenig commented Jan 9, 2023

Took a quick skim over what would be involved - we might run into trouble with the embedded Vault client in consul-template - AFAICT there isn't (yet) a way to configure UA/custom headers.

In other cases where Nomad creates the Vault client and we can call client.SetHeaders.

@shoenig shoenig self-assigned this Jan 10, 2023
@shoenig shoenig mentioned this issue Jan 10, 2023
Merged
@shoenig
Copy link
Contributor

shoenig commented Apr 11, 2023

Looks like Consul Template just added the missing piece to get user agent threaded through

https://github.com/hashicorp/consul-template/blob/main/CHANGELOG.md

@shoenig shoenig closed this as completed Dec 6, 2023
@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 3, 2025

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 3, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@shoenig shoenig

Labels
theme/vault type/bug
Projects
Nomad - Community Issues Triage
Archived in project
Milestone
No milestone
Development

No branches or pull requests
3 participants
@shoenig @tgross @noahehall