docs: expand documentation on node pools

[lgfa29]

Add new Concepts page about node pools and expand the Architecture page to note planning client configuration. Also include some smaller changes where node pools may affect the scheduling outcome.

Preview links for the biggest changes:
https://nomad-git-docs-node-pools-hashicorp.vercel.app/nomad/docs/concepts/architecture#client-organization
https://nomad-git-docs-node-pools-hashicorp.vercel.app/nomad/docs/concepts/node-pools

[tgross]

This is looking great @lgfa29! I'm about halfway thru but wanted to leave comments and then come back to finish the rest of my read-thru later today.

[tgross]

Suggested change

	Using affinities and constraints has the downside of only allowing allocations
	to gravitate towards certain nodes, but it does not prevent placements of jobs
	where the rules don't apply.
	Using affinities and constraints has the downside of only allowing allocations
	to gravitate towards certain nodes, but it does not prevent placements of other
	jobs for which the rules don't apply.

I think?

[Juanadelacuesta]

Im a little confused here: constraints are hard requirements and "Job placement fails if a constraint cannot be satisfied." but at the same time " it does not prevent placements of jobs where the rules don't apply" ?

[lgfa29]

I think?

Yup, that's right. Thanks!

Im a little confused here: constraints are hard requirements and "Job placement fails if a constraint cannot be satisfied." but at the same time " it does not prevent placements of jobs where the rules don't apply" ?

Yeah, it's kind of weird to describe in text, so maybe I should put an example.

But imagine you have two jobs, one with a constraint and one without any constraints:

job "app"  {
  constraint {
    attribute = "${meta.env}"
    value      = "prod"
  }
  # ...
}

job "db"  {
  # ...
}

The app job is only allowed to run in clients with metadata env=prod, but the db is free to run anywhere, including clients with env=prod.

So if you want to reserve the env=prod clients to only run app then you need a negative constraint in all other jobs:

job "db"  {
  constraint {
    attribute = "${meta.env}"
    operator  = "!="
    value      = "prod"
  }
  # ...
}

So constraints are kind of unidirectional in this sense: you can restrict a job to specific nodes, but you can't restrict a node to specific jobs.

Does this help?

[lgfa29]

Updated this paragraph as I moved this section to the Scheduler section

nomad/website/content/docs/concepts/scheduling/placement.mdx

Lines 36 to 38 in 4f4ba9f

	One restriction of using affinities and constraints is that they only express
	relationships from jobs to nodes, so it is not possible to use them to restrict
	a node to only receive allocations for specific jobs.

Is this better or should I add an example to help illustrate the problem?

[tgross]

This looks great @lgfa29! I've left a few more comments but I think if we resolve those we're good-to-go here.

[Juanadelacuesta]

when can it be different?

[shoenig]

I kind of disagree with noting that it may be used interchangeably (even though it often is). Node is intended to refer to the machine (and operating system, etc.) while Client refers to the Nomad agent running in client mode. We should avoid conflating the terms in our own docs.

[lgfa29]

when can it be different?

"Node" is a very generic term that different people use it to mean different things 😅

For example, a node in a network context can be anything sending or receiving traffic, like a computer, a router, a switch, or maybe even an allocation, like a sidecar proxy. So if you're talking about Nomad's network topology, "node" may not necessarily mean this specific definition.

But to @shoenig's point, I think I didn't express myself well here. I meant more like "you may see or hear people using them interchangeably", which I have in informal conversations, issues, community blog posts etc.

Maybe this sentence would be better like this:

Despite being different concepts, you may find the term "node" being used interchangeably with "client" in some materials.

Node is intended to refer to the machine (and operating system, etc.)

To the machine running a Nomad client right? I don't recall us using "node" to refer to machines running servers. For example, the nomad node commands and /v1/node/ endpoints are only about clients.

[lgfa29]

I ping-ponged a bit on how to phrase this and ended up here:

nomad/website/content/docs/concepts/architecture.mdx

Lines 116 to 120 in efc2e37

	#### Node
	A more generic term used to refer to machines running Nomad agents in client
	mode. Despite being different concepts, you may find "node" being used
	interchangeably with "client" in some materials and informal content.

Is this a better description? Or maybe I should just leave this part out?

[Juanadelacuesta]

If this paragraph is used to introduce Affinities and Constraints, Datacenter, Nodepools... it would be worth mentioning them as the settings

[lgfa29]

Hum...I don't quite follow. I think the confusion may be that I'm using "setting" as the gerund of the verb "to set", not as "configuration".

But this paragraph is gone once I moved this section to the new Scheduling -> Placements page 😅

[Juanadelacuesta]

Not directly related, but I couldn't find what authoritative regions are, maybe add a little phrase about it on the glossary on the region?

[lgfa29]

Ah good point, I added an entry to the glossary about authoritative and non-authoritative regions. I used the same entry because they're opposites so it was kind of weird to explain one without explaining the other 😅

https://github.com/hashicorp/nomad/pull/18109/files#diff-81af9e6ca20054c572009d730ea51e6a6563ef34befcc20a398fac0c3a6d68d9R96-R106

[Juanadelacuesta]

This phrase is very clear and useful, I would add it in the node pools description

[lgfa29]

Thanks! Which node pool description are you referring to?