-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
8 changed files
with
308 additions
and
5 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,116 @@ | ||
--- | ||
layout: "guides" | ||
page_title: "Automatically Bootstrapping a Nomad Cluster" | ||
sidebar_current: "guides-cluster-automatic" | ||
description: |- | ||
Learn how to automatically bootstrap a Nomad cluster using Consul. By having | ||
a Consul agent installed on each host, Nomad can automatically discover other | ||
clients and servers to bootstrap the cluster without operator involvement. | ||
--- | ||
|
||
# Automatic Bootstrapping | ||
|
||
To automatically bootstrap a Nomad cluster, we must leverage another HashiCorp | ||
open source tool, [Consul](https://www.consul.io/). Bootstrapping Nomad is | ||
easiest against an existing Consul cluster. The Nomad servers and clients | ||
will become informed of each other's existence when the Consul agent is | ||
installed and configured on each host. As an added benefit, integrating Consul | ||
with Nomad provides service and health check registration for applications which | ||
later run under Nomad. | ||
|
||
Consul models infrastructures as datacenters and multiple Consul datacenters can | ||
be connected over the WAN so that clients can discover nodes in other | ||
datacenters. Since Nomad regions can encapsulate many datacenters, we recommend | ||
running a Consul cluster in every Nomad datacenter and connecting them over the | ||
WAN. Please refer to the Consul guide for both | ||
[bootstrapping](https://www.consul.io/docs/guides/bootstrapping.html) a single | ||
datacenter and [connecting multiple Consul clusters over the | ||
WAN](https://www.consul.io/docs/guides/datacenters.html). | ||
|
||
If a Consul agent is installed on the host prior to Nomad starting, the Nomad | ||
agent will register with Consul and discover other nodes. | ||
|
||
For servers, we must inform the cluster how many servers we expect to have. This | ||
is required to form the initial quorum, since Nomad is unaware of how many peers | ||
to expect. For example, to form a region with three Nomad servers, you would use | ||
the following Nomad configuration file: | ||
|
||
```hcl | ||
# /etc/nomad.d/server.hcl | ||
server { | ||
enabled = true | ||
bootstrap_expect = 3 | ||
} | ||
``` | ||
|
||
This configuration would be saved to disk and then run: | ||
|
||
```shell | ||
$ nomad agent -config=/etc/nomad.d/server.hcl | ||
``` | ||
|
||
A similar configuration is available for Nomad clients: | ||
|
||
```hcl | ||
# /etc/nomad.d/client.hcl | ||
datacenter = "dc1" | ||
client { | ||
enabled = true | ||
} | ||
``` | ||
|
||
The agent is started in a similar manner: | ||
|
||
```shell | ||
$ nomad agent -config=/etc/nomad.d/client.hcl | ||
``` | ||
|
||
As you can see, the above configurations include no IP or DNS addresses between | ||
the clients and servers. This is because Nomad detected the existence of Consul | ||
and utilized service discovery to form the cluster. | ||
|
||
## Internals | ||
|
||
~> This section discusses the internals of the Consul and Nomad integration at a | ||
very high level. Reading is only recommended for those curious to the | ||
implementation. | ||
|
||
As discussed in the previous section, Nomad merges multiple configuration files | ||
together, so the `-config` may be specified more than once: | ||
|
||
```shell | ||
$ nomad agent -config=base.hcl -config=server.hcl | ||
``` | ||
|
||
In addition to merging configuration on the command line, Nomad also maintains | ||
its own internal configurations (called "default configs") which include sane | ||
base defaults. One of those default configurations includes a "consul" block, | ||
which specifies sane defaults for connecting to and integrating with Consul. In | ||
essence, this configuration file resembles the following: | ||
|
||
```hcl | ||
# You do not need to add this to your configuration file. This is an example | ||
# that is part of Nomad's internal default configuration for Consul integration. | ||
consul { | ||
# The address to the Consul agent. | ||
address = "127.0.0.1:8500" | ||
# The service name to register the server and client with Consul. | ||
server_service_name = "nomad" | ||
client_service_name = "nomad-client" | ||
# Enables automatically registering the services. | ||
auto_advertise = true | ||
# Enabling the server and client to bootstrap using Consul. | ||
server_auto_join = true | ||
client_auto_join = true | ||
} | ||
``` | ||
|
||
Please refer to the [Consul | ||
documentation](/docs/agent/configuration/consul.html) for the complete set of | ||
configuration options. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
--- | ||
layout: "guides" | ||
page_title: "Bootstrapping a Nomad Cluster" | ||
sidebar_current: "guides-cluster-bootstrap" | ||
description: |- | ||
Learn how to bootstrap a Nomad cluster. | ||
--- | ||
|
||
# Bootstrapping a Nomad Cluster | ||
|
||
Nomad models infrastructure into regions and datacenters. Servers reside at the | ||
regional layer and manage all state and scheduling decisions for that region. | ||
Regions contain multiple datacenters, and clients are registered to a single | ||
datacenter (and thus a region that contains that datacenter). For more details on | ||
the architecture of Nomad and how it models infrastructure see the [architecture | ||
page](/docs/internals/architecture.html). | ||
|
||
There are two strategies for bootstrapping a Nomad cluster: | ||
|
||
1. <a href="/guides/cluster/automatic.html">Automatic bootstrapping</a> | ||
1. <a href="/guides/cluster/manual.html">Manual bootstrapping</a> | ||
|
||
Please refer to the specific documentation links above or in the sidebar for | ||
more detailed information about each strategy. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
--- | ||
layout: "guides" | ||
page_title: "Federating a Nomad Cluster" | ||
sidebar_current: "guides-cluster-federation" | ||
description: |- | ||
Learn how to join Nomad servers across multiple regions so users can submit | ||
jobs to any server in any region using global federation. | ||
--- | ||
|
||
# Federating a Cluster | ||
|
||
Because Nomad operates at a regional level, federation is part of Nomad core. | ||
Federation enables users to submit jobs or interact with the HTTP API targeting | ||
any region, from any server, even if that server resides in a different region. | ||
|
||
Federating multiple Nomad clusters is as simple as joining servers. From any | ||
server in one region, issue a join command to a server in a remote region: | ||
|
||
```shell | ||
$ nomad server-join 1.2.3.4:4648 | ||
``` | ||
|
||
Note that only one join command is required per region. Servers across regions | ||
discover other servers in the cluster via the gossip protocol and hence it's | ||
enough to join just one known server. | ||
|
||
If bootstrapped via Consul and the Consul clusters in the Nomad regions are | ||
federated, then federation occurs automatically. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,65 @@ | ||
--- | ||
layout: "guides" | ||
page_title: "Manually Bootstrapping a Nomad Cluster" | ||
sidebar_current: "guides-cluster-manual" | ||
description: |- | ||
Learn how to manually bootstrap a Nomad cluster using the server-join | ||
command. This section also discusses Nomad federation across multiple | ||
datacenters and regions. | ||
--- | ||
|
||
# Manual Bootstrapping | ||
|
||
Manually bootstrapping a Nomad cluster does not rely on additional tooling, but | ||
does require operator participation in the cluster formation process. When | ||
bootstrapping, Nomad servers and clients must be started and informed with the | ||
address of at least one Nomad server. | ||
|
||
As you can tell, this creates a chicken-and-egg problem where one server must | ||
first be fully bootstrapped and configured before the remaining servers and | ||
clients can join the cluster. This requirement can add additional provisioning | ||
time as well as ordered dependencies during provisioning. | ||
|
||
First, we bootstrap a single Nomad server and capture its IP address. After we | ||
have that nodes IP address, we place this address in the configuration. | ||
|
||
For Nomad servers, this configuration may look something like this: | ||
|
||
```hcl | ||
server { | ||
enabled = true | ||
bootstrap_expect = 3 | ||
# This is the IP address of the first server we provisioned | ||
retry_join = ["<known-address>:4648"] | ||
} | ||
``` | ||
|
||
Alternatively, the address can be supplied after the servers have all been | ||
started by running the [`server-join` command](/docs/commands/server-join.html) | ||
on the servers individual to cluster the servers. All servers can join just one | ||
other server, and then rely on the gossip protocol to discover the rest. | ||
|
||
``` | ||
$ nomad server-join <known-address> | ||
``` | ||
|
||
For Nomad clients, the configuration may look something like: | ||
|
||
```hcl | ||
client { | ||
enabled = true | ||
servers = ["<known-address>:4647"] | ||
} | ||
``` | ||
|
||
At this time, there is no equivalent of the <tt>server-join</tt> command for | ||
Nomad clients. | ||
|
||
The port corresponds to the RPC port. If no port is specified with the IP | ||
address, the default RCP port of `4647` is assumed. | ||
|
||
As servers are added or removed from the cluster, this information is pushed to | ||
the client. This means only one server must be specified because, after initial | ||
contact, the full set of servers in the client's region are shared with the | ||
client. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,71 @@ | ||
--- | ||
layout: "guides" | ||
page_title: "Nomad Client and Server Requirements" | ||
sidebar_current: "guides-cluster-requirements" | ||
description: |- | ||
Learn about Nomad client and server requirements such as memory and CPU | ||
recommendations, network topologies, and more. | ||
--- | ||
|
||
# Cluster Requirements | ||
|
||
## Resources (RAM, CPU, etc.) | ||
|
||
**Nomad servers** may need to be run on large machine instances. We suggest | ||
having 8+ cores, 32 GB+ of memory, 80 GB+ of disk and significant network | ||
bandwidth. The core count and network recommendations are to ensure high | ||
throughput as Nomad heavily relies on network communication and as the Servers | ||
are managing all the nodes in the region and performing scheduling. The memory | ||
and disk requirements are due to the fact that Nomad stores all state in memory | ||
and will store two snapshots of this data onto disk. Thus disk should be at | ||
least 2 times the memory available to the server when deploying a high load | ||
cluster. | ||
|
||
**Nomad clients** support reserving resources on the node that should not be | ||
used by Nomad. This should be used to target a specific resource utilization per | ||
node and to reserve resources for applications running outside of Nomad's | ||
supervision such as Consul and the operating system itself. | ||
|
||
Please see the [reservation configuration](/docs/agent/configuration/client.html#reserved) for | ||
more detail. | ||
|
||
## Network Topology | ||
|
||
**Nomad servers** are expected to have sub 10 millisecond network latencies | ||
between each other to ensure liveness and high throughput scheduling. Nomad | ||
servers can be spread across multiple datacenters if they have low latency | ||
connections between them to achieve high availability. | ||
|
||
For example, on AWS every region comprises of multiple zones which have very low | ||
latency links between them, so every zone can be modeled as a Nomad datacenter | ||
and every Zone can have a single Nomad server which could be connected to form a | ||
quorum and a region. | ||
|
||
Nomad servers uses Raft for state replication and Raft being highly consistent | ||
needs a quorum of servers to function, therefore we recommend running an odd | ||
number of Nomad servers in a region. Usually running 3-5 servers in a region is | ||
recommended. The cluster can withstand a failure of one server in a cluster of | ||
three servers and two failures in a cluster of five servers. Adding more servers | ||
to the quorum adds more time to replicate state and hence throughput decreases | ||
so we don't recommend having more than seven servers in a region. | ||
|
||
**Nomad clients** do not have the same latency requirements as servers since they | ||
are not participating in Raft. Thus clients can have 100+ millisecond latency to | ||
their servers. This allows having a set of Nomad servers that service clients | ||
that can be spread geographically over a continent or even the world in the case | ||
of having a single "global" region and many datacenter. | ||
|
||
## Ports Used | ||
|
||
Nomad requires 3 different ports to work properly on servers and 2 on clients, | ||
some on TCP, UDP, or both protocols. Below we document the requirements for each | ||
port. | ||
|
||
* HTTP API (Default 4646). This is used by clients and servers to serve the HTTP | ||
API. TCP only. | ||
|
||
* RPC (Default 4647). This is used by servers and clients to communicate amongst | ||
each other. TCP only. | ||
|
||
* Serf WAN (Default 4648). This is used by servers to gossip over the WAN to | ||
other servers. TCP and UDP. |