Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test flake in TestAutopilot_RollingUpdate #13931

Closed
tgross opened this issue Jul 28, 2022 · 1 comment · Fixed by #14475
Closed

test flake in TestAutopilot_RollingUpdate #13931

tgross opened this issue Jul 28, 2022 · 1 comment · Fixed by #14475
Assignees
@tgross
Labels
theme/flaky-tests type/bug
Milestone
1.4.0

Comments

@tgross
Copy link
Member

tgross commented Jul 28, 2022 •
edited

Example run https://app.circleci.com/pipelines/github/hashicorp/nomad/28734/workflows/9c27bd15-423e-437e-be7e-eaf456d23624/jobs/317342. CircleCI has identified this test as flaky with multiple failures on main in the last 30 days.

test logs

Failed
=== RUN TestAutopilot_RollingUpdate
2022-07-27T11:17:24.636Z [TRACE] eventer/eventer.go:74: plugin_loader.exec: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.636Z [TRACE] eventer/eventer.go:74: plugin_loader.mock_driver: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.636Z [TRACE] eventer/eventer.go:74: plugin_loader.raw_exec: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.636Z [TRACE] eventer/eventer.go:74: plugin_loader.java: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.636Z [TRACE] eventer/eventer.go:74: plugin_loader.qemu: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.636Z [TRACE] eventer/eventer.go:74: plugin_loader.docker: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.636Z [TRACE] eventer/eventer.go:74: plugin_loader.docker: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.636Z [TRACE] eventer/eventer.go:74: plugin_loader.mock_driver: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.636Z [TRACE] eventer/eventer.go:74: plugin_loader.raw_exec: task event loop shutdown: plugin_dir=""
node-063 2022-07-27T11:17:24.637Z [TRACE] nomad/worker.go:330: worker: changed workload status: worker_id=32cec098-6878-ee8d-8777-6169e91b319c from=WaitingToDequeue to=Paused
2022-07-27T11:17:24.637Z [TRACE] eventer/eventer.go:74: plugin_loader.qemu: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.637Z [TRACE] eventer/eventer.go:74: plugin_loader.exec: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.637Z [TRACE] eventer/eventer.go:74: plugin_loader.java: task event loop shutdown: plugin_dir=""
node-063 2022-07-27T11:17:24.637Z [DEBUG] nomad/encrypter.go:409: nomad.keyring.replicator: exiting key replication
node-068 2022-07-27T11:17:24.637Z [INFO] raft@v1.3.5/api.go:570: nomad.raft: initial configuration: index=0 servers=[]
node-068 2022-07-27T11:17:24.637Z [INFO] raft@v1.3.5/raft.go:152: nomad.raft: entering follower state: follower="Node at 127.0.0.1:9136 [Follower]" leader=
node-068 2022-07-27T11:17:24.638Z [INFO] go-hclog@v1.2.0/stdlog.go:57: nomad: serf: EventMemberJoin: nomad-068.global 127.0.0.1
node-068 2022-07-27T11:17:24.638Z [INFO] nomad/server.go:1734: nomad: starting scheduling worker(s): num_workers=4 schedulers=["batch", "system", "sysbatch", "noop", "service", "_core"]
node-068 2022-07-27T11:17:24.638Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=c55e2cbd-f8ca-de7a-0950-e1a5b93b7e7f index=1 of=4
node-068 2022-07-27T11:17:24.638Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=713527ec-db64-4ba1-50e3-ef25e7962875 index=2 of=4
node-068 2022-07-27T11:17:24.638Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=96867b49-2a02-eabe-80f2-1fff56d73276 index=3 of=4
node-068 2022-07-27T11:17:24.638Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=1f86f651-fe6a-7391-567f-04b56b9de657 index=4 of=4
node-068 2022-07-27T11:17:24.638Z [INFO] nomad/server.go:1746: nomad: started scheduling worker(s): num_workers=4 schedulers=["batch", "system", "sysbatch", "noop", "service", "_core"]
node-068 2022-07-27T11:17:24.639Z [INFO] nomad/serf.go:60: nomad: adding server: server="nomad-068.global (Addr: 127.0.0.1:9136) (DC: dc1)"
node-068 2022-07-27T11:17:24.639Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=c55e2cbd-f8ca-de7a-0950-e1a5b93b7e7f from=Starting to=Started
node-068 2022-07-27T11:17:24.639Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=c55e2cbd-f8ca-de7a-0950-e1a5b93b7e7f from=UnknownStatus to=Running
node-068 2022-07-27T11:17:24.639Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=c55e2cbd-f8ca-de7a-0950-e1a5b93b7e7f
node-068 2022-07-27T11:17:24.639Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=c55e2cbd-f8ca-de7a-0950-e1a5b93b7e7f from=Running to=WaitingToDequeue
node-068 2022-07-27T11:17:24.639Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=713527ec-db64-4ba1-50e3-ef25e7962875 from=Starting to=Started
node-068 2022-07-27T11:17:24.639Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=713527ec-db64-4ba1-50e3-ef25e7962875 from=UnknownStatus to=Running
node-068 2022-07-27T11:17:24.639Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=713527ec-db64-4ba1-50e3-ef25e7962875
node-068 2022-07-27T11:17:24.639Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=713527ec-db64-4ba1-50e3-ef25e7962875 from=Running to=WaitingToDequeue
node-068 2022-07-27T11:17:24.639Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=96867b49-2a02-eabe-80f2-1fff56d73276 from=Starting to=Started
node-068 2022-07-27T11:17:24.639Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=96867b49-2a02-eabe-80f2-1fff56d73276 from=UnknownStatus to=Running
node-068 2022-07-27T11:17:24.639Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=96867b49-2a02-eabe-80f2-1fff56d73276
node-068 2022-07-27T11:17:24.639Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=96867b49-2a02-eabe-80f2-1fff56d73276 from=Running to=WaitingToDequeue
node-068 2022-07-27T11:17:24.639Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=1f86f651-fe6a-7391-567f-04b56b9de657 from=Starting to=Started
node-068 2022-07-27T11:17:24.639Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=1f86f651-fe6a-7391-567f-04b56b9de657 from=UnknownStatus to=Running
node-068 2022-07-27T11:17:24.639Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=1f86f651-fe6a-7391-567f-04b56b9de657
node-068 2022-07-27T11:17:24.639Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=1f86f651-fe6a-7391-567f-04b56b9de657 from=Running to=WaitingToDequeue
node-068 2022-07-27T11:17:24.639Z [DEBUG] nomad/encrypter.go:397: nomad.keyring.replicator: starting encryption key replication
2022-07-27T11:17:24.639Z [TRACE] eventer/eventer.go:74: plugin_loader.mock_driver: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.639Z [TRACE] eventer/eventer.go:74: plugin_loader.exec: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.639Z [TRACE] eventer/eventer.go:74: plugin_loader.qemu: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.639Z [TRACE] eventer/eventer.go:74: plugin_loader.java: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.639Z [TRACE] eventer/eventer.go:74: plugin_loader.docker: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.639Z [TRACE] eventer/eventer.go:74: plugin_loader.qemu: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.639Z [TRACE] eventer/eventer.go:74: plugin_loader.raw_exec: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.639Z [TRACE] eventer/eventer.go:74: plugin_loader.exec: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.639Z [TRACE] eventer/eventer.go:74: plugin_loader.docker: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.639Z [TRACE] eventer/eventer.go:74: plugin_loader.java: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.639Z [TRACE] eventer/eventer.go:74: plugin_loader.mock_driver: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.639Z [TRACE] eventer/eventer.go:74: plugin_loader.raw_exec: task event loop shutdown: plugin_dir=""
node-064 2022-07-27T11:17:24.640Z [DEBUG] nomad/encrypter.go:409: nomad.keyring.replicator: exiting key replication
node-069 2022-07-27T11:17:24.640Z [INFO] raft@v1.3.5/api.go:570: nomad.raft: initial configuration: index=0 servers=[]
node-069 2022-07-27T11:17:24.640Z [INFO] raft@v1.3.5/raft.go:152: nomad.raft: entering follower state: follower="Node at 127.0.0.1:9138 [Follower]" leader=
node-069 2022-07-27T11:17:24.640Z [INFO] go-hclog@v1.2.0/stdlog.go:57: nomad: serf: EventMemberJoin: nomad-069.global 127.0.0.1
node-069 2022-07-27T11:17:24.640Z [INFO] nomad/server.go:1734: nomad: starting scheduling worker(s): num_workers=4 schedulers=["service", "batch", "system", "sysbatch", "noop", "_core"]
node-069 2022-07-27T11:17:24.640Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=d0ae57ff-e69f-e538-00a2-894d6a219b02 index=1 of=4
node-069 2022-07-27T11:17:24.640Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=aa635903-efeb-67a5-f787-33803b7db752 index=2 of=4
node-069 2022-07-27T11:17:24.640Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=aa635903-efeb-67a5-f787-33803b7db752 from=Starting to=Started
node-069 2022-07-27T11:17:24.640Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=aa635903-efeb-67a5-f787-33803b7db752 from=UnknownStatus to=Running
node-069 2022-07-27T11:17:24.640Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=aa635903-efeb-67a5-f787-33803b7db752
node-069 2022-07-27T11:17:24.640Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=aa635903-efeb-67a5-f787-33803b7db752 from=Running to=WaitingToDequeue
node-069 2022-07-27T11:17:24.640Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=7a5f1006-fad0-ec22-328b-66b28c72ca21 index=3 of=4
node-069 2022-07-27T11:17:24.640Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=b78bff0e-ea34-e3c0-f841-99e7205ce9a8 index=4 of=4
node-069 2022-07-27T11:17:24.640Z [INFO] nomad/server.go:1746: nomad: started scheduling worker(s): num_workers=4 schedulers=["service", "batch", "system", "sysbatch", "noop", "_core"]
node-070 2022-07-27T11:17:24.643Z [INFO] raft@v1.3.5/api.go:570: nomad.raft: initial configuration: index=0 servers=[]
node-070 2022-07-27T11:17:24.643Z [INFO] go-hclog@v1.2.0/stdlog.go:57: nomad: serf: EventMemberJoin: nomad-070.global 127.0.0.1
node-070 2022-07-27T11:17:24.643Z [INFO] nomad/server.go:1734: nomad: starting scheduling worker(s): num_workers=4 schedulers=["service", "batch", "system", "sysbatch", "noop", "_core"]
node-070 2022-07-27T11:17:24.643Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=70f494ae-3ab7-ae39-7f07-b8883d222033 index=1 of=4
node-070 2022-07-27T11:17:24.643Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=3dd28ea8-b03b-c519-2102-44e1b63cbfb8 index=2 of=4
node-070 2022-07-27T11:17:24.643Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=45d8f36f-5e1e-2de1-c678-fa70945c1989 index=3 of=4
node-070 2022-07-27T11:17:24.643Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=26e3ddf7-9887-a130-6f17-ba751c007bed index=4 of=4
node-070 2022-07-27T11:17:24.643Z [INFO] nomad/server.go:1746: nomad: started scheduling worker(s): num_workers=4 schedulers=["service", "batch", "system", "sysbatch", "noop", "_core"]
node-069 2022-07-27T11:17:24.640Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=7a5f1006-fad0-ec22-328b-66b28c72ca21 from=Starting to=Started
node-069 2022-07-27T11:17:24.643Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=7a5f1006-fad0-ec22-328b-66b28c72ca21 from=UnknownStatus to=Running
node-069 2022-07-27T11:17:24.643Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=7a5f1006-fad0-ec22-328b-66b28c72ca21
node-069 2022-07-27T11:17:24.643Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=7a5f1006-fad0-ec22-328b-66b28c72ca21 from=Running to=WaitingToDequeue
node-069 2022-07-27T11:17:24.640Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=d0ae57ff-e69f-e538-00a2-894d6a219b02 from=Starting to=Started
node-069 2022-07-27T11:17:24.643Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=d0ae57ff-e69f-e538-00a2-894d6a219b02 from=UnknownStatus to=Running
node-069 2022-07-27T11:17:24.643Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=d0ae57ff-e69f-e538-00a2-894d6a219b02
node-069 2022-07-27T11:17:24.643Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=d0ae57ff-e69f-e538-00a2-894d6a219b02 from=Running to=WaitingToDequeue
node-069 2022-07-27T11:17:24.643Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=b78bff0e-ea34-e3c0-f841-99e7205ce9a8 from=Starting to=Started
node-069 2022-07-27T11:17:24.643Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=b78bff0e-ea34-e3c0-f841-99e7205ce9a8 from=UnknownStatus to=Running
node-069 2022-07-27T11:17:24.643Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=b78bff0e-ea34-e3c0-f841-99e7205ce9a8
node-069 2022-07-27T11:17:24.643Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=b78bff0e-ea34-e3c0-f841-99e7205ce9a8 from=Running to=WaitingToDequeue
node-069 2022-07-27T11:17:24.643Z [INFO] nomad/serf.go:60: nomad: adding server: server="nomad-069.global (Addr: 127.0.0.1:9138) (DC: dc1)"
node-069 2022-07-27T11:17:24.643Z [DEBUG] nomad/encrypter.go:397: nomad.keyring.replicator: starting encryption key replication
2022-07-27T11:17:24.643Z [TRACE] eventer/eventer.go:74: plugin_loader.java: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.643Z [TRACE] eventer/eventer.go:74: plugin_loader.docker: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.643Z [TRACE] eventer/eventer.go:74: plugin_loader.mock_driver: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.643Z [TRACE] eventer/eventer.go:74: plugin_loader.raw_exec: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.643Z [TRACE] eventer/eventer.go:74: plugin_loader.exec: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.643Z [TRACE] eventer/eventer.go:74: plugin_loader.qemu: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.643Z [TRACE] eventer/eventer.go:74: plugin_loader.docker: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.643Z [TRACE] eventer/eventer.go:74: plugin_loader.mock_driver: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.643Z [TRACE] eventer/eventer.go:74: plugin_loader.raw_exec: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.643Z [TRACE] eventer/eventer.go:74: plugin_loader.exec: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.643Z [TRACE] eventer/eventer.go:74: plugin_loader.qemu: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.643Z [TRACE] eventer/eventer.go:74: plugin_loader.java: task event loop shutdown: plugin_dir=""
node-070 2022-07-27T11:17:24.643Z [INFO] raft@v1.3.5/raft.go:152: nomad.raft: entering follower state: follower="Node at 127.0.0.1:9140 [Follower]" leader=
node-070 2022-07-27T11:17:24.644Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=70f494ae-3ab7-ae39-7f07-b8883d222033 from=Starting to=Started
node-070 2022-07-27T11:17:24.644Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=70f494ae-3ab7-ae39-7f07-b8883d222033 from=UnknownStatus to=Running
node-070 2022-07-27T11:17:24.644Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=70f494ae-3ab7-ae39-7f07-b8883d222033
node-070 2022-07-27T11:17:24.644Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=70f494ae-3ab7-ae39-7f07-b8883d222033 from=Running to=WaitingToDequeue
node-070 2022-07-27T11:17:24.644Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=3dd28ea8-b03b-c519-2102-44e1b63cbfb8 from=Starting to=Started
node-070 2022-07-27T11:17:24.644Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=3dd28ea8-b03b-c519-2102-44e1b63cbfb8 from=UnknownStatus to=Running
node-070 2022-07-27T11:17:24.644Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=3dd28ea8-b03b-c519-2102-44e1b63cbfb8
node-070 2022-07-27T11:17:24.644Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=3dd28ea8-b03b-c519-2102-44e1b63cbfb8 from=Running to=WaitingToDequeue
node-070 2022-07-27T11:17:24.644Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=45d8f36f-5e1e-2de1-c678-fa70945c1989 from=Starting to=Started
node-070 2022-07-27T11:17:24.644Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=45d8f36f-5e1e-2de1-c678-fa70945c1989 from=UnknownStatus to=Running
node-070 2022-07-27T11:17:24.644Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=45d8f36f-5e1e-2de1-c678-fa70945c1989
node-070 2022-07-27T11:17:24.644Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=45d8f36f-5e1e-2de1-c678-fa70945c1989 from=Running to=WaitingToDequeue
node-070 2022-07-27T11:17:24.644Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=26e3ddf7-9887-a130-6f17-ba751c007bed from=Starting to=Started
node-070 2022-07-27T11:17:24.644Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=26e3ddf7-9887-a130-6f17-ba751c007bed from=UnknownStatus to=Running
node-070 2022-07-27T11:17:24.644Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=26e3ddf7-9887-a130-6f17-ba751c007bed
node-070 2022-07-27T11:17:24.644Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=26e3ddf7-9887-a130-6f17-ba751c007bed from=Running to=WaitingToDequeue
node-070 2022-07-27T11:17:24.644Z [INFO] nomad/serf.go:60: nomad: adding server: server="nomad-070.global (Addr: 127.0.0.1:9140) (DC: dc1)"
node-070 2022-07-27T11:17:24.644Z [DEBUG] nomad/encrypter.go:397: nomad.keyring.replicator: starting encryption key replication
node-065 2022-07-27T11:17:24.644Z [DEBUG] nomad/encrypter.go:409: nomad.keyring.replicator: exiting key replication
node-069 2022-07-27T11:17:24.644Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: memberlist: Initiating push/pull sync with: 127.0.0.1:9137
node-068 2022-07-27T11:17:24.644Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: memberlist: Stream connection from=127.0.0.1:35108
node-068 2022-07-27T11:17:24.644Z [INFO] go-hclog@v1.2.0/stdlog.go:57: nomad: serf: EventMemberJoin: nomad-069.global 127.0.0.1
node-068 2022-07-27T11:17:24.644Z [INFO] nomad/serf.go:60: nomad: adding server: server="nomad-069.global (Addr: 127.0.0.1:9138) (DC: dc1)"
node-069 2022-07-27T11:17:24.645Z [INFO] go-hclog@v1.2.0/stdlog.go:57: nomad: serf: EventMemberJoin: nomad-068.global 127.0.0.1
node-069 2022-07-27T11:17:24.645Z [INFO] nomad/serf.go:60: nomad: adding server: server="nomad-068.global (Addr: 127.0.0.1:9136) (DC: dc1)"
node-070 2022-07-27T11:17:24.645Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: memberlist: Initiating push/pull sync with: 127.0.0.1:9137
node-068 2022-07-27T11:17:24.645Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: memberlist: Stream connection from=127.0.0.1:35110
node-068 2022-07-27T11:17:24.645Z [INFO] go-hclog@v1.2.0/stdlog.go:57: nomad: serf: EventMemberJoin: nomad-070.global 127.0.0.1
node-070 2022-07-27T11:17:24.645Z [INFO] go-hclog@v1.2.0/stdlog.go:57: nomad: serf: EventMemberJoin: nomad-069.global 127.0.0.1
node-070 2022-07-27T11:17:24.645Z [INFO] go-hclog@v1.2.0/stdlog.go:57: nomad: serf: EventMemberJoin: nomad-068.global 127.0.0.1
node-068 2022-07-27T11:17:24.645Z [INFO] nomad/serf.go:60: nomad: adding server: server="nomad-070.global (Addr: 127.0.0.1:9140) (DC: dc1)"
node-070 2022-07-27T11:17:24.645Z [INFO] nomad/serf.go:60: nomad: adding server: server="nomad-069.global (Addr: 127.0.0.1:9138) (DC: dc1)"
node-070 2022-07-27T11:17:24.645Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: memberlist: Initiating push/pull sync with: 127.0.0.1:9139
node-069 2022-07-27T11:17:24.646Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: memberlist: Stream connection from=127.0.0.1:56032
node-069 2022-07-27T11:17:24.646Z [INFO] go-hclog@v1.2.0/stdlog.go:57: nomad: serf: EventMemberJoin: nomad-070.global 127.0.0.1
node-069 2022-07-27T11:17:24.646Z [INFO] nomad/serf.go:60: nomad: adding server: server="nomad-070.global (Addr: 127.0.0.1:9140) (DC: dc1)"
node-070 2022-07-27T11:17:24.648Z [INFO] nomad/serf.go:225: nomad: found expected number of peers, attempting to bootstrap cluster...: peers="127.0.0.1:9140,127.0.0.1:9138,127.0.0.1:9136"
node-070 2022-07-27T11:17:24.648Z [INFO] nomad/serf.go:60: nomad: adding server: server="nomad-068.global (Addr: 127.0.0.1:9136) (DC: dc1)"
node-068 2022-07-27T11:17:24.648Z [INFO] nomad/serf.go:189: nomad: disabling bootstrap mode because existing Raft peers being reported by peer: peer_name=nomad-070.global peer_address=127.0.0.1:9140
node-069 2022-07-27T11:17:24.648Z [INFO] nomad/serf.go:225: nomad: found expected number of peers, attempting to bootstrap cluster...: peers="127.0.0.1:9136,127.0.0.1:9140,127.0.0.1:9138"
node-063 2022-07-27T11:17:24.648Z [WARN] nomad/stats_fetcher.go:98: nomad.stats_fetcher: failed retrieving server health: server=nomad-066.global error="context deadline exceeded"
node-063 2022-07-27T11:17:24.648Z [WARN] nomad/stats_fetcher.go:98: nomad.stats_fetcher: failed retrieving server health: server=nomad-067.global error="context deadline exceeded"
node-063 2022-07-27T11:17:24.648Z [WARN] nomad/stats_fetcher.go:98: nomad.stats_fetcher: failed retrieving server health: server=nomad-063.global error="context deadline exceeded"
node-063 2022-07-27T11:17:24.648Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=32cec098-6878-ee8d-8777-6169e91b319c from=Paused to=Resuming
node-063 2022-07-27T11:17:24.648Z [TRACE] nomad/worker.go:340: worker: changed workload status: worker_id=32cec098-6878-ee8d-8777-6169e91b319c from=Paused to=WaitingToDequeue
node-063 2022-07-27T11:17:24.649Z [TRACE] nomad/worker.go:345: worker: changed worker status: worker_id=32cec098-6878-ee8d-8777-6169e91b319c from=Resuming to=Started
node-063 2022-07-27T11:17:24.649Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=32cec098-6878-ee8d-8777-6169e91b319c from=Started to=Stopped
node-063 2022-07-27T11:17:24.649Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=32cec098-6878-ee8d-8777-6169e91b319c from=WaitingToDequeue to=Stopped
node-063 2022-07-27T11:17:24.649Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=32cec098-6878-ee8d-8777-6169e91b319c
node-063 2022-07-27T11:17:24.648Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=548cb4fd-fd2d-a827-b127-22380bb3f292 from=WaitingToDequeue to=Backoff
node-063 2022-07-27T11:17:24.649Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=548cb4fd-fd2d-a827-b127-22380bb3f292 from=Pausing to=Stopped
node-063 2022-07-27T11:17:24.649Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=548cb4fd-fd2d-a827-b127-22380bb3f292 from=Backoff to=Stopped
node-063 2022-07-27T11:17:24.649Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=548cb4fd-fd2d-a827-b127-22380bb3f292
node-063 2022-07-27T11:17:24.648Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=976b9738-c634-9381-384f-7f0d8fbea4b8 from=WaitingToDequeue to=Backoff
node-063 2022-07-27T11:17:24.649Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=976b9738-c634-9381-384f-7f0d8fbea4b8 from=Started to=Stopped
node-063 2022-07-27T11:17:24.649Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=976b9738-c634-9381-384f-7f0d8fbea4b8 from=Backoff to=Stopped
node-063 2022-07-27T11:17:24.649Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=976b9738-c634-9381-384f-7f0d8fbea4b8
node-063 2022-07-27T11:17:24.648Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=af9a04fa-057b-a2b2-3f53-8f2813f447c2 from=WaitingToDequeue to=Backoff
node-063 2022-07-27T11:17:24.649Z [INFO] nomad/leader.go:86: nomad: cluster leadership lost
node-063 2022-07-27T11:17:24.649Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=af9a04fa-057b-a2b2-3f53-8f2813f447c2 from=Pausing to=Stopped
node-063 2022-07-27T11:17:24.649Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=af9a04fa-057b-a2b2-3f53-8f2813f447c2 from=Backoff to=Stopped
node-063 2022-07-27T11:17:24.649Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=af9a04fa-057b-a2b2-3f53-8f2813f447c2
node-067 2022-07-27T11:17:24.654Z [DEBUG] nomad/encrypter.go:409: nomad.keyring.replicator: exiting key replication
node-068 2022-07-27T11:17:24.693Z [WARN] raft@v1.3.5/raft.go:205: nomad.raft: no known peers, aborting election
node-070 2022-07-27T11:17:24.711Z [WARN] raft@v1.3.5/raft.go:217: nomad.raft: heartbeat timeout reached, starting election: last-leader=
node-070 2022-07-27T11:17:24.711Z [INFO] raft@v1.3.5/raft.go:255: nomad.raft: entering candidate state: node="Node at 127.0.0.1:9140 [Candidate]" term=2
node-070 2022-07-27T11:17:24.711Z [DEBUG] raft@v1.3.5/raft.go:273: nomad.raft: votes: needed=2
node-070 2022-07-27T11:17:24.711Z [DEBUG] raft@v1.3.5/raft.go:292: nomad.raft: vote granted: from=c61f03dc-5505-9fbc-7d27-c7cf83915347 term=2 tally=1
node-068 2022-07-27T11:17:24.711Z [DEBUG] raft@v1.3.5/raft.go:1488: nomad.raft: lost leadership because received a requestVote with a newer term
node-069 2022-07-27T11:17:24.711Z [DEBUG] raft@v1.3.5/raft.go:1488: nomad.raft: lost leadership because received a requestVote with a newer term
node-070 2022-07-27T11:17:24.711Z [DEBUG] raft@v1.3.5/raft.go:292: nomad.raft: vote granted: from=d7f46dd1-5d25-3c67-8a91-ac25c836c2fe term=2 tally=2
node-070 2022-07-27T11:17:24.711Z [INFO] raft@v1.3.5/raft.go:297: nomad.raft: election won: tally=2
node-070 2022-07-27T11:17:24.711Z [INFO] raft@v1.3.5/raft.go:369: nomad.raft: entering leader state: leader="Node at 127.0.0.1:9140 [Leader]"
node-070 2022-07-27T11:17:24.711Z [INFO] raft@v1.3.5/raft.go:490: nomad.raft: added peer, starting replication: peer=d7f46dd1-5d25-3c67-8a91-ac25c836c2fe
node-070 2022-07-27T11:17:24.711Z [INFO] raft@v1.3.5/raft.go:490: nomad.raft: added peer, starting replication: peer=22e1a9ff-a48b-4a15-eac6-83c02a5e92ae
node-070 2022-07-27T11:17:24.712Z [INFO] nomad/leader.go:73: nomad: cluster leadership acquired
node-068 2022-07-27T11:17:24.712Z [WARN] raft@v1.3.5/raft.go:1328: nomad.raft: failed to get previous log: previous-index=1 last-index=0 error="log not found"
node-070 2022-07-27T11:17:24.712Z [WARN] raft@v1.3.5/replication.go:259: nomad.raft: appendEntries rejected, sending older logs: peer="{Voter 22e1a9ff-a48b-4a15-eac6-83c02a5e92ae 127.0.0.1:9136}" next=1
node-070 2022-07-27T11:17:24.712Z [INFO] raft@v1.3.5/replication.go:441: nomad.raft: pipelining replication: peer="{Voter d7f46dd1-5d25-3c67-8a91-ac25c836c2fe 127.0.0.1:9138}"
node-070 2022-07-27T11:17:24.712Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=70f494ae-3ab7-ae39-7f07-b8883d222033 from=Started to=Pausing
node-070 2022-07-27T11:17:24.712Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=3dd28ea8-b03b-c519-2102-44e1b63cbfb8 from=Started to=Pausing
node-070 2022-07-27T11:17:24.712Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=45d8f36f-5e1e-2de1-c678-fa70945c1989 from=Started to=Pausing
node-070 2022-07-27T11:17:24.712Z [INFO] raft@v1.3.5/replication.go:441: nomad.raft: pipelining replication: peer="{Voter 22e1a9ff-a48b-4a15-eac6-83c02a5e92ae 127.0.0.1:9136}"
node-070 2022-07-27T11:17:24.712Z [TRACE] nomad/leader.go:1717: nomad.core: initializing keyring
node-070 2022-07-27T11:17:24.712Z [INFO] nomad/leader.go:1737: nomad.core: initialized keyring: id=cd3433db-3c85-02a7-e89c-c49b183137b4
node-070 2022-07-27T11:17:24.717Z [TRACE] nomad/fsm.go:357: nomad.fsm: ClusterSetMetadata: cluster_id=66a8eaa6-317c-e587-68f3-d4e076e802b1 create_time=1658920644713717011
node-070 2022-07-27T11:17:24.718Z [INFO] nomad/leader.go:1753: nomad.core: established cluster id: cluster_id=66a8eaa6-317c-e587-68f3-d4e076e802b1 create_time=1658920644713717011
node-070 2022-07-27T11:17:24.718Z [INFO] nomad/leader.go:1805: nomad: eval broker status modified: paused=false
node-070 2022-07-27T11:17:24.718Z [INFO] nomad/leader.go:1812: nomad: blocked evals status modified: paused=false
node-070 2022-07-27T11:17:24.719Z [TRACE] drainer/watch_jobs.go:150: nomad.drain.job_watcher: getting job allocs at index: index=1
autopilot_test.go:224: adding server s4
2022-07-27T11:17:24.723Z [TRACE] eventer/eventer.go:74: plugin_loader.docker: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.723Z [TRACE] eventer/eventer.go:74: plugin_loader.mock_driver: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.723Z [TRACE] eventer/eventer.go:74: plugin_loader.exec: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.723Z [TRACE] eventer/eventer.go:74: plugin_loader.qemu: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.723Z [TRACE] eventer/eventer.go:74: plugin_loader.java: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.723Z [TRACE] eventer/eventer.go:74: plugin_loader.raw_exec: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.723Z [TRACE] eventer/eventer.go:74: plugin_loader.mock_driver: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.723Z [TRACE] eventer/eventer.go:74: plugin_loader.raw_exec: task event loop shutdown: plugin_dir=""
node-071 2022-07-27T11:17:24.725Z [INFO] raft@v1.3.5/api.go:570: nomad.raft: initial configuration: index=0 servers=[]
2022-07-27T11:17:24.725Z [TRACE] eventer/eventer.go:74: plugin_loader.exec: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.725Z [TRACE] eventer/eventer.go:74: plugin_loader.qemu: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.725Z [TRACE] eventer/eventer.go:74: plugin_loader.java: task event loop shutdown: plugin_dir=""
2022-07-27T11:17:24.725Z [TRACE] eventer/eventer.go:74: plugin_loader.docker: task event loop shutdown: plugin_dir=""
node-071 2022-07-27T11:17:24.725Z [INFO] raft@v1.3.5/raft.go:152: nomad.raft: entering follower state: follower="Node at 127.0.0.1:9142 [Follower]" leader=
node-071 2022-07-27T11:17:24.725Z [INFO] go-hclog@v1.2.0/stdlog.go:57: nomad: serf: EventMemberJoin: nomad-071.global 127.0.0.1
node-071 2022-07-27T11:17:24.725Z [INFO] nomad/server.go:1734: nomad: starting scheduling worker(s): num_workers=4 schedulers=["service", "batch", "system", "sysbatch", "noop", "_core"]
node-071 2022-07-27T11:17:24.725Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=67deb249-1d1d-6b2d-4042-e23eb10e0697 index=1 of=4
node-071 2022-07-27T11:17:24.725Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=92fd3d9e-7d5b-a854-706f-b54ad7c4f00a index=2 of=4
node-071 2022-07-27T11:17:24.725Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=7400da47-1812-ec99-7e91-2dc29a6c3a1d index=3 of=4
node-071 2022-07-27T11:17:24.725Z [DEBUG] nomad/server.go:1741: nomad: started scheduling worker: id=0cfa8a16-4163-2395-bc86-3ef387e9c885 index=4 of=4
node-071 2022-07-27T11:17:24.725Z [INFO] nomad/server.go:1746: nomad: started scheduling worker(s): num_workers=4 schedulers=["service", "batch", "system", "sysbatch", "noop", "_core"]
node-071 2022-07-27T11:17:24.726Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=67deb249-1d1d-6b2d-4042-e23eb10e0697 from=Starting to=Started
node-071 2022-07-27T11:17:24.726Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=67deb249-1d1d-6b2d-4042-e23eb10e0697 from=UnknownStatus to=Running
node-071 2022-07-27T11:17:24.726Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=67deb249-1d1d-6b2d-4042-e23eb10e0697
node-071 2022-07-27T11:17:24.726Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=67deb249-1d1d-6b2d-4042-e23eb10e0697 from=Running to=WaitingToDequeue
node-071 2022-07-27T11:17:24.726Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=92fd3d9e-7d5b-a854-706f-b54ad7c4f00a from=Starting to=Started
node-071 2022-07-27T11:17:24.726Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=92fd3d9e-7d5b-a854-706f-b54ad7c4f00a from=UnknownStatus to=Running
node-071 2022-07-27T11:17:24.726Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: memberlist: Initiating push/pull sync with: 127.0.0.1:9137
node-071 2022-07-27T11:17:24.726Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=7400da47-1812-ec99-7e91-2dc29a6c3a1d from=Starting to=Started
node-071 2022-07-27T11:17:24.726Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=7400da47-1812-ec99-7e91-2dc29a6c3a1d from=UnknownStatus to=Running
node-071 2022-07-27T11:17:24.726Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=7400da47-1812-ec99-7e91-2dc29a6c3a1d
node-071 2022-07-27T11:17:24.726Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=7400da47-1812-ec99-7e91-2dc29a6c3a1d from=Running to=WaitingToDequeue
node-068 2022-07-27T11:17:24.726Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: memberlist: Stream connection from=127.0.0.1:35138
node-071 2022-07-27T11:17:24.726Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=0cfa8a16-4163-2395-bc86-3ef387e9c885 from=Starting to=Started
node-071 2022-07-27T11:17:24.726Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=0cfa8a16-4163-2395-bc86-3ef387e9c885 from=UnknownStatus to=Running
node-071 2022-07-27T11:17:24.726Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=0cfa8a16-4163-2395-bc86-3ef387e9c885
node-071 2022-07-27T11:17:24.726Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=0cfa8a16-4163-2395-bc86-3ef387e9c885 from=Running to=WaitingToDequeue
node-071 2022-07-27T11:17:24.726Z [DEBUG] nomad/worker.go:391: worker: running: worker_id=92fd3d9e-7d5b-a854-706f-b54ad7c4f00a
node-071 2022-07-27T11:17:24.726Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=92fd3d9e-7d5b-a854-706f-b54ad7c4f00a from=Running to=WaitingToDequeue
node-071 2022-07-27T11:17:24.726Z [INFO] nomad/serf.go:60: nomad: adding server: server="nomad-071.global (Addr: 127.0.0.1:9142) (DC: dc1)"
node-071 2022-07-27T11:17:24.726Z [DEBUG] nomad/encrypter.go:397: nomad.keyring.replicator: starting encryption key replication
node-068 2022-07-27T11:17:24.726Z [INFO] go-hclog@v1.2.0/stdlog.go:57: nomad: serf: EventMemberJoin: nomad-071.global 127.0.0.1
node-068 2022-07-27T11:17:24.726Z [INFO] nomad/serf.go:60: nomad: adding server: server="nomad-071.global (Addr: 127.0.0.1:9142) (DC: dc1)"
node-071 2022-07-27T11:17:24.726Z [INFO] go-hclog@v1.2.0/stdlog.go:57: nomad: serf: EventMemberJoin: nomad-069.global 127.0.0.1
node-071 2022-07-27T11:17:24.726Z [INFO] go-hclog@v1.2.0/stdlog.go:57: nomad: serf: EventMemberJoin: nomad-070.global 127.0.0.1
node-071 2022-07-27T11:17:24.726Z [INFO] go-hclog@v1.2.0/stdlog.go:57: nomad: serf: EventMemberJoin: nomad-068.global 127.0.0.1
node-071 2022-07-27T11:17:24.726Z [INFO] nomad/serf.go:60: nomad: adding server: server="nomad-069.global (Addr: 127.0.0.1:9138) (DC: dc1)"
node-071 2022-07-27T11:17:24.727Z [INFO] nomad/serf.go:189: nomad: disabling bootstrap mode because existing Raft peers being reported by peer: peer_name=nomad-068.global peer_address=127.0.0.1:9136
node-071 2022-07-27T11:17:24.727Z [INFO] nomad/serf.go:60: nomad: adding server: server="nomad-070.global (Addr: 127.0.0.1:9140) (DC: dc1)"
node-071 2022-07-27T11:17:24.727Z [INFO] nomad/serf.go:60: nomad: adding server: server="nomad-068.global (Addr: 127.0.0.1:9136) (DC: dc1)"
node-069 2022-07-27T11:17:24.739Z [INFO] go-hclog@v1.2.0/stdlog.go:57: nomad: serf: EventMemberJoin: nomad-071.global 127.0.0.1
node-069 2022-07-27T11:17:24.739Z [INFO] nomad/serf.go:60: nomad: adding server: server="nomad-071.global (Addr: 127.0.0.1:9142) (DC: dc1)"
node-068 2022-07-27T11:17:24.739Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-071.global
node-068 2022-07-27T11:17:24.739Z [TRACE] nomad/encrypter.go:435: nomad.keyring.replicator: replicating new key: id=cd3433db-3c85-02a7-e89c-c49b183137b4
node-068 2022-07-27T11:17:24.739Z [TRACE] nomad/encrypter.go:473: nomad.keyring.replicator: added key: key=cd3433db-3c85-02a7-e89c-c49b183137b4
node-071 2022-07-27T11:17:24.741Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-069.global
node-068 2022-07-27T11:17:24.741Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-069.global
node-069 2022-07-27T11:17:24.741Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-071.global
node-069 2022-07-27T11:17:24.744Z [TRACE] nomad/encrypter.go:435: nomad.keyring.replicator: replicating new key: id=cd3433db-3c85-02a7-e89c-c49b183137b4
node-070 2022-07-27T11:17:24.744Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-071.global
node-069 2022-07-27T11:17:24.744Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-070.global
node-069 2022-07-27T11:17:24.744Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-070.global
node-069 2022-07-27T11:17:24.744Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-070.global
node-069 2022-07-27T11:17:24.744Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-070.global
node-069 2022-07-27T11:17:24.744Z [TRACE] nomad/encrypter.go:473: nomad.keyring.replicator: added key: key=cd3433db-3c85-02a7-e89c-c49b183137b4
node-069 2022-07-27T11:17:24.781Z [TRACE] nomad/fsm.go:357: nomad.fsm: ClusterSetMetadata: cluster_id=66a8eaa6-317c-e587-68f3-d4e076e802b1 create_time=1658920644713717011
node-068 2022-07-27T11:17:24.786Z [TRACE] nomad/fsm.go:357: nomad.fsm: ClusterSetMetadata: cluster_id=66a8eaa6-317c-e587-68f3-d4e076e802b1 create_time=1658920644713717011
node-071 2022-07-27T11:17:24.810Z [WARN] raft@v1.3.5/raft.go:205: nomad.raft: no known peers, aborting election
node-069 2022-07-27T11:17:24.827Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-069.global
node-069 2022-07-27T11:17:24.827Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-071.global
node-069 2022-07-27T11:17:24.827Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-069.global
node-071 2022-07-27T11:17:24.827Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-071.global
node-071 2022-07-27T11:17:24.827Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-070.global
node-069 2022-07-27T11:17:24.839Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-069.global
node-071 2022-07-27T11:17:24.839Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-071.global
node-071 2022-07-27T11:17:24.839Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-069.global
node-068 2022-07-27T11:17:24.839Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-071.global
node-070 2022-07-27T11:17:24.841Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-070.global
node-069 2022-07-27T11:17:24.841Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-070.global
node-068 2022-07-27T11:17:24.844Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-071.global
node-068 2022-07-27T11:17:24.927Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-070.global
node-069 2022-07-27T11:17:24.938Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-070.global
node-069 2022-07-27T11:17:24.938Z [DEBUG] go-hclog@v1.2.0/stdlog.go:55: nomad: serf: messageJoinType: nomad-070.global
node-070 2022-07-27T11:17:25.368Z [TRACE] nomad/worker.go:330: worker: changed workload status: worker_id=3dd28ea8-b03b-c519-2102-44e1b63cbfb8 from=WaitingToDequeue to=Paused
node-070 2022-07-27T11:17:25.449Z [TRACE] nomad/worker.go:330: worker: changed workload status: worker_id=45d8f36f-5e1e-2de1-c678-fa70945c1989 from=WaitingToDequeue to=Paused
node-070 2022-07-27T11:17:25.450Z [TRACE] nomad/worker.go:330: worker: changed workload status: worker_id=70f494ae-3ab7-ae39-7f07-b8883d222033 from=WaitingToDequeue to=Paused
node-071 2022-07-27T11:17:29.756Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=67deb249-1d1d-6b2d-4042-e23eb10e0697 from=WaitingToDequeue to=Backoff
node-071 2022-07-27T11:17:29.764Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=92fd3d9e-7d5b-a854-706f-b54ad7c4f00a from=WaitingToDequeue to=Backoff
node-071 2022-07-27T11:17:29.774Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=0cfa8a16-4163-2395-bc86-3ef387e9c885 from=WaitingToDequeue to=Backoff
node-071 2022-07-27T11:17:29.777Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=67deb249-1d1d-6b2d-4042-e23eb10e0697 from=Backoff to=WaitingToDequeue
node-071 2022-07-27T11:17:29.785Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=92fd3d9e-7d5b-a854-706f-b54ad7c4f00a from=Backoff to=WaitingToDequeue
node-071 2022-07-27T11:17:29.794Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=0cfa8a16-4163-2395-bc86-3ef387e9c885 from=Backoff to=WaitingToDequeue
node-071 2022-07-27T11:17:29.836Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=7400da47-1812-ec99-7e91-2dc29a6c3a1d from=WaitingToDequeue to=Backoff
node-071 2022-07-27T11:17:29.856Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=7400da47-1812-ec99-7e91-2dc29a6c3a1d from=Backoff to=WaitingToDequeue
autopilot_test.go:230: autopilot_test.go:231: didn't find map[89cdb0a1-dab4-c1d2-09d2-7ce7e2c111f9:true] in []raft.ServerID{"c61f03dc-5505-9fbc-7d27-c7cf83915347", "d7f46dd1-5d25-3c67-8a91-ac25c836c2fe", "22e1a9ff-a48b-4a15-eac6-83c02a5e92ae"}

node-071 2022-07-27T11:17:31.742Z [INFO] nomad/server.go:665: nomad: shutting down server
node-071 2022-07-27T11:17:31.742Z [WARN] go-hclog@v1.2.0/stdlog.go:59: nomad: serf: Shutdown without a Leave
node-071 2022-07-27T11:17:31.742Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=7400da47-1812-ec99-7e91-2dc29a6c3a1d from=WaitingToDequeue to=Backoff
node-071 2022-07-27T11:17:31.742Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=7400da47-1812-ec99-7e91-2dc29a6c3a1d from=Started to=Stopped
node-071 2022-07-27T11:17:31.742Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=7400da47-1812-ec99-7e91-2dc29a6c3a1d from=Backoff to=Stopped
node-071 2022-07-27T11:17:31.742Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=7400da47-1812-ec99-7e91-2dc29a6c3a1d
node-071 2022-07-27T11:17:31.742Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=0cfa8a16-4163-2395-bc86-3ef387e9c885 from=WaitingToDequeue to=Backoff
node-071 2022-07-27T11:17:31.742Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=0cfa8a16-4163-2395-bc86-3ef387e9c885 from=Started to=Stopped
node-071 2022-07-27T11:17:31.742Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=0cfa8a16-4163-2395-bc86-3ef387e9c885 from=Backoff to=Stopped
node-071 2022-07-27T11:17:31.742Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=0cfa8a16-4163-2395-bc86-3ef387e9c885
node-071 2022-07-27T11:17:31.742Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=67deb249-1d1d-6b2d-4042-e23eb10e0697 from=WaitingToDequeue to=Backoff
node-071 2022-07-27T11:17:31.742Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=67deb249-1d1d-6b2d-4042-e23eb10e0697 from=Started to=Stopped
node-071 2022-07-27T11:17:31.742Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=67deb249-1d1d-6b2d-4042-e23eb10e0697 from=Backoff to=Stopped
node-071 2022-07-27T11:17:31.742Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=67deb249-1d1d-6b2d-4042-e23eb10e0697
node-071 2022-07-27T11:17:31.742Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=92fd3d9e-7d5b-a854-706f-b54ad7c4f00a from=WaitingToDequeue to=Backoff
node-071 2022-07-27T11:17:31.742Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=92fd3d9e-7d5b-a854-706f-b54ad7c4f00a from=Started to=Stopped
node-071 2022-07-27T11:17:31.742Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=92fd3d9e-7d5b-a854-706f-b54ad7c4f00a from=Backoff to=Stopped
node-071 2022-07-27T11:17:31.742Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=92fd3d9e-7d5b-a854-706f-b54ad7c4f00a
node-070 2022-07-27T11:17:31.743Z [INFO] nomad/server.go:665: nomad: shutting down server
node-070 2022-07-27T11:17:31.743Z [WARN] go-hclog@v1.2.0/stdlog.go:59: nomad: serf: Shutdown without a Leave
node-070 2022-07-27T11:17:31.743Z [DEBUG] nomad/leader.go:82: nomad: shutting down leader loop
node-070 2022-07-27T11:17:31.743Z [INFO] raft@v1.3.5/replication.go:489: nomad.raft: aborting pipeline replication: peer="{Voter 22e1a9ff-a48b-4a15-eac6-83c02a5e92ae 127.0.0.1:9136}"
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=70f494ae-3ab7-ae39-7f07-b8883d222033 from=Paused to=Resuming
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=3dd28ea8-b03b-c519-2102-44e1b63cbfb8 from=Paused to=Resuming
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=45d8f36f-5e1e-2de1-c678-fa70945c1989 from=Paused to=Resuming
node-070 2022-07-27T11:17:31.743Z [INFO] raft@v1.3.5/replication.go:489: nomad.raft: aborting pipeline replication: peer="{Voter d7f46dd1-5d25-3c67-8a91-ac25c836c2fe 127.0.0.1:9138}"
node-069 2022-07-27T11:17:31.743Z [INFO] nomad/server.go:665: nomad: shutting down server
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=26e3ddf7-9887-a130-6f17-ba751c007bed from=WaitingToDequeue to=Backoff
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=26e3ddf7-9887-a130-6f17-ba751c007bed from=Started to=Stopped
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=26e3ddf7-9887-a130-6f17-ba751c007bed from=Backoff to=Stopped
node-070 2022-07-27T11:17:31.743Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=26e3ddf7-9887-a130-6f17-ba751c007bed
node-070 2022-07-27T11:17:31.743Z [INFO] nomad/leader.go:86: nomad: cluster leadership lost
node-068 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=713527ec-db64-4ba1-50e3-ef25e7962875 from=WaitingToDequeue to=Backoff
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:340: worker: changed workload status: worker_id=70f494ae-3ab7-ae39-7f07-b8883d222033 from=Paused to=WaitingToDequeue
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:345: worker: changed worker status: worker_id=70f494ae-3ab7-ae39-7f07-b8883d222033 from=Resuming to=Started
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=70f494ae-3ab7-ae39-7f07-b8883d222033 from=Started to=Stopped
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=70f494ae-3ab7-ae39-7f07-b8883d222033 from=WaitingToDequeue to=Stopped
node-070 2022-07-27T11:17:31.743Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=70f494ae-3ab7-ae39-7f07-b8883d222033
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:340: worker: changed workload status: worker_id=3dd28ea8-b03b-c519-2102-44e1b63cbfb8 from=Paused to=WaitingToDequeue
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:345: worker: changed worker status: worker_id=3dd28ea8-b03b-c519-2102-44e1b63cbfb8 from=Resuming to=Started
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=3dd28ea8-b03b-c519-2102-44e1b63cbfb8 from=Started to=Stopped
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=3dd28ea8-b03b-c519-2102-44e1b63cbfb8 from=WaitingToDequeue to=Stopped
node-070 2022-07-27T11:17:31.743Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=3dd28ea8-b03b-c519-2102-44e1b63cbfb8
node-068 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=1f86f651-fe6a-7391-567f-04b56b9de657 from=WaitingToDequeue to=Backoff
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:340: worker: changed workload status: worker_id=45d8f36f-5e1e-2de1-c678-fa70945c1989 from=Paused to=WaitingToDequeue
node-068 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=c55e2cbd-f8ca-de7a-0950-e1a5b93b7e7f from=WaitingToDequeue to=Backoff
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:345: worker: changed worker status: worker_id=45d8f36f-5e1e-2de1-c678-fa70945c1989 from=Resuming to=Started
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=45d8f36f-5e1e-2de1-c678-fa70945c1989 from=Started to=Stopped
node-070 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=45d8f36f-5e1e-2de1-c678-fa70945c1989 from=WaitingToDequeue to=Stopped
node-070 2022-07-27T11:17:31.743Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=45d8f36f-5e1e-2de1-c678-fa70945c1989
node-069 2022-07-27T11:17:31.743Z [WARN] go-hclog@v1.2.0/stdlog.go:59: nomad: serf: Shutdown without a Leave
node-068 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=96867b49-2a02-eabe-80f2-1fff56d73276 from=WaitingToDequeue to=Backoff
node-069 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=aa635903-efeb-67a5-f787-33803b7db752 from=WaitingToDequeue to=Backoff
node-069 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=aa635903-efeb-67a5-f787-33803b7db752 from=Started to=Stopped
node-069 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=aa635903-efeb-67a5-f787-33803b7db752 from=Backoff to=Stopped
node-069 2022-07-27T11:17:31.743Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=aa635903-efeb-67a5-f787-33803b7db752
node-069 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=7a5f1006-fad0-ec22-328b-66b28c72ca21 from=WaitingToDequeue to=Backoff
node-069 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=7a5f1006-fad0-ec22-328b-66b28c72ca21 from=Started to=Stopped
node-069 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=7a5f1006-fad0-ec22-328b-66b28c72ca21 from=Backoff to=Stopped
node-069 2022-07-27T11:17:31.743Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=7a5f1006-fad0-ec22-328b-66b28c72ca21
node-069 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=b78bff0e-ea34-e3c0-f841-99e7205ce9a8 from=WaitingToDequeue to=Backoff
node-069 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=b78bff0e-ea34-e3c0-f841-99e7205ce9a8 from=Started to=Stopped
node-069 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=b78bff0e-ea34-e3c0-f841-99e7205ce9a8 from=Backoff to=Stopped
node-069 2022-07-27T11:17:31.743Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=b78bff0e-ea34-e3c0-f841-99e7205ce9a8
node-069 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=d0ae57ff-e69f-e538-00a2-894d6a219b02 from=WaitingToDequeue to=Backoff
node-069 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=d0ae57ff-e69f-e538-00a2-894d6a219b02 from=Started to=Stopped
node-069 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=d0ae57ff-e69f-e538-00a2-894d6a219b02 from=Backoff to=Stopped
node-069 2022-07-27T11:17:31.743Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=d0ae57ff-e69f-e538-00a2-894d6a219b02
node-068 2022-07-27T11:17:31.743Z [INFO] nomad/server.go:665: nomad: shutting down server
node-068 2022-07-27T11:17:31.743Z [WARN] go-hclog@v1.2.0/stdlog.go:59: nomad: serf: Shutdown without a Leave
node-069 2022-07-27T11:17:31.743Z [DEBUG] nomad/encrypter.go:409: nomad.keyring.replicator: exiting key replication
node-068 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=c55e2cbd-f8ca-de7a-0950-e1a5b93b7e7f from=Started to=Stopped
node-068 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=c55e2cbd-f8ca-de7a-0950-e1a5b93b7e7f from=Backoff to=Stopped
node-068 2022-07-27T11:17:31.743Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=c55e2cbd-f8ca-de7a-0950-e1a5b93b7e7f
node-068 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=713527ec-db64-4ba1-50e3-ef25e7962875 from=Started to=Stopped
node-068 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=713527ec-db64-4ba1-50e3-ef25e7962875 from=Backoff to=Stopped
node-068 2022-07-27T11:17:31.743Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=713527ec-db64-4ba1-50e3-ef25e7962875
node-068 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=96867b49-2a02-eabe-80f2-1fff56d73276 from=Started to=Stopped
node-068 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=96867b49-2a02-eabe-80f2-1fff56d73276 from=Backoff to=Stopped
node-068 2022-07-27T11:17:31.743Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=96867b49-2a02-eabe-80f2-1fff56d73276
node-068 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:237: worker: changed worker status: worker_id=1f86f651-fe6a-7391-567f-04b56b9de657 from=Started to=Stopped
node-068 2022-07-27T11:17:31.743Z [TRACE] nomad/worker.go:260: worker: changed workload status: worker_id=1f86f651-fe6a-7391-567f-04b56b9de657 from=Backoff to=Stopped
node-068 2022-07-27T11:17:31.743Z [DEBUG] nomad/worker.go:369: worker: stopped: worker_id=1f86f651-fe6a-7391-567f-04b56b9de657
--- FAIL: TestAutopilot_RollingUpdate (7.11s)

This test exercises similar code paths to #13930
@tgross tgross mentioned this issue Jul 28, 2022
Closed
@tgross tgross mentioned this issue Sep 1, 2022
Merged
@tgross tgross self-assigned this Sep 1, 2022
@tgross tgross added this to the 1.4.0 milestone Sep 1, 2022
@tgross tgross mentioned this issue Sep 6, 2022
Merged
@github-actions
Copy link

github-actions bot commented Jan 6, 2023

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 6, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@tgross tgross

Labels
theme/flaky-tests type/bug
Projects
None yet
Milestone
1.4.0
Development

Successfully merging a pull request may close this issue.

autopilot: deflake tests hashicorp/nomad
1 participant
@tgross