docker: command not found error on a loader node, while on others nodes of loaders set it is OK

[dimakr]

longevity-1tb-5days-azure-test build on 2024.1.10 failed with the error:

2024-09-18 17:41:45.945: (TestFrameworkEvent Severity.ERROR) period_type=one-time event_id=af9ca0b3-3f55-48e3-b419-a6ea37f3dc8b, source=LongevityTest.test_custom_time (longevity_test.LongevityTest)() message=Traceback (most recent call last):
File "/home/ubuntu/scylla-cluster-tests/longevity_test.py", line 144, in test_custom_time
self.run_prepare_write_cmd()
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/loader_utils.py", line 240, in run_prepare_write_cmd
self._run_all_stress_cmds(write_queue, params={'stress_cmd': prepare_write_cmd,
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/loader_utils.py", line 113, in _run_all_stress_cmds
stress_queue.append(self.run_stress_thread(**stress_params))
File "/home/ubuntu/scylla-cluster-tests/sdcm/tester.py", line 1874, in run_stress_thread
return self.run_stress_cassandra_thread(**params)
File "/home/ubuntu/scylla-cluster-tests/sdcm/tester.py", line 1916, in run_stress_cassandra_thread
cs_thread = CassandraStressThread(loader_set=self.loaders,
File "/home/ubuntu/scylla-cluster-tests/sdcm/stress_thread.py", line 98, in __init__
super().__init__(loader_set=loader_set, stress_cmd=stress_cmd, timeout=timeout,
File "/home/ubuntu/scylla-cluster-tests/sdcm/stress/base.py", line 57, in __init__
RemoteDocker.pull_image(loader, self.docker_image_name)
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/docker_remote.py", line 146, in pull_image
node.remoter.run(
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 614, in run
result = _run()
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/decorators.py", line 70, in inner
return func(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 605, in _run
return self._run_execute(cmd, timeout, ignore_status, verbose, new_session, watchers)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 538, in _run_execute
result = connection.run(**command_kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/libssh2_client/__init__.py", line 620, in run
return self._complete_run(channel, exception, timeout_reached, timeout, result, warn, stdout, stderr)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/libssh2_client/__init__.py", line 655, in _complete_run
raise UnexpectedExit(result)
sdcm.remote.libssh2_client.exceptions.UnexpectedExit: Encountered a bad command exit code!
Command: 'sudo docker pull scylladb/cassandra-stress:3.13.0'
Exit code: 1
Stdout:
Stderr:
sudo: docker: command not found

As per system log on loader nodes the docker was started on all loader nodes:

❯ grep -Ein 'started.*docker' loader-set-d5d8458d/longevity-tls-1tb-7d-2024-1-loader-node-d5d8458d-eastus-*/system.log
loader-set-d5d8458d/longevity-tls-1tb-7d-2024-1-loader-node-d5d8458d-eastus-1/system.log:2491:Sep 18 17:21:40 longevity-tls-1tb-7d-2024-1-loader-node-d5d8458d-eastus-1 systemd[1]: Started Docker Application Container Engine.
loader-set-d5d8458d/longevity-tls-1tb-7d-2024-1-loader-node-d5d8458d-eastus-2/system.log:2482:Sep 18 17:21:41 longevity-tls-1tb-7d-2024-1-loader-node-d5d8458d-eastus-2 systemd[1]: Started Docker Application Container Engine.
loader-set-d5d8458d/longevity-tls-1tb-7d-2024-1-loader-node-d5d8458d-eastus-3/system.log:2547:Sep 18 17:21:37 longevity-tls-1tb-7d-2024-1-loader-node-d5d8458d-eastus-3 systemd[1]: Started Docker Application Container Engine.
loader-set-d5d8458d/longevity-tls-1tb-7d-2024-1-loader-node-d5d8458d-eastus-4/system.log:2525:Sep 18 17:21:36 longevity-tls-1tb-7d-2024-1-loader-node-d5d8458d-eastus-4 systemd[1]: Started Docker Application Container Engine.

But docker pull command was successful only on loaders 1 and 2:

❯ grep -Ein 'pull' loader-set-d5d8458d/longevity-tls-1tb-7d-2024-1-loader-node-d5d8458d-eastus-*/system.log
loader-set-d5d8458d/longevity-tls-1tb-7d-2024-1-loader-node-d5d8458d-eastus-1/system.log:2560:Sep 18 17:40:45 longevity-tls-1tb-7d-2024-1-loader-node-d5d8458d-eastus-1 sudo[18891]:   ubuntu : TTY=unknown ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/docker pull scylladb/cassandra-stress:3.13.0
loader-set-d5d8458d/longevity-tls-1tb-7d-2024-1-loader-node-d5d8458d-eastus-2/system.log:2549:Sep 18 17:40:58 longevity-tls-1tb-7d-2024-1-loader-node-d5d8458d-eastus-2 sudo[18816]:   ubuntu : TTY=unknown ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/docker pull scylladb/cassandra-stress:3.13.0

Installation details

Cluster size: 4 nodes (Standard_L16s_v3)

Scylla Nodes used in this run:

• longevity-tls-1tb-7d-2024-1-db-node-d5d8458d-eastus-4 (48.217.82.213 | 10.0.0.8) (shards: 14)
• longevity-tls-1tb-7d-2024-1-db-node-d5d8458d-eastus-3 (48.217.81.152 | 10.0.0.7) (shards: 14)
• longevity-tls-1tb-7d-2024-1-db-node-d5d8458d-eastus-2 (48.217.81.22 | 10.0.0.6) (shards: 14)
• longevity-tls-1tb-7d-2024-1-db-node-d5d8458d-eastus-1 (48.217.84.24 | 10.0.0.5) (shards: 14)

OS / Image: /subscriptions/6c268694-47ab-43ab-b306-3c5514bc4112/resourceGroups/scylla-images/providers/Microsoft.Compute/images/scylla-2024.1.10-x86_64-2024-09-18T18-52-41 (azure: undefined_region)

Test: longevity-1tb-5days-azure-test
Test id: d5d8458d-890a-4042-a75e-9871edafad72
Test name: enterprise-2024.1/longevity/longevity-1tb-5days-azure-test
Test method: longevity_test.LongevityTest.test_custom_time
Test config file(s):

• longevity-1TB-5days-authorization-and-tls-ssl.yaml

Logs and commands

• Restore Monitor Stack command: $ hydra investigate show-monitor d5d8458d-890a-4042-a75e-9871edafad72
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs d5d8458d-890a-4042-a75e-9871edafad72

Logs:

• db-cluster-d5d8458d.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/d5d8458d-890a-4042-a75e-9871edafad72/20240918_174232/db-cluster-d5d8458d.tar.gz
• sct-runner-events-d5d8458d.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/d5d8458d-890a-4042-a75e-9871edafad72/20240918_174232/sct-runner-events-d5d8458d.tar.gz
• sct-d5d8458d.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/d5d8458d-890a-4042-a75e-9871edafad72/20240918_174232/sct-d5d8458d.log.tar.gz
• loader-set-d5d8458d.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/d5d8458d-890a-4042-a75e-9871edafad72/20240918_174232/loader-set-d5d8458d.tar.gz
• monitor-set-d5d8458d.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/d5d8458d-890a-4042-a75e-9871edafad72/20240918_174232/monitor-set-d5d8458d.tar.gz

Jenkins job URL
Argus