Test stuck during db node_setup (apt-get update)

[soyacz]

Issue description

• This issue is a regression.
• It is unknown if this issue is a regression.

During db node-4 preparation, apt-get update got frozen (while installing scylla manager agent):

2024-08-03T03:13:47.007+00:00 longevity-twcs-48h-master-db-node-9e80e135-4   !NOTICE | sudo[8491]: scyllaadm : PWD=/home/scyllaadm ; USER=root ; COMMAND=/usr/bin/apt-get update
2024-08-03T03:13:47.007+00:00 longevity-twcs-48h-master-db-node-9e80e135-4     !INFO | sudo[8491]: pam_unix(sudo:session): session opened for user root(uid=0) by scyllaadm(uid=1000)
...

And process sudo[8491] never ends.
culprit line is

scylla-cluster-tests/sdcm/cluster.py

Line 1794 in 090519e

self.remoter.sudo("apt-get update", ignore_status=True)

but actually it could happen in other cases too and we have no protection against frozen setup.

We could add timeout to all apt update and/or add protection on higher level around

scylla-cluster-tests/sdcm/tester.py

Line 1021 in 090519e

with ThreadPoolExecutor(max_workers=4) as executor:

Impact

Frozen test, lost time and money

How frequently does it reproduce?

First time seen

Installation details

Cluster size: 4 nodes (i3en.2xlarge)

Scylla Nodes used in this run:

• longevity-twcs-48h-master-db-node-9e80e135-4 (34.241.150.172 | 10.4.9.233) (shards: -1)
• longevity-twcs-48h-master-db-node-9e80e135-3 (18.201.19.226 | 10.4.10.183) (shards: 7)
• longevity-twcs-48h-master-db-node-9e80e135-2 (18.201.166.170 | 10.4.10.140) (shards: 7)
• longevity-twcs-48h-master-db-node-9e80e135-1 (63.33.69.199 | 10.4.11.8) (shards: 7)

OS / Image: ami-00355186dfc821610 (aws: undefined_region)

Test: longevity-twcs-48h-test
Test id: 9e80e135-c117-4b9b-a87d-15aeb5dc363c
Test name: scylla-master/tier1/longevity-twcs-48h-test
Test method: longevity_twcs_test.TWCSLongevityTest.test_custom_time
Test config file(s):

• longevity-twcs-48h.yaml

Logs and commands

• Restore Monitor Stack command: $ hydra investigate show-monitor 9e80e135-c117-4b9b-a87d-15aeb5dc363c
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs 9e80e135-c117-4b9b-a87d-15aeb5dc363c

Logs:

• db-cluster-9e80e135.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/9e80e135-c117-4b9b-a87d-15aeb5dc363c/20240804_092049/db-cluster-9e80e135.tar.gz
• sct-runner-events-9e80e135.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/9e80e135-c117-4b9b-a87d-15aeb5dc363c/20240804_092049/sct-runner-events-9e80e135.tar.gz
• sct-9e80e135.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/9e80e135-c117-4b9b-a87d-15aeb5dc363c/20240804_092049/sct-9e80e135.log.tar.gz
• loader-set-9e80e135.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/9e80e135-c117-4b9b-a87d-15aeb5dc363c/20240804_092049/loader-set-9e80e135.tar.gz
• monitor-set-9e80e135.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/9e80e135-c117-4b9b-a87d-15aeb5dc363c/20240804_092049/monitor-set-9e80e135.tar.gz

Jenkins job URL
Argus

[fruch]

@soyacz

let keep track of similar cases, I don't remember seeing lots of such cases.

[fruch]

on more then I've seen, don't know if related, seem that there are places we scan the scylla repo in apt update commands ,while the we didn't update the key.

we introduced this new key for signing deb packages recently, it might be connected.

[vponomaryov]

And process sudo[8491] never ends. culprit line is https://github.com/scylladb/scylla-cluster-tests/blob/master/sdcm/cluster.py#L1794 but actually it could happen in other cases too and we have no protection against frozen setup.

@soyacz please, use commit IDs, not branch names when you refer to some specific code line.
Moreover such a popular file for changes.
In some short period of time it will move significantly.