CI Failure (Set changed size during iteration) in CloudStorageChunkReadTest.test_read_when_cache_smaller_than_segment_size

[ztlpn]

https://buildkite.com/redpanda/vtools/builds/9259#018a4d33-bbf5-4d66-93c4-3fd78903abcd

Module: rptest.tests.cloud_storage_chunk_read_path_test
Class:  CloudStorageChunkReadTest
Method: test_read_when_cache_smaller_than_segment_size

test_id:    rptest.tests.cloud_storage_chunk_read_path_test.CloudStorageChunkReadTest.test_read_when_cache_smaller_than_segment_size
status:     FAIL
run time:   55.521 seconds


    RuntimeError('Set changed size during iteration')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 82, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/cloud_storage_chunk_read_path_test.py", line 377, in test_read_when_cache_smaller_than_segment_size
    for file in chunks_found:
RuntimeError: Set changed size during iteration

[abhijat]

reclassified as sev-low because this is a test bug. the cache observer runs on a thread and the following sequence happened here:

1. test starts cache observer
2. cache observer checks variable in loop, sleeps and mutates the data set
3. test issues stop to cache observer
4. test starts reading from data set
5. the observer may have been asleep, it wakes up and mutates the data set before stopping