stuck cloud-storage readers with chunked reads enabled

[jason-da-redpanda]

Version & Environment

23.2.24

What went wrong?

cloud-storage readers were getting stuck..

More in depth analysis has been done internally (please add/also please change title if it ends up not being chunked reads)..

cloud-storage is creating a reader and this reader is trying to acquire a segment_reader_units semaphore.
This semaphore limits number of segment readers that can be simultaneously created.

Disabling chunk reads via
cloud_storage_disable_chunk_reads=true
and restarting the broker cleared the issue

What should have happened instead?

Don't get stuck when chunked reads are enabled

How to reproduce the issue?

Have not reproduced at this point

Additional information

Messages such as these were observed with DEBUG :
DEBUG 2024-02-02 08:39:49,166 [shard 20] cloud_storage - materialized_resources.cc:379 - Materialized segment {kafka/request/95} base-offset 321713857 is not stale: 2 readers=0

JIRA Link: CORE-1752

[jason-da-redpanda]

related to finding the cause
cloud_storage: Add watchdog utility and use it to detect stuck eviction loop #16466

[jason-da-redpanda]

Variation of this seen ..in recent issue

 ERROR 2024-03-26 13:46:13,269 [shard 4:main] cloud_storage - remote_segment.cc:1470 - remote_segment_batch_reader {"927653cc/kafka/my-topic.traces/1_16024049/14914798-14952049-9074378-23-v1.log.23"} stop operation stuck
  ERROR 2024-03-26 13:46:13,269 [shard 4:main] cloud_storage - remote_partition.cc:920 - Eviction loop for partition {kafka/my-topic.traces/1} stuck
2024-03-26 13:46:13.269	

Disabling chunked reads again mitigates

[piyushredpanda]

We now RCAed this fully. Closing.