CI Failure (assert offset == o) in TimeQueryTest.test_timequery

[vbotbuildovich]

https://buildkite.com/redpanda/vtools/builds/12363

Module: rptest.tests.timequery_test
Class: TimeQueryTest
Method: test_timequery
Arguments: {
    "spillover": true,
    "batch_cache": false,
    "cloud_storage": true
}

test_id:    TimeQueryTest.test_timequery
status:     FAIL
run time:   33.746 seconds

AssertionError()
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 269, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 104, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/timequery_test.py", line 372, in test_timequery
    self._do_test_timequery(cloud_storage, batch_cache, spillover)
  File "/home/ubuntu/redpanda/tests/rptest/tests/timequery_test.py", line 346, in _do_test_timequery
    self._test_timequery(cluster=self.redpanda,
  File "/home/ubuntu/redpanda/tests/rptest/tests/timequery_test.py", line 210, in _test_timequery
    assert offset == o
AssertionError

JIRA Link: CORE-1895

[michael-redpanda]

Note to developer: please provide a message with the assertion when addressing this issue

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12600

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12629

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12629

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12827
*https://buildkite.com/redpanda/vtools/builds/12843

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12856

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12890

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/13345

[WillemKauf]

Without going into the details of the CI failure itself, whats interesting about this is that it has only been triggered in CDT over the last few months, and every report of it failing also has a number of other failed ducktapes with it- usually node operation or partition moving tests.

Most specifically, this seems to fail quite often with rptest.tests.cloud_storage_timing_stress_test::CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves and rptest.tests.node_pool_migration_test::NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.

So, that leads me to ask the question, is ducktape parallelization and the interaction of these tests responsible for the failures seen in the TimeQueryTest?

Timequeries seem very stable on their own.

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/14763
*https://buildkite.com/redpanda/vtools/builds/14777

[piyushredpanda]

Closing older-bot-filed CI issues as we transition to a more reliable system.