CI Failure (AssertionError: Transformed partitions must the same or larger watermark than source) in TxAtomicProduceConsumeTest.test_basic_tx_consumer_transform_produce

[vbotbuildovich]

https://buildkite.com/redpanda/redpanda/builds/50784

Module: rptest.transactions.tx_atomic_produce_consume_test
Class: TxAtomicProduceConsumeTest
Method: test_basic_tx_consumer_transform_produce

test_id:    TxAtomicProduceConsumeTest.test_basic_tx_consumer_transform_produce
status:     FAIL
run time:   70.140 seconds

AssertionError('Transformed partitions must the same or larger watermark than source')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 105, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/transactions/tx_atomic_produce_consume_test.py", line 370, in test_basic_tx_consumer_transform_produce
    transformer.validate_transform()
  File "/root/tests/rptest/transactions/tx_atomic_produce_consume_test.py", line 271, in validate_transform
    assert dst_high_watermarks[
AssertionError: Transformed partitions must the same or larger watermark than source

JIRA Link: CORE-4602

[vbotbuildovich]

*https://buildkite.com/redpanda/redpanda/builds/50831

[bharathv]

I suspect this is not fully fixed yet.

Example: https://buildkite.com/redpanda/redpanda/builds/51066#01907ade-af1f-4fa5-9bf4-303c4bf4f226