PartitionClient may silently read from replica

[crepererum]

When a cluster re-balances, the existing partition leader may become a replica, but will not actively report that to the PartitionClient. Even worse, the read lag may trigger this Redpanda quirk even under Kafka:

rskafka/src/client/partition.rs

Lines 149 to 159 in 822186d

	// Redpanda never sends OffsetOutOfRange even when it should. "Luckily" it does not support deletions so we can
	// implement a simple heuristic.
	if partition.high_watermark.0 < offset {
	warn!(
	"This message looks like Redpanda wants to report a OffsetOutOfRange but doesn't."
	);
	return Err(Error::ServerError(
	ProtocolError::OffsetOutOfRange,
	String::from("Offset out of range"),
	));
	}

[alamb]

"we saw this happen on our production system at InfluxData" so it is not a theoretical issue ;)