Stop recovery when follower offset was already updated

[mmaslankaprv]

Recovery and replicate stms are not synchronized. It may be the case
when both of stms are active at the same time that the same batch is
delivered to the follower twice. In general this batch duplication is
harmless as Raft is not vulnerable for messages redelivery but it may
cause unnecessary truncation and latency increase.

Added a check validating expected log end offset right before sending
recovery append entries request. This will prevent sending the same set
of batches twice to the follower.

Fixes: #14413

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.2.x
• v23.1.x
• v22.3.x

Release Notes

• none

[vbotbuildovich]

/backport v23.2.x

[vbotbuildovich]

/backport v23.1.x

[vbotbuildovich]

Failed to create a backport PR to v23.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-15049-v23.2.x-941 remotes/upstream/v23.2.x
git cherry-pick -x cb4e6075344b50cc3606e4d7946a4ad985404c4c 7bb54d3c5e747f0809e7d9fc74a36e2a8a34e25a

Workflow run logs.

[vbotbuildovich]

Failed to create a backport PR to v23.1.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-15049-v23.1.x-120 remotes/upstream/v23.1.x
git cherry-pick -x cb4e6075344b50cc3606e4d7946a4ad985404c4c 7bb54d3c5e747f0809e7d9fc74a36e2a8a34e25a

Workflow run logs.