[v23.3.x] [CORE-4443] Pandaproxy: Avoid large allocations whilst serializing JSON

[vbotbuildovich]

Backport #20827 to branch v23.3.x. Note that this issue was created as a placeholder, since the original PR's commit(s) could not be automatically cherry-picked.
Here are the commands to execute:

git checkout v23.3.x
git checkout -b vbotbuildovich/backport-20827-v23.3.x-825
git cherry-pick -x 7cea48d60091e79ff8e85a0f2f7aacd07b30eef2 ebc4c1347ef8cb9955dd671fb69a7bfbe103a34c 9fcb311283019d0dd359cb89dfd99a7edf289438 a455d2374ec799c14a627989df09ff320ddf0447 5d90218887aec7e665626a8ba1ed5deb40ab3a21 d74ec2aef846f895d0ef5f6e5b05c08120baeb14 f17567fe898c37960896ef303e123ef4c43bcd17 744b0f01067395b993a48005387dae8d356c9f25 5f3236c0ead0f8e9c439e957e04bc342f0f136de c676f6e8b7294019cd4d517dd93a6c4002473c4f 240c682d9fa655d3b19525260d31f5326fb48f54 8560e0ab3653ea3235f170043741a213dca1a66b 6795578e7cd2d171b9733e4da95e829dc0476105 c93475720e00301e2c192bd2a44eac8b9a564c6e ac7c7dd8f2a7177e88a886753c687f46e84c3f1b

git push origin vbotbuildovich/backport-20827-v23.3.x-825
gh pr create \
  --title "[v23.3.x] [CORE-4443] Pandaproxy: Avoid large allocations whilst serializing JSON" \
  --base "v23.3.x" \
  --label "kind/backport" \
  --head "vbotbuildovich/backport-20827-v23.3.x-825" \
  --draft \
  --repo "redpanda-data/redpanda" \
  --milestone "v23.3.x-next" \
  --body "Backport of PR https://github.com/redpanda-data/redpanda/pull/20827 "

JIRA Link: [CORE-5539](https://redpandadata.atlassian.net/browse/CORE-5539)

[BenPope]

Fixed by #21469