feat: instrument failed chunk encoding/decoding

[trevorwhitney]

What this PR does / why we need it:

This is related to problems we've seen with corrupted chunks. These corruptions are very rare, making it hard to troubleshoot. This instrumentation should allow us to alert on and track occurrences to help us investigate when they happen.

[grafanabot]

This PR must be merged before a backport PR will be created.

[grafanabot]

This PR must be merged before a backport PR will be created.

[grafanabot]

The backport to k212 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-13684-to-k212 origin/k212
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 5a87ccb648ee3bf48a3704643ae9923d64651aed

When the conflicts are resolved, stage and commit the changes:

git add . && git cherry-pick --continue

If you have the GitHub CLI installed:

# Push the branch to GitHub:
git push --set-upstream origin backport-13684-to-k212
# Create the PR body template
PR_BODY=$(gh pr view 13684 --json body --template 'Backport 5a87ccb648ee3bf48a3704643ae9923d64651aed from #13684{{ "\n\n---\n\n" }}{{ index . "body" }}')
# Create the PR on GitHub
echo "${PR_BODY}" | gh pr create --title 'chore: [k212] feat: instrument failed chunk encoding/decoding' --body-file - --label 'size/L' --label 'product-approved' --label 'backport' --base k212 --milestone k212 --web

Or, if you don't have the GitHub CLI installed (we recommend you install it!):

# Push the branch to GitHub:
git push --set-upstream origin backport-13684-to-k212

# Create a pull request where the `base` branch is `k212` and the `compare`/`head` branch is `backport-13684-to-k212`.

# Remove the local backport branch
git switch main
git branch -D backport-13684-to-k212