Backport of report intermediate error messages during request forwarding into release/1.11.x

[hc-github-team-secure-vault-core]

Backport

This PR is auto-generated from #20643 to be assessed for backporting due to the inclusion of the label backport/1.11.x.

🚨

Warning automatic cherry-pick of commits failed. If the first commit failed,
you will see a blank no-op commit below. If at least one commit succeeded, you
will see the cherry-picked commits up to, not including, the commit where
the merge conflict occurred.

The person who merged in the original PR is:
@hghaf099
This person should manually cherry-pick the original PR into a new backport PR,
and close this one when the manual backport PR is merged in.

merge conflict error: POST https://api.github.com/repos/hashicorp/vault/merges: 409 Merge conflict []

The below text is copied from the body of the original PR.

---

OSS version of https://github.com/hashicorp/vault-enterprise/pull/3937

Addresses VAULT-15375
Core elides intermediate error messages during request forwarding.

There are two main places where Alex and I find that we need to report the errors considering request forwarding. One is in the request handling where we initially forward the request upon getting a read only error. Two is in the replication code here.

Having aggregate the errors in the above places, we noticed that the reported error has many duplicates. Below is a sample of the existing behaviour:

    backend_revocation_queue_ent_test.go:391: error revoking leaf cert, Error making API request.
        
        Namespace: ns1/
        URL: PUT https://127.0.0.1:59296/v1/pki/revoke
        Code: 500. Errors:
        
        * 2 errors occurred:
        	* errors from both primary and secondary; primary error was 2 errors occurred:
        	* 2 errors occurred:
        	* error from primary active: failed to write WAL entries for Delta CRLs: failed to write cross-cluster delta WAL entry: error saving delta CRL WAL entry: forwarded writer lacked replication client: cannot write to readonly storage
        	* error from perf secondary active: error persisting cross-cluster revocation request: refusing to write to write-forwarded storage when not the active node: cannot write to readonly storage
        This may occur when the active node of the primary performance replication cluster is unavailable.
        
        
        	* error from a standby node: error persisting cross-cluster revocation request: refusing to write to write-forwarded storage when not the active node: cannot write to readonly storage
        This may occur when the active node of the primary performance replication cluster is unavailable.
        
        ; secondary errors follow
        	* 2 errors occurred:
        	* 2 errors occurred:
        	* error from primary active: failed to write WAL entries for Delta CRLs: failed to write cross-cluster delta WAL entry: error saving delta CRL WAL entry: forwarded writer lacked replication client: cannot write to readonly storage
        	* error from perf secondary active: error persisting cross-cluster revocation request: refusing to write to write-forwarded storage when not the active node: cannot write to readonly storage
        This may occur when the active node of the primary performance replication cluster is unavailable.
        
        
        	* error from a standby node: error persisting cross-cluster revocation request: refusing to write to write-forwarded storage when not the active node: cannot write to readonly storage
        This may occur when the active node of the primary performance replication cluster is unavailable.

It would be very confusing to report such an error to the client. We found that the issue is in respondErrorCommon code where the call back function passed in the errwrap.Walk function will aggregate errors multiple times. Fixing the issue will result in the following reported error:

 backend_revocation_queue_ent_test.go:391: error revoking leaf cert, Error making API request.
        
        Namespace: ns1/
        URL: PUT https://127.0.0.1:63885/v1/pki/revoke
        Code: 500. Errors:
        
         errors from both primary and secondary; primary error was 2 errors occurred:
        	* error from primary active: failed to write WAL entries for Delta CRLs: failed to write cross-cluster delta WAL entry: error saving delta CRL WAL entry: forwarded writer lacked replication client: cannot write to readonly storage
        	* error from perf secondary active: error persisting cross-cluster revocation request: refusing to write to write-forwarded storage when not the active node: cannot write to readonly storage
        
        ; secondary errors follow: error from a standby node: error persisting cross-cluster revocation request: refusing to write to write-forwarded storage when not the active node: cannot write to readonly storage

---

Overview of commits

• db5f416

[hashicorp-cla]

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

---

temp seems not to be a GitHub user.
You need a GitHub account to be able to sign the CLA. If you already have a GitHub account, please add the email address used for this commit to your account.

_{Have you signed the CLA already but the status is still pending? Recheck it.}