🩹 Add http2 transport configuration to work around http2 upstream bug on tcp drop

[serbrech]

TL;DR:

When a TCP connection is dropped, there is a bug in the http2 implementation in go that misses to mark the connection as failed. as a result, the connection stays in the pool, and the calls using that connection hit a context deadline.
The connection can stay in the pool fur up to 17 min (with default linux kernel settings).

This configures the transport to enable ReadIdleTimeout and Ping Timeout.
internaslly, it adds the h2 middleware on the transport which will issue http2 ping frames on the connections when no reads are occuring. if the server does not answer the ping (for example, the tcp connection was dropped), the connection is effectively closed, working around the golang upstream issue.
on the next call, a connection will be re-opened, and no context.Timeout is experienced at the app level.

for more details, see :

• azure sdk issue: Outbound calls timing out with context deadline exceeded when TCP connection is been drop azure-sdk-for-go#21346 (comment)
• go upstream issue: x/net/http2: raciness in stream close prevents closing of failed connections, blocking subsequent requests golang/go#59690

other context:

We have experienced sporadic context deadline on cosmosdb operations that we attributed to server side issues.
I think this might be the actual root cause, as Julien Strohecker experienced this issue using cosmosdb sdk track 2 in a use case with higher scale than ours.

This same fix is applied in our service already.

[codecov-commenter]

Codecov Report

Merging #29 (63da122) into main (3d961b8) will increase coverage by 1.63%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main      #29      +/-   ##
==========================================
+ Coverage   81.77%   83.40%   +1.63%     
==========================================
  Files           4        4              
  Lines         203      223      +20     
==========================================
+ Hits          166      186      +20     
  Misses         32       32              
  Partials        5        5              

Files	Coverage Δ
pkg/middleware/default_http_client.go	100.00% <100.00%> (ø)

[Bryce-Soghigian]

LGTM 🚀🚀🚀🚀🚀

Thanks for contributing lets make this a good holiday project