Create port bindings instead of relying on docker random port bindings

[colin-axner]

Based on a lot of investigation, we believe there to be a bug in docker which is causing non-deterministic failures in our e2e's due to incorrect packet routing. The problem arises when relying on docker to do the port bindings for the rpc/grpc/p2p ports. I propose we provide in the host config for creating a container, the port bindings.

from the issue above, here is the summarized problem:

in either a workstation or in a shared CI agent or in a production docker cluster, any process can bind an arbitrary high port to itself, accidentally triggering this issue if the port is in the sequence that docker expects to be free.

and the summarized solution:

If a stable, predictable port is important, then specifying the port-mapping is the correct approach.

---

We will need to determine a starting port address and then increment by X for each chain and by Y for each node. For example:

Start at 32000.
Increment by 1000 per chain
Increment by 5 per node

chain a val 0: 32000, 32001, 32002 etc
chain a val 1: 32005, etc
chain b val 0: 33000, etc

The cosmos chain node type can be modified to store the full port bindings (it currently stores the host rpc and grpc ports). I believe the chain interface would need to be modified to support incrementing the starting port by its index. Maybe there's an alternative approach?

[colin-axner]

Another option I explored was trying to prevent the port binding on [::]:<port>, but I could not seem to find a way to do this. Disabling IPv6 in the created docker network did not affect the container port bindings

[agouin]

Thank you for the investigation @colin-axner !
It would be wonderful if this is truly the root cause of the dreaded intermittent malformed HTTP response errors.

I took a stab at introducing the explicit port bindings while keeping automatic port allocation in #444