fix: remove race condition on start container retries

[colin-axner]

While investigating cosmos/ibc-go#2827 I came across an issue locally. If I created an interchain test with many validators/full nodes, I would get indeterminate errors on configuration startup. One issue was actually an address being returned twice in the standard output and then being reused for gentx, {address}\n{address} was the string being passed around. Another was the genesis file already existing (trying to execute simd init twice)

It became apparent to me that on a rare occasion, some of the config commands would get executed twice. With the help of @chatton, we determined it was likely the retry in StartContainer. I believe while a node was processing a request, the timeout would occur. Since this is a request being sent to a server, there's no way to ensure that request doesn't get executed. A retry with the same request would be sent and then I would see an error roughly saying that this tx already executed.

The proposed changes removes the retry entirely. I'm not sure what use it currently provides. I propose using the calling context's timeout value, if it does not exist, use a standard timeout (30 seconds in the current pr, fine with whatever, but should likely be a defined variable). The only scenario I think a retry could be useful is if the first request attempted is expected to fail and the second may succeed, but the current code doesn't seem to handler this case as the first error would be marked as unrecoverable. How likely is this? Should a defensive retry be used if the timeout hasn't been reached? Are there any common errors returned which warrant a retry? Like a connection being reset?

Is there any context I am missing on why the code is the way it is? Thanks for taking to review this, we would love to patch up these indeterminacy issues.

ref #402

[colin-axner]

Marking as r4r. The CI on ibc-go doesn't run into this issue which is why I haven't re-reviewed this until now, but I noticed this failure, which seems like it's very possible a command is being executed twice given the error:

found multiple connections on chain-a for client 07-tendermint-0

[agouin]

Thank you! I like the approach of the longer timeout and no retries to prevent potential double execution.