Provider and Consumer's OnAcknowledgementPacket should only log an error

[shaspitz]

Problem

OnAcknowledgementPacket currently closes the CCV channel in the event that an IBC err ack is returned from the provider. If CCV is working as intended, these paths should never be executed. Therefore a returned err ack is probably a symptom of a deeper bug that needs to be addressed manually.

Closing criteria

Either only log an error in OnAcknowledgementPacket, or halt the consumer. Let users know that there's a critical bug to be addressed, without polluting state by executing automatic recovery logic.

[shaspitz]

Edit: this issue also applies to the provider's OnAcknowledgementPacket method, and should be changed for the same reasons

[mpoke]

@jtremback This is a product discussion. Should a critical bug (e.g., marshaling errors) result in the consumer shutting down?

[shaspitz]

Note if we're able to implement epochs, we could elegantly halt the consumer upon receiving an err ack, be able to patch the consumer binary with no state migrations, and resume the consumer once fixed. Any epoch packets constructed before/during the halt would be queued up, and ready to send once the consumer is back up

[mpoke]

Closing as non-issue. The behavior is as intended.