Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ICS4: Fix Sequencing logic in UpgradeTry #5419

Merged
merged 15 commits into from
Dec 20, 2023
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 64 additions & 39 deletions modules/core/04-channel/keeper/upgrade.go
Original file line number Diff line number Diff line change
Expand Up @@ -99,44 +99,6 @@ func (k Keeper) ChanUpgradeTry(
)
}

// construct counterpartyChannel from existing information and provided counterpartyUpgradeSequence
// create upgrade fields from counterparty proposed upgrade and own verified connection hops
proposedUpgradeFields := types.UpgradeFields{
Ordering: counterpartyUpgradeFields.Ordering,
ConnectionHops: proposedConnectionHops,
Version: counterpartyUpgradeFields.Version,
}

var (
err error
upgrade types.Upgrade
)

// NOTE: if an upgrade exists (crossing hellos) then use existing upgrade fields
// otherwise, run the upgrade init sub-protocol
upgrade, found = k.GetUpgrade(ctx, portID, channelID)
if found {
proposedUpgradeFields = upgrade.Fields
} else {
// NOTE: OnChanUpgradeInit will not be executed by the application
upgrade, err = k.ChanUpgradeInit(ctx, portID, channelID, proposedUpgradeFields)
if err != nil {
return types.Channel{}, types.Upgrade{}, errorsmod.Wrap(err, "failed to initialize upgrade")
}

channel, upgrade = k.WriteUpgradeInitChannel(ctx, portID, channelID, upgrade, upgrade.Fields.Version)

// if the counterparty sequence is greater than the current sequence, we fast-forward to the counterparty sequence.
if counterpartyUpgradeSequence > channel.UpgradeSequence {
channel.UpgradeSequence = counterpartyUpgradeSequence
k.SetChannel(ctx, portID, channelID, channel)
}
}

if err := k.checkForUpgradeCompatibility(ctx, proposedUpgradeFields, counterpartyUpgradeFields); err != nil {
return types.Channel{}, types.Upgrade{}, errorsmod.Wrap(err, "failed upgrade compatibility check")
}

// construct expected counterparty channel from information in state
// only the counterpartyUpgradeSequence is provided by the relayer
counterpartyConnectionHops := []string{connection.GetCounterparty().GetConnectionID()}
Expand All @@ -161,8 +123,39 @@ func (k Keeper) ChanUpgradeTry(
return types.Channel{}, types.Upgrade{}, errorsmod.Wrap(err, "failed to verify counterparty channel state")
}

var (
err error
upgrade types.Upgrade
isCrossingHello bool
)

upgrade, isCrossingHello = k.GetUpgrade(ctx, portID, channelID)
if counterpartyUpgradeSequence < channel.UpgradeSequence {
return channel, upgrade, types.NewUpgradeError(channel.UpgradeSequence-1, errorsmod.Wrapf(
// In this case, the counterparty upgrade is outdated. We want to force the counterparty
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

delicious documentation @AdityaSripal, very helpful

// to abort their upgrade and come back to sync with our own upgrade sequence.
var upgradeSequence uint64
if isCrossingHello {
// In the crossing hello case, we already have an upgrade but it is at a higher sequence than the counterparty.
// Thus, our upgrade should take priority. We force the counterparty to abort their upgrade by invalidating all counterparty
// upgrade attempts below our own sequence by setting errorReceipt to upgradeSequence - 1.
// The upgrade handshake may then proceed on the counterparty with our sequence
upgradeSequence = channel.UpgradeSequence - 1
} else {
// In the case, that we are in a non-crossing hello (i.e. upgrade does not exist on our side),
// the sequence on both sides should move to a fresh sequence on the next upgrade attempt.
// Thus, we write an error receipt with our own upgrade sequence which will cause the counterparty
// to cancel their upgrade and move to the same sequence. When a new upgrade attempt is started from either
// side, it will be a fresh sequence for both sides (i.e. channel.upgradeSequence + 1)
upgradeSequence = channel.UpgradeSequence
}

// NOTE: Two possible outcomes may occur in this scenario.
// The ChanUpgradeCancel datagram may reach the counterparty first, which will cause the counterparty to cancel. The counterparty
// may then receive a TRY with our channel upgrade sequence and correctly increment their sequence to become synced with our upgrade attempt.
// The ChanUpgradeTry message may arrive first, in this case, **IF** the upgrade fields are mutually compatible; the counterparty will simply
// fast forward their sequence to our own and continue the upgrade. The following ChanUpgradeCancel message will be rejected as it is below the current sequence.

return channel, upgrade, types.NewUpgradeError(upgradeSequence, errorsmod.Wrapf(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its intensional that the channel and upgrade are returned here instead of empty structs, right?

This is the only place where an upgrade error is return from the try handler and it looks like the channel returned is used for emitting error receipt events in this case:
https://github.com/cosmos/ibc-go/blob/04-channel-upgrades/modules/core/keeper/msg_server.go#L818-L820

types.ErrInvalidUpgradeSequence, "counterparty upgrade sequence < current upgrade sequence (%d < %d)", counterpartyUpgradeSequence, channel.UpgradeSequence,
))
}
Expand All @@ -179,6 +172,38 @@ func (k Keeper) ChanUpgradeTry(
return types.Channel{}, types.Upgrade{}, errorsmod.Wrap(err, "failed to verify counterparty upgrade")
}

// construct counterpartyChannel from existing information and provided counterpartyUpgradeSequence
// create upgrade fields from counterparty proposed upgrade and own verified connection hops
proposedUpgradeFields := types.UpgradeFields{
Ordering: counterpartyUpgradeFields.Ordering,
ConnectionHops: proposedConnectionHops,
Version: counterpartyUpgradeFields.Version,
}

// NOTE: if an upgrade exists (crossing hellos) then use existing upgrade fields
// otherwise, run the upgrade init sub-protocol
if isCrossingHello {
proposedUpgradeFields = upgrade.Fields
} else {
// NOTE: OnChanUpgradeInit will not be executed by the application
upgrade, err = k.ChanUpgradeInit(ctx, portID, channelID, proposedUpgradeFields)
if err != nil {
return types.Channel{}, types.Upgrade{}, errorsmod.Wrap(err, "failed to initialize upgrade")
}

channel, upgrade = k.WriteUpgradeInitChannel(ctx, portID, channelID, upgrade, upgrade.Fields.Version)
}

if err := k.checkForUpgradeCompatibility(ctx, proposedUpgradeFields, counterpartyUpgradeFields); err != nil {
return types.Channel{}, types.Upgrade{}, errorsmod.Wrap(err, "failed upgrade compatibility check")
}

// if the counterparty sequence is greater than the current sequence, we fast-forward to the counterparty sequence.
if counterpartyUpgradeSequence > channel.UpgradeSequence {
channel.UpgradeSequence = counterpartyUpgradeSequence
k.SetChannel(ctx, portID, channelID, channel)
}

if err := k.startFlushing(ctx, portID, channelID, &upgrade); err != nil {
return types.Channel{}, types.Upgrade{}, err
}
Expand Down
189 changes: 182 additions & 7 deletions modules/core/04-channel/keeper/upgrade_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ func (suite *KeeperTestSuite) TestChanUpgradeTry() {
func() {
counterpartyUpgrade.Fields.ConnectionHops = []string{ibctesting.InvalidID}
},
types.ErrIncompatibleCounterpartyUpgrade,
commitmenttypes.ErrInvalidProof,
},
{
"fails due to incompatible upgrades, chainB proposes a new connection hop that does not match counterparty",
Expand All @@ -252,9 +252,57 @@ func (suite *KeeperTestSuite) TestChanUpgradeTry() {
channel.UpgradeSequence = 5
path.EndpointB.SetChannel(channel)
},
// channel sequence - 1 will be returned (upgrade sequence is bumped in init as this is non-crossing hellos case)
// channel sequence will be returned so that counterparty inits on completely fresh sequence for both sides
types.NewUpgradeError(5, types.ErrInvalidUpgradeSequence),
},
{
// ChainA(Sequence: 0, mock-version-v2), ChainB(Sequence: 0, mock-version-v3)
// ChainA.INIT(Sequence: 1)
// ChainB.INIT(Sequence: 1)
// ChainA.TRY => error (incompatible versions)
// ChainB.TRY => error (incompatible versions)
"crossing hellos: fails due to incompatible version",
func() {
// use incompatible version
path.EndpointB.ChannelConfig.ProposedUpgrade.Fields.Version = fmt.Sprintf("%s-v3", mock.Version)
proposedUpgrade = path.EndpointB.GetProposedUpgrade()

err := path.EndpointB.ChanUpgradeInit()
suite.Require().NoError(err)

err = path.EndpointA.ChanUpgradeTry()
suite.Require().Error(err)
suite.Require().ErrorContains(err, "incompatible counterparty upgrade")
suite.Require().Equal(uint64(1), path.EndpointA.GetChannel().UpgradeSequence)
},
types.ErrIncompatibleCounterpartyUpgrade,
},
{
// ChainA(Sequence: 0, mock-version-v2), ChainB(Sequence: 4, mock-version-v3)
// ChainA.INIT(Sequence: 1)
// ChainB.INIT(Sequence: 5)
// ChainA.TRY => error (incompatible versions)
// ChainB.TRY(ErrorReceipt: 4)
"crossing hellos: upgrade starts with mismatching upgrade sequences and try fails on counterparty due to incompatible version",
func() {
channel := path.EndpointB.GetChannel()
channel.UpgradeSequence = 4
path.EndpointB.SetChannel(channel)

// use incompatible version
path.EndpointB.ChannelConfig.ProposedUpgrade.Fields.Version = fmt.Sprintf("%s-v3", mock.Version)
proposedUpgrade = path.EndpointB.GetProposedUpgrade()

err := path.EndpointB.ChanUpgradeInit()
suite.Require().NoError(err)

err = path.EndpointA.ChanUpgradeTry()
suite.Require().Error(err)
suite.Require().ErrorContains(err, "incompatible counterparty upgrade")
suite.Require().Equal(uint64(1), path.EndpointA.GetChannel().UpgradeSequence)
},
types.NewUpgradeError(4, types.ErrInvalidUpgradeSequence),
},
}

for _, tc := range testCases {
Expand Down Expand Up @@ -311,6 +359,133 @@ func (suite *KeeperTestSuite) TestChanUpgradeTry() {
}
}

// TestChanUpgrade_CrossingHellos_UpgradeSucceeds_AfterCancel verifies that under crossing hellos if upgrade
// sequences become out of sync, the upgrade can still be performed successfully after the upgrade is cancelled.
// ChainA(Sequence: 0), ChainB(Sequence 4)
// ChainA.INIT(Sequence: 1)
// ChainB.INIT(Sequence: 5)
// ChainB.TRY(ErrorReceipt: 4)
// ChainA.Cancel(Sequence: 4)
// ChainA.TRY(Sequence: 5) // fastforward
// ChainB.ACK => Success
// ChainA.Confirm => Success
// ChainB.Open => Success
func (suite *KeeperTestSuite) TestChanUpgrade_CrossingHellos_UpgradeSucceeds_AfterCancel() {
path := ibctesting.NewPath(suite.chainA, suite.chainB)
suite.coordinator.Setup(path)

path.EndpointA.ChannelConfig.ProposedUpgrade.Fields.Version = mock.UpgradeVersion
path.EndpointB.ChannelConfig.ProposedUpgrade.Fields.Version = mock.UpgradeVersion

suite.Require().NoError(path.EndpointA.ChanUpgradeInit())

channel := path.EndpointB.GetChannel()
channel.UpgradeSequence = 4
path.EndpointB.SetChannel(channel)

suite.Require().NoError(path.EndpointB.ChanUpgradeInit())
suite.Require().NoError(path.EndpointB.ChanUpgradeTry())
damiannolan marked this conversation as resolved.
Show resolved Hide resolved

suite.Require().NoError(path.EndpointA.ChanUpgradeCancel())
channel = path.EndpointA.GetChannel()
suite.Require().Equal(types.OPEN, channel.State)
suite.Require().Equal(uint64(4), channel.UpgradeSequence)

suite.Require().NoError(path.EndpointA.ChanUpgradeTry())
channel = path.EndpointA.GetChannel()
suite.Require().Equal(types.FLUSHING, channel.State)
suite.Require().Equal(uint64(5), channel.UpgradeSequence)

suite.T().Run("successfully completes upgrade", func(t *testing.T) {
suite.Require().NoError(path.EndpointB.ChanUpgradeAck())
suite.Require().NoError(path.EndpointA.ChanUpgradeConfirm())
suite.Require().NoError(path.EndpointB.ChanUpgradeOpen())
})

suite.T().Run("channel in expected state", func(t *testing.T) {
channel := path.EndpointA.GetChannel()
suite.Require().Equal(types.OPEN, channel.State, "channel should be in OPEN state")
suite.Require().Equal(mock.UpgradeVersion, channel.Version, "version should be correctly upgraded")
suite.Require().Equal(mock.UpgradeVersion, path.EndpointB.GetChannel().Version, "version should be correctly upgraded")
suite.Require().Equal(uint64(5), channel.UpgradeSequence, "upgrade sequence should be incremented")
suite.Require().Equal(uint64(5), path.EndpointB.GetChannel().UpgradeSequence, "upgrade sequence should be incremented on counterparty")
})
}

// TestChanUpgrade_CrossingHellos_UpgradeSucceeds_AfterCancelErrors verifies that under crossing hellos if upgrade
// sequences become out of sync, the upgrade can still be performed successfully after the cancel fails.
// ChainA(Sequence: 0), ChainB(Sequence 4)
// ChainA.INIT(Sequence: 1)
// ChainB.INIT(Sequence: 5)
// ChainA.TRY(Sequence: 5) // fastforward
// ChainB.TRY(ErrorReceipt: 4)
// ChainA.Cancel => Error (errorReceipt.Sequence < channel.UpgradeSequence)
// ChainB.ACK => Success
// ChainA.Confirm => Success
// ChainB.Open => Success
func (suite *KeeperTestSuite) TestChanUpgrade_CrossingHellos_UpgradeSucceeds_AfterCancelErrors() {
path := ibctesting.NewPath(suite.chainA, suite.chainB)
suite.coordinator.Setup(path)

path.EndpointA.ChannelConfig.ProposedUpgrade.Fields.Version = mock.UpgradeVersion
path.EndpointB.ChannelConfig.ProposedUpgrade.Fields.Version = mock.UpgradeVersion

suite.Require().NoError(path.EndpointA.ChanUpgradeInit())

channel := path.EndpointB.GetChannel()
channel.UpgradeSequence = 4
path.EndpointB.SetChannel(channel)

suite.Require().NoError(path.EndpointB.ChanUpgradeInit())

suite.coordinator.CommitBlock(suite.chainA, suite.chainB)
suite.Require().NoError(path.EndpointB.UpdateClient())

// use proofs when chain A has not executed TRY yet and use them when executing TRY on chain B
historicalChannelProof, historicalUpgradeProof, proofHeight := path.EndpointA.QueryChannelUpgradeProof()

suite.Require().NoError(path.EndpointA.ChanUpgradeTry())
_, _, err := suite.chainB.GetSimApp().GetIBCKeeper().ChannelKeeper.ChanUpgradeTry(
suite.chainB.GetContext(),
path.EndpointB.ChannelConfig.PortID,
path.EndpointB.ChannelID,
path.EndpointB.GetChannelUpgrade().Fields.ConnectionHops,
path.EndpointA.GetChannelUpgrade().Fields,
1,
historicalChannelProof,
historicalUpgradeProof,
proofHeight,
)
suite.Require().Error(err)
suite.assertUpgradeError(err, types.NewUpgradeError(4, types.ErrInvalidUpgradeSequence))
errorReceipt := err.(*types.UpgradeError).GetErrorReceipt()
suite.chainB.GetSimApp().IBCKeeper.ChannelKeeper.SetUpgradeErrorReceipt(suite.chainB.GetContext(), path.EndpointB.ChannelConfig.PortID, path.EndpointB.ChannelID, errorReceipt)
suite.coordinator.CommitBlock(suite.chainB)

err = path.EndpointA.ChanUpgradeCancel()
suite.Require().Error(err)
suite.Require().ErrorContains(err, "invalid upgrade sequence")

channel = path.EndpointA.GetChannel()
suite.Require().Equal(types.FLUSHING, channel.State)
suite.Require().Equal(uint64(5), channel.UpgradeSequence)

suite.T().Run("successfully completes upgrade", func(t *testing.T) {
suite.Require().NoError(path.EndpointB.ChanUpgradeAck())
suite.Require().NoError(path.EndpointA.ChanUpgradeConfirm())
suite.Require().NoError(path.EndpointB.ChanUpgradeOpen())
})

suite.T().Run("channel in expected state", func(t *testing.T) {
channel := path.EndpointA.GetChannel()
suite.Require().Equal(types.OPEN, channel.State, "channel should be in OPEN state")
suite.Require().Equal(mock.UpgradeVersion, channel.Version, "version should be correctly upgraded")
suite.Require().Equal(mock.UpgradeVersion, path.EndpointB.GetChannel().Version, "version should be correctly upgraded")
suite.Require().Equal(uint64(5), channel.UpgradeSequence, "upgrade sequence should be incremented")
suite.Require().Equal(uint64(5), path.EndpointB.GetChannel().UpgradeSequence, "upgrade sequence should be incremented on counterparty")
})
}

func (suite *KeeperTestSuite) TestWriteUpgradeTry() {
var (
path *ibctesting.Path
Expand Down Expand Up @@ -1410,7 +1585,7 @@ func (suite *KeeperTestSuite) TestChanUpgrade_UpgradeSucceeds_AfterCancel() {
path.EndpointA.SetChannel(channel)

channel = path.EndpointB.GetChannel()
channel.UpgradeSequence = 2
channel.UpgradeSequence = 5
path.EndpointB.SetChannel(channel)

suite.Require().NoError(path.EndpointA.UpdateClient())
Expand Down Expand Up @@ -1441,8 +1616,8 @@ func (suite *KeeperTestSuite) TestChanUpgrade_UpgradeSucceeds_AfterCancel() {
channel := path.EndpointA.GetChannel()
suite.Require().Equal(types.OPEN, channel.State)

suite.T().Run("verify upgrade sequences are synced", func(t *testing.T) {
suite.Require().Equal(uint64(2), channel.UpgradeSequence)
suite.T().Run("verify upgrade sequence fastforwards to channelB sequence", func(t *testing.T) {
suite.Require().Equal(uint64(5), channel.UpgradeSequence)
})
})

Expand All @@ -1459,8 +1634,8 @@ func (suite *KeeperTestSuite) TestChanUpgrade_UpgradeSucceeds_AfterCancel() {
suite.Require().Equal(types.OPEN, channel.State, "channel should be in OPEN state")
suite.Require().Equal(mock.UpgradeVersion, channel.Version, "version should be correctly upgraded")
suite.Require().Equal(mock.UpgradeVersion, path.EndpointB.GetChannel().Version, "version should be correctly upgraded")
suite.Require().Equal(uint64(3), channel.UpgradeSequence, "upgrade sequence should be incremented")
suite.Require().Equal(uint64(3), path.EndpointB.GetChannel().UpgradeSequence, "upgrade sequence should be incremented on counterparty")
suite.Require().Equal(uint64(6), channel.UpgradeSequence, "upgrade sequence should be incremented")
suite.Require().Equal(uint64(6), path.EndpointB.GetChannel().UpgradeSequence, "upgrade sequence should be incremented on counterparty")
})
}

Expand Down
14 changes: 8 additions & 6 deletions testing/chain.go
Original file line number Diff line number Diff line change
Expand Up @@ -344,6 +344,14 @@ func (chain *TestChain) SendMsgs(msgs ...sdk.Msg) (*abci.ExecTxResult, error) {
// ensure the chain has the latest time
chain.Coordinator.UpdateTimeForChain(chain)

// increment acc sequence regardless of success or failure tx execution
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great find @damiannolan 👀

defer func() {
err := chain.SenderAccount.SetSequence(chain.SenderAccount.GetSequence() + 1)
if err != nil {
panic(err)
}
}()

resp, err := simapp.SignAndDeliver(
chain.TB,
chain.TxConfig,
Expand All @@ -370,12 +378,6 @@ func (chain *TestChain) SendMsgs(msgs ...sdk.Msg) (*abci.ExecTxResult, error) {
return txResult, fmt.Errorf("%s/%d: %q", txResult.Codespace, txResult.Code, txResult.Log)
}

// increment sequence for successful transaction execution
err = chain.SenderAccount.SetSequence(chain.SenderAccount.GetSequence() + 1)
if err != nil {
return nil, err
}

chain.Coordinator.IncrementTime()

return txResult, nil
Expand Down
Loading