Impossible to create IBC channels after on-chain upgrade #9800
Comments
|
Is the "simple on-chain upgrade” is an export upgrade, or upgrade module upgrade? |
It used the |
|
Do you have the following lines in your https://github.com/cosmos/gaia/blob/main/app/app.go#L455 |
Of course. You can see the post-upgrade files here |
|
Hmm that should have reinitialized the in-memory capabilities. I will test this issue, thanks for the writeup |
|
Hi @RiccardoM , @colin-axner just tried to reproduce your issue and did not run into any issues.
There was no errors with this workflow. Could you provide more information on what you did beyond the steps above? |
|
@AdityaSripal Aside from those operations, we did nothing. We had a running chain with different channels opened, we performed the upgrade and I tried to create a new channel after the upgrade. If I query the previously created channels I can see them being there, but if I try to create a new one I end up with the above mentioned error. For a better understanding, we had a chain running with the following details:
and we upgraded to use Desmos The upgrade was successful and most of the validators used |
|
@AdityaSripal @colin-axner If you have any idea how I might debug this (maybe something particular to look at?) let me know please cause this is currently blocking our testnet from having any integration with other chains over IBC |
If so, could you print the following info:
If you could then show the outputs here that would be great. Ideally we want to see that on a node restart these are called and correctly filled in. |
|
This was already tested before IBC launch and we haven't changed that code. Don't know exactly what could be the issue but it would be useful to at least rule out some things |
|
Also are you able to send/receive packets on an already existing channel? |
|
Also have you been using statesync in any way? |
I can't. I've tried using This is the channel details
No state sync has been used for this node. It has been synced from genesis and is currently using |
I've added the logs inside this commit. Here are the results: Looks like |
|
Does this impact the 0.43 release as well? Is anyone working on a PR? |
No they shouldn't but they should be called when you try to open a channel. The logs I see seem to show that we are correctly filling in memory on restart. Could you also print out what you see in the logs once you try opening a channel? |
|
@AdityaSripal Here are the logs when I try creating a channel Command used: It appears that the node in which I executed the command (running the new binary with logs enabled) successfully created it. But from its logs we can see that the majority of the nodes didn't: |
|
@RiccardoM Does this issue only arise when interacting with |
Yes, the I can also confirm that opening channels for both |
|
Interesting so, the network itself is capable of opening new channels, but just this node is having issues? |
No, the opposite. The network is not able, but this node is now (after replacing the binary with the logger one) apparently. |
|
Just noticed: one difference is that I've now started the |
Have you observed any error from cosmovisor? |
No we did the test with cosmovisor both without upgrading and with upgrading and ran into no issues. Doesn't seem like cosmovisor is related to issue at all. |
|
Thanks to @colin-axner for finding the potential bug, and to @RiccardoM for helping to debug today. We think we understand what the problem is. As mentioned in the original statesync fix #9392 , we introduced a global However, if the first tx that calls Thus, in this case we reach an inconsistent state in the capability module and worse still, the global This would explain the behaviour we are seeing. It explains why when debugging with @RiccardoM today we could not reproduce because the first tx we tried after restart was successful. It also explains the issue with nodes on other networks where the first tx passes the capability checks and fails later down the line, triggering the problem described above and then subsequent tx's fail on the capability check. It also explains the non-deterministic behaviour of the problem since it will only arise for nodes that have statesynced (or restarted) to a height right before a failing channel tx. |
|
Given the above understanding this is the next steps for the IBC team.
|
<!-- The default pull request template is for types feat, fix, or refactor. For other templates, add one of the following parameters to the url: - template=docs.md - template=other.md --> ## Description Closes: #9800 The following is a breaking fix to #9800. In the non-breaking fix, the initialization logic was moved out of `InitializeAndSeal` but the API was preserved. I've now removed `InitializeAndSeal` and replaced it with just the `Seal` function, which may be called much more cleanly in `app.go` --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [x] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [x] added a changelog entry to `CHANGELOG.md` - [x] included comments for [documenting Go code](https://blog.golang.org/godoc) - [x] updated the relevant documentation or specification - [x] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)
<!-- The default pull request template is for types feat, fix, or refactor. For other templates, add one of the following parameters to the url: - template=docs.md - template=other.md --> ## Description Closes: #9800 The following is a breaking fix to #9800. In the non-breaking fix, the initialization logic was moved out of `InitializeAndSeal` but the API was preserved. I've now removed `InitializeAndSeal` and replaced it with just the `Seal` function, which may be called much more cleanly in `app.go` --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [x] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [x] added a changelog entry to `CHANGELOG.md` - [x] included comments for [documenting Go code](https://blog.golang.org/godoc) - [x] updated the relevant documentation or specification - [x] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) (cherry picked from commit 7f316ad) # Conflicts: # CHANGELOG.md
…#9841) * fix!: Capability Issue on Restart, Backport to v0.43 (#9836) <!-- The default pull request template is for types feat, fix, or refactor. For other templates, add one of the following parameters to the url: - template=docs.md - template=other.md --> ## Description Closes: #9800 The following is a breaking fix to #9800. In the non-breaking fix, the initialization logic was moved out of `InitializeAndSeal` but the API was preserved. I've now removed `InitializeAndSeal` and replaced it with just the `Seal` function, which may be called much more cleanly in `app.go` --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [x] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [x] added a changelog entry to `CHANGELOG.md` - [x] included comments for [documenting Go code](https://blog.golang.org/godoc) - [x] updated the relevant documentation or specification - [x] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) (cherry picked from commit 7f316ad) # Conflicts: # CHANGELOG.md * solve conflicts Co-authored-by: Aditya <adityasripal@gmail.com> Co-authored-by: Robert Zaremba <robert@zaremba.ch>
…#9836) (cosmos#9841) * fix!: Capability Issue on Restart, Backport to v0.43 (cosmos#9836) <!-- The default pull request template is for types feat, fix, or refactor. For other templates, add one of the following parameters to the url: - template=docs.md - template=other.md --> ## Description Closes: cosmos#9800 The following is a breaking fix to cosmos#9800. In the non-breaking fix, the initialization logic was moved out of `InitializeAndSeal` but the API was preserved. I've now removed `InitializeAndSeal` and replaced it with just the `Seal` function, which may be called much more cleanly in `app.go` --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [x] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [x] added a changelog entry to `CHANGELOG.md` - [x] included comments for [documenting Go code](https://blog.golang.org/godoc) - [x] updated the relevant documentation or specification - [x] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) (cherry picked from commit 7f316ad) # Conflicts: # CHANGELOG.md * solve conflicts Co-authored-by: Aditya <adityasripal@gmail.com> Co-authored-by: Robert Zaremba <robert@zaremba.ch>
…#9836) (cosmos#9841) * fix!: Capability Issue on Restart, Backport to v0.43 (cosmos#9836) <!-- The default pull request template is for types feat, fix, or refactor. For other templates, add one of the following parameters to the url: - template=docs.md - template=other.md --> ## Description Closes: cosmos#9800 The following is a breaking fix to cosmos#9800. In the non-breaking fix, the initialization logic was moved out of `InitializeAndSeal` but the API was preserved. I've now removed `InitializeAndSeal` and replaced it with just the `Seal` function, which may be called much more cleanly in `app.go` --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [x] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [x] added a changelog entry to `CHANGELOG.md` - [x] included comments for [documenting Go code](https://blog.golang.org/godoc) - [x] updated the relevant documentation or specification - [x] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) (cherry picked from commit 7f316ad) # Conflicts: # CHANGELOG.md * solve conflicts Co-authored-by: Aditya <adityasripal@gmail.com> Co-authored-by: Robert Zaremba <robert@zaremba.ch>
<!-- The default pull request template is for types feat, fix, or refactor. For other templates, add one of the following parameters to the url: - template=docs.md - template=other.md --> ## Description Closes: cosmos#9800 The following is a breaking fix to cosmos#9800. In the non-breaking fix, the initialization logic was moved out of `InitializeAndSeal` but the API was preserved. I've now removed `InitializeAndSeal` and replaced it with just the `Seal` function, which may be called much more cleanly in `app.go` --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [x] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [x] added a changelog entry to `CHANGELOG.md` - [x] included comments for [documenting Go code](https://blog.golang.org/godoc) - [x] updated the relevant documentation or specification - [x] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)
…#9836) (cosmos#9841) * fix!: Capability Issue on Restart, Backport to v0.43 (cosmos#9836) <!-- The default pull request template is for types feat, fix, or refactor. For other templates, add one of the following parameters to the url: - template=docs.md - template=other.md --> ## Description Closes: cosmos#9800 The following is a breaking fix to cosmos#9800. In the non-breaking fix, the initialization logic was moved out of `InitializeAndSeal` but the API was preserved. I've now removed `InitializeAndSeal` and replaced it with just the `Seal` function, which may be called much more cleanly in `app.go` --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [x] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [x] added a changelog entry to `CHANGELOG.md` - [x] included comments for [documenting Go code](https://blog.golang.org/godoc) - [x] updated the relevant documentation or specification - [x] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) (cherry picked from commit 7f316ad) # Conflicts: # CHANGELOG.md * solve conflicts Co-authored-by: Aditya <adityasripal@gmail.com> Co-authored-by: Robert Zaremba <robert@zaremba.ch>
Summary of Bug
After running an on-chain upgrade, it is impossible to create new IBC channels.
Version
v0.42.7Steps to Reproduce
The following error will be displayed:
This happens with both custom modules and the
transfermodule.The error started showing up only after the upgrade, while before the upgrade itself it was possible to create correct channels that worked without any issue.
The upgrade has been tested on our testnet and did not have any change apart from the IBC version checking being deleted since we do not need it.
For Admin Use
The text was updated successfully, but these errors were encountered: