Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backport v0.38.5: IAVL Bump (RC) & Pruning Refactor #6505

Merged
merged 16 commits into from
Jun 26, 2020

Conversation

alexanderbez
Copy link
Contributor

@alexanderbez alexanderbez commented Jun 24, 2020

Backport:


Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

  • Targeted PR against correct branch (see CONTRIBUTING.md)
  • Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
  • Code follows the module structure standards.
  • Wrote unit and integration tests
  • Updated relevant documentation (docs/) or specification (x/<module>/spec/)
  • Added relevant godoc comments.
  • Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
  • Re-reviewed Files changed in the Github PR explorer

Copy link
Collaborator

@fedekunze fedekunze left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK. Pending tests and lint fixes

@alexanderbez alexanderbez changed the title Backport v0.38.5: IAVL Bump (RC) Backport v0.38.5: IAVL Bump (RC) & Pruning Refactor Jun 24, 2020
@zmanian
Copy link
Member

zmanian commented Jun 24, 2020

So my mental model here is

This is breaking from a config api point of view but a node from this branch should be to sync from genesis on existing 0.38.5 network like Kava.

@alexanderbez
Copy link
Contributor Author

@zmanian that is correct, yes. A few (go) client-facing breaking API changes, but the state machine is obviously intact. @erikgrinaker plans to write something up about how to make sure an upgrade goes smoothly without any data loss or corruption.

@codecov
Copy link

codecov bot commented Jun 24, 2020

Codecov Report

Merging #6505 into backport/release/v0.38.x will increase coverage by 0.09%.
The diff coverage is 63.57%.

@@                     Coverage Diff                      @@
##           backport/release/v0.38.x    #6505      +/-   ##
============================================================
+ Coverage                     51.50%   51.60%   +0.09%     
============================================================
  Files                           336      337       +1     
  Lines                         20577    20624      +47     
============================================================
+ Hits                          10599    10642      +43     
- Misses                         9175     9176       +1     
- Partials                        803      806       +3     

@alexanderbez alexanderbez self-assigned this Jun 24, 2020
Copy link
Contributor

@colin-axner colin-axner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, left an small observance - feel free to resolve if you don't think it needs to be addressed

store/types/pruning.go Show resolved Hide resolved
@erikgrinaker
Copy link
Contributor

I've written a function that can repair faulty orphans in databases written with IAVL 0.13.x: cosmos/iavl#282

It must be run after the database is closed with 0.13.x and before it is opened with 0.14 or later.

We can consider running this automatically, but it's potentially scary to automatically delete a bunch of stuff on IAVL load. It also cannot be run directly on a bare SDK database since it uses a PrefixDB on top. Maybe we should add a script or something that node operators can run?

@alexanderbez
Copy link
Contributor Author

@erikgrinaker that's excellent! AFAIU, this patching would be voluntary and not necessary for a fully functional upgrade, it's just to be safe, right?

@erikgrinaker
Copy link
Contributor

erikgrinaker commented Jun 25, 2020

@erikgrinaker that's excellent! AFAIU, this patching would be voluntary and not necessary for a fully functional upgrade, it's just to be safe, right?

It depends on what the user does after the upgrade.

With KeepEvery: 10000, let's say the user stops at height 37000, upgrades, and resumes from height 30000 - if the user at any point deletes version 30000, then versions between 30000 and 37000 are very likely to experience data loss and corruption. If the values written in version 30000-37000 are different from what they were when they were processed before the upgrade then all versions after 30000 are vulnerable to data corruption.

If the user is able to stop at 30000, without ever having saved versions above that, upgrading is safe.

@alexanderbez
Copy link
Contributor Author

Ok I think we definitely need very clear communication and documentation on this. We should encourage the use of this patch script OR have operators run archive mode for a bit before upgrading...or both?

@erikgrinaker
Copy link
Contributor

I'll need to read up on the new pruning mechanism and defaults, and doublecheck a couple of things. There are configurations that would eventually flush out the invalid data on its own, without any corruption, as long as the user doesn't change anything. Will get back to you.

* store/types: Fix pruning opts validation

* store/types: Add case
Copy link
Contributor

@erikgrinaker erikgrinaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tested this with Kava, all the upgrade instructions are accurate and the failure modes are as expected. No failures were found after any successful upgrade following upgrade instructions.

Let's cut an RC!

@ethanfrey ethanfrey mentioned this pull request Jun 26, 2020
4 tasks
@alexanderbez alexanderbez merged commit 65b6030 into backport/release/v0.38.x Jun 26, 2020
@alexanderbez alexanderbez deleted the bez/backport/iavl-bump branch June 26, 2020 16:14
@alexanderbez alexanderbez added this to the 0.39.0 [launchpad] milestone Jul 6, 2020
Raumo0 pushed a commit to mapofzones/cosmos-sdk that referenced this pull request Feb 13, 2022
* v0.38.4-RC1

* add release date

* client/keys/parse: honor config changes (cosmos#6340)

`keys parse` uses the global configuration before
before client applications have had a chance to
apply their settings.

This change adds a `GetSealedConfig()` helper
that waits for the config to be sealed before
returning it.

Fixes: cosmos#5091
Origin: cosmos@4e328d7
Author: Adam Bozanich <adam.boz@gmail.com>
Reviewed-by: Alessio Treglia <alessio@tendermint.com>

* Merge PR cosmos#6505: Backport v0.38.5: IAVL Bump (RC) & Pruning Refactor

* Fix cl

* Fix cl

* Merge PR cosmos#6551: Ethanfrey/fix trace flag

* Merge PR cosmos#6552: Add sender info to bank transfer event

* Merge PR cosmos#6581: v0.38.5-RC1

* Add release date

* deps: bump IAVL to 0.14

* cl++

* Merge PR cosmos#6618: backport 0.39.0 (launchpad): cherry pick cosmos#5839

* launchpad: bump tendermint to v0.33.6 (cosmos#6673)

* launchpad: bump tendermint to v0.33.6

* cha-cha-cha

* fix types.ChainAnteDecorators() panic (cosmos#5742) (cosmos#6669)

ChainAnteDecorators() panics when no arguments are supplied.
This change its behaviour and the function now returns a nil
AnteHandler in case no AnteDecorator instances are supplied.

Closes: cosmos#5741

* launchpad: register MsgFundCommunityPool to distribution codec (cosmos#6675)

Closes: cosmos#6210

* client: backport IBC additions (cosmos#6682)

* launchpad: backport cliCtx.QueryABCI

* add prove flag

* Save account for multi sending (cosmos#6674)

Include changes from PR cosmos#6283

Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
Co-authored-by: Alessio Treglia <alessio@tendermint.com>

* baseapp: fix sender events accumulation (cosmos#6683)

closes: cosmos#6306
original PR: cosmos#6307

* run go mod tidy

* add changelog line re: issue that was not included in v0.38.5

* Fix changelog

* add release notes

* update CODEOWNERS as per STABLE_RELEASES.md

Launchpad's Stable Release Managers team's current composition as
approved by @okwme (ICF):

* @alessio
* @clevinson
* @ethanfrey

* Add milestone's URL

* Revert "update CODEOWNERS as per STABLE_RELEASES.md"

This reverts commit f384592.

* fix typo

* add example patch

* launchpad: backport account sequence stuck (cosmos#6721)

Launchpad fix for cosmos#6287

* update release notes

* make explicit that the regression is fixed in 0.39

* update release notes

* Update CHANGELOG.md

* update release notes

@ethanfrey

* remove pruning info from changelog

They have been moved to NEWS.md

* Update NEWS.md

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* Mege PR cosmos#6749: auth: remove custom JSON marshaling

* mv NEWS.md -> RELEASE_NOTES.md

Self-explanatory naming convention.

* Update changelog

* add reference to gaia software upgrade

* x/staking: add call to iterator Close() method (cosmos#6794)

Original pull request: cosmos#6791

* [ci] fix linter (cosmos#6795)

* Merge PR cosmos#6842: Remove custom acc JSON marshl

* Backport cosmos#5671: Add a Flag for CORS (cosmos#6853)

This adds the --unsafe-cors flag functionality.

From: cosmos#5671
Co-authored-by: Marko <marbar3778@yahoo.com>

* Merge PR cosmos#6855: Allow passing through Content-Type (needed for POST)

* Merge PR cosmos#6861: Revert "Backport 0.39.1: Remove Custom JSON Marshaling for Accounts"

* Merge PR cosmos#6869: Backport 0.39.1: Launchpad Migration cosmos#6829

* Update changelog and release notes

* Update x/auth/legacy/v0_39/types.go

* Update x/auth/legacy/v0_38/types.go

* Merge PR cosmos#6938: Fix v0.39 migrate test

* Merge PR cosmos#6937: Bump Tendermint to v0.33.7

* Update x/auth/legacy/v0_39/types.go

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* update RELEASE_NOTES.md

* update changelog

* test case with real genesis data (cosmos#6995)

* incorporate Ethan Frey's suggestion

* make format

* Update go mod

* Remove some ci files

* Modify data for test

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com>
Co-authored-by: Alessio Treglia <alessio@tendermint.com>
Co-authored-by: Ethan Frey <ethanfrey@users.noreply.github.com>
Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
Co-authored-by: Jonathan Gimeno <jgimeno@gmail.com>
Co-authored-by: Marko <marbar3778@yahoo.com>
Raumo0 pushed a commit to mapofzones/cosmos-sdk that referenced this pull request Feb 13, 2022
* v0.38.4-RC1

* add release date

* client/keys/parse: honor config changes (cosmos#6340)

`keys parse` uses the global configuration before
before client applications have had a chance to
apply their settings.

This change adds a `GetSealedConfig()` helper
that waits for the config to be sealed before
returning it.

Fixes: cosmos#5091
Origin: cosmos@4e328d7
Author: Adam Bozanich <adam.boz@gmail.com>
Reviewed-by: Alessio Treglia <alessio@tendermint.com>

* Merge PR cosmos#6505: Backport v0.38.5: IAVL Bump (RC) & Pruning Refactor

* Fix cl

* Fix cl

* Merge PR cosmos#6551: Ethanfrey/fix trace flag

* Merge PR cosmos#6552: Add sender info to bank transfer event

* Merge PR cosmos#6581: v0.38.5-RC1

* Add release date

* deps: bump IAVL to 0.14

* cl++

* Merge PR cosmos#6618: backport 0.39.0 (launchpad): cherry pick cosmos#5839

* launchpad: bump tendermint to v0.33.6 (cosmos#6673)

* launchpad: bump tendermint to v0.33.6

* cha-cha-cha

* fix types.ChainAnteDecorators() panic (cosmos#5742) (cosmos#6669)

ChainAnteDecorators() panics when no arguments are supplied.
This change its behaviour and the function now returns a nil
AnteHandler in case no AnteDecorator instances are supplied.

Closes: cosmos#5741

* launchpad: register MsgFundCommunityPool to distribution codec (cosmos#6675)

Closes: cosmos#6210

* client: backport IBC additions (cosmos#6682)

* launchpad: backport cliCtx.QueryABCI

* add prove flag

* Save account for multi sending (cosmos#6674)

Include changes from PR cosmos#6283

Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
Co-authored-by: Alessio Treglia <alessio@tendermint.com>

* baseapp: fix sender events accumulation (cosmos#6683)

closes: cosmos#6306
original PR: cosmos#6307

* run go mod tidy

* add changelog line re: issue that was not included in v0.38.5

* Fix changelog

* add release notes

* update CODEOWNERS as per STABLE_RELEASES.md

Launchpad's Stable Release Managers team's current composition as
approved by @okwme (ICF):

* @alessio
* @clevinson
* @ethanfrey

* Add milestone's URL

* Revert "update CODEOWNERS as per STABLE_RELEASES.md"

This reverts commit f384592.

* fix typo

* add example patch

* launchpad: backport account sequence stuck (cosmos#6721)

Launchpad fix for cosmos#6287

* update release notes

* make explicit that the regression is fixed in 0.39

* update release notes

* Update CHANGELOG.md

* update release notes

@ethanfrey

* remove pruning info from changelog

They have been moved to NEWS.md

* Update NEWS.md

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* Mege PR cosmos#6749: auth: remove custom JSON marshaling

* mv NEWS.md -> RELEASE_NOTES.md

Self-explanatory naming convention.

* Update changelog

* add reference to gaia software upgrade

* x/staking: add call to iterator Close() method (cosmos#6794)

Original pull request: cosmos#6791

* [ci] fix linter (cosmos#6795)

* Merge PR cosmos#6842: Remove custom acc JSON marshl

* Backport cosmos#5671: Add a Flag for CORS (cosmos#6853)

This adds the --unsafe-cors flag functionality.

From: cosmos#5671
Co-authored-by: Marko <marbar3778@yahoo.com>

* Merge PR cosmos#6855: Allow passing through Content-Type (needed for POST)

* Merge PR cosmos#6861: Revert "Backport 0.39.1: Remove Custom JSON Marshaling for Accounts"

* Merge PR cosmos#6869: Backport 0.39.1: Launchpad Migration cosmos#6829

* Update changelog and release notes

* Update x/auth/legacy/v0_39/types.go

* Update x/auth/legacy/v0_38/types.go

* Merge PR cosmos#6938: Fix v0.39 migrate test

* Merge PR cosmos#6937: Bump Tendermint to v0.33.7

* Update x/auth/legacy/v0_39/types.go

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* update RELEASE_NOTES.md

* update changelog

* test case with real genesis data (cosmos#6995)

* incorporate Ethan Frey's suggestion

* make format

* Update go mod

* Remove some ci files

* Modify data for test

* API server

* Port telemetry

* Initial metrics

* Fix measure

* Merge PR cosmos#6761: telemetry: use UTC() in wrappers

* Remove file

* Make format

* Revert changes to client

* Revert changes to client/lcd/root

* Revert renames in client

* Fix int64 panics

* Revert "Revert renames in client"

This reverts commit cc0a95d14c3212fa1c49f93789dd94e167427a57.

* Revert "Revert changes to client/lcd/root"

This reverts commit e3bb87bbacae13676c3a1f86f8d441d576b1f2ba.

* Revert "Revert changes to client"

This reverts commit 332cacba3242e503bcc4bffc3f6b25c1906efca4.

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com>
Co-authored-by: Alessio Treglia <alessio@tendermint.com>
Co-authored-by: Ethan Frey <ethanfrey@users.noreply.github.com>
Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
Co-authored-by: Jonathan Gimeno <jgimeno@gmail.com>
Co-authored-by: Marko <marbar3778@yahoo.com>
Co-authored-by: Adam Bozanich <adam.boz@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants