Make it easy to pull changes from upstream and cherry-pick celestia-core onto tendermint

[evan-forbes]

Summary

Now that celestia-core will be significantly closer to tendermint, it makes even more sense to make it easier to pull changes from tendermint. This will mainly involve reducing merge conflicts, but also include using a bot to frequently create PRs to pull in changes.

Details

It has been decided that we will do everything that we can to save time pulling changes from upstream, both in tendermint and the sdk. To further that goal, we will stop declaring the module for celestia-core, and declare it once again as tendermint. This will dramatically reduce the number of merge conflicts when pulling changes from upstream simply because the imports for both repos will be the same.

While we're doing this, we're also planning on updating to the latest version of tendermint by "resetting" (see #527) and then progressively re-implementing the remaining necessary changes back in. The idea being that this will actually be easier than merging the latest version, as we can test individual components as we add them, instead of having to do that all at once. The changes that we do add in should be a relatively straightforward process, as they are already implemented, and we have already finished work that simplifies the remaining changes mentioned in #491 (see #505). It should also further reduce the number of conflicts between the two repos.

Action Items

Core

• Reset repo to upstream Reset to tendermint master #534

  • use celestia specific readme #529
  • copy over the pkg directory
  • add celestia specific consts #530
  • cherry-pick once finished Refactor how to create DataAvailabilityHeaders #524
  • add the DataHash to the Header
  • Reimplement PreProcessTxs

• use a bot to frequently make PRs that incorporate changes from upstream (previously discussed in git sync #180)

SDK

• Pull changes from upstream and use version of celestia-core that is based on tendermint v0.34.12 cosmos-sdk#33
• create optimint master branch based on the updated sdk
• use a bot to frequently make PRs that incorporate changes from upstream

App

• regeneration of app celestia-app#127
• Reimplement/copy-paste PFM and client code

References

Huge shoutout to @marbar3778 who's previous and current efforts pulling changes from upstream were crucial in informing this decision. #526

this issue replaces #492
tracking issue #491

[liamsi]

regarding the bot, this was suggested in the past: #180

[evan-forbes]

not only will we have to make it easy to pull changes from tendermint master, but we should also try to make it easy to port our changes to tagged versions of tendermint. This way our fork of the cosmos-sdk can replace those specific versions.

ref #549

[evan-forbes]

per celestiaorg/cosmos-sdk#34 (comment)

we also need to use the same bot to keep track of the most recent release versions of tendermint and the sdk and document this somewhere clearly.

[evan-forbes]

we have made it significantly easier to upgrade from upstream, and imho can close this issue. We can still reopen this issue if other think other wise. Originally I was waiting on merging a bot that pull upstream before closing, but as shown in #558, this will still result in us having to handle conflicts manually anyway.