[Epic] Database layer performance and stability

[czarcas7ic]

Completed

• IAVL state export --height bug
  • Appears to have fixed the app hash mismatch being observed
• Memory leak #1037
• Backport 7.0.3 IAVL changes to v6.4.0
• OOM during prune causing state hash conflict on restart #1042
• fatal error: concurrent map read and map write iavl#22
• Backport concurrent gateway to osmosis-sdk v0.45 and test with fast storage #1065
• IAVL snapshot - pruning race issue tracking #932
• Implement query to search hash by KVStoreKey at specific heights #1137
• fix: commit info data race cosmos-sdk#155
• Refactor fastNodeCache and nodeCache to be separate modules with synchrony iavl#24
• Remove fast node cache and benchmark #1041
• Upstream iavl and sdk database changes #1175
• Investigate flushing pruning heights metadata once it is added to memory cosmos-sdk#149

Blocked

• tendermint wrong Block.Header.AppHash bug #1090
  • Planning to not spend too much time on investigating on nodes, until we get an improved mainnet node setup automation, via ansible & prometheus. Otherwise we spend all the time in node setup / attempts at reproducing
  • We also want to try reproducing in the e2e test suite once its further along

[p0mvn]

As discussed in sprint planning, this epic is getting closed. #1163 and osmosis-labs/cosmos-sdk#148 are getting moved as frogs to the current sprint while all the remaining issues are moved to be tracked #1172 without specific priority assigned to them