perf: Speedup cachekv iterator on large deletions & IBC v2 upgrade logic

[ValarDragon]

Description

This PR provides extreme speedups to the iterator creation pattern used in upgrades from IBC v1 to v2. On Osmosis, executing an upgrade from a state export was taking us over 2 hours due to this bug. (We never let it finish) After this fix, it takes under 8 minutes. (There is then ~20-30 min time overhead for IAVL commit, but thats orthogonal)

Essentially the CacheKVStore previously would iterate over every deleted item in cache when constructing an iterator. In the upgrade logic, bank first migrates the entire store, rewriting everything. (And thus adding 100k + deletes, other upgrades added far more deletes as well. Guestimmated that Osmosis had around 500k deletes) Then in IBC's upgrade, you do at least two iterator constructions per IBC client, which on Osmosis there are ~2000. This adds up to the at least two hours we were experiencing.

This PR adds a benchmark to highlight the speed improvement, RAM improvement, and fixes this.

<Benchmark name> <num benchmark iterations> <time taken> <Heap I/O> <New Memory allocated>
Before:
BenchmarkIteratorOnParentWith1MDeletes-16              1        1112540940 ns/op        101743776 B/op     38196 allocs/op
After:
BenchmarkIteratorOnParentWith1MDeletes-16         154346              7570 ns/op             331 B/op          3 allocs/op

Thus this fix really does solve the problem

---

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

• included the correct type prefix in the PR title
• added ! to the type prefix if API or client breaking change
• targeted the correct branch (see PR Targeting)
• provided a link to the relevant issue or specification
• followed the guidelines for building modules
• included the necessary unit and integration tests
• added a changelog entry to CHANGELOG.md
• included comments for documenting Go code
• updated the relevant documentation or specification
• reviewed "Files changed" and left comments if necessary
• confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

• confirmed the correct type prefix in the PR title
• confirmed ! in the type prefix if API or client breaking change
• confirmed all author checklist items have been addressed
• reviewed state machine logic
• reviewed API design and naming
• reviewed documentation is accurate
• reviewed tests and test coverage
• manually tested (if applicable)

[ValarDragon]

This bug is not related to this PR, I just noticed it in constructing the benchmark / working with the relevant logic.

This can be spot checked if folks want by just checking out the first commit in this PR, and removing the [1:].

This should probably moved to its own issue where we discuss the iterator structure, but its relatively low priority given all the other module / app writing improvements in the works 🙏

[tac0turtle]

utACK

[odeke-em]

Awesome work, LGTM and thank you @ValarDragon for the change and for the keen eyes!

[alexanderbez]

🎉