Backport tendermint-v0.34.20 into main (#642)

* rpc: avoid leaking threads (#8329)

Co-authored-by: Sam Kleinman <garen@tychoish.com>

* (fixup) Revert the CheckTxSync callback parameter from https://github.com/Finschia/ostracon/pull/163/files#diff-eafa5360d0b71c14dba6ad5b29be23f69f0fe93e0751fe89fe208a5dafcfbbcf

* (fixup) Test the reverted CheckTxSync callback parameter from https://github.com/Finschia/ostracon/pull/163/files#diff-eafa5360d0b71c14dba6ad5b29be23f69f0fe93e0751fe89fe208a5dafcfbbcf

* (fixup) Add tests for code coverage: rpc/core/mempool_test.go

* Set a cap on the length of subscription queries. (#8349)

A manual backport of #7263.

As a safety measure, don't allow a query string to be unreasonably long. The
query filter is not especially efficient, so a query that needs more than basic
detail should filter coarsely in the subscriber and refine on the client side.

This affects Subscribe and TxSearch queries.

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* blocksync: validate block before persisting it (backport #8493) (#8495)

* Update default config docs for PSQL indexer settings. (#8728)

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* cmd: add tool for compaction of goleveldb (backport #8564) (#8674)

* (fixup) Add compact_test for improving coverage

* makefile: change buf to use tools.go ; backport v0.34 (#8852)

* makefile: update buf commands to use tools.go (#8609)

This will keep the version of `buf` consistent between all developer machines.

Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com>

* (fixup) Improve to work for Ostracon with tendermint proto files

* fix: "Lazy" Stringers to defer Sprintf and Hash until logs print (#8845)

Co-authored-by: Joe Abbey <joe.abbey@gmail.com>

* makefile: buf  setup backport v0.34 (#8863)

Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com>

* (fixup) Remove proto.yml since don't work and remove the next version of tendermint-v0.34.21: tendermint/tendermint@c723357

* (fixup) Remove unused import: buf-lint rule: IMPORT_USED

* (fixup) Remove duplicated a rule of `FILE_LOWER_SNAKE_CASE` since BASIC includes it

* (fixup) Use mockery latest v2.31.4 since v2.30.17 doesn't release: https://github.com/vektra/mockery/releases

* (fixup) Re-generate mocks with mockery-v2.30.18

* feat: v0.34.x Prioritized Mempool (#8695)

* Updated mocks

* add reactor tests

* add v1 reactor tests

* Fix fuzz test for priority mempool

* e2e adapted to mempool v1; prio pool is default now

* Reverted default mempool to be fifo

* Changed buf version

* Added priority mempool to ci testnet

* Fixed linter

* Updated makefile

* Aligned makefile changes to v0.34.x

* Added go install for proto

* Add log message to warn about prioritized mempool bug

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Changelog message

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com>
Co-authored-by: Callum Waters <cmwaters19@gmail.com>
Co-authored-by: Sam Kleinman <garen@tychoish.com>
Co-authored-by: Thane Thomson <connect@thanethomson.com>

* release: prepare v0.34.20-rc0 (#8888)

Co-authored-by: Callum Waters <cmwaters19@gmail.com>

* Backport priority mempool fixes from v0.35.x to v0.34.x. (#8962)

This is a manual backport of the changes from these commits:

- bc49f66 Add more unit tests for the priority mempool. (#8961)
- 9b02094 Fix unbounded heap growth in the priority mempool. (#8944)

Imports and type signatures have been updated to match the v0.34 usage.

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* mempool: minor cleanup after backport from v0.35 (#8971)

- Remove warning log for issue #8775.
- Fix call to FlushAsync (no error is reported).
- Don't log on rechecks, it's the default (manual backport of #8969).

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* mempool: reduce lock contention during CheckTx (backport #8983) (#8985)

A manual cherry-pick of 9e64c95.

The way this was originally structured, we reacquired the lock after issuing
the initial ABCI CheckTx call, only to immediately release it. Restructure the
code so that this redundant acquire is no longer necessary.

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* mempool: release lock during app connection flush (#8986)

A manual backport of #8984.

This case is symmetric to what we did for CheckTx calls, where we release the
mempool mutex to ensure callbacks can fire during call setup.  We also need
this behaviour for application flush, for the same reason: The caller holds the
lock by contract from the Mempool interface.

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* config: remove obsolete mempool v1 warning (#8987)

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* mempool: ensure evicted transactions are removed from the cache (backport #9000) (#9004)

This is a manual cherry-pick of commit b94470a6a42e8ffe7e7467521de5f51eb937c454.

In the original implementation transactions evicted for priority were also
removed from the cache. In addition, remove expired transactions from the
cache.

Related:

- Add Has method to cache implementations.
- Update tests to exercise this condition.

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>

* mempool: ensure async requests are flushed to the server (#9010)

In the v0.34 line, the socket and gRPC clients require explicit flushes to
ensure that the client and server have received an async request.  Add these
calls explicitly where required in the backport of the priority mempool.

In addition, the gRPC client's flush plumbing was not fully hooked up in the
v0.34 line, so this change includes that update as well.

Co-authored: M. J. Fromberger <fromberger@interchain.io>

* Upgrade to tendermint/tendermint v0.34.20

* (fixup) Improve libraries to work well

---------

Co-authored-by: Sam Kleinman <garen@tychoish.com>
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com>
Co-authored-by: Joe Abbey <joe.abbey@gmail.com>
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Callum Waters <cmwaters19@gmail.com>
Co-authored-by: Thane Thomson <connect@thanethomson.com>

.github/workflows/check-generated.yml

@@ -0,0 +1,75 @@
+# Verify that generated code is up-to-date.
+#
+# Note that we run these checks regardless whether the input files have
+# changed, because generated code can change in response to toolchain updates
+# even if no files in the repository are modified.
+name: Check generated code
+on:
+  pull_request:
+    branches:
+      - main
+
+permissions:
+  contents: read
+
+jobs:
+  check-mocks:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v3
+        with:
+          go-version: '1.18'
+
+      - uses: actions/checkout@v3
+
+      - name: "Check generated mocks"
+        run: |
+          set -euo pipefail
+
+          readonly MOCKERY=2.30.18  # N.B. no leading "v"
+          curl -sL "https://github.com/vektra/mockery/releases/download/v${MOCKERY}/mockery_${MOCKERY}_Linux_x86_64.tar.gz" | tar -C /usr/local/bin -xzf -
+          make mockery 2>/dev/null
+
+          if ! git diff --stat --exit-code ; then
+            echo ">> ERROR:"
+            echo ">>"
+            echo ">> Generated mocks require update (either Mockery or source files may have changed)."
+            echo ">> Ensure your tools are up-to-date, re-run 'make mockery' and update this PR."
+            echo ">>"
+            exit 1
+          fi
+
+  check-proto:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v3
+        with:
+          go-version: '1.18'
+
+      - uses: actions/checkout@v3
+        with:
+          fetch-depth: 1  # we need a .git directory to run git diff
+
+      - name: "Check protobuf generated code"
+        run: |
+          set -euo pipefail
+
+          # Install buf and gogo tools, so that differences that arise from
+          # toolchain differences are also caught.
+          readonly tools="$(mktemp -d)"
+          export PATH="${PATH}:${tools}/bin"
+          export GOBIN="${tools}/bin"
+
+          go install github.com/bufbuild/buf/cmd/buf
+          go install github.com/gogo/protobuf/protoc-gen-gogofaster@latest
+
+          make proto-gen
+
+          if ! git diff --stat --exit-code ; then
+            echo ">> ERROR:"
+            echo ">>"
+            echo ">> Protobuf generated code requires update (either tools or .proto files may have changed)."
+            echo ">> Ensure your tools are up-to-date, re-run 'make proto-gen' and update this PR."
+            echo ">>"
+            exit 1
+          fi

.github/workflows/proto-lint.yml

@@ -0,0 +1,19 @@
+name: Protobuf Lint
+on:
+  pull_request:
+    paths:
+      - 'proto/**'
+  push:
+    paths:
+      - 'proto/**'
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - uses: actions/checkout@v3
+      - uses: bufbuild/buf-setup-action@v1.5.0
+      - uses: bufbuild/buf-lint-action@v1
+        with:
+          input: 'proto'

Makefile

@@ -8,7 +8,6 @@ VERSION := $(shell git describe --always)
 LD_FLAGS = -X github.com/Finschia/ostracon/version.OCCoreSemVer=$(VERSION)
 BUILD_FLAGS = -mod=readonly -ldflags "$(LD_FLAGS)"
 HTTPS_GIT := https://github.com/Finschia/ostracon.git
-DOCKER_BUF := docker run -v $(shell pwd):/workspace --workdir /workspace bufbuild/buf
 CGO_ENABLED ?= 0
 TARGET_OS ?= $(shell go env GOOS)
 TARGET_ARCH ?= $(shell go env GOARCH)
@@ -77,42 +76,48 @@ mockery:
 ###                                Protobuf                                 ###
 ###############################################################################
 
-containerProtoVer=v0.2
-containerProtoImage=tendermintdev/sdk-proto-gen:$(containerProtoVer)
-
-###
-# https://github.com/protocolbuffers/protobuf
-# https://developers.google.com/protocol-buffers/docs/gotutorial
-# Should install
-### go install
-# go install google.golang.org/protobuf/cmd/protoc-gen-go
-### Docker for Protocol Buffer
-# https://hub.docker.com/r/bufbuild/buf
+check-proto-deps:
+ifeq (,$(shell which protoc-gen-gogofaster))
+	@go install github.com/gogo/protobuf/protoc-gen-gogofaster@latest
+endif
+.PHONY: check-proto-deps
 
-proto-all: proto-gen proto-lint proto-check-breaking
-.PHONY: proto-all
+check-proto-format-deps:
+ifeq (,$(shell which clang-format))
+	$(error "clang-format is required for Protobuf formatting. See instructions for your platform on how to install it.")
+endif
+.PHONY: check-proto-format-deps
 
-proto-gen:
-	@docker pull -q tendermintdev/docker-build-proto
+proto-gen: check-proto-deps
 	@echo "Generating Protobuf files"
-	@docker run --rm -v $(shell pwd):/workspace --workdir /workspace $(containerProtoImage) sh ./scripts/protocgen.sh
+	@go run github.com/bufbuild/buf/cmd/buf generate
+	@mv ./proto/ostracon/abci/types.pb.go ./abci/types/
+	@mv ./proto/ostracon/rpc/grpc/types.pb.go ./rpc/grpc/
+	@rm -rf ./proto/tendermint
 .PHONY: proto-gen
 
-proto-lint:
-	@$(DOCKER_BUF) lint --error-format=json
+# These targets are provided for convenience and are intended for local
+# execution only.
+proto-lint: check-proto-deps
+	@echo "Linting Protobuf files"
+	@go run github.com/bufbuild/buf/cmd/buf lint
 .PHONY: proto-lint
 
-proto-format:
+proto-format: check-proto-format-deps
 	@echo "Formatting Protobuf files"
-	docker run --rm -v $(shell pwd):/workspace --workdir /workspace tendermintdev/docker-build-proto find ./ -not -path "./third_party/*" -name *.proto -exec clang-format -i {} \;
+	@find . -name '*.proto' -path "./proto/*" -exec clang-format -i {} \;
 .PHONY: proto-format
 
-proto-check-breaking:
-	@$(DOCKER_BUF) breaking --against .git#branch=main
+proto-check-breaking: check-proto-deps
+	@echo "Checking for breaking changes in Protobuf files against local branch"
+	@echo "Note: This is only useful if your changes have not yet been committed."
+	@echo "      Otherwise read up on buf's \"breaking\" command usage:"
+	@echo "      https://docs.buf.build/breaking/usage"
+	@go run github.com/bufbuild/buf/cmd/buf breaking --against ".git"
 .PHONY: proto-check-breaking
 
 proto-check-breaking-ci:
-	@$(DOCKER_BUF) breaking --against $(HTTPS_GIT)#branch=main
+	@go run github.com/bufbuild/buf/cmd/buf breaking --against ".git"#branch=main
 .PHONY: proto-check-breaking-ci
 
 ###############################################################################

abci/client/grpc_client.go

@@ -310,6 +310,7 @@ func (cli *grpcClient) finishAsyncCall(req *ocabci.Request, res *ocabci.Response
 }
 
 // ----------------------------------------
+
 func (cli *grpcClient) FlushSync() (*types.ResponseFlush, error) {
 	reqres := cli.FlushAsync(nil)
 	reqres.Wait()