Skip to content

Commit

Permalink
refactor(tm2): split pkg/db into sub-packages (gnolang#1602)
Browse files Browse the repository at this point in the history
Changes mostly involve shuffling code around.

The `db` package now has its basic types and helper functions in
`pkg/db`, and then sub-packages for each DB which is actually used. This
allows a package using it to only import the database code it needs.

The end goal I wanted was to remove the transitive dependency of
`cmd/gno` on `goleveldb`. Turns out this is more complicated than I
thought. In any case, I already did it in another branch; will make a
second PR shortly after this one.

Notes for reviewing:

- `pkg/db` has also the helper packages `_all` and `_tags`.
- `_all` imports all the databases while respecting the `cgo` build
tag.[^1] (This is automatically set by the go compiler depending on
whether cgo is enabled).
- The `_tags` package, instead, imports the databases specified by the
build tags. I meant this for end-user binaries, like gno.land eventually
(currently it only supports GoLevelDB), so a user can compile with the
given build tags.
- I removed `badger` and `gorocksdb`. They were so important that they
didn't compile and nobody noticed.
- `badger` probably makes sense to be re-added and tried out eventually,
but it's outside of the scope of this PR.
- `gorocksdb` is unmaintained and will not compile with the latest
version of RocksDB. In fact, we already had a DB implementation for
[grocksdb](https://github.com/linxGnu/grocksdb), the maintained fork. I
fixed some code there too to actually make it compile.
- Removing the two dependencies made our `go.mods` a bunch lighter.
:tada:
- Some changes also involve the CI. 
- I removed splitting the build for each database being tested, as the
tests on the package run quickly anyway so there is no real need to
split them.
- I also severely downgraded `grocksdb` so that its version is
compatible with the version of RocksDB present on ubuntu 22.04's
repositories. There could be a variety of different changes to the CI to
allow building on a more recent version of the database; but after
having seen that the compilation of RocksDB on a beefy machine with
`make -j4` is already beyond 5 mintues, I decided we don't need a slow
CI for a feature literally no-one uses, so here we are.
- This PR also fixes the benchmarks in `tm2/pkg/iavl/benchmarks` (fixes
gnolang#908). [The fix itself is
stupid.](gnolang@87ea39d#diff-112673453ecf482fb6bfd8004ebc11eff7536b812cc48bd6dff3328767550908R227-R229)

[^1]: A while ago I was upset that my go compilation was taking long and
then found out that it was using cgo for no apparent reason -- it's
actually because a few places in the standard library will by default
use cgo unless it's disabled. For this reason I actually set
`CGO_ENABLED=0` in my `.profile`.

---------

Co-authored-by: Antonio Navarro <antnavper@gmail.com>
  • Loading branch information
2 people authored and leohhhn committed Mar 6, 2024
1 parent 3356c7a commit 8cc113e
Show file tree
Hide file tree
Showing 94 changed files with 1,267 additions and 2,263 deletions.
10 changes: 5 additions & 5 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,14 @@ updates:
golang-x:
patterns:
- "golang.org/x/*"
dbs:
patterns:
- "github.com/linxGnu/grocksdb"
- "go.etcd.io/bbolt"
- "github.com/dgraph-io/badger/v3"
everything-else:
patterns:
- "*"
exclude-patterns:
# NOTE: grocksdb should be updated manually, to match the version
# available on Ubuntu's latest LTS release; and updated in conjunction
# with the ubuntu version on .github/workflows/db-tests.yml
- "github.com/linxGnu/grocksdb"
open-pull-requests-limit: 10
pull-request-branch-name:
separator: "-"
Expand Down
29 changes: 11 additions & 18 deletions .github/workflows/db-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,20 @@ concurrency:

jobs:
test:
runs-on: ubuntu-latest
# NOTE: this job uses a specific version of ubuntu as the version of grocksb
# we use is related to the version of RocksDB in ubuntu's repositories.
# If updating this to a later ubuntu release, update the grocksdb version
# accordingly:
# https://github.com/linxGnu/grocksdb/releases
# https://pkgs.org/search/?q=rocksdb-dev
runs-on: ubuntu-22.04
timeout-minutes: 5
strategy:
fail-fast: false
matrix:
goversion:
- "1.21.x"
- "1.22.x"
tags:
- cleveldb
- memdb
- fsdb
- boltdb
steps:
- uses: actions/checkout@v4

Expand All @@ -36,16 +37,8 @@ jobs:
with:
go-version: ${{ matrix.goversion }}

# leveldb
- name: install leveldb
if: ${{ matrix.tags=='cleveldb' }}
run: |
sudo apt-get install libsnappy1v5
mkdir -p /tmp/leveldb
cd /tmp/leveldb
wget http://ftp.us.debian.org/debian/pool/main/l/leveldb/libleveldb1d_1.22-3_amd64.deb
wget http://ftp.us.debian.org/debian/pool/main/l/leveldb/libleveldb-dev_1.22-3_amd64.deb
sudo dpkg -i *.deb
- name: install database dependencies
run: sudo apt-get install -y libleveldb-dev librocksdb-dev

- name: Set environment variables for debug mode
if: env.ACTIONS_STEP_DEBUG == 'true'
Expand All @@ -55,7 +48,7 @@ jobs:
echo "LOG_LEVEL=debug" >> $GITHUB_ENV
echo "LOG_PATH_DIR=$LOG_PATH_DIR" >> $GITHUB_ENV
# test ./pkgs/db
- name: test ./tm2/pkg/db
run: go test -tags ${{ matrix.tags }} ./tm2/pkg/db/...
run: go test -v ./tm2/pkg/db/...
5 changes: 4 additions & 1 deletion .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ jobs:

- name: Lint
uses: golangci/golangci-lint-action@v4
env:
# Don't attempt to compile/resolve C packages (grocksdb, cleveldb).
CGO_ENABLED: 0
with:
# sync with misc/devdeps/go.mod
version: v1.54
Expand Down Expand Up @@ -79,6 +82,6 @@ jobs:
run: |
# Ensure Make is installed
make --version
# Run the tidy target
make tidy
1 change: 1 addition & 0 deletions .github/workflows/tm2.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ jobs:
- _test.pkg.amino
- _test.pkg.bft
- _test.pkg.others
# _test.pkg.db needs special dependencies -- see db-tests.
runs-on: ubuntu-latest
timeout-minutes: 21
steps:
Expand Down
28 changes: 7 additions & 21 deletions contribs/gnodev/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -9,60 +9,46 @@ require (
github.com/gnolang/gno v0.0.0-00010101000000-000000000000
github.com/gorilla/websocket v1.5.1
go.uber.org/zap v1.26.0
golang.org/x/term v0.16.0
golang.org/x/term v0.17.0
)

require (
dario.cat/mergo v1.0.0 // indirect
github.com/btcsuite/btcd/btcec/v2 v2.3.2 // indirect
github.com/btcsuite/btcd/btcutil v1.1.3 // indirect
github.com/cespare/xxhash v1.1.0 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/cockroachdb/apd/v3 v3.2.1 // indirect
github.com/cosmos/ledger-cosmos-go v0.13.3 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect
github.com/dgraph-io/badger/v3 v3.2103.5 // indirect
github.com/dgraph-io/ristretto v0.1.1 // indirect
github.com/dustin/go-humanize v1.0.0 // indirect
github.com/gnolang/goleveldb v0.0.9 // indirect
github.com/gnolang/overflow v0.0.0-20170615021017-4d914c927216 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/glog v1.1.0 // indirect
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/flatbuffers v1.12.1 // indirect
github.com/gorilla/mux v1.8.1 // indirect
github.com/gorilla/securecookie v1.1.1 // indirect
github.com/gorilla/sessions v1.2.1 // indirect
github.com/gotuna/gotuna v0.6.0 // indirect
github.com/jaekwon/testify v1.6.1 // indirect
github.com/jmhodges/levigo v1.0.0 // indirect
github.com/klauspost/compress v1.12.3 // indirect
github.com/kr/pretty v0.2.1 // indirect
github.com/libp2p/go-buffer-pool v0.1.0 // indirect
github.com/linxGnu/grocksdb v1.8.11 // indirect
github.com/pelletier/go-toml v1.9.5 // indirect
github.com/peterbourgon/ff/v3 v3.4.0 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/rogpeppe/go-internal v1.11.0 // indirect
github.com/rs/cors v1.10.1 // indirect
github.com/stretchr/testify v1.8.4 // indirect
github.com/tecbot/gorocksdb v0.0.0-20191217155057-f0fad39f321c // indirect
github.com/zondax/hid v0.9.2 // indirect
github.com/zondax/ledger-go v0.14.3 // indirect
go.etcd.io/bbolt v1.3.8 // indirect
go.opencensus.io v0.22.5 // indirect
go.uber.org/multierr v1.10.0 // indirect
go.uber.org/zap/exp v0.2.0 // indirect
golang.org/x/crypto v0.18.0 // indirect
golang.org/x/mod v0.14.0 // indirect
golang.org/x/net v0.20.0 // indirect
golang.org/x/sys v0.16.0 // indirect
golang.org/x/crypto v0.19.0 // indirect
golang.org/x/exp v0.0.0-20240222234643-814bf88cf225 // indirect
golang.org/x/mod v0.15.0 // indirect
golang.org/x/net v0.21.0 // indirect
golang.org/x/sys v0.17.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/tools v0.17.0 // indirect
golang.org/x/tools v0.18.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20231009173412-8bfb1ae86b6c // indirect
google.golang.org/grpc v1.58.3 // indirect
google.golang.org/protobuf v1.31.0 // indirect
Expand Down
Loading

0 comments on commit 8cc113e

Please sign in to comment.