Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: adopt testcontainers-go for Postgres, MySQL, MSSQL and MongoDB #1515

Open
wants to merge 44 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
4ed3649
chore: use testcontainers-go in mongodb store
mdelapenya Aug 29, 2024
0e060e1
chore: use testcontainers-go in mssql store
mdelapenya Aug 29, 2024
2f17493
chore: use testcontainers-go in mysql store
mdelapenya Aug 29, 2024
2f5263e
chore: use testcontainers-go in postgres store
mdelapenya Aug 29, 2024
689759c
fix: wrong image name for mongo
mdelapenya Aug 29, 2024
ff25433
fix: proper Go version matrix
mdelapenya Aug 29, 2024
fc01fc5
fix: remove toolchain in go mod
mdelapenya Aug 29, 2024
4ad4b03
chor: consistent Go version
mdelapenya Aug 29, 2024
b53f02b
fix: remove toolchain
mdelapenya Aug 29, 2024
de8f798
chore: consistent Go versions matrix
mdelapenya Aug 29, 2024
471a6ac
fix: remove more postgres options
mdelapenya Sep 16, 2024
c80f126
chore: adjust Go matrix versions to use 1.20 as the minimum version
mdelapenya Sep 16, 2024
1d6c0c3
chore: set Go 1.21 for modules
mdelapenya Sep 16, 2024
4d7a172
chore: use Ubuntu 22.04 base image for MSSQL
mdelapenya Sep 18, 2024
75c8525
chore: close store for previous modules
mdelapenya Sep 18, 2024
1bb4d92
chore: close store
mdelapenya Sep 18, 2024
f962d64
chore: terminate container on end
mdelapenya Sep 18, 2024
1c23d53
chore: bump MSSQL image to not crash in the GH workers
mdelapenya Sep 19, 2024
7bbad9a
fix: swap values in mysql constants
mdelapenya Sep 19, 2024
efbc89f
fix: do not close store twice in tests
mdelapenya Sep 19, 2024
153c837
fix: reorder
mdelapenya Sep 19, 2024
43a47ef
fix: do not close twice in mysql test
mdelapenya Sep 19, 2024
3cd89db
fix: terminate second mysql container
mdelapenya Sep 19, 2024
60be628
chore: bump couchbase image
mdelapenya Sep 19, 2024
a4849c8
Revert "fix: terminate second mysql container"
mdelapenya Sep 19, 2024
b3a2e40
chore: bump testcontainers-go to v0.34.0
mdelapenya Oct 23, 2024
db3ab9a
Merge branch 'main' into tc-go-adoption-2
mdelapenya Dec 11, 2024
78da7fc
chore(ci): run modules in Go 1.21 and above
mdelapenya Dec 11, 2024
49eeb1d
chore: add verbose flag to benchmarks
mdelapenya Dec 11, 2024
1a5e6f9
chore: include Go 1.23 for clickhouse tests
mdelapenya Dec 11, 2024
cdf78c2
chore: tune the benchmark worker
mdelapenya Dec 11, 2024
a7a26b8
fix: run with sudo
mdelapenya Dec 11, 2024
39be258
chore: remove tuning
mdelapenya Dec 11, 2024
8ee8130
chore: bump MSSQL image
mdelapenya Dec 11, 2024
bff5333
chore: simplify testing panics
mdelapenya Dec 12, 2024
8cb7bc5
chore: increase sysctl values for the MSSQL container
mdelapenya Dec 12, 2024
15865d5
Revert "chore: increase sysctl values for the MSSQL container"
mdelapenya Dec 12, 2024
4f1577a
fix: use ubuntu-22.04 as GH runner for the benchmarks
mdelapenya Dec 12, 2024
478ea67
Revert "fix: use ubuntu-22.04 as GH runner for the benchmarks"
mdelapenya Dec 12, 2024
b074e83
chore: try MSSQL 2022 latest image
mdelapenya Dec 12, 2024
cac4fd5
chore: use cleanup function
mdelapenya Dec 12, 2024
ae3a2de
chore: revert MSSQL work
mdelapenya Dec 12, 2024
c061165
chor: increase go test timeout
mdelapenya Dec 12, 2024
af52fa3
chore: define timeout for the workflow
mdelapenya Dec 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 8 additions & 55 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@ permissions:
name: Benchmark
jobs:
Compare:
runs-on: ubuntu-latest
# Using 22.04 until latest receive the changes in https://github.com/actions/runner-images/issues/10649
runs-on: ubuntu-22.04
services:
arangodb:
image: 'arangodb:latest'
Expand All @@ -34,49 +35,6 @@ jobs:
image: 'memcached:latest'
ports:
- '11211:11211'
mongo:
image: 'mongo:latest'
ports:
- '27017:27017'
mssql:
image: 'mcmoe/mssqldocker:latest'
ports:
- '1433:1433'
env:
ACCEPT_EULA: Y
SA_PASSWORD: MsSql!1234
MSSQL_DB: master
MSSQL_USER: sa
MSSQL_PASSWORD: MsSql!1234
options: >-
--health-cmd "/opt/mssql-tools/bin/sqlcmd -U sa -P $SA_PASSWORD -Q 'select 1' -b -o /dev/null"
--health-interval 1s
--health-timeout 30s
--health-start-period 10s
--health-retries 20
mysql:
image: 'mysql:latest'
env:
MYSQL_DATABASE: fiber
MYSQL_USER: username
MYSQL_PASSWORD: password
MYSQL_ROOT_PASSWORD: password
ports:
- '3306:3306'
options: >-
--health-cmd "mysqladmin ping" --health-interval 10s --health-timeout
5s --health-retries 5
postgres:
image: 'postgres:latest'
ports:
- '5432:5432'
env:
POSTGRES_DB: fiber
POSTGRES_USER: username
POSTGRES_PASSWORD: "pass#w%rd"
options: >-
--health-cmd pg_isready --health-interval 10s --health-timeout 5s
--health-retries 5

steps:
- name: Fetch Repository
Expand Down Expand Up @@ -147,23 +105,18 @@ jobs:

cd "$d"
echo "Bench dir: $d"
go test ./... -benchmem -run=^$ -bench . | tee -a ../output.txt
go test ./... -v -benchmem -run=^$ -bench . | tee -a ../output.txt
cd ..
done
shell: bash
env:
MSSQL_DATABASE: master
MSSQL_USERNAME: sa
MSSQL_PASSWORD: MsSql!1234
MYSQL_USERNAME: username
MYSQL_PASSWORD: password
MYSQL_DATABASE: fiber
POSTGRES_DATABASE: fiber
POSTGRES_USERNAME: username
POSTGRES_PASSWORD: "pass#w%rd"
TEST_CLICKHOUSE_IMAGE: "clickhouse/clickhouse-server:23-alpine"
TEST_COUCHBASE_IMAGE: "couchbase:enterprise-7.1.1"
TEST_COUCHBASE_IMAGE: "couchbase:enterprise-7.6.3"
TEST_MINIO_IMAGE: "docker.io/minio/minio:RELEASE.2024-08-17T01-24-54Z"
TEST_MONGODB_IMAGE: "docker.io/mongo:7"
TEST_MSSQL_IMAGE: "mcr.microsoft.com/mssql/server:2022-CU16-ubuntu-22.04"
TEST_MYSQL_IMAGE: "docker.io/mysql:9"
TEST_POSTGRES_IMAGE: "docker.io/postgres:16-alpine"

- name: Get Previous Benchmark Results
uses: actions/cache@v4
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/test-clickhouse.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ jobs:
go-version:
- 1.21.x
- 1.22.x
- 1.23.x
steps:
- name: Fetch Repository
uses: actions/checkout@v4
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test-couchbase.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,5 @@ jobs:
go-version: '${{ matrix.go-version }}'
- name: Run Test
env:
TEST_COUCHBASE_IMAGE: couchbase:enterprise-7.1.1
TEST_COUCHBASE_IMAGE: couchbase:enterprise-7.6.3
run: cd ./couchbase && go test ./... -v -race
11 changes: 4 additions & 7 deletions .github/workflows/test-mongodb.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,17 +12,12 @@ name: "Tests Mongodb"
jobs:
Tests:
runs-on: ubuntu-latest
services:
mongo:
image: 'mongo:latest'
ports:
- '27017:27017'
strategy:
matrix:
go-version:
- 1.19.x
- 1.20.x
- 1.21.x
- 1.22.x
mdelapenya marked this conversation as resolved.
Show resolved Hide resolved
- 1.23.x
steps:
- name: Fetch Repository
uses: actions/checkout@v4
Expand All @@ -31,5 +26,7 @@ jobs:
with:
go-version: '${{ matrix.go-version }}'
- name: Run Test
env:
TEST_MONGODB_IMAGE: docker.io/mongo:7
run: cd ./mongodb && go test ./... -v -race

27 changes: 4 additions & 23 deletions .github/workflows/test-mssql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,29 +12,12 @@ name: "Tests MSSQL"
jobs:
Tests:
runs-on: ubuntu-latest
services:
mssql:
image: 'mcmoe/mssqldocker:latest'
ports:
- '1433:1433'
env:
ACCEPT_EULA: Y
SA_PASSWORD: MsSql!1234
MSSQL_DB: master
MSSQL_USER: sa
MSSQL_PASSWORD: MsSql!1234
options: >-
--health-cmd "/opt/mssql-tools/bin/sqlcmd -U sa -P $SA_PASSWORD -Q 'select 1' -b -o /dev/null"
--health-interval 1s
--health-timeout 30s
--health-start-period 10s
--health-retries 20
strategy:
matrix:
go-version:
- 1.19.x
- 1.20.x
- 1.21.x
- 1.22.x
- 1.23.x
steps:
- name: Fetch Repository
uses: actions/checkout@v4
Expand All @@ -43,8 +26,6 @@ jobs:
with:
go-version: '${{ matrix.go-version }}'
- name: Run Test
run: cd ./mssql && go test ./... -v -race
env:
MSSQL_DATABASE: master
MSSQL_USERNAME: sa
MSSQL_PASSWORD: MsSql!1234
TEST_MSSQL_IMAGE: "mcr.microsoft.com/mssql/server:2022-CU16-ubuntu-22.04"
run: cd ./mssql && go test ./... -v -race
23 changes: 4 additions & 19 deletions .github/workflows/test-mysql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,25 +12,12 @@ name: "Tests MySQL"
jobs:
Tests:
runs-on: ubuntu-latest
services:
mysql:
image: 'mysql:latest'
env:
MYSQL_DATABASE: fiber
MYSQL_USER: username
MYSQL_PASSWORD: password
MYSQL_ROOT_PASSWORD: password
ports:
- '3306:3306'
options: >-
--health-cmd "mysqladmin ping" --health-interval 10s --health-timeout
5s --health-retries 5
strategy:
matrix:
go-version:
- 1.19.x
- 1.20.x
- 1.21.x
- 1.22.x
mdelapenya marked this conversation as resolved.
Show resolved Hide resolved
- 1.23.x
steps:
- name: Fetch Repository
uses: actions/checkout@v4
Expand All @@ -39,8 +26,6 @@ jobs:
with:
go-version: '${{ matrix.go-version }}'
- name: Run Test
run: cd ./mysql && go test ./... -v -race
env:
MYSQL_USERNAME: username
MYSQL_PASSWORD: password
MYSQL_DATABASE: fiber
TEST_MYSQL_IMAGE: docker.io/mysql:9
run: cd ./mysql && go test ./... -v -race
20 changes: 3 additions & 17 deletions .github/workflows/test-postgres.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,24 +12,12 @@ name: "Tests Postgres"
jobs:
Tests:
runs-on: ubuntu-latest
services:
postgres:
image: 'postgres:latest'
ports:
- '5432:5432'
env:
POSTGRES_DB: fiber
POSTGRES_USER: username
POSTGRES_PASSWORD: "pass#w%rd"
options: >-
--health-cmd pg_isready --health-interval 10s --health-timeout 5s
--health-retries 5
strategy:
matrix:
go-version:
- 1.20.x
- 1.21.x
- 1.22.x
- 1.23.x
steps:
- name: Fetch Repository
uses: actions/checkout@v4
Expand All @@ -38,8 +26,6 @@ jobs:
with:
go-version: '${{ matrix.go-version }}'
- name: Run Test
run: cd ./postgres && go test ./... -v -race
env:
POSTGRES_DATABASE: fiber
POSTGRES_USERNAME: username
POSTGRES_PASSWORD: "pass#w%rd"
TEST_POSTGRES_IMAGE: "docker.io/postgres:16-alpine"
run: cd ./postgres && go test ./... -v -race
21 changes: 19 additions & 2 deletions clickhouse/clickhouse_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,11 @@ func getTestConnection(t testing.TB, cfg Config) (*Storage, error) {
if err != nil {
return nil, err
}
t.Cleanup(func() {
if c != nil {
require.NoError(t, c.Terminate(ctx))
}
})

hostPort, err := c.ConnectionHost(ctx)
if err != nil {
Expand All @@ -67,13 +72,14 @@ func getTestConnection(t testing.TB, cfg Config) (*Storage, error) {
}

func Test_Connection(t *testing.T) {
_, err := getTestConnection(t, Config{
client, err := getTestConnection(t, Config{
Engine: Memory,
Table: "test_table",
Clean: true,
})

require.NoError(t, err)

defer client.Close()
}

func Test_Set(t *testing.T) {
Expand Down Expand Up @@ -191,6 +197,17 @@ func Test_Reset(t *testing.T) {
assert.Equal(t, []byte{}, value)
}

func TestClose_ShouldReturn_NoError(t *testing.T) {
client, err := getTestConnection(t, Config{
Engine: Memory,
Table: "test_table",
Clean: true,
})
require.NoError(t, err)

require.NoError(t, client.Close())
}

func Benchmark_Clickhouse_Set(b *testing.B) {
b.ReportAllocs()
b.ResetTimer()
Expand Down
8 changes: 4 additions & 4 deletions clickhouse/go.mod
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
module github.com/gofiber/storage/clickhouse

go 1.21
go 1.22

require (
github.com/ClickHouse/clickhouse-go/v2 v2.26.0
github.com/stretchr/testify v1.9.0
github.com/testcontainers/testcontainers-go/modules/clickhouse v0.33.0
github.com/testcontainers/testcontainers-go/modules/clickhouse v0.34.0
)

require (
Expand All @@ -18,7 +18,7 @@ require (
github.com/containerd/containerd v1.7.18 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v0.2.1 // indirect
github.com/cpuguy83/dockercfg v0.3.1 // indirect
github.com/cpuguy83/dockercfg v0.3.2 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/docker v27.1.1+incompatible // indirect
Expand Down Expand Up @@ -53,7 +53,7 @@ require (
github.com/shoenig/go-m1cpu v0.1.6 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/testcontainers/testcontainers-go v0.33.0 // indirect
github.com/testcontainers/testcontainers-go v0.34.0 // indirect
github.com/tklauser/go-sysconf v0.3.12 // indirect
github.com/tklauser/numcpus v0.6.1 // indirect
github.com/yusufpapurcu/wmi v1.2.3 // indirect
Expand Down
14 changes: 8 additions & 6 deletions clickhouse/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I=
github.com/containerd/log v0.1.0/go.mod h1:VRRf09a7mHDIRezVKTRCrOq78v577GXq3bSa3EhrzVo=
github.com/containerd/platforms v0.2.1 h1:zvwtM3rz2YHPQsF2CHYM8+KtB5dvhISiXh5ZpSBQv6A=
github.com/containerd/platforms v0.2.1/go.mod h1:XHCb+2/hzowdiut9rkudds9bE5yJ7npe7dG/wG+uFPw=
github.com/cpuguy83/dockercfg v0.3.1 h1:/FpZ+JaygUR/lZP2NlFI2DVfrOEMAIKP5wWEJdoYe9E=
github.com/cpuguy83/dockercfg v0.3.1/go.mod h1:sugsbF4//dDlL/i+S+rtpIWp+5h0BHJHfjj5/jFyUJc=
github.com/cpuguy83/dockercfg v0.3.2 h1:DlJTyZGBDlXqUZ2Dk2Q3xHs/FtnooJJVaad2S9GKorA=
github.com/cpuguy83/dockercfg v0.3.2/go.mod h1:sugsbF4//dDlL/i+S+rtpIWp+5h0BHJHfjj5/jFyUJc=
github.com/creack/pty v1.1.18 h1:n56/Zwd5o6whRC5PMGretI4IdRLlmBXYNjScPaBgsbY=
github.com/creack/pty v1.1.18/go.mod h1:MOBLtS5ELjhRRrroQr9kyvTxUAFNvYEK993ew/Vr4O4=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
Expand Down Expand Up @@ -122,17 +122,19 @@ github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVs
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/testcontainers/testcontainers-go v0.33.0 h1:zJS9PfXYT5O0ZFXM2xxXfk4J5UMw/kRiISng037Gxdw=
github.com/testcontainers/testcontainers-go v0.33.0/go.mod h1:W80YpTa8D5C3Yy16icheD01UTDu+LmXIA2Keo+jWtT8=
github.com/testcontainers/testcontainers-go/modules/clickhouse v0.33.0 h1:YbB5DBkpgY+GlGPFqTSV1hzWPm3ZHirEyooZrj+ZXK0=
github.com/testcontainers/testcontainers-go/modules/clickhouse v0.33.0/go.mod h1:qJuMPl9yWIWasmdBILM2uDk1Ny1kdeigcKMJ6A8PZz0=
github.com/testcontainers/testcontainers-go v0.34.0 h1:5fbgF0vIN5u+nD3IWabQwRybuB4GY8G2HHgCkbMzMHo=
github.com/testcontainers/testcontainers-go v0.34.0/go.mod h1:6P/kMkQe8yqPHfPWNulFGdFHTD8HB2vLq/231xY2iPQ=
github.com/testcontainers/testcontainers-go/modules/clickhouse v0.34.0 h1:1CO1aV5kFXWvFCTxgIVjcMXaXwO8cznw58F+V8Kp+Fg=
github.com/testcontainers/testcontainers-go/modules/clickhouse v0.34.0/go.mod h1:TTecjXxuNwIyNfSKo6X071LoUkMIa3+unU5yam8Db+s=
github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
github.com/tklauser/go-sysconf v0.3.12 h1:0QaGUFOdQaIVdPgfITYzaTegZvdCjmYO52cSFAEVmqU=
github.com/tklauser/go-sysconf v0.3.12/go.mod h1:Ho14jnntGE1fpdOqQEEaiKRpvIavV0hSfmBq8nJbHYI=
Expand Down
Loading
Loading