Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support/db: Add benchmark for BatchInsertBuilder #4086

Merged
merged 1 commit into from
Nov 22, 2021
Merged

support/db: Add benchmark for BatchInsertBuilder #4086

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 45 additions & 6 deletions support/db/batch_insert_builder_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package db

import (
"context"
"fmt"
"testing"

"github.com/stellar/go/support/db/dbtest"
Expand All @@ -20,6 +21,44 @@ type invalidHungerRow struct {
LastName string `db:"last_name"`
}

func BenchmarkBatchInsertBuilder(b *testing.B) {
b.StopTimer()
// In order to show SQL queries
// log.SetLevel(logrus.DebugLevel)
db := dbtest.Postgres(b).Load(testSchema)
defer db.Close()
sess := &Session{DB: db.Open()}
defer sess.DB.Close()
ctx := context.Background()
maxBatchSize := 1000
insertBuilder := &BatchInsertBuilder{
Table: sess.GetTable("people"),
MaxBatchSize: maxBatchSize,
}
b.StartTimer()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it make sense to use b.ResetTimer() here instead of stoping and then starting?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

auto-merge merged before I could resolve this, sorry.

I thought @leighmcculloch changed the repo config so that all comments had to be resolved before auto-merged kicked in?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tamirms you are right, it's a better option. I will change it later on 👍

Copy link
Member

@leighmcculloch leighmcculloch Nov 22, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though terraform does not support configuring the wait for conversations to be resolved, it appears it is being cleared on multiple repos, this one and the starlight repo. Maybe the terraform integration is clearing it on every deploy. I think this means we can't use the wait for conversations feature until integrations/terraform-provider-github#868 is resolved.

It might be worth holding off on using auto-merge until this is resolved. The GitHub terraform provider usually ships new features quickly so we might not be waiting that long.

cc @stellar/ops-team

for i := 0; i < b.N; i++ {
for j := 0; j < maxBatchSize; j++ {
err := insertBuilder.RowStruct(ctx, hungerRow{
Name: fmt.Sprintf("bubba%d", i*maxBatchSize+j),
HungerLevel: "1202",
})
require.NoError(b, err)
}
}
err := insertBuilder.Exec(ctx)

b.StopTimer()
assert.NoError(b, err)
var count []int
err = sess.SelectRaw(ctx,
&count,
"SELECT COUNT(*) FROM people",
)
assert.NoError(b, err)
preexistingCount := 3
assert.Equal(b, b.N*maxBatchSize+preexistingCount, count[0])
}

func TestBatchInsertBuilder(t *testing.T) {
db := dbtest.Postgres(t).Load(testSchema)
defer db.Close()
Expand Down Expand Up @@ -87,8 +126,8 @@ func TestBatchInsertBuilder(t *testing.T) {
t,
found,
[]person{
person{Name: "bubba", HungerLevel: "120"},
person{Name: "bubba2", HungerLevel: "1202"},
{Name: "bubba", HungerLevel: "120"},
{Name: "bubba2", HungerLevel: "1202"},
},
)

Expand Down Expand Up @@ -122,8 +161,8 @@ func TestBatchInsertBuilder(t *testing.T) {
t,
found,
[]person{
person{Name: "bubba", HungerLevel: "120"},
person{Name: "bubba2", HungerLevel: "1202"},
{Name: "bubba", HungerLevel: "120"},
{Name: "bubba2", HungerLevel: "1202"},
},
)

Expand All @@ -145,8 +184,8 @@ func TestBatchInsertBuilder(t *testing.T) {
t,
found,
[]person{
person{Name: "bubba2", HungerLevel: "1202"},
person{Name: "bubba", HungerLevel: "1"},
{Name: "bubba2", HungerLevel: "1202"},
{Name: "bubba", HungerLevel: "1"},
},
)
}
6 changes: 3 additions & 3 deletions support/db/dbtest/db.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ type DB struct {
Dialect string
DSN string
dbName string
t *testing.T
t testing.TB
closer func()
closed bool
}
Expand Down Expand Up @@ -101,7 +101,7 @@ func (db *DB) Version() (major int) {
return major
}

func execStatement(t *testing.T, pguser, query string) {
func execStatement(t testing.TB, pguser, query string) {
db, err := sqlx.Open("postgres", fmt.Sprintf("postgres://%s@localhost/?sslmode=disable", pguser))
require.NoError(t, err)
_, err = db.Exec(query)
Expand All @@ -113,7 +113,7 @@ func execStatement(t *testing.T, pguser, query string) {
// of the running process. It assumes that you have postgres running on the
// default port, have the command line postgres tools installed, and that the
// current user has access to the server. It panics on the event of a failure.
func Postgres(t *testing.T) *DB {
func Postgres(t testing.TB) *DB {
var result DB
result.dbName = randomName()
result.Dialect = "postgres"
Expand Down