Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: introduce load table total rows #3851

Merged
merged 31 commits into from
Sep 29, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
bca9936
chore: load table should respond with total rows affected
achettyiitr Sep 6, 2023
47fafb6
Merge branch 'master' of github.com:rudderlabs/rudder-server into cho…
achettyiitr Sep 10, 2023
f685e2c
chore: master pull
achettyiitr Sep 15, 2023
100e85e
Merge branch 'master' of github.com:rudderlabs/rudder-server into cho…
achettyiitr Sep 15, 2023
a9af702
Merge branch 'master' of github.com:rudderlabs/rudder-server into cho…
achettyiitr Sep 19, 2023
b4593f9
Merge branch 'master' of github.com:rudderlabs/rudder-server into cho…
achettyiitr Sep 19, 2023
a66eafb
chore: review comments
achettyiitr Sep 19, 2023
d0718f0
chore: master pull
achettyiitr Sep 20, 2023
324fafb
chore: review comments
achettyiitr Sep 20, 2023
6784931
chore: adding test cases for load table statistics
achettyiitr Sep 21, 2023
4216bbf
chore: adding test cases for load table statistics
achettyiitr Sep 21, 2023
8201b92
chore: master pull
achettyiitr Sep 21, 2023
9467b62
chore: master pull
achettyiitr Sep 22, 2023
5771d73
Merge branch 'master' of github.com:rudderlabs/rudder-server into cho…
achettyiitr Sep 22, 2023
4212717
chore: master pull
achettyiitr Sep 22, 2023
0e57e81
chore: some more changes
achettyiitr Sep 22, 2023
3b4c0b2
chore: some more changes
achettyiitr Sep 22, 2023
a95bdbf
chore: some more changes
achettyiitr Sep 24, 2023
cdc2dea
Merge commit 'a95bdbff0' into chore.total-rows
achettyiitr Sep 24, 2023
69ebdf2
Merge branch 'master' of github.com:rudderlabs/rudder-server into cho…
achettyiitr Sep 25, 2023
2463e11
chore: some more changes
achettyiitr Sep 26, 2023
4daa91b
Merge branch 'master' into chore.total-rows
achettyiitr Sep 27, 2023
d41db9b
chore: disable race
achettyiitr Sep 27, 2023
f6d4a45
Merge branch 'master' into chore.total-rows
achettyiitr Sep 27, 2023
462667c
chore: master pull
achettyiitr Sep 28, 2023
8d3b470
fix: use dedup on new records for deltalake
achettyiitr Sep 28, 2023
2a47cef
chore: added some more test cases
achettyiitr Sep 28, 2023
14f919c
Merge branch 'master' of github.com:rudderlabs/rudder-server into cho…
achettyiitr Sep 29, 2023
bbb82e2
chore: errors.Is during sentinel errors check.
achettyiitr Sep 29, 2023
6ef788e
chore: review comments
achettyiitr Sep 29, 2023
9633b50
Merge branch 'master' of github.com:rudderlabs/rudder-server into cho…
achettyiitr Sep 29, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/tools/matrixchecker/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ var IgnorePackages = []string{
"warehouse/integrations/testhelper",
"warehouse/integrations/testdata",
"warehouse/integrations/config",
"warehouse/integrations/types",
}

func main() {
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ endif
test-warehouse-integration:
$(eval TEST_PATTERN = 'TestIntegration')
$(eval TEST_CMD = SLOW=1 go test)
$(eval TEST_OPTIONS = -v -p 8 -timeout 30m -count 1 -race -run $(TEST_PATTERN) -coverprofile=profile.out -covermode=atomic -coverpkg=./...)
$(eval TEST_OPTIONS = -v -p 8 -timeout 30m -count 1 -run $(TEST_PATTERN) -coverprofile=profile.out -covermode=atomic -coverpkg=./...)
$(TEST_CMD) $(TEST_OPTIONS) $(package) && touch $(TESTFILE) || true

test-warehouse: test-warehouse-integration test-teardown
Expand Down
6 changes: 3 additions & 3 deletions warehouse/client/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -82,10 +82,10 @@ func (cl *Client) bqQuery(statement string) (result warehouseutils.QueryResult,
for {
var row []bigquery.Value
err = it.Next(&row)
if err == iterator.Done {
break
}
if err != nil {
if errors.Is(err, iterator.Done) {
break
}
return
}
var stringRow []string
Expand Down
3 changes: 2 additions & 1 deletion warehouse/identity/identity.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"compress/gzip"
"context"
"database/sql"
"errors"
"fmt"
"io"
"os"
Expand Down Expand Up @@ -243,7 +244,7 @@ func (idr *Identity) addRules(txn *sqlmiddleware.Tx, loadFileNames []string, gzW
var record []string
record, err = eventReader.Read(columnNames)
if err != nil {
if err == io.EOF {
if errors.Is(err, io.EOF) {
break
}
pkgLogger.Errorf("IDR: Error while reading merge rule file %s for loading in staging table locally:%s: %v", loadFileName, mergeRulesStagingTable, err)
Expand Down
Loading