feat: Arrow migration #87

yevgenypats · 2023-03-05T18:56:09Z

but ready for initial review and discussion. I can do a short walkthrough for anyone up for review.

Apache arrow fork is here (We use it until this is merged): https://github.com/cloudquery/arrow/tree/feat_extension_builder.

Some more notes:

Right now we are just migrating the json writer/reader to use apache arrow so we can roll format by format and see if there are any real world issues before we roll this to everywhere instead of our own type system.

internal/cqarrow/arrow.go

erezrokah

Looks good (didn't test). So the TDLR for this PR is that we use Apache arrow type system instead of ours (extending types they don't support)?

A walkthrough will be great, and also seeing them accept some of our extended types as contributions

erezrokah · 2023-03-06T09:39:13Z

internal/cqarrow/arrow.go

+	case schema.TypeIntArray:
+		typ = arrow.ListOf(arrow.PrimitiveTypes.Int64)
+	case schema.TypeTimestamp:
+		typ = arrow.FixedWidthTypes.Timestamp_us


Maybe add a comment that us means micros seconds (and not US timezone)

arrow type is named badly, should've been arrow.FixedWidthTypes.TimestampMicros

internal/cqarrow/inet.go

hermanschaaf · 2023-03-06T10:54:34Z

internal/cqarrow/mac.go

+	return b.Unmarshal(dec)
+}
+
+// UUIDArray is a simple array which is a FixedSizeBinary(16)


Same here: It looks like all the comments in this file that refer to FixedSizeBinary or UUID need to be updated (they were copied from UUID)

internal/cqarrow/json.go

internal/cqarrow/inet_test.go

internal/cqarrow/json_test.go

internal/cqarrow/mac.go

This should unblock cloudquery/filetypes#87 Add supports for timestamp to decode arrow string and for byte to decode base64 (backward compat is saved)

Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>

Co-authored-by: Kemal <223029+disq@users.noreply.github.com>

Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>

hermanschaaf · 2023-03-06T14:04:54Z

internal/cqarrow/arrow_test.go

+	"github.com/google/go-cmp/cmp"
+)
+
+func DiffSchema(sc, o *arrow.Schema) (string, bool) {


I think this is some dead code that can be removed?

hermanschaaf · 2023-03-06T14:05:09Z

internal/cqarrow/arrow_test.go

+	if diff := cmp.Diff(arrowSchema.String(), expecetdSchema.String()); diff != "" {
+		t.Errorf(diff)
+	}
+	// if diff, _ := DiffSchema(arrowSchema, expecetdSchema); diff != "" {


(The only use of DiffSchema is here and it's commented out)

hermanschaaf · 2023-03-06T14:07:13Z

internal/cqarrow/json.go

+		b.AppendNull()
+		return
+	}
+	bytes, err := json.Marshal(v)


I'm wondering if we need to use json.Encoder here as well, like we had to in plugin-sdk (https://github.com/cloudquery/plugin-sdk/pull/714/files)?

hermanschaaf

LGTM - I tested it against the old json filetypes plugin for a couple of sources and the output was the byte-for-byte the same 👍 We may want to consider using json.Encoder but I think it's not a blocker

🤖 I have created a release *beep* *boop* --- ## [1.5.0](v1.4.2...v1.5.0) (2023-03-06) ### Features * Arrow migration ([#87](#87)) ([135f022](135f022)) ### Bug Fixes * **deps:** Update golang.org/x/xerrors digest to 04be3eb ([#78](#78)) ([5d47135](5d47135)) * **deps:** Update module github.com/apache/thrift to v0.18.0 ([#83](#83)) ([bce51fa](bce51fa)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.38.1 ([#72](#72)) ([0eadcfd](0eadcfd)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.38.2 ([#74](#74)) ([90bdbba](90bdbba)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.39.0 ([#75](#75)) ([b65db2e](b65db2e)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.39.1 ([#76](#76)) ([c03025e](c03025e)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.40.0 ([#77](#77)) ([7ec1df0](7ec1df0)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.41.0 ([#86](#86)) ([5b89f33](5b89f33)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.42.0 ([#88](#88)) ([99da3b2](99da3b2)) * **deps:** Update module github.com/klauspost/compress to v1.16.0 ([#84](#84)) ([a9c9da3](a9c9da3)) * **deps:** Update module github.com/pierrec/lz4/v4 to v4.1.17 ([#79](#79)) ([fbc42d4](fbc42d4)) * **deps:** Update module github.com/stretchr/testify to v1.8.2 ([#80](#80)) ([b080141](b080141)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

yevgenypats requested review from erezrokah and hermanschaaf March 5, 2023 18:56

github-actions bot added the feat label Mar 5, 2023

yevgenypats mentioned this pull request Mar 6, 2023

feat: Add arrow support for timestamp and bytea cloudquery/plugin-sdk#724

Merged

yevgenypats marked this pull request as ready for review March 6, 2023 07:43

github-actions bot added feat and removed feat labels Mar 6, 2023

disq reviewed Mar 6, 2023

View reviewed changes

internal/cqarrow/arrow.go Outdated Show resolved Hide resolved

erezrokah approved these changes Mar 6, 2023

View reviewed changes

hermanschaaf reviewed Mar 6, 2023

View reviewed changes

internal/cqarrow/inet.go Outdated Show resolved Hide resolved

hermanschaaf reviewed Mar 6, 2023

View reviewed changes

internal/cqarrow/json.go Outdated Show resolved Hide resolved

hermanschaaf reviewed Mar 6, 2023

View reviewed changes

internal/cqarrow/json.go Outdated Show resolved Hide resolved

hermanschaaf reviewed Mar 6, 2023

View reviewed changes

internal/cqarrow/inet_test.go Outdated Show resolved Hide resolved

hermanschaaf reviewed Mar 6, 2023

View reviewed changes

internal/cqarrow/json_test.go Outdated Show resolved Hide resolved

hermanschaaf reviewed Mar 6, 2023

View reviewed changes

internal/cqarrow/mac.go Outdated Show resolved Hide resolved

yevgenypats and others added 9 commits March 6, 2023 15:39

feat: Arrow migration

e535ebb

fmt

a8b8a83

Update internal/cqarrow/mac.go

5962c81

Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>

Update internal/cqarrow/arrow.go

9d304f6

Co-authored-by: Kemal <223029+disq@users.noreply.github.com>

Update internal/cqarrow/json_test.go

eccdfe3

Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>

Update internal/cqarrow/json.go

5278eb6

Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>

review

a8dd373

mod tidy

28b383a

tidy

2ce1b28

yevgenypats force-pushed the feat/arrow_migration branch from cded2e3 to 2ce1b28 Compare March 6, 2023 13:40

hermanschaaf reviewed Mar 6, 2023

View reviewed changes

hermanschaaf approved these changes Mar 6, 2023

View reviewed changes

yevgenypats added 3 commits March 6, 2023 20:14

fix null

f74cf5b

fmt

118a515

fmt

96b2efa

yevgenypats added the automerge Add to automerge PRs once requirements are met label Mar 6, 2023

kodiakhq bot merged commit 135f022 into main Mar 6, 2023

kodiakhq bot deleted the feat/arrow_migration branch March 6, 2023 18:47

cq-bot mentioned this pull request Mar 6, 2023

chore(main): Release v1.5.0 #73

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Arrow migration #87

feat: Arrow migration #87

yevgenypats commented Mar 5, 2023 •

edited

Loading

erezrokah left a comment

erezrokah Mar 6, 2023

disq Mar 6, 2023

hermanschaaf Mar 6, 2023

hermanschaaf Mar 6, 2023

hermanschaaf Mar 6, 2023

hermanschaaf Mar 6, 2023

hermanschaaf left a comment

feat: Arrow migration #87

feat: Arrow migration #87

Conversation

yevgenypats commented Mar 5, 2023 • edited Loading

erezrokah left a comment

Choose a reason for hiding this comment

erezrokah Mar 6, 2023

Choose a reason for hiding this comment

disq Mar 6, 2023

Choose a reason for hiding this comment

hermanschaaf Mar 6, 2023

Choose a reason for hiding this comment

hermanschaaf Mar 6, 2023

Choose a reason for hiding this comment

hermanschaaf Mar 6, 2023

Choose a reason for hiding this comment

hermanschaaf Mar 6, 2023

Choose a reason for hiding this comment

hermanschaaf left a comment

Choose a reason for hiding this comment

yevgenypats commented Mar 5, 2023 •

edited

Loading