Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Presto Connector Support #314

Merged
merged 94 commits into from
Sep 1, 2023
Merged

Presto Connector Support #314

merged 94 commits into from
Sep 1, 2023

Conversation

clouddea
Copy link
Contributor

@clouddea clouddea commented Sep 1, 2023 •
edited by xuchen-plus
Loading

Presto engine was integrated in this PR!
Close #286

  • can read local and s3 tables
  • can read common supported data types
  • support tables with primary key and without primary key
  • passed mysql to lakesoul CDC test

maosen and others added 30 commits May 29, 2023 18:05
fix flink stream read bug for same bucketid
df6a8e5 
Signed-off-by: maosen <sunmaosen@dmetasoul.com>
fix flink stream read bug for same bucketid
96d1bf8 
Signed-off-by: maosen <sunmaosen@dmetasoul.com>
fix flink stream read bug for same bucketid
689eb25 
Signed-off-by: maosen <sunmaosen@dmetasoul.com>
fix flink stream read bug for same bucketid
0c31ee8 
Signed-off-by: maosen <sunmaosen@dmetasoul.com>
add bucketnum for stream assigner
75ba7df 
Signed-off-by: maosen <sunmaosen@dmetasoul.com>
@moresun
Merge branch 'lakesoul-io:main' into main
ce28535
test StreamReadSuite with variant BUCKET_NUM_AND_PARALLELISM tuple
27469e8 
Signed-off-by: zenghua <huazeng@dmetasoul.com>
Merge remote-tracking branch 'maosen/main' into dev/flink_split_enume…
54413a6 
…rator
add concurrent hash set
0f8732a 
Signed-off-by: maosen <sunmaosen@dmetasoul.com>
Merge remote-tracking branch 'origin/main'
1e20409
@moresun
Merge branch 'lakesoul-io:main' into main
0fb77bd
@moresun
Merge branch 'lakesoul-io:main' into main
f4f4ffc
Merge remote-tracking branch 'origin/main'
3333fe7
Merge remote-tracking branch 'origin/main'
62048b3
Merge remote-tracking branch 'origin/main'
df51b8e
Merge remote-tracking branch 'origin/main'
6015092
Merge remote-tracking branch 'origin/main'
e6a9125
Merge remote-tracking branch 'origin/main'
c2fddf8
add lakesoul connectors for presto
61653a0 
Signed-off-by: maosen <sunmaosen@dmetasoul.com>
add table info
4c0e34d 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
get table info and table columns ok
537b3ad 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
@moresun
Merge branch 'lakesoul-io:main' into presto
6dbdcf7
add select mock data
36f52ec 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
Merge remote-tracking branch 'moresun/presto' into presto
d336310
add table layout handle
cb0d030 
Signed-off-by: maosen <sunmaosen@dmetasoul.com>
get splits ok
832bb1b 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
init native-io reader
0a8d262 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
add basic type conversion
7fbf50d 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
add int filter
ac16fd4 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
add lakesoul split reader
57ada9a 
Signed-off-by: maosen <sunmaosen@dmetasoul.com>
yuanf and others added 22 commits August 30, 2023 14:51
update yml12
dc1b102 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
update yml13
ed1c118 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
update yml14
4294466 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
update yml15
962354c 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
test other type but found some bugs
a34a294 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
add varbinary type
2b72b14 
Signed-off-by: maosen <sunmaosen@dmetasoul.com>
add flink presto talble generator
7e86629 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
add flink presto talble generator2
ff1cb15 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
test ok with tables generated by flink
3399075 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
fix single verification bug
6ec6271 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
drop fink cdc change column
cf4b0d1 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
drop cdc column
a00327c 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
add more tip info
5db08f3 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
add more tip info2
a506c70 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
delete some steps
b24568c 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
change cursor
c2d2d0e 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
filter cdc deleted row
ece0ac9 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
solve null problem
463a9ce 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
clean the code
14f65df 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
merge main
5ace477 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
merge main2
b80643a 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
update
6562f91 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
@xuchen-plus xuchen-plus added enhancement New feature or request presto labels Sep 1, 2023
fix merge bug
a095033 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
fix yaml
73136ae 
Signed-off-by: yuanf <yuanf@dmetasoul.com>
xuchen-plus
xuchen-plus approved these changes Sep 1, 2023
View reviewed changes
Copy link
Contributor

@xuchen-plus xuchen-plus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge this for now. Will open new PRs for issues after more thorough tests.

@xuchen-plus xuchen-plus merged commit 93f7cbf into lakesoul-io:main Sep 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xuchen-plus xuchen-plus xuchen-plus approved these changes

@Ceng23333 Ceng23333 Awaiting requested review from Ceng23333

@moresun moresun Awaiting requested review from moresun

Assignees
No one assigned
Labels
enhancement New feature or request presto
Projects
2.4.0 Release
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Presto Connector
3 participants
@clouddea @xuchen-plus @moresun