Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improve] mongodb connector v2 add source query capability #3697

Merged
merged 14 commits into from
Dec 26, 2022

Conversation

MonsterChenzhuo
Copy link
Contributor

@MonsterChenzhuo MonsterChenzhuo commented Dec 12, 2022

Purpose of this pull request

close #3567,#3607
I was going to finish refactoring the mongodb linker in this pr, but considering the number of changes to refactoring, I split it up. #3698

Check list

Copy link
Member

@hailin0 hailin0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hailin0
Copy link
Member

hailin0 commented Dec 12, 2022

Please check ci error

@hailin0
Copy link
Member

hailin0 commented Dec 13, 2022

please fix ci error

@MonsterChenzhuo
Copy link
Contributor Author

please fix ci error

Thanks for your review,I have made the changes.

@@ -66,6 +71,7 @@ mongodb {
uri = "mongodb://username:password@127.0.0.1:27017/mypost?retryWrites=true&writeConcern=majority"
database = "mydatabase"
collection = "mycollection"
matchQuery = "{"id":3}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure the json in hocon value can be parse normally, maybe you should change a way, like use object

matchQuery = {
     "id"=3
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This place is a string, except that this string has to conform to a certain format in order to be transformed into a BsonDocument

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core is this method:
.find(Optional.ofNullable(params.getMatchQuery()).isPresent() ? BsonDocument.parse(params.getMatchQuery()) : new BsonDocument())
I have tested in my development env example mode, it is ok

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core is this method: .find(Optional.ofNullable(params.getMatchQuery()).isPresent() ? BsonDocument.parse(params.getMatchQuery()) : new BsonDocument()) I have tested in my development env example mode, it is ok

Because I consider one thing is: the query statement is best to conform to the native syntax of mongodb, this will be very friendly to the developer, he does not need to learn anything extra, directly paste the history of mongodb query statement can be used

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok for me, please update e2e test case. Thanks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your review,I have made the changes.

@MonsterChenzhuo
Copy link
Contributor Author

cc @TyrantLucifer PTAL, thanks.

Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@EricJoy2048 EricJoy2048 merged commit 8a7fe6f into apache:dev Dec 26, 2022
@TyrantLucifer
Copy link
Member

@MonsterChenzhuo Hi, #3607 Looks like you didn't finish. Do you want to keep doing this?

lhyundeadsoul pushed a commit to lhyundeadsoul/incubator-seatunnel that referenced this pull request Jan 3, 2023
* [Improve] mongodb connector v2 add source query capability
lhyundeadsoul pushed a commit to lhyundeadsoul/incubator-seatunnel that referenced this pull request Jan 3, 2023
* [Improve] mongodb connector v2 add source query capability
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][Connector V2] Add conditions for mongodb source incremental pull
5 participants