-
Notifications
You must be signed in to change notification settings - Fork 314
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix bigquery connector #1111
fix bigquery connector #1111
Conversation
46cfdea
to
0cb1821
Compare
Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>
0cb1821
to
89cb653
Compare
* | ||
* @return | ||
*/ | ||
private Optional<String> getBigQueryTableName(BigQueryRelation relation) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the main change of the PR: call both methods tableName
and getTableName
on BigQueryRelation
to retrieve name regardless of version.
@@ -40,7 +40,6 @@ repositories { | |||
archivesBaseName='openlineage-spark-spark3' | |||
|
|||
ext { | |||
bigqueryVersion = '0.21.1' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bigquery dependencies are not needed for spark2
, spark3
and spark32
subprojects. Instead of upgrading, I remove them,
Signed-off-by: Pawel Leszczynski leszczynski.pawel@gmail.com
Problem
Spark integration fails with spark-bigquery-connector >=0.25.0
Closes: #1105
Solution
If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports
S3
andGCS
filesystem operations, tested with AWS EMR).Checklist
CHANGELOG.md
with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)