-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🐛 Destination S3: Fix Parquet LZO compression #26284
Conversation
This reverts commit 1241569.
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
/test connector=connectors/destination-s3
Build PassedTest summary info:
|
Affected Connector ReportNOTE
|
Connector | Version | Changelog | Publish |
---|
- See "Actionable Items" below for how to resolve warnings and errors.
❌ Destinations (10)
Connector | Version | Changelog | Publish |
---|---|---|---|
destination-bigquery |
1.4.3 |
✅ | ✅ |
destination-bigquery-denormalized |
1.4.1 |
✅ | ✅ |
destination-databricks |
1.0.2 |
✅ | ✅ |
destination-gcs |
0.3.0 |
✅ | ✅ |
destination-r2 |
0.1.0 |
✅ | ✅ |
destination-redshift |
0.4.7 |
✅ | ✅ |
destination-s3 |
0.4.1 |
✅ | ❌ (diff seed version) |
destination-s3-glue |
0.1.7 |
✅ | ✅ |
destination-snowflake |
1.0.4 |
✅ | ✅ |
destination-starburst-galaxy |
0.0.1 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Other Modules (0)
Actionable Items
(click to expand)
Category | Status | Actionable Item |
---|---|---|
Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
Publish | ⚠ not in seed |
The connector is not in the cloud or oss registry, so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that you have added a metadata.yaml file and the expected registries are enabled. |
hm, test is passing but I'm still seeing errors when I try running a sync in oss :/ |
nope, I'm just dumb and built the image from the wrong branch 🤦 @airbytehq/destinations ptal :D |
/test connector=connectors/destination-gcs
Build PassedTest summary info:
|
* Only verifies that it runs successfully, which is sufficient to catch any issues with installing the lzo libraries. | ||
*/ | ||
@Test | ||
public void testLzoCompression() throws Exception { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
/test connector=connectors/destination-s3
Build PassedTest summary info:
|
merging without publish because we have publish on merge! |
but there's (presumably?) a diff between the dockerfile and how dagger is configured :( lzo sync still failing on cloud https://cloud.airbyte.com/workspaces/46b4844a-94a3-496f-af53-2050bf4141af/connections/79b4d19f-2077-461a-9d4b-cfaa6c9538dc/job-history#2292715::0 |
* Revert "Move hadoop-lzo to test dependency (#21085)" This reverts commit 1241569. * add basic test * Automated Change * version bumps, changelog * Automated Change * unused import * Ran ./gradlew :spotlessJavaApply to trigger GitHub build * regenerate registry --------- Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: ryankfu <ryan.fu@airbyte.io>
* Revert "Move hadoop-lzo to test dependency (airbytehq#21085)" This reverts commit 1241569. * add basic test * Automated Change * version bumps, changelog * Automated Change * unused import * Ran ./gradlew :spotlessJavaApply to trigger GitHub build * regenerate registry --------- Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: ryankfu <ryan.fu@airbyte.io>
revert #21085 + add a test case to actually run an lzo compression sync. seen in https://github.com/airbytehq/oncall/issues/2028
the test fails without the revert + passes after the revert.
it's still broken in destination-gcs, but I don't want to install java 8 + lzo libraries there... issue to do something about this https://github.com/airbytehq/airbyte-internal-issues/issues/1774