-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BEAM-2954] update shade configurations in extension/sql #3848
Conversation
R: @aviemzur |
The shading config looks like it should work. However, we need to make sure we can include the classes from these artifacts as part of our distribution, based on their license. Calcite is Apache License 2.0 so should be fine |
Thanks @aviemzur. Some licensing comments:
Happy to help -- let me know if you need something else. |
@davorbonaci thanks for comments. For #3, could you help with a draft for the NOTICE? |
Based on all my tests, here’re the yes or no cases:
Note: Given the choices, I would prefer 1 > 2, no 3; @aviemzur @davorbonaci any comments? |
As mentioned in the ticket we cannot opt-out of shading in this module as we cannot leak Guava to our users. |
+1 to shading and relocating |
Absolutely open to any workable solution. According to my test(I may miss something, and is asking @aviemzur to rerun my test code), shade+relocate fails as some code hard code java class path(like below).
If none of them are workable/acceptable, the last option would be remove |
update relocation settings in shade, Also add NOTICE and Janino license for distribution. @davorbonaci @aviemzur please help with a review to make sure we follow the license requirement. |
retest this please |
add NOTICE and include Janino license;
CC @reuvenlax @jbonofre |
retest this please |
R: @jbonofre |
hey guys, any updates? |
License looks good to me. @aviemzur @davorbonaci does the shading look ok? |
@jbonofre @davorbonaci @reuvenlax re: licenses, looks like Protobuf was added to the shading as well Suggestion: Instead of adding a |
@aviemzur, do you mean the settings as below?
A |
FYI this PR is one of the last remaining blockers for the 2.2.0 cut, so please let me know if you need any more help! |
IMO the current shade configuration is good to go. I can make a trick as previous comment to avoid file @aviemzur let me know if anything I missed? |
The configuration looks OK, but I agree that if possible you should use shading rather than add the properties file. Taking a peek at the contents of the jar is helpful. I am surprised there are no changes to the |
Taking a look, you probably want a couple more surface tests. The test you have is OK for ensuring that the actual public API surface is still going to work, but it does not ensure that you don't have forbidden residual dependencies. I actually think it needs to be an IT to operate on the shaded jar, right? And then it should test |
To add some more context:
|
let's summary the discussions together:
I intent to include that file. If we want to avoid it, I suppose my previous comment is one option, which would package a non-relocated class file
Is WordCountIT the right example? I think BeamSqlExample can be used. Thoughts?
The existing test lists the 4 public classes, except |
FYI, create https://issues.apache.org/jira/browse/BEAM-2998 for 2) |
So what's left here? Do we still need more tests? |
can our CI job run on shade libraries? I take a quick look at If it supports, can anyone take the task or share some notes on how to add a IT test? |
I don't understand what is going on with the properties file. It seems like you had a way to make it automated with the shade plugin. |
while, let me make it clear about the properties file, now we have two choices:
I prefer to 1), mostly because I don't want to have a non-relocated class file; |
Can we do both? |
Do you mean the two shade options? I don't think so, they're mutually exclusive. |
Yes, I mean to have the identical result in the built jar, but automated. |
IMO it cannot be automated, as shade plugin doesn't check and update the properties file. |
any conclusion here? |
Also how critical is this for 2.2.0, as I would like to get a cut out. |
I would label it critical as I see failures when running queries which have join/aggregation. (Test environment: Kafka+MySQL+FlinkRunner) |
The lack of automation is an issue for maintainability, but doesn't affect users. So we should merge it for the release and file a ticket against 2.3.0 to find a way to automate it. |
We need to explore the world of shading configs and perhaps other resource-manipulating maven plugins so that we aren't making funky hacked versions of properties files for our dependencies. |
Agree with an automate configuration (also how to test shaded libraries), this is a workaround that doesn't impact end users, we could look for other options later. |
ok, I'll go ahead and merge. Please file a JIRA for automation. |
Retest this please. |
Follow this checklist to help us incorporate your contribution quickly and easily:
[BEAM-XXX] Fixes bug in ApproximateQuantiles
, where you replaceBEAM-XXX
with the appropriate JIRA issue.mvn clean verify
to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.