This repository has been archived by the owner on Sep 18, 2023. It is now read-only.
[NSE-800] Pack the classes into one single jar #1002
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PHILO-HE
force-pushed
the
dependency-refine
branch
from
4fe922e
to
24d0a40
Compare
zhouyuan
reviewed
Jun 30, 2022
docs/User-Guide.md
Outdated
| @@ -62,11 +64,20 @@ And for spark 3.2.x, the jar whose `<spark-version>` is `spark321` should be use | |||
|
|
|||
| 4. `spark-sql-columnar-shims-<spark-version>-<version>-SNAPSHOT.jar` | |||
|
|
|||
| Please notice the files are fat jars shipped with our custom Arrow library and pre-compiled from our server(using GCC 9.3.0 and LLVM 7.0.1), which means you will require to pre-install GCC 9.3.0 and LLVM 7.0.1 in your system for normal usage. | |||
| #### After 1.4.0 (inclusively) | |||
There was a problem hiding this comment.
"starts from" sounds better
There was a problem hiding this comment.
Corrected. Thanks!
This reverts commit f78d5dd.
|
@zhouyuan, this patch will NOT change the behavior of building separate module. Actually, the build is almost as same as before except that an assembly step is added after all required modules are built. |
zhouyuan
approved these changes
Jun 30, 2022
zhouyuan
pushed a commit
to zhouyuan/native-sql-engine
that referenced
this pull request
Jun 30, 2022
* Port from PR#825 * Generate two jars * Use provided scope for scala and exclude meta-inf for jackson * Add classifier * Append spark version and pack to one jar * Update user guide * Revert "Add classifier" This reverts commit f78d5dd. * Refine the doc
zhouyuan
pushed a commit
to zhouyuan/native-sql-engine
that referenced
this pull request
Jun 30, 2022
* Port from PR#825 * Generate two jars * Use provided scope for scala and exclude meta-inf for jackson * Add classifier * Append spark version and pack to one jar * Update user guide * Revert "Add classifier" This reverts commit f78d5dd. * Refine the doc Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
zhouyuan
added a commit
that referenced
this pull request
Jun 30, 2022
* Port from PR#825 * Generate two jars * Use provided scope for scala and exclude meta-inf for jackson * Add classifier * Append spark version and pack to one jar * Update user guide * Revert "Add classifier" This reverts commit f78d5dd. * Refine the doc Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> Co-authored-by: PHILO-HE <feilong.he@intel.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We found that even though core source code can be shared for different spark versions, the jar built for core source code can NOT be shared. So the jar is only applicable to the spark version specified in the building. To distinguish them, the dist jar's name is appended with spark version.
The final jar is put into
gazelle-dist/target/. Currently, the jar can be eithergazelle-plugin-1.4.0-spark-3.1.1.jarorgazelle-plugin-1.4.0-spark-3.2.1.jar, depending on the spark version specified.