[NSE-62]Fixing issue0062 for package arrow dependencies in jar with refresh2 #172
Conversation
|
|
|
It looks like the test has been passed, check if the formatting check can be ignored and any other concern under review, then I can merge the PR. |
There was a problem hiding this comment.
looks like ARROW_ROOT is not correctly passed to cpp building
`mvn clean package -pl native-sql-engine/core -am -DskipTests -Dcpp_tests=ON -Dbuild_arrow=OFF -DSTATIC_ARROW=OFF -DBUILD_PROTOBUF=ON -DARROW_ROOT=/home/sparkuser/miniconda3/envs/pyarrow-dev/
....
[INFO] --- exec-maven-plugin:1.6.0:exec (Build cpp) @ spark-columnar-core ---
CMAKE Arguments:
TESTS=ON
BUILD_ARROW=OFF
STATIC_ARROW=OFF
BUILD_PROTOBUF=ON
ARROW_ROOT=/usr/local
`
| @@ -13,6 +13,11 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON) | |||
|
|
|||
| set(CMAKE_BUILD_TYPE "Release") | |||
|
|
|||
| set(ARROW_ROOT "/usr/local") | |||
There was a problem hiding this comment.
| set(ARROW_ROOT "/usr/local") | |
| set(ARROW_ROOT "/usr/local" CACHE PATH "Arrow root dir") |
There was a problem hiding this comment.
This is an upper/lower case issue, please try to use "lower cases in mvn" and "UPPER CASES in cpp".
it is a good finding, let me try if we can use the same cases in both mvn and cpp.
There was a problem hiding this comment.
Done to fix the issue for no default value in parameters.
The case sensitive usage for maven and cpp will be kept since coding style concern:
mvn -> lowercase, cpp -> UPPERCASE
|
|
|
|
|
@weiting-chen the new building process looks good for me. one question: as native sql engine depends arrow data source, we'll need to apply the same building process for |
Per discussion, lets merge the PR first and backport it to v1.1 branch since this PR also have to pass Gold Deck testing. |
* [NSE-130] support decimal round and abs (#166) * support decimal round and abs * remove duplicate cast in multiply/divide * [NSE-161] adding format check (#165) * adding format check Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * formating code Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * adding google format Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * reformat with new style Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * lower clang version to 10 Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * adding script to format code Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * [NSE-170]improve sort shuffle code (#171) * improve sort shuffle code Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * fix format Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * pass by ref in builder Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * fix string/decimal builder Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * [NSE-62]Fixing issue0062 for package arrow dependencies in jar with refresh2 (#172) * Add arrow build and dependency support * Add compress.sh default value * Fix bug for parameter's default value * Add CACHE PATH * fix copy bitmap in InplaceSort (#174) * [NSE-153] fix window results (#175) * fix window sort memory Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * remove unused code Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * fix windown w/o avg Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * fix format Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * fix decimal sort Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * Fix issue 179 for arrow include directory (#181) * Fix issue0191 for .so file copy to tmp. (#192) * Following NSE-153, optimize fallback conditions for columnar window (#189) * Fix q14a/b segfault (#193) Signed-off-by: Chendi Xue <chendi.xue@intel.com> * [NSE-194]Turn on several Arrow parameters (#195) * Turn on several Arrow parameters * Change SIMD Level Setting * Hashmap build opt for semi/anti/exists join (#197) Signed-off-by: Chendi Xue <chendi.xue@intel.com> * [NSE-198] support the month() and dayofmonth() functions with DateType (#199) * [NSE-206] fix doc link, update limitations (#205) * fix doc link Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * update arrow data source Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * adding limitations Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * mention limits Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * [NSE-170] using unsafe appender (#203) This patch adds an unsafe appender which will reserve space before builder array. The boolean builder is not touched as only malloc small sized memory The string builder are not touched as it's diffcult to pre-allocate the space * using unsafe appender Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * fix format Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * update arrow branch Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> Co-authored-by: Rui Mo <rui.mo@intel.com> Co-authored-by: Wei-Ting Chen <weiting.chen@intel.com> Co-authored-by: Hongze Zhang <hongze.zhang@intel.com> Co-authored-by: Chendi.Xue <chendi.xue@intel.com> Co-authored-by: JiaKe <ke.a.jia@intel.com>
What changes were proposed in this pull request?
Close #62
Close #66
The previous one will be drop, please use the new PR with refresh2
How was this patch tested?
The PR has been tested in my developing server.