[NSE-304] Upgrade to Arrow 4.0.0 #307
Conversation
|
Native SQL engine will be based on https://github.com/oap-project/arrow/tree/arrow-4.0.0-oap after applying this. |
zhztheplayer
force-pushed
the
NSE-304
branch
2 times, most recently
from
657a8b4
to
1b436f3
Compare
|
@github-actions ram-usage-test |
zhztheplayer
force-pushed
the
NSE-304
branch
2 times, most recently
from
1c3d262
to
7d10fea
Compare
| <artifactId>log4j</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| <artifactId>hadoop-client</artifactId> |
There was a problem hiding this comment.
not sure if this will break v2 @rui-mo
There was a problem hiding this comment.
If executions are required they were added to the parent pom https://github.com/oap-project/native-sql-engine/pull/307/files/7a39548a3dedc3c8f89127f4cac712a4c20864c6#diff-9c5fb3d1b7e3b0f54bc5c4182965c4fe1f9023d449017cece3005d3f90e8e4d8R145-R170
There was a problem hiding this comment.
And is there other reason v2 is breaked?
There was a problem hiding this comment.
the issue are mostly due to spark pre-build binary with hadoop 3.2 - which packages some specific version of depdencies other than those in hadoop-common-3.2.0. will do a quick test on my env to verify
https://github.com/apache/spark/blob/v3.1.1/pom.xml#L172
https://github.com/apache/spark/blob/v3.1.1/pom.xml#L121
There was a problem hiding this comment.
@zhouyuan it is ok on my env
|
|
|
|
|
|
|
thanks @zhztheplayer ! |
* [NSE-304] Upgrade to Arrow 4.0.0 * use .400 * .400 * fix * fix
* [NSE-304] Upgrade to Arrow 4.0.0 (#307) * [NSE-304] Upgrade to Arrow 4.0.0 * use .400 * .400 * fix * fix * [NSE-318]check schema before reuse exchange (#328) * check schema before reuse exchange Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * remove debug log Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * bump version to 1.1.1 Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * update arrow branch to use arrow-4.0.0-oap-1.1.1 Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * update GHA arrow branch Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * [NSE-329] fix out partitioning in BHJ and SHJ (#335) * fix out partitioning for SHJ and BHJ * refine * [NSE-325] fix incremental compile issue with 4.5.x scala-maven-plugin (#332) the new release of scala-maven-plugin(4.5.x) seems buggy on incremental compile, using 4.4.0 instead Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> Co-authored-by: Hongze Zhang <hongze.zhang@intel.com> Co-authored-by: Rui Mo <rui.mo@intel.com>
No description provided.