Add hive 4.0 image #218
Add hive 4.0 image #218
Conversation
mayankvadariya
requested review from
ebyhr,
hashhar,
anusudarsan and
findinpath
mayankvadariya
force-pushed
the
mayank/hive4-using-apache-hive-image
branch
11 times, most recently
from
9e19ed7 to
314e24d
Compare
| @@ -35,6 +35,9 @@ jobs: | |||
| - image: hive3.1-hive | |||
| platforms: linux/amd64,linux/arm64 | |||
| test: hive3.1-hive | |||
| - image: hive4.0-hive | |||
| platforms: linux/amd64 | |||
There was a problem hiding this comment.
Haven't aded linux/arm64 platform as test image had been complaining about different arch
The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested
There was a problem hiding this comment.
on ci this is expected, what happens when you build the image locally? if it builds then you should allow all platforms here - CI will use emulation to build the arm version of the image but then devs who have arm machines will get arm images instead of having to emulate amd64 images.
on ci the node running the build is not an arm runner hence the warning.
There was a problem hiding this comment.
ya I realized about the runner spec. However, its just not about warning but image test is failing with arm. Please advise if there is a way to run only amd64 test and not arm.
There was a problem hiding this comment.
feel free to merge without arm image.
The fix isn't to skip tests on arm but rather to find what is failing. The logs here are almost useless, I'll take a look at this and submit follow up PR.
testing/hive4.0-hive/Dockerfile
Outdated
|
|
||
| FROM apache/hive:4.0.0 | ||
|
|
||
| # TODO replace with aws sdk v2 |
There was a problem hiding this comment.
What are we missing to go ahead now with aws sdk v2 ?
Can you pls link an issue on Hive that needs to be done to be able to use aws sdk v2?
There was a problem hiding this comment.
I tried it but /tmp/root/hive.log doesn't provide enough information on what is causing sdk v2 to not work.
|
Add appropriate description explaining why do we need to customize the vanilla apache hive 4 docker image. |
mayankvadariya
force-pushed
the
mayank/hive4-using-apache-hive-image
branch
from
314e24d to
b90a41d
Compare
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| FROM apache/hive:4.0.0 |
There was a problem hiding this comment.
does the base image include some arch-specific code? if not we can make this multi-arch by inlining the base image.
I can take a look at this as follow-up.
There was a problem hiding this comment.
hmmm, the hive image is actually multi-arch already
|
@hashhar thanks for offering to take care of arm build. Would you please merge this PR? |
|
@mayankvadariya Could you fix Polaris failure? Otherwise, we can't release a new version. |
|
|
Running a release workflow: https://github.com/trinodb/docker-images/actions/runs/12289019120 |
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| FROM apache/hive:4.0.0 |
There was a problem hiding this comment.
Nothing speaks against it but I am not sure on what major/minor releases do we test.
If we want to use the latest one then, I can change it to 4.0.1.
Build custom hive4 image with access to S3 by using plain apache hive4 image. Plain hive image offers an advantage of not having to tweak multiple configuration. However, currently created image doesn't include yarn services which will be required to run queries concurrently. We may need to build a new image to accommodate yarn services as we run different tests using Hive4.