Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dockerized Native Image build failing #1580

Closed
strophy opened this issue Sep 22, 2023 · 4 comments
Closed

Dockerized Native Image build failing #1580

strophy opened this issue Sep 22, 2023 · 4 comments

Comments

@strophy
Copy link

strophy commented Sep 22, 2023

Hello, following up on #1461 where I received excellent help in building multi-arch ScalaPB for inclusion in https://github.com/rvolosatovs/docker-protobuf

Our build recently started failing with:

> [protoc_gen_scala 5/8] RUN gu install native-image:
0.240 Downloading: Release index file from oca.opensource.oracle.com
2.121 Error: Error reading component list: https://oca.opensource.oracle.com/gds/meta-data.json

This is likely due to the removal of GraalVM Updater from their images. Native Image should now be included with GraalVM directly, so I have removed the gu install native-image step, but I'm still unable to get the build to work under x86_64 using the following minimal Dockerfile:

FROM sbtscala/scala-sbt:graalvm-ce-21.3.0-java17_1.8.0_3.2.1 as protoc_gen_scala
RUN mkdir -p /scala-protobuf
ARG PROTOC_GEN_SCALA_VERSION=v0.11.13
RUN curl -sSL https://api.github.com/repos/scalapb/ScalaPB/tarball/${PROTOC_GEN_SCALA_VERSION} | tar xz --strip 1 -C /scala-protobuf
WORKDIR /scala-protobuf
# RUN gu install native-image
# Make sbt use the version of native-image installed by gu instead of downloading a separate version
ARG NATIVE_IMAGE_INSTALLED=true
RUN ./make_reflect_config.sh
RUN sbt protocGenScalaNativeImage/nativeImage
RUN install -D /scala-protobuf/target/protoc-gen-scala /out/usr/bin/protoc-gen-scala

The build eventually errors in the RUN ./make_reflect_config.sh step with:

83.62 [info] done compiling
83.76 [info] compiling 2 Scala sources to /scala-protobuf/scalapbc/target/jvm-2.12/classes ...
85.95 [info] done compiling
86.04 [info] running scalapb.scripts.scalapbc --plugin=protoc-gen-scalaref=/tmp/tmp.L1dX3d6Me1/plugin.sh --scalaref_out=/tmp/tmp.L1dX3d6Me1 e2e/src/main/protobuf/any.proto e2e/src/main/protobuf/bytes_proto2.proto e2e/src/main/protobuf/comments.proto e2e/src/main/protobuf/custom_options.proto e2e/src/main/protobuf/custom_options_p3.proto e2e/src/main/protobuf/custom_options_use.proto e2e/src/main/protobuf/defaults.proto e2e/src/main/protobuf/enum_options.proto e2e/src/main/protobuf/enum_options2.proto e2e/src/main/protobuf/enum_options3.proto e2e/src/main/protobuf/enum_options4.proto e2e/src/main/protobuf/extensions.proto e2e/src/main/protobuf/flat.proto e2e/src/main/protobuf/google/protobuf/fromstruct_two_level_nesting.proto e2e/src/main/protobuf/google/protobuf/unittest.proto e2e/src/main/protobuf/google/protobuf/unittest_import.proto e2e/src/main/protobuf/google/protobuf/unittest_import_public.proto e2e/src/main/protobuf/issue102.proto e2e/src/main/protobuf/issue1226.proto e2e/src/main/protobuf/issue1354.proto e2e/src/main/protobuf/issue1431.proto e2e/src/main/protobuf/issue286.proto e2e/src/main/protobuf/issue286_flat.proto e2e/src/main/protobuf/issue286_flat2.proto e2e/src/main/protobuf/issue320.proto e2e/src/main/protobuf/issue356.proto e2e/src/main/protobuf/issue547.proto e2e/src/main/protobuf/nocode/Issue86.proto e2e/src/main/protobuf/nocode/bytes.proto e2e/src/main/protobuf/nocode/custom_annotation.proto e2e/src/main/protobuf/nocode/custom_names.proto e2e/src/main/protobuf/nocode/custom_object_name.proto e2e/src/main/protobuf/nocode/hellogoodbye.proto e2e/src/main/protobuf/nocode/issue-287.proto e2e/src/main/protobuf/nocode/issue1168.proto e2e/src/main/protobuf/nocode/issue176.proto e2e/src/main/protobuf/nocode/issue212.proto e2e/src/main/protobuf/nocode/issue220_proto2.proto e2e/src/main/protobuf/nocode/issue220_proto3.proto e2e/src/main/protobuf/nocode/issue26.proto e2e/src/main/protobuf/nocode/issue364.proto e2e/src/main/protobuf/nocode/issue472.proto e2e/src/main/protobuf/nocode/issue499.mode.override.v1.proto e2e/src/main/protobuf/nocode/issue581a.proto e2e/src/main/protobuf/nocode/issue581b.proto e2e/src/main/protobuf/nocode/issue695.proto e2e/src/main/protobuf/nocode/issue782.proto e2e/src/main/protobuf/nocode/java_multi.proto e2e/src/main/protobuf/nocode/names.dot.proto e2e/src/main/protobuf/nocode/names3.dot.proto e2e/src/main/protobuf/nocode/no_getters.proto e2e/src/main/protobuf/nocode/no_getters_or_lenses.proto e2e/src/main/protobuf/nocode/no_lenses.proto e2e/src/main/protobuf/nocode/same_name.proto e2e/src/main/protobuf/nocode/sealed_trait.proto e2e/src/main/protobuf/nodef/file.proto e2e/src/main/protobuf/nodef/message.proto e2e/src/main/protobuf/one_of.proto e2e/src/main/protobuf/oneof_named_field.proto e2e/src/main/protobuf/proto3_presence.proto e2e/src/main/protobuf/repeatables.proto e2e/src/main/protobuf/reqs.proto e2e/src/main/protobuf/scalapb_ext.proto e2e/src/main/protobuf/scoped/order.proto e2e/src/main/protobuf/scoped/package.proto e2e/src/main/protobuf/scoped/test.proto e2e/src/main/protobuf/sealed_oneof.proto e2e/src/main/protobuf/sealed_oneof_companion_extends.proto e2e/src/main/protobuf/sealed_oneof_extends.proto e2e/src/main/protobuf/sealed_oneof_extends_nested.proto e2e/src/main/protobuf/sealed_oneof_extends_universal.proto e2e/src/main/protobuf/sealed_oneof_in_oneof.proto e2e/src/main/protobuf/sealed_oneof_in_oneof_nested.proto e2e/src/main/protobuf/sealed_oneof_nested.proto e2e/src/main/protobuf/sealed_oneof_nested_single_file.proto e2e/src/main/protobuf/sealed_oneof_optional.proto e2e/src/main/protobuf/sealed_oneof_single_file.proto e2e/src/main/protobuf/transformations/field_transformations.proto e2e/src/main/protobuf/transformations/option.proto e2e/src/main/protobuf/type_level.proto e2e/src/main/protobuf/unknown_fields.proto e2e/src/main/protobuf/well_known.proto --experimental_allow_proto3_optional -I third_party -I /scala-protobuf/protobuf -I e2e/src/main/protobuf -I e2e-withjava/src/main/protobuf
86.11 issue286.proto:3:1: warning: Import scalapb/scalapb.proto is unused.
86.12 scoped/test.proto:5:1: warning: Import scalapb/scalapb.proto is unused.
86.15 --scalaref_out: protoc-gen-scalaref: Plugin failed with status code 1.

I've tried setting NATIVE_IMAGE_INSTALLED to true, false and omitting it entirely. Could you help me get the protoc plugin build working again? Thanks!

@strophy strophy changed the title Dockerized native-image build failing Dockerized Native Image build failing Sep 23, 2023
@thesamet
Copy link
Contributor

I could make it work with a more recent version of graalvm. From whatever reason gu install native-image failed on the version you have used.

FROM sbtscala/scala-sbt:graalvm-ce-22.3.3-b1-java17_1.9.6_3.3.1 as protoc_gen_scala
RUN mkdir -p /scala-protobuf
ARG PROTOC_GEN_SCALA_VERSION=v0.11.13
RUN curl -sSL https://api.github.com/repos/scalapb/ScalaPB/tarball/${PROTOC_GEN_SCALA_VERSION} | tar xz --strip 1 -C /scala-protobuf
WORKDIR /scala-protobuf
RUN gu install native-image
# Make sbt use the version of native-image installed by gu instead of downloading a separate version
ARG NATIVE_IMAGE_INSTALLED=true
RUN ./make_reflect_config.sh
RUN sbt protocGenScalaNativeImage/nativeImage
RUN install -D /scala-protobuf/target/protoc-gen-scala /out/usr/bin/protoc-gen-scala

When running the Dockerfile you supplied, the reason for the failure (native-image not being found) is hidden. To expose it, you can create a file like this

tmp_reflect_config.sh

#!/usr/bin/env bash
set -e
TMPDIR=$(mktemp -d)
PROTOFILES=$(find e2e/src/main/protobuf -name "*.proto" -print)

sbt "set ThisBuild / version := \"SNAPSHOT\"" protoc-gen-scala-unix/assembly
printf "#!/usr/bin/env bash\nset -e\n" > $TMPDIR/plugin.sh
echo export JAVA_OPTS=-agentlib:native-image-agent=config-output-dir=protoc-gen-scala-native-image/native-image-config >> $TMPDIR/plugin.sh
echo $PWD/.protoc-gen-scala-unix/target/scala-2.12/protoc-gen-scala-unix-assembly-SNAPSHOT.jar >> $TMPDIR/plugin.sh

chmod +x $TMPDIR/plugin.sh

mkdir -p protoc-gen-scala-native-image/native-image-config

$TMPDIR/plugin.sh

And try this Dockerfile

FROM sbtscala/scala-sbt:graalvm-ce-22.3.3-b1-java17_1.9.6_3.3.1 as protoc_gen_scala
RUN mkdir -p /scala-protobuf
ARG PROTOC_GEN_SCALA_VERSION=v0.11.13
RUN curl -sSL https://api.github.com/repos/scalapb/ScalaPB/tarball/${PROTOC_GEN_SCALA_VERSION} | tar xz --strip 1 -C /scala-protobuf
WORKDIR /scala-protobuf
RUN gu install native-image
# Make sbt use the version of native-image installed by gu instead of downloading a separate version
ARG NATIVE_IMAGE_INSTALLED=true
ADD tmp_reflect_config.sh /scala-protobuf/
RUN ./tmp_reflect_config.sh

The script above tries to run `plugin.sh directly, so the error about missing native-image becomes visible.

Hope this helps!

@strophy
Copy link
Author

strophy commented Sep 29, 2023

Thanks for the help @thesamet! Before we go with this update, it looks like gu won't exist for much longer though, which might be why that oracle.com URL in my first post stopped working? Also, am I correct in understanding the b1 in the new version indicates it is a beta release?

@thesamet
Copy link
Contributor

thesamet commented Sep 29, 2023

Hi @strophy , my intent was to show that the only thing that was needed to make your Dockerfile work was to update to a more recent version of the docker image you have been using. I am not sure what b1 stands for, but you may be able to find a different stable version that works, though it's beyond the support I can provide as a maintainer of ScalaPB - the answers for this should come from graalvm team or https://github.com/sbt/docker-sbt projects as this is not ScalaPB specific.

@strophy
Copy link
Author

strophy commented Oct 4, 2023

I ran into some more problems running the build for ARM64 in QEMU, but updating to the latest tonistiigi/binfmt to update QEMU resolved it. Build is working again, thanks!

@strophy strophy closed this as completed Oct 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants