[flink] add 1.12 ; remove 1.10 #9249
Conversation
|
This inserted + exec $(drop_privs_cmd) bash -c "${args[@]}"$ # current
$ docker run -it flink native-k8s 'some-command --with --its arguments && other stuff'
$ # missing quotes would mean the arguments to "some-command" would be interpreted as arguments to "bash"
$ docker run -it flink native-k8s some-command --with --its arguments
$ # vs if you drop the bash -c
$ docker run -it flink native-k8s some-command --with --its arguments
$ docker run -it flink native-k8s bash -c 'some-command --with --its arguments && other stuff' |
|
Thank you for your review! I agree that the |
|
@tianon @rmetzger Yes, the |
Seems a little strange for the Flink client to generate a command reliant on interpreted env vars, shouldn't it just generate some args for |
|
@yosifkit Thanks for your comments and suggestions. Just as you said, we could move the Another alternative is we could rename the BTW, could we defer this improvement in the following release version(Flink 1.12.1)? Both the two solutions require us to update the Flink source codes, but we have released the Flink 1.12.0 and only one step left(publishing the image to dockerhub). |
|
thanks @wangyang0918 for submitting the |
|
Hi @rogaha @tianon @yosifkit, do you think the new |
Deprecate command `native-k8` and add preparation for pass-through mode. This is to address the comments in the following docker-library/official-images PR. docker-library/official-images#9249
Deprecate command `native-k8` and add preparation for pass-through mode. This is to address the comments in the following docker-library/official-images PR. docker-library/official-images#9249 This closes #50
|
(Commenting here instead of on #9345 because this is where the previous discussion is and I'd really appreciate if we didn't splinter it.) I've given this more thought, and figured out what's got me so confused/conflicted. I see that there exist a lot of (useful looking) scripts in https://github.com/apache/flink/tree/2eaee8344bdcab961799127c7e72d3f4abe75726/flink-dist/src/main/flink-bin that I can only assume exist in this directory because they are part of Flink releases, and are how users are expected to use Flink directly (outside containers), correct? My main concern (and confusion) here is that we have a separate custom script that's specific to the Dockerization, and it keeps growing, and it feels like it's really duplicating the efforts that exist there. If this had been implemented as part of those scripts and been part of the Flink release artifacts, we wouldn't have even noticed their addition, their usage would still fall under the "consistent interface" guidelines we are looking for, and we wouldn't still be sitting here with a PR that both adds and deprecates a feature in the same new release, which feels really off. |
|
@tianon Your assessment is somewhat correct. All deployments of Flink makes use of the the flink-bin scripts in some form; for YARN or non-containerized setups users are directly using the contained scripts to create a cluster. For docker deployments, users work against the interface provided by the docker entrypoint, which ultimately just wraps the flink-bin scripts which just add some additional docker-specific logic. Finally, the native kubernetes integration has users interact with the flink-bin scripts to issue the creation of a cluster, where Flink requests containers and generates a command to be executed within the container through the docker-entrypoint.sh. In other words the Flink docker image is used in 2 ways; directly by users and in some essentially-internal fashion where we're just passing a command through. Re deprecating a new feature: The annoying reality is that the 1.12.0 code expects the (now deprecated) command to exists, and we would like 1.12.0 clients to continue working with 1.12.1 images. (The used images are not hard-wired to a specific release) If I understand you correctly, then it would be fine for us to move everything in the docker-entrypoint.sh into a new scripts within distribution, and just call that scripts from docker-entrypoint.sh with all passed arguments. We had the same idea on our side, and would probably be fine with doing that starting from 1.13. |
Yes, I'd love to see this (anything that decreases the delta between a "standard" Flink install and a "Docker" Flink install is a big win IMO) -- the part about this particular instance that feels so off is that this feature appears to have been worked on for months with the intention of adding this additional code to the bits of the image we review (which is one of the core purposes of this program), but we weren't looped into the conversation until it was already too late to discuss it, which is why we're having this conversation. 😬 😄 |
|
@tianon How could we move the Flink 1.12.* official images publishing forward? Is marking the |
This is TBD. There may be other ways to solve this, but let's not have this discussion here. |
|
|
||
| Tags: 1.10.2-scala_2.11, 1.10-scala_2.11 | ||
| Tags: 1.12.0-scala_2.12-java8, 1.12-scala_2.12-java8, scala_2.12-java8, 1.12.0-scala_2.12, 1.12-scala_2.12, scala_2.12, 1.12.0-java8, 1.12-java8, java8, 1.12.0, 1.12, latest |
There was a problem hiding this comment.
What is the reason to allow java8 images?
There was a problem hiding this comment.
Java 8 is the lowest version required and the maven target version for Flink, and is AFAIK used by the majority of Flink developers and users.
Diff for 4bb82a9:diff --git a/_bashbrew-cat b/_bashbrew-cat
index 2817ce3..c02207a 100644
--- a/_bashbrew-cat
+++ b/_bashbrew-cat
@@ -1,14 +1,6 @@
Maintainers: Patrick Lucas <me@patricklucas.com> (@patricklucas), Ismaël Mejía <iemejia@gmail.com> (@iemejia)
GitRepo: https://github.com/apache/flink-docker.git
-Tags: 1.10.2-scala_2.11, 1.10-scala_2.11
-GitCommit: 58a29fca7c6ff05999fad4371638d16335f7e93e
-Directory: 1.10/scala_2.11-debian
-
-Tags: 1.10.2-scala_2.12, 1.10-scala_2.12, 1.10.2, 1.10
-GitCommit: 58a29fca7c6ff05999fad4371638d16335f7e93e
-Directory: 1.10/scala_2.12-debian
-
Tags: 1.11.3-scala_2.11-java8, 1.11-scala_2.11-java8, 1.11.3-scala_2.11, 1.11-scala_2.11
GitCommit: 7035f03679b11352f2fdecd9f6a9bb0ec8bc2022
Directory: 1.11/scala_2.11-java8-debian
@@ -24,3 +16,19 @@ Directory: 1.11/scala_2.12-java8-debian
Tags: 1.11.3-scala_2.12-java11, 1.11-scala_2.12-java11, 1.11.3-java11, 1.11-java11
GitCommit: 7035f03679b11352f2fdecd9f6a9bb0ec8bc2022
Directory: 1.11/scala_2.12-java11-debian
+
+Tags: 1.12.0-scala_2.11-java8, 1.12-scala_2.11-java8, scala_2.11-java8, 1.12.0-scala_2.11, 1.12-scala_2.11, scala_2.11
+GitCommit: 4d7cb3068d670bef274975c9412d5e40317fba3f
+Directory: 1.12/scala_2.11-java8-debian
+
+Tags: 1.12.0-scala_2.11-java11, 1.12-scala_2.11-java11, scala_2.11-java11
+GitCommit: 4d7cb3068d670bef274975c9412d5e40317fba3f
+Directory: 1.12/scala_2.11-java11-debian
+
+Tags: 1.12.0-scala_2.12-java8, 1.12-scala_2.12-java8, scala_2.12-java8, 1.12.0-scala_2.12, 1.12-scala_2.12, scala_2.12, 1.12.0-java8, 1.12-java8, java8, 1.12.0, 1.12, latest
+GitCommit: 4d7cb3068d670bef274975c9412d5e40317fba3f
+Directory: 1.12/scala_2.12-java8-debian
+
+Tags: 1.12.0-scala_2.12-java11, 1.12-scala_2.12-java11, scala_2.12-java11, 1.12.0-java11, 1.12-java11, java11
+GitCommit: 4d7cb3068d670bef274975c9412d5e40317fba3f
+Directory: 1.12/scala_2.12-java11-debian
diff --git a/_bashbrew-list b/_bashbrew-list
index f4a0459..9fe01b8 100644
--- a/_bashbrew-list
+++ b/_bashbrew-list
@@ -1,9 +1,3 @@
-flink:1.10
-flink:1.10-scala_2.11
-flink:1.10-scala_2.12
-flink:1.10.2
-flink:1.10.2-scala_2.11
-flink:1.10.2-scala_2.12
flink:1.11
flink:1.11-java8
flink:1.11-java11
@@ -22,3 +16,30 @@ flink:1.11.3-scala_2.11-java11
flink:1.11.3-scala_2.12
flink:1.11.3-scala_2.12-java8
flink:1.11.3-scala_2.12-java11
+flink:1.12
+flink:1.12-java8
+flink:1.12-java11
+flink:1.12-scala_2.11
+flink:1.12-scala_2.11-java8
+flink:1.12-scala_2.11-java11
+flink:1.12-scala_2.12
+flink:1.12-scala_2.12-java8
+flink:1.12-scala_2.12-java11
+flink:1.12.0
+flink:1.12.0-java8
+flink:1.12.0-java11
+flink:1.12.0-scala_2.11
+flink:1.12.0-scala_2.11-java8
+flink:1.12.0-scala_2.11-java11
+flink:1.12.0-scala_2.12
+flink:1.12.0-scala_2.12-java8
+flink:1.12.0-scala_2.12-java11
+flink:java8
+flink:java11
+flink:latest
+flink:scala_2.11
+flink:scala_2.11-java8
+flink:scala_2.11-java11
+flink:scala_2.12
+flink:scala_2.12-java8
+flink:scala_2.12-java11
diff --git a/flink_1.11-java11/Dockerfile b/flink_java11/Dockerfile
similarity index 90%
copy from flink_1.11-java11/Dockerfile
copy to flink_java11/Dockerfile
index 8b27b61..1a6e48a 100644
--- a/flink_1.11-java11/Dockerfile
+++ b/flink_java11/Dockerfile
@@ -21,7 +21,7 @@ FROM openjdk:11-jre
# Install dependencies
RUN set -ex; \
apt-get update; \
- apt-get -y install libsnappy1v5 gettext-base; \
+ apt-get -y install libsnappy1v5 gettext-base libjemalloc-dev; \
rm -rf /var/lib/apt/lists/*
# Grab gosu for easy step-down from root
@@ -44,9 +44,9 @@ RUN set -ex; \
gosu nobody true
# Configure Flink version
-ENV FLINK_TGZ_URL=https://www.apache.org/dyn/closer.cgi?action=download&filename=flink/flink-1.11.3/flink-1.11.3-bin-scala_2.12.tgz \
- FLINK_ASC_URL=https://www.apache.org/dist/flink/flink-1.11.3/flink-1.11.3-bin-scala_2.12.tgz.asc \
- GPG_KEY=F8E419AA0B60C28879E876859DFF40967ABFC5A4 \
+ENV FLINK_TGZ_URL=https://archive.apache.org/dist/flink/flink-1.12.0/flink-1.12.0-bin-scala_2.12.tgz \
+ FLINK_ASC_URL=https://archive.apache.org/dist/flink/flink-1.12.0/flink-1.12.0-bin-scala_2.12.tgz.asc \
+ GPG_KEY=43CE299BC305AFF8B912AA95183F6944D9839159 \
CHECK_GPG=true
# Prepare environment
diff --git a/flink_1.10/docker-entrypoint.sh b/flink_java11/docker-entrypoint.sh
similarity index 65%
copy from flink_1.10/docker-entrypoint.sh
copy to flink_java11/docker-entrypoint.sh
index 6084f06..275488e 100755
--- a/flink_1.10/docker-entrypoint.sh
+++ b/flink_java11/docker-entrypoint.sh
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/usr/bin/env bash
###############################################################################
# Licensed to the Apache Software Foundation (ASF) under one
@@ -19,6 +19,9 @@
###############################################################################
COMMAND_STANDALONE="standalone-job"
+# Deprecated, should be remove in Flink release 1.13
+COMMAND_NATIVE_KUBERNETES="native-k8s"
+COMMAND_HISTORY_SERVER="history-server"
# If unspecified, the hostname of the container is taken as the JobManager address
JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
@@ -73,17 +76,13 @@ set_config_option() {
fi
}
-set_common_options() {
+prepare_configuration() {
set_config_option jobmanager.rpc.address ${JOB_MANAGER_RPC_ADDRESS}
set_config_option blob.server.port 6124
set_config_option query.server.port 6125
-}
-
-prepare_job_manager_start() {
- echo "Starting Job Manager"
- copy_plugins_if_required
- set_common_options
+ TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-1}
+ set_config_option taskmanager.numberOfTaskSlots ${TASK_MANAGER_NUMBER_OF_TASK_SLOTS}
if [ -n "${FLINK_PROPERTIES}" ]; then
echo "${FLINK_PROPERTIES}" >> "${CONF_FILE}"
@@ -91,35 +90,65 @@ prepare_job_manager_start() {
envsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp" "${CONF_FILE}"
}
+maybe_enable_jemalloc() {
+ if [ "${DISABLE_JEMALLOC:-false}" == "false" ]; then
+ export LD_PRELOAD=$LD_PRELOAD:/usr/lib/x86_64-linux-gnu/libjemalloc.so
+ fi
+}
+
+maybe_enable_jemalloc
+
+copy_plugins_if_required
+
+prepare_configuration
+
+args=("$@")
if [ "$1" = "help" ]; then
- echo "Usage: $(basename "$0") (jobmanager|${COMMAND_STANDALONE}|taskmanager|help)"
+ printf "Usage: $(basename "$0") (jobmanager|${COMMAND_STANDALONE}|taskmanager|${COMMAND_HISTORY_SERVER})\n"
+ printf " Or $(basename "$0") help\n\n"
+ printf "By default, Flink image adopts jemalloc as default memory allocator. This behavior can be disabled by setting the 'DISABLE_JEMALLOC' environment variable to 'true'.\n"
exit 0
elif [ "$1" = "jobmanager" ]; then
- shift 1
- prepare_job_manager_start
+ args=("${args[@]:1}")
+
+ echo "Starting Job Manager"
- exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "${args[@]}"
elif [ "$1" = ${COMMAND_STANDALONE} ]; then
- shift 1
- prepare_job_manager_start
+ args=("${args[@]:1}")
- exec $(drop_privs_cmd) "$FLINK_HOME/bin/standalone-job.sh" start-foreground "$@"
+ echo "Starting Job Manager"
+
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/standalone-job.sh" start-foreground "${args[@]}"
+elif [ "$1" = ${COMMAND_HISTORY_SERVER} ]; then
+ args=("${args[@]:1}")
+
+ echo "Starting History Server"
+
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/historyserver.sh" start-foreground "${args[@]}"
elif [ "$1" = "taskmanager" ]; then
- shift 1
+ args=("${args[@]:1}")
+
echo "Starting Task Manager"
- copy_plugins_if_required
- TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground "${args[@]}"
+elif [ "$1" = "$COMMAND_NATIVE_KUBERNETES" ]; then
+ args=("${args[@]:1}")
- set_common_options
- set_config_option taskmanager.numberOfTaskSlots ${TASK_MANAGER_NUMBER_OF_TASK_SLOTS}
+ export _FLINK_HOME_DETERMINED=true
+ . $FLINK_HOME/bin/config.sh
+ export FLINK_CLASSPATH="`constructFlinkClassPath`:$INTERNAL_HADOOP_CLASSPATHS"
+ # Start commands for jobmanager and taskmanager are generated by Flink internally.
+ echo "Start command: ${args[@]}"
+ exec $(drop_privs_cmd) bash -c "${args[@]}"
+fi
- if [ -n "${FLINK_PROPERTIES}" ]; then
- echo "${FLINK_PROPERTIES}" >> "${CONF_FILE}"
- fi
- envsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp" "${CONF_FILE}"
+args=("${args[@]}")
- exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground "$@"
-fi
+# Set the Flink related environments
+export _FLINK_HOME_DETERMINED=true
+. $FLINK_HOME/bin/config.sh
+export FLINK_CLASSPATH="`constructFlinkClassPath`:$INTERNAL_HADOOP_CLASSPATHS"
-exec "$@"
+# Running command in pass-through mode
+exec $(drop_privs_cmd) "${args[@]}"
diff --git a/flink_1.10/Dockerfile b/flink_latest/Dockerfile
similarity index 90%
rename from flink_1.10/Dockerfile
rename to flink_latest/Dockerfile
index 29afb1a..48164fd 100644
--- a/flink_1.10/Dockerfile
+++ b/flink_latest/Dockerfile
@@ -21,7 +21,7 @@ FROM openjdk:8-jre
# Install dependencies
RUN set -ex; \
apt-get update; \
- apt-get -y install libsnappy1v5 gettext-base; \
+ apt-get -y install libsnappy1v5 gettext-base libjemalloc-dev; \
rm -rf /var/lib/apt/lists/*
# Grab gosu for easy step-down from root
@@ -44,9 +44,9 @@ RUN set -ex; \
gosu nobody true
# Configure Flink version
-ENV FLINK_TGZ_URL=https://www.apache.org/dyn/closer.cgi?action=download&filename=flink/flink-1.10.2/flink-1.10.2-bin-scala_2.12.tgz \
- FLINK_ASC_URL=https://www.apache.org/dist/flink/flink-1.10.2/flink-1.10.2-bin-scala_2.12.tgz.asc \
- GPG_KEY=C63E230EFFF519A5BBF2C9AE6767487CD505859C \
+ENV FLINK_TGZ_URL=https://archive.apache.org/dist/flink/flink-1.12.0/flink-1.12.0-bin-scala_2.12.tgz \
+ FLINK_ASC_URL=https://archive.apache.org/dist/flink/flink-1.12.0/flink-1.12.0-bin-scala_2.12.tgz.asc \
+ GPG_KEY=43CE299BC305AFF8B912AA95183F6944D9839159 \
CHECK_GPG=true
# Prepare environment
diff --git a/flink_1.10-scala_2.11/docker-entrypoint.sh b/flink_latest/docker-entrypoint.sh
similarity index 65%
rename from flink_1.10-scala_2.11/docker-entrypoint.sh
rename to flink_latest/docker-entrypoint.sh
index 6084f06..275488e 100755
--- a/flink_1.10-scala_2.11/docker-entrypoint.sh
+++ b/flink_latest/docker-entrypoint.sh
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/usr/bin/env bash
###############################################################################
# Licensed to the Apache Software Foundation (ASF) under one
@@ -19,6 +19,9 @@
###############################################################################
COMMAND_STANDALONE="standalone-job"
+# Deprecated, should be remove in Flink release 1.13
+COMMAND_NATIVE_KUBERNETES="native-k8s"
+COMMAND_HISTORY_SERVER="history-server"
# If unspecified, the hostname of the container is taken as the JobManager address
JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
@@ -73,17 +76,13 @@ set_config_option() {
fi
}
-set_common_options() {
+prepare_configuration() {
set_config_option jobmanager.rpc.address ${JOB_MANAGER_RPC_ADDRESS}
set_config_option blob.server.port 6124
set_config_option query.server.port 6125
-}
-
-prepare_job_manager_start() {
- echo "Starting Job Manager"
- copy_plugins_if_required
- set_common_options
+ TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-1}
+ set_config_option taskmanager.numberOfTaskSlots ${TASK_MANAGER_NUMBER_OF_TASK_SLOTS}
if [ -n "${FLINK_PROPERTIES}" ]; then
echo "${FLINK_PROPERTIES}" >> "${CONF_FILE}"
@@ -91,35 +90,65 @@ prepare_job_manager_start() {
envsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp" "${CONF_FILE}"
}
+maybe_enable_jemalloc() {
+ if [ "${DISABLE_JEMALLOC:-false}" == "false" ]; then
+ export LD_PRELOAD=$LD_PRELOAD:/usr/lib/x86_64-linux-gnu/libjemalloc.so
+ fi
+}
+
+maybe_enable_jemalloc
+
+copy_plugins_if_required
+
+prepare_configuration
+
+args=("$@")
if [ "$1" = "help" ]; then
- echo "Usage: $(basename "$0") (jobmanager|${COMMAND_STANDALONE}|taskmanager|help)"
+ printf "Usage: $(basename "$0") (jobmanager|${COMMAND_STANDALONE}|taskmanager|${COMMAND_HISTORY_SERVER})\n"
+ printf " Or $(basename "$0") help\n\n"
+ printf "By default, Flink image adopts jemalloc as default memory allocator. This behavior can be disabled by setting the 'DISABLE_JEMALLOC' environment variable to 'true'.\n"
exit 0
elif [ "$1" = "jobmanager" ]; then
- shift 1
- prepare_job_manager_start
+ args=("${args[@]:1}")
+
+ echo "Starting Job Manager"
- exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "${args[@]}"
elif [ "$1" = ${COMMAND_STANDALONE} ]; then
- shift 1
- prepare_job_manager_start
+ args=("${args[@]:1}")
- exec $(drop_privs_cmd) "$FLINK_HOME/bin/standalone-job.sh" start-foreground "$@"
+ echo "Starting Job Manager"
+
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/standalone-job.sh" start-foreground "${args[@]}"
+elif [ "$1" = ${COMMAND_HISTORY_SERVER} ]; then
+ args=("${args[@]:1}")
+
+ echo "Starting History Server"
+
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/historyserver.sh" start-foreground "${args[@]}"
elif [ "$1" = "taskmanager" ]; then
- shift 1
+ args=("${args[@]:1}")
+
echo "Starting Task Manager"
- copy_plugins_if_required
- TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground "${args[@]}"
+elif [ "$1" = "$COMMAND_NATIVE_KUBERNETES" ]; then
+ args=("${args[@]:1}")
- set_common_options
- set_config_option taskmanager.numberOfTaskSlots ${TASK_MANAGER_NUMBER_OF_TASK_SLOTS}
+ export _FLINK_HOME_DETERMINED=true
+ . $FLINK_HOME/bin/config.sh
+ export FLINK_CLASSPATH="`constructFlinkClassPath`:$INTERNAL_HADOOP_CLASSPATHS"
+ # Start commands for jobmanager and taskmanager are generated by Flink internally.
+ echo "Start command: ${args[@]}"
+ exec $(drop_privs_cmd) bash -c "${args[@]}"
+fi
- if [ -n "${FLINK_PROPERTIES}" ]; then
- echo "${FLINK_PROPERTIES}" >> "${CONF_FILE}"
- fi
- envsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp" "${CONF_FILE}"
+args=("${args[@]}")
- exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground "$@"
-fi
+# Set the Flink related environments
+export _FLINK_HOME_DETERMINED=true
+. $FLINK_HOME/bin/config.sh
+export FLINK_CLASSPATH="`constructFlinkClassPath`:$INTERNAL_HADOOP_CLASSPATHS"
-exec "$@"
+# Running command in pass-through mode
+exec $(drop_privs_cmd) "${args[@]}"
diff --git a/flink_1.11-java11/Dockerfile b/flink_scala_2.11-java11/Dockerfile
similarity index 90%
copy from flink_1.11-java11/Dockerfile
copy to flink_scala_2.11-java11/Dockerfile
index 8b27b61..70310d6 100644
--- a/flink_1.11-java11/Dockerfile
+++ b/flink_scala_2.11-java11/Dockerfile
@@ -21,7 +21,7 @@ FROM openjdk:11-jre
# Install dependencies
RUN set -ex; \
apt-get update; \
- apt-get -y install libsnappy1v5 gettext-base; \
+ apt-get -y install libsnappy1v5 gettext-base libjemalloc-dev; \
rm -rf /var/lib/apt/lists/*
# Grab gosu for easy step-down from root
@@ -44,9 +44,9 @@ RUN set -ex; \
gosu nobody true
# Configure Flink version
-ENV FLINK_TGZ_URL=https://www.apache.org/dyn/closer.cgi?action=download&filename=flink/flink-1.11.3/flink-1.11.3-bin-scala_2.12.tgz \
- FLINK_ASC_URL=https://www.apache.org/dist/flink/flink-1.11.3/flink-1.11.3-bin-scala_2.12.tgz.asc \
- GPG_KEY=F8E419AA0B60C28879E876859DFF40967ABFC5A4 \
+ENV FLINK_TGZ_URL=https://archive.apache.org/dist/flink/flink-1.12.0/flink-1.12.0-bin-scala_2.11.tgz \
+ FLINK_ASC_URL=https://archive.apache.org/dist/flink/flink-1.12.0/flink-1.12.0-bin-scala_2.11.tgz.asc \
+ GPG_KEY=43CE299BC305AFF8B912AA95183F6944D9839159 \
CHECK_GPG=true
# Prepare environment
diff --git a/flink_1.10/docker-entrypoint.sh b/flink_scala_2.11-java11/docker-entrypoint.sh
similarity index 65%
copy from flink_1.10/docker-entrypoint.sh
copy to flink_scala_2.11-java11/docker-entrypoint.sh
index 6084f06..275488e 100755
--- a/flink_1.10/docker-entrypoint.sh
+++ b/flink_scala_2.11-java11/docker-entrypoint.sh
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/usr/bin/env bash
###############################################################################
# Licensed to the Apache Software Foundation (ASF) under one
@@ -19,6 +19,9 @@
###############################################################################
COMMAND_STANDALONE="standalone-job"
+# Deprecated, should be remove in Flink release 1.13
+COMMAND_NATIVE_KUBERNETES="native-k8s"
+COMMAND_HISTORY_SERVER="history-server"
# If unspecified, the hostname of the container is taken as the JobManager address
JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
@@ -73,17 +76,13 @@ set_config_option() {
fi
}
-set_common_options() {
+prepare_configuration() {
set_config_option jobmanager.rpc.address ${JOB_MANAGER_RPC_ADDRESS}
set_config_option blob.server.port 6124
set_config_option query.server.port 6125
-}
-
-prepare_job_manager_start() {
- echo "Starting Job Manager"
- copy_plugins_if_required
- set_common_options
+ TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-1}
+ set_config_option taskmanager.numberOfTaskSlots ${TASK_MANAGER_NUMBER_OF_TASK_SLOTS}
if [ -n "${FLINK_PROPERTIES}" ]; then
echo "${FLINK_PROPERTIES}" >> "${CONF_FILE}"
@@ -91,35 +90,65 @@ prepare_job_manager_start() {
envsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp" "${CONF_FILE}"
}
+maybe_enable_jemalloc() {
+ if [ "${DISABLE_JEMALLOC:-false}" == "false" ]; then
+ export LD_PRELOAD=$LD_PRELOAD:/usr/lib/x86_64-linux-gnu/libjemalloc.so
+ fi
+}
+
+maybe_enable_jemalloc
+
+copy_plugins_if_required
+
+prepare_configuration
+
+args=("$@")
if [ "$1" = "help" ]; then
- echo "Usage: $(basename "$0") (jobmanager|${COMMAND_STANDALONE}|taskmanager|help)"
+ printf "Usage: $(basename "$0") (jobmanager|${COMMAND_STANDALONE}|taskmanager|${COMMAND_HISTORY_SERVER})\n"
+ printf " Or $(basename "$0") help\n\n"
+ printf "By default, Flink image adopts jemalloc as default memory allocator. This behavior can be disabled by setting the 'DISABLE_JEMALLOC' environment variable to 'true'.\n"
exit 0
elif [ "$1" = "jobmanager" ]; then
- shift 1
- prepare_job_manager_start
+ args=("${args[@]:1}")
+
+ echo "Starting Job Manager"
- exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "${args[@]}"
elif [ "$1" = ${COMMAND_STANDALONE} ]; then
- shift 1
- prepare_job_manager_start
+ args=("${args[@]:1}")
- exec $(drop_privs_cmd) "$FLINK_HOME/bin/standalone-job.sh" start-foreground "$@"
+ echo "Starting Job Manager"
+
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/standalone-job.sh" start-foreground "${args[@]}"
+elif [ "$1" = ${COMMAND_HISTORY_SERVER} ]; then
+ args=("${args[@]:1}")
+
+ echo "Starting History Server"
+
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/historyserver.sh" start-foreground "${args[@]}"
elif [ "$1" = "taskmanager" ]; then
- shift 1
+ args=("${args[@]:1}")
+
echo "Starting Task Manager"
- copy_plugins_if_required
- TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground "${args[@]}"
+elif [ "$1" = "$COMMAND_NATIVE_KUBERNETES" ]; then
+ args=("${args[@]:1}")
- set_common_options
- set_config_option taskmanager.numberOfTaskSlots ${TASK_MANAGER_NUMBER_OF_TASK_SLOTS}
+ export _FLINK_HOME_DETERMINED=true
+ . $FLINK_HOME/bin/config.sh
+ export FLINK_CLASSPATH="`constructFlinkClassPath`:$INTERNAL_HADOOP_CLASSPATHS"
+ # Start commands for jobmanager and taskmanager are generated by Flink internally.
+ echo "Start command: ${args[@]}"
+ exec $(drop_privs_cmd) bash -c "${args[@]}"
+fi
- if [ -n "${FLINK_PROPERTIES}" ]; then
- echo "${FLINK_PROPERTIES}" >> "${CONF_FILE}"
- fi
- envsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp" "${CONF_FILE}"
+args=("${args[@]}")
- exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground "$@"
-fi
+# Set the Flink related environments
+export _FLINK_HOME_DETERMINED=true
+. $FLINK_HOME/bin/config.sh
+export FLINK_CLASSPATH="`constructFlinkClassPath`:$INTERNAL_HADOOP_CLASSPATHS"
-exec "$@"
+# Running command in pass-through mode
+exec $(drop_privs_cmd) "${args[@]}"
diff --git a/flink_1.10-scala_2.11/Dockerfile b/flink_scala_2.11/Dockerfile
similarity index 90%
rename from flink_1.10-scala_2.11/Dockerfile
rename to flink_scala_2.11/Dockerfile
index 7c07b15..8c1291a 100644
--- a/flink_1.10-scala_2.11/Dockerfile
+++ b/flink_scala_2.11/Dockerfile
@@ -21,7 +21,7 @@ FROM openjdk:8-jre
# Install dependencies
RUN set -ex; \
apt-get update; \
- apt-get -y install libsnappy1v5 gettext-base; \
+ apt-get -y install libsnappy1v5 gettext-base libjemalloc-dev; \
rm -rf /var/lib/apt/lists/*
# Grab gosu for easy step-down from root
@@ -44,9 +44,9 @@ RUN set -ex; \
gosu nobody true
# Configure Flink version
-ENV FLINK_TGZ_URL=https://www.apache.org/dyn/closer.cgi?action=download&filename=flink/flink-1.10.2/flink-1.10.2-bin-scala_2.11.tgz \
- FLINK_ASC_URL=https://www.apache.org/dist/flink/flink-1.10.2/flink-1.10.2-bin-scala_2.11.tgz.asc \
- GPG_KEY=C63E230EFFF519A5BBF2C9AE6767487CD505859C \
+ENV FLINK_TGZ_URL=https://archive.apache.org/dist/flink/flink-1.12.0/flink-1.12.0-bin-scala_2.11.tgz \
+ FLINK_ASC_URL=https://archive.apache.org/dist/flink/flink-1.12.0/flink-1.12.0-bin-scala_2.11.tgz.asc \
+ GPG_KEY=43CE299BC305AFF8B912AA95183F6944D9839159 \
CHECK_GPG=true
# Prepare environment
diff --git a/flink_1.10/docker-entrypoint.sh b/flink_scala_2.11/docker-entrypoint.sh
similarity index 65%
rename from flink_1.10/docker-entrypoint.sh
rename to flink_scala_2.11/docker-entrypoint.sh
index 6084f06..275488e 100755
--- a/flink_1.10/docker-entrypoint.sh
+++ b/flink_scala_2.11/docker-entrypoint.sh
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/usr/bin/env bash
###############################################################################
# Licensed to the Apache Software Foundation (ASF) under one
@@ -19,6 +19,9 @@
###############################################################################
COMMAND_STANDALONE="standalone-job"
+# Deprecated, should be remove in Flink release 1.13
+COMMAND_NATIVE_KUBERNETES="native-k8s"
+COMMAND_HISTORY_SERVER="history-server"
# If unspecified, the hostname of the container is taken as the JobManager address
JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
@@ -73,17 +76,13 @@ set_config_option() {
fi
}
-set_common_options() {
+prepare_configuration() {
set_config_option jobmanager.rpc.address ${JOB_MANAGER_RPC_ADDRESS}
set_config_option blob.server.port 6124
set_config_option query.server.port 6125
-}
-
-prepare_job_manager_start() {
- echo "Starting Job Manager"
- copy_plugins_if_required
- set_common_options
+ TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-1}
+ set_config_option taskmanager.numberOfTaskSlots ${TASK_MANAGER_NUMBER_OF_TASK_SLOTS}
if [ -n "${FLINK_PROPERTIES}" ]; then
echo "${FLINK_PROPERTIES}" >> "${CONF_FILE}"
@@ -91,35 +90,65 @@ prepare_job_manager_start() {
envsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp" "${CONF_FILE}"
}
+maybe_enable_jemalloc() {
+ if [ "${DISABLE_JEMALLOC:-false}" == "false" ]; then
+ export LD_PRELOAD=$LD_PRELOAD:/usr/lib/x86_64-linux-gnu/libjemalloc.so
+ fi
+}
+
+maybe_enable_jemalloc
+
+copy_plugins_if_required
+
+prepare_configuration
+
+args=("$@")
if [ "$1" = "help" ]; then
- echo "Usage: $(basename "$0") (jobmanager|${COMMAND_STANDALONE}|taskmanager|help)"
+ printf "Usage: $(basename "$0") (jobmanager|${COMMAND_STANDALONE}|taskmanager|${COMMAND_HISTORY_SERVER})\n"
+ printf " Or $(basename "$0") help\n\n"
+ printf "By default, Flink image adopts jemalloc as default memory allocator. This behavior can be disabled by setting the 'DISABLE_JEMALLOC' environment variable to 'true'.\n"
exit 0
elif [ "$1" = "jobmanager" ]; then
- shift 1
- prepare_job_manager_start
+ args=("${args[@]:1}")
+
+ echo "Starting Job Manager"
- exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "${args[@]}"
elif [ "$1" = ${COMMAND_STANDALONE} ]; then
- shift 1
- prepare_job_manager_start
+ args=("${args[@]:1}")
- exec $(drop_privs_cmd) "$FLINK_HOME/bin/standalone-job.sh" start-foreground "$@"
+ echo "Starting Job Manager"
+
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/standalone-job.sh" start-foreground "${args[@]}"
+elif [ "$1" = ${COMMAND_HISTORY_SERVER} ]; then
+ args=("${args[@]:1}")
+
+ echo "Starting History Server"
+
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/historyserver.sh" start-foreground "${args[@]}"
elif [ "$1" = "taskmanager" ]; then
- shift 1
+ args=("${args[@]:1}")
+
echo "Starting Task Manager"
- copy_plugins_if_required
- TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
+ exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground "${args[@]}"
+elif [ "$1" = "$COMMAND_NATIVE_KUBERNETES" ]; then
+ args=("${args[@]:1}")
- set_common_options
- set_config_option taskmanager.numberOfTaskSlots ${TASK_MANAGER_NUMBER_OF_TASK_SLOTS}
+ export _FLINK_HOME_DETERMINED=true
+ . $FLINK_HOME/bin/config.sh
+ export FLINK_CLASSPATH="`constructFlinkClassPath`:$INTERNAL_HADOOP_CLASSPATHS"
+ # Start commands for jobmanager and taskmanager are generated by Flink internally.
+ echo "Start command: ${args[@]}"
+ exec $(drop_privs_cmd) bash -c "${args[@]}"
+fi
- if [ -n "${FLINK_PROPERTIES}" ]; then
- echo "${FLINK_PROPERTIES}" >> "${CONF_FILE}"
- fi
- envsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp" "${CONF_FILE}"
+args=("${args[@]}")
- exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground "$@"
-fi
+# Set the Flink related environments
+export _FLINK_HOME_DETERMINED=true
+. $FLINK_HOME/bin/config.sh
+export FLINK_CLASSPATH="`constructFlinkClassPath`:$INTERNAL_HADOOP_CLASSPATHS"
-exec "$@"
+# Running command in pass-through mode
+exec $(drop_privs_cmd) "${args[@]}"
|
|
(Reviewing again given this was updated and I'm assuming it's ready? 😅) Per apache/flink#14630 (comment), this jemalloc usage looks much better; thank you. Looking forward to improved/extracted k8s integration for 1.13 (https://issues.apache.org/jira/browse/FLINK-21128). 👍 (Happy to merge now following confirmation that this is indeed "ready" from your perspective) |
|
@tianon Yes, from our side it is ready to go 👍 |
This might be a replacement for #9243