-
Notifications
You must be signed in to change notification settings - Fork 31
Pilot executes cassandra
directly, bypassing the container image entry point
#347
base: master
Are you sure you want to change the base?
Changes from all commits
69a59e5
20deca2
350c038
330437b
2426e2c
a25029d
74bb244
641310c
015c813
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,3 +6,4 @@ | |
.generate_exes | ||
.get_deps | ||
bin/ | ||
**/.test/ |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -25,6 +25,8 @@ All the C* nodes (pods) in a ``nodepool`` have the same configuration and the fo | |
|
||
.. include:: configure-scheduler.rst | ||
|
||
.. _availability-zones-cassandra: | ||
|
||
Cassandra Across Multiple Availability Zones | ||
-------------------------------------------- | ||
|
||
|
@@ -240,6 +242,37 @@ Navigator will add C* nodes, one at a time, until the desired number of nodes is | |
|
||
You can look at ``CassandraCluster.Status.NodePools[<nodepoolname>].ReadyReplicas`` to see the current number of healthy C* nodes in each ``nodepool``. | ||
|
||
Pilots and Cassandra Docker Images | ||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
||
By default, Navigator will use the `Cassandra Docker images from DockerHub <https://hub.docker.com/_/cassandra/>`_. | ||
It will use an image with a tag matching the supplied ``CassandraCluster.Spec.Version`` field. | ||
If you prefer to use your own container image you should configure the ``CassandraCluster.Spec.Image`` fields. | ||
|
||
Navigator installs a ``navigator-pilot-cassandra`` executable into each Pod at the path ``/pilot``. | ||
This ``pilot`` process connects to the API server to: | ||
get extra configuration settings, | ||
report the status of this C* node, and to | ||
perform leader election of a single pilot in the cluster. | ||
|
||
The ``pilot`` overrides the following keys in the default ``/etc/cassandra/cassandra.yaml`` file: | ||
|
||
* ``cluster_name``: This will be set to match the name of the corresponding ``CassandraCluster`` resource in the API server. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just a thought: is this safe to do, given two clusters may have the same name (in two different namespaces) |
||
* ``listen_address`` / ``listen_interface`` / ``broadcast_address`` / ``rpc_address`` / ``broadcast_rpc_address``: These keys will be set to ``null``. | ||
This ensures that Cassandra process listens and communicates using the IP address currently associated with the fully qualified domain name of the Pod. | ||
This is important if the Pod is moved to another node and is assigned a different IP address. | ||
Removing these settings from the Configuration file ensures that Cassandra uses the most recent IP address that Kubernetes has assigned to the Pod and that other C* nodes in the cluster are notified of the change of IP address. | ||
* ``seed_provider``: This is set to ``io.jetstack.cassandra.KubernetesSeedProvider`` which allows Cassandra to look up the seed IP addresses from a Kubernetes service. | ||
The ``seed_provider.*.seeds`` sub key will be emptied. | ||
This is to avoid the risk of nodes mistakenly joining the cluster as seeds if the seed provider service is temporarily unavailable. | ||
|
||
The ``pilot`` also overwrites ``cassandra-rackdc.properties`` with values derived from the ``CassandraCluster.Spec.Nodepools`` (see :ref:`availability-zones-cassandra`). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
||
Finally the ``pilot`` executes ``/usr/sbin/cassandra`` directly. | ||
|
||
.. note:: | ||
The default entry point (e.g. `/docker-entrypoint.sh <https://github.com/docker-library/cassandra/blob/master/3.11/docker-entrypoint.sh>`_ is ignored. | ||
|
||
Supported Versions | ||
------------------ | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -31,3 +31,5 @@ spec: | |
pilotImage: | ||
repository: "quay.io/jetstack/navigator-pilot-cassandra" | ||
tag: "v0.1.0" | ||
securityContext: | ||
runAsUser: 999 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As per discussion earlier - what happens if this field is not specified on a Cassandra resource? If we are not careful, this PR could become a breaking change as older versions of Navigator handled setting the uid automatically (i.e. via the entrypoint script) before forking to the cassandra subprocess. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
#!/bin/bash | ||
# | ||
# Download the binaries needed by the sigs.k8s.io/testing_frameworks/integration package. | ||
# etcd, kube-apiserver, kubectl | ||
# XXX: There is already a script to do this: | ||
# * sigs.k8s.io/testing_frameworks/integration/scripts/download-binaries.sh | ||
# But it currently downloads kube-apiserver v1.10.0-alpha.1 which doesn't support ``CustomResourceSubresources``. | ||
# See https://github.com/kubernetes-sigs/testing_frameworks/issues/44 | ||
|
||
set -o errexit | ||
set -o nounset | ||
set -o pipefail | ||
set -o xtrace | ||
|
||
# Close stdin | ||
exec 0<&- | ||
|
||
ROOT_DIR="$(git rev-parse --show-toplevel)" | ||
|
||
ETCD_VERSION=v3.2.10 | ||
ETCD_URL="https://storage.googleapis.com/etcd" | ||
|
||
KUBE_VERSION_URL="https://storage.googleapis.com/kubernetes-release/release/stable-1.10.txt" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we pin this to a specific patch release of 1.10? Or otherwise, clearly print out the exact version returned here in the test logs, so we can debug failures caused by patch version changes. |
||
KUBE_VERSION=$(curl --fail --silent "${KUBE_VERSION_URL}") | ||
KUBE_BIN_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBE_VERSION}/bin/linux/amd64" | ||
|
||
ASSETS_DIR="${ROOT_DIR}/vendor/sigs.k8s.io/testing_frameworks/integration/assets/bin" | ||
|
||
mkdir -p "${ASSETS_DIR}" | ||
|
||
curl --fail --silent ${ETCD_URL}/${ETCD_VERSION}/etcd-${ETCD_VERSION}-linux-amd64.tar.gz | \ | ||
tar --extract --gzip --directory="${ASSETS_DIR}" --strip-components=1 --wildcards '*/etcd' | ||
curl --fail --silent --output "${ASSETS_DIR}/kube-apiserver" "${KUBE_BIN_URL}/kube-apiserver" | ||
curl --fail --silent --output "${ASSETS_DIR}/kubectl" "${KUBE_BIN_URL}/kubectl" | ||
|
||
chmod +x ${ASSETS_DIR}/* |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
package testfs | ||
|
||
import ( | ||
"fmt" | ||
"io/ioutil" | ||
"os" | ||
"path" | ||
"path/filepath" | ||
"testing" | ||
|
||
"github.com/stretchr/testify/require" | ||
) | ||
|
||
type TestFs struct { | ||
t *testing.T | ||
d string | ||
} | ||
|
||
func New(t *testing.T) *TestFs { | ||
d := fmt.Sprintf(".test/%s", t.Name()) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there a reason for specifying a custom base, instead of relying on |
||
d, err := filepath.Abs(d) | ||
require.NoError(t, err) | ||
err = os.RemoveAll(d) | ||
if err != nil && !os.IsNotExist(err) { | ||
t.Fatalf("Error while removing old test directory: %s", err) | ||
} | ||
|
||
err = os.MkdirAll(d, os.ModePerm) | ||
require.NoError(t, err) | ||
|
||
return &TestFs{ | ||
t: t, | ||
d: d, | ||
} | ||
} | ||
|
||
func (tfs *TestFs) TempPath(name string) string { | ||
outPath := path.Join(tfs.d, name) | ||
tmpFile, err := ioutil.TempFile(tfs.d, name) | ||
require.NoError(tfs.t, err) | ||
err = tmpFile.Close() | ||
require.NoError(tfs.t, err) | ||
err = os.Rename(tmpFile.Name(), outPath) | ||
require.NoError(tfs.t, err) | ||
return outPath | ||
} | ||
|
||
func (tfs *TestFs) TempDir(name string) string { | ||
outPath := path.Join(tfs.d, name) | ||
err := os.MkdirAll(outPath, os.ModePerm) | ||
require.NoError(tfs.t, err) | ||
return outPath | ||
} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What's the reasoning for this package? It seems to wrap |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(nit) Bulleted list 😄