-
Notifications
You must be signed in to change notification settings - Fork 2
/
skewer.yaml
170 lines (158 loc) · 7.44 KB
/
skewer.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
title: Accessing Kafka using Skupper
subtitle: Use public cloud resources to process data from a private Kafka cluster
overview: |
This example is a simple Kafka application that shows how you can
use Skupper to access a Kafka cluster at a remote site without
exposing it to the public internet.
It contains two services:
* A Kafka cluster named "cluster1" running in a private data center.
The cluster has a topic named "topic1".
* A Kafka client running in the public cloud. It sends 10 messages
to "topic1" and then receives them back.
To set up the Kafka cluster, this example uses the Kubernetes
operator from the [Strimzi][strimzi] project. The Kafka client is a
Java application built using [Quarkus][quarkus].
The example uses two Kubernetes namespaces, "private" and "public",
to represent the private data center and public cloud.
[strimzi]: https://strimzi.io/
[quarkus]: https://quarkus.io/
sites:
public:
title: Public
platform: kubernetes
namespace: public
env:
KUBECONFIG: ~/.kube/config-public
private:
title: Private
platform: kubernetes
namespace: private
env:
KUBECONFIG: ~/.kube/config-private
steps:
- standard: install_the_skupper_command_line_tool
- standard: kubernetes/set_up_your_namespaces
- title: Deploy the Kafka cluster
preamble: |
In Private, use the `kubectl create` and `kubectl apply`
commands with the listed YAML files to install the operator and
deploy the cluster and topic.
commands:
private:
- run: kubectl create -f server/strimzi.yaml
output: |
customresourcedefinition.apiextensions.k8s.io/kafkas.kafka.strimzi.io created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-entity-operator-delegation created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-topic-operator-delegation created
customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkarebalances.kafka.strimzi.io created
deployment.apps/strimzi-cluster-operator created
customresourcedefinition.apiextensions.k8s.io/kafkamirrormaker2s.kafka.strimzi.io created
clusterrole.rbac.authorization.k8s.io/strimzi-entity-operator created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-global created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-broker-delegation created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-namespaced created
clusterrole.rbac.authorization.k8s.io/strimzi-topic-operator created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-client-delegation created
clusterrole.rbac.authorization.k8s.io/strimzi-kafka-client created
serviceaccount/strimzi-cluster-operator created
clusterrole.rbac.authorization.k8s.io/strimzi-kafka-broker created
customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkabridges.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkaconnectors.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkaconnects2is.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkaconnects.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkamirrormakers.kafka.strimzi.io created
configmap/strimzi-cluster-operator created
- run: kubectl apply -f server/cluster1.yaml
output: |
kafka.kafka.strimzi.io/cluster1 created
kafkatopic.kafka.strimzi.io/topic1 created
- await_resource: kafka/cluster1
- run: kubectl wait --for condition=ready --timeout 900s kafka/cluster1
output: |
kafka.kafka.strimzi.io/cluster1 condition met
postamble: |
**Note:**
By default, the Kafka bootstrap server returns broker addresses
that include the Kubernetes namespace in their domain name.
When, as in this example, the Kafka client is running in a
namespace with a different name from that of the Kafka cluster,
this prevents the client from resolving the Kafka brokers.
To make the Kafka brokers reachable, set the `advertisedHost`
property of each broker to a domain name that the Kafka client
can resolve at the remote site. In this example, this is
achieved with the following listener configuration:
~~~ yaml
spec:
kafka:
listeners:
- name: plain
port: 9092
type: internal
tls: false
configuration:
brokers:
- broker: 0
advertisedHost: cluster1-kafka-0.cluster1-kafka-brokers
~~~
See [Advertised addresses for brokers][advertised-addresses] for
more information.
[advertised-addresses]: https://strimzi.io/docs/operators/in-development/configuring.html#property-listener-config-broker-reference
- standard: kubernetes/create_your_sites
- standard: kubernetes/link_your_sites
- title: Expose the Kafka cluster
preamble: |
In Private, use `skupper expose` with the `--headless` option to
expose the Kafka cluster as a headless service on the Skupper
network.
Then, in Public, use the `kubectl get service` command to check
that the `cluster1-kafka-brokers` service appears after a
moment.
commands:
private:
- await_resource: statefulset/cluster1-kafka
- run: skupper expose statefulset/cluster1-kafka --headless --port 9092
output: |
statefulset cluster1-kafka exposed as cluster1-kafka-brokers
public:
- await_resource: service/cluster1-kafka-brokers
- run: kubectl get service/cluster1-kafka-brokers
output: |
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cluster1-kafka-brokers ClusterIP None <none> 9092/TCP 2s
- title: Run the client
preamble: |
Use the `kubectl run` command to execute the client program in
Public.
commands:
public:
- await_resource: service/cluster1-kafka-brokers
- run: kubectl run client --attach --rm --restart Never --image quay.io/skupper/kafka-example-client --env BOOTSTRAP_SERVERS=cluster1-kafka-brokers:9092
output: |
[...]
Received message 1
Received message 2
Received message 3
Received message 4
Received message 5
Received message 6
Received message 7
Received message 8
Received message 9
Received message 10
Result: OK
[...]
postamble: |
To see the client code, look in the [client directory](client)
of this project.
- standard: cleaning_up
commands:
private:
- run: skupper delete
- run: kubectl delete -f server/cluster1.yaml
- run: kubectl delete -f server/strimzi.yaml
public:
- run: skupper delete