OpenShift Service Mesh federation configuration for IBM System Z/LinuxONE and IBM Power systems running OpenShift 4.6+ clusters.
derived from https://github.com/maistra/istio/pkg/servicemesh/federation/example/config-poc
OSSM Federation connects groups of microservices managed under two different service mesh control planes.
The practical use case of this is to do more advanced load sharing and management on e.g. two geographically separated clusters.
Never the less, for development and test purposes, it makes sense to configure your two flights of micro-services on the same cluster initially, as this case is the simplest to configure the environment for.
Configuring two clusters' environments to support OSSM federation between them is also relatively simple in cloud environments, as you can use the LoadBalancer service type to route your applications' data within the federated data plane, and routing between the clusters is handled by the cloud provider's load balancer.
However, in clusters provisioned on 'bare metal' systems, and virtualized on IBM System Z and Power systems, you need to use the NodePort service type to route your applications' data within the federated data planes, but between the nodes of the clusters. This means you need to make sure to open the necessary firewall ports, and make sure data can be routed between the nodes of one cluster, to the nodes of the other.
This can mean taking a good hard look at your DNS settings, and
provisioning a load balancer in your environment, external to the
two clusters. The example scripts for these cases assume you are
using haproxy
for the load balancer and firewalld
for your firewall;
for deployment in production environments, please feel free to use these
scripts as examples of what you'll need to do in your own environments.
The prerequisites for all installations of OSSM federation are:
- Install the Service Mesh, Kiali and Jaeger operators on the clusters
- Use the 'install to all namespaces' option
- Set environment variables MESH1-KUBECONFIG and MESH2-KUBECONFIG.
These are the kubeconfig files for the
two clusters you're going to run federated service mesh
installations on. These files can be found, typically,
stashed away in the auth
directory of the cluster installation artifacts.
e.g.:
export MESH1_KUBECONFIG=/root/ipi/pipe-2/auth/kubeconfig
export MESH2_KUBECONFIG=/root/second_cluster/kubeconfig
- Run the ./tools/mod-bookinfo-images.sh to update the bookinfo images.
Note: For different hosts, copy kubeconfig of second cluster on the host having first cluster using scp command at path "/root/second_cluster/kubeconfig"
If you are testing federated services on two different service meshes on the
same cluster, you can use the install_onecluster.sh
script without any
further system configuration.
If you are running multi-cluster service mesh federation on bare-metal, libvirt or UPI OCP installations such as z/VM, you will (a) need to use NodePort rather than LoadBalancer service types to federate, and consequently (b) you will need to route the traffic somehow.
Here we use a proxy server -- haproxy.
To install haproxy
on RHEL 8, sudo yum install -y haproxy
In this case:
- set the PROXY_HOST environment variable if both clusters on same host
- set
MESH1_ADDRESS
andMESH2_ADDRESS
two clusters are on different hosts. - Edit the
OPTIONS
field in/etc/sysconfig/haproxy
file:
# specifying multiple configuration files with multiple -f options.
# See haproxy(1) for a complete list of options.
OPTIONS="-f /etc/haproxy/federation.cfg"
-
Run ./install-libvirt.sh - This script creates: - mesh1-system - mesh1-bookinfo on cluster1 - mesh2-system,mesh2-bookinfo on cluster2. - creates smcp and smmr in mesh1-system and mesh2-system namespaces. - Installs the NodePort services - Opens the appropriate firewalls in the libvirt zone - Generates & installs the proxy configuration file federation.cfg - Restarts haproxy with the additional proxy configuration - sets up the federation plane - deploys the two bookinfo projects that will be federated in the two clusters
-
Check the federation configuration is successfully copied to location /etc/haproxy/
# cat /etc/haproxy/federation.cfg
- Check the firewall is opened for discovery and service ports
# grep -w '<discovery_port/service_port>/tcp' /etc/services
e.g. # grep -w '32568/tcp' /etc/services
Note: If firewall is not opened, manually copy front end and
backend sections from federation.cfg to /etc/haproxy/haproxy
and restart haproxy service
- Check the two bookinfo projects, are up and running
# oc get pods -n mesh1-bookinfo
# oc get pods -n mesh2-bookinfo
This can apply to any of: IPI libvirt, PowerVM or z/VM, or bare metal installs
- Exchange public keys using ssh-copy-id
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@<host_ip>
- passwordless ssh from the host you're running the install scripts
- Run the
install-multihost.sh
script instead of./install-libvirt.sh
- check the results for respective cluster on respective host as in step 2.
./tools/test-federation.sh
is the script which gives the federation status,
services imported from mesh1 to mesh2 and deploy v2 ratings system into
mesh1 and mesh2.
To check federation manually:
- To check the status of federation
# oc -n mesh1-system get servicemeshpeer mesh2 -o json | jq .status # on cluster1
# oc -n mesh2-system get servicemeshpeer mesh1 -o json | jq .status # on cluster2
- To check services imported from mesh1 into mesh2
# oc -n mesh2-system get importedservicesets mesh1 -o json | jq .status # on cluster2
- On mesh1 cluster, run: oc logs -n mesh1-bookinfo deploy/ratings-v2-mysql -f
- On mesh2 cluster: oc logs -n mesh2-bookinfo deploy/ratings-v2-mysql -f
- Open
http://$(oc -n mesh2-system get route istio-ingressgateway -o json | jq -r .spec.host)/productpage
- Refresh the page several times
- observe requests hitting either the mesh1 or the mesh2 cluster.
The script tools/uninstall-federation.sh
can be run to delete all
of the federation test artifacts.