Develop (#12)

* first commit of ingress tests

* Add info on using custom helm chart

* add ingress for fadi services

* Update deploy.sh to include traefik

* update documentation about Traefik

* fix urls

* Feature/ingress (#10)

* first commit of ingress tests

* add ingress for fadi services

* Update deploy.sh to include traefik

* update documentation about Traefik

* fix urls

* update doc

* Enable ingress addon in minikube

INSTALL.md

@@ -72,6 +72,7 @@ Launch the Helm script, this will deploy all the FADI services on the Minikube c
 
 ```bash
 kubectl config set-context minikube
+minikube addons enable ingress
 cd helm
 # you can edit values.yaml file to customise the stack
 ./deploy.sh
@@ -82,13 +83,21 @@ You can check everything is deploying/-ed in the Kubernetes dashboard:
 
 ![Kubernetes FADI dashboard](doc/images/installation/minikube_fadi_dashboard.png)
 
+To access a service in your browser, you need to setup entries in your /etc/hosts file to route [http://fadi.minikube](http://fadi.minikube) to our cluster.
 
-To access a service in your browser, type for instance:
+In production you would want to set up real DNS entries. You can get the IP address of your minikube instance by running minikube ip:
 
-```bash
-minikube service list
-minikube service -n fadi fadi-nifi
 ```
+echo "$(minikube ip) fadi.minikube" | sudo tee -a /etc/hosts
+```
+
+Reply this operation for [superset](https://superset.incubator.apache.org/).
+
+```
+echo "$(minikube ip) fadi-superset.minikube" | sudo tee -a /etc/hosts
+```
+
+We should now be able to visit the different services behind [http://fadi.minikube](http://fadi.minikube). For instance, you can access Grafana in the browser at [http://fadi.minikube/grafana](http://fadi.minikube/grafana).
 
 To update the FADI stack, re-type:
 

USERGUIDE.md

@@ -58,11 +58,7 @@ measure_ts,temperature
 
 First, setup the datalake by creating a table in the postgresql database. 
 
-Head to the pgadmin interface and execute the [table creation script](examples/basic/create_datalake_tables.sql).
-
-```
-minikube service fadi-pgadmin -n fadi
-```
+Head to the pgadmin interface ([http://fadi.minikube/pgadmin](http://fadi.minikube/pgadmin)) and execute the [table creation script](examples/basic/create_datalake_tables.sql).
 
 (the default credentials are `pgadmin4@pgadmin.org`/`admin`):
 
@@ -81,11 +77,7 @@ kubectl cp ./postgresql-42.2.6.jar fadi/fadi-nifi-0:/opt/nifi/postgresql-42.2.6.
 rm postgresql-42.2.6.jar
 ```
 
-Then head to the Nifi web interface, if you are using the local installation with Minikube you can run the following command:
-
- ```
- minikube service fadi-nifi-load-balancer -n fadi
- ```
+Then head to the Nifi web interface ([http://fadi.minikube/nifi](http://fadi.minikube/nifi)), if you are using the local installation with Minikube).
 
 ![Nifi web interface](examples/basic/images/nifi_interface.png)
 
@@ -120,9 +112,7 @@ Once the measurements are stored in the database, we will want to display the re
 
 [Grafana](http://grafana.com/) provides a dashboard and alerting interface.
 
-Head to the Grafana interface (the default credentials are `admin`/`password`): 
-
-```minikube service fadi-grafana -n fadi```
+Head to the Grafana interface at [http://fadi.minikube/grafana](http://fadi.minikube/grafana) (the default credentials are `admin`/`password`): 
 
 ![Grafana web interface](examples/basic/images/grafana_interface.png)
 
@@ -154,9 +144,7 @@ For more information on how to use Grafana, see the [official Grafana user guide
 
 [Apache Superset](https://superset.incubator.apache.org) provides some interesting features to explore your data and build basic dashboards.
 
-Head to the Superset interface (the default credentials are `admin`/`admin`): 
-
-```minikube service fadi-superset -n fadi```
+Head to the Superset interface at [http://fadi-superset.minikube](http://fadi-superset.minikube) (the default credentials are `admin`/`admin`): 
 
 First we will define the datasource:
 
@@ -192,9 +180,7 @@ For more information on how to use Superset, see the [official Superset user gui
 
 In this simple use case, we will try to access the data that is stored in the data lake.
 
-Head to the Jupyter notebook interface (the default credentials are `admin`/`password`): 
-
-```minikube service proxy-public -n fadi```
+Head to the Jupyter notebook interface at [http://fadi.minikube/jupyterhub](http://fadi.minikube/jupyterhub) (the default credentials are `admin`/`password`):
 
 ![Jupyter web interface](examples/basic/images/jupyter_interface.png)
 
@@ -212,4 +198,4 @@ For more information on how to use Superset, see the [official Jupyter documenta
 
 In this use case, we have demonstrated a simple configuration for FADI, where we use various services to ingest, store, analyse, explore and provide dashboards and alerts 
 
-You can find the various resources for this sample use case (Nifi flowfile, Grafana dashboards, ...) in the [examples folder](examples/basic)
+You can find the various resources for this sample use case (Nifi flowfile, Grafana dashboards, ...) in the [examples folder](examples/basic)

helm/README.md

@@ -2,4 +2,25 @@
 
 ![](/doc/images/logos/helm.png)
 
-This folder contains the installation scripts for FADI.
+This folder contains the installation scripts for FADI.
+
+See the FADI helm chart for more information.
+
+Using a custom Helm chart
+-------
+
+To use a custom Helm chart:
+
+* copy the FADI helm chart in this folder
+* modify it to your needs 
+* adapt `deploy.sh` and `teardown.sh`:
+
+    ```bash
+    helm upgrade --install ${NAMESPACE} cetic/fadi -f ./values.yaml --namespace ${NAMESPACE} --tiller-namespace tiller
+    ```
+
+    should become 
+
+    ```bash
+    helm upgrade --install ${NAMESPACE} . -f ./values.yaml --namespace ${NAMESPACE} --tiller-namespace tiller
+    ```

helm/deploy.sh

@@ -30,6 +30,12 @@ printf "\n\nHelm all the things!...\n"
 helm repo add cetic https://cetic.github.io/helm-charts/
 helm repo update
 
+# create clusterrole for traefik
+kubectl get clusterrole traefik-ingress-controller 2> /dev/null || kubectl create -f ./traefik/rbac-config.yaml
+
+# install/upgrade traefik
+helm upgrade --install traefik stable/traefik -f ./traefik/values.yaml --namespace ${NAMESPACE} --tiller-namespace tiller
+
 # install/upgrade FADI
 helm upgrade --install ${NAMESPACE} cetic/fadi -f ./values.yaml --namespace ${NAMESPACE} --tiller-namespace tiller
 

helm/teardown.sh

@@ -17,11 +17,18 @@ printf "\n\nDeleting FADI release...\n"
 
 helm delete --purge ${NAMESPACE} --tiller-namespace tiller || true
 
+printf "\n\nDeleting Traefik release...\n"
+
+helm delete --purge traefik --tiller-namespace tiller || true
+
 printf "\n\nDeleting Tiller and namespaces...\n"
 
 # Delete sa for tiller
 kubectl delete -f ./tiller/rbac-config.yaml
 
+# Delete clusterrole for traefik
+kubectl delete -f ./traefik/rbac-config.yaml
+
 # Delete namespaces
 kubectl delete namespace tiller
 kubectl delete namespace ${NAMESPACE}

helm/traefik/rbac-config.yaml

@@ -0,0 +1,38 @@
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: traefik-ingress-controller
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - services
+      - endpoints
+      - secrets
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - extensions
+    resources:
+      - ingresses
+    verbs:
+      - get
+      - list
+      - watch
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: traefik-ingress-controller
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: traefik-ingress-controller
+subjects:
+- kind: ServiceAccount
+  name: traefik-ingress-controller
+  namespace: kube-system
+

helm/traefik/values.yaml

@@ -0,0 +1,9 @@
+dashboard:
+  enabled: true
+  domain: fadi.minikube
+  serviceType: NodePort
+kubernetes:
+  namespaces:
+    - default
+    - kube-system
+

helm/values.yaml

@@ -19,8 +19,13 @@ superset:
   persistence:
     enabled: true
   service:
-    type: LoadBalancer
-
+    type: ClusterIP
+  ingress:
+    enabled: true
+    annotations: {}
+    path: /
+    hosts: [fadi-superset.minikube]
+
 postgresql:
   enabled: true
   persistence:
@@ -31,13 +36,28 @@ minio:
   persistence:
     enabled: true
   service:
-    type: LoadBalancer
+    type: ClusterIP
+  ingress:
+    enabled: true
+    path: /minio
+    hosts: [fadi.minikube]
 
 grafana:
   enabled: true
+  image:
+    tag: 6.3.0-beta2
   #LoadBalancer service
   service:
-    type: LoadBalancer
+    type: ClusterIP
+  ingress:
+    enabled: true
+    path: /grafana/
+    hosts:
+      - fadi.minikube
+  grafana.ini:
+    server:
+      root_url: http://fadi.minikube/grafana/
+      serve_from_sub_path: true
   # Enable persistence
   persistence:
     enabled: true
@@ -50,6 +70,7 @@ jupyterhub:
   proxy:
     secretToken: 'af83775ec3bfaf0507ce596df51d491e7ed54450adc454038fa7405495465f19'
   hub:
+    baseUrl: /jupyterhub
     db:
       type: sqlite-memory
   rbac:
@@ -94,10 +115,34 @@ jupyterhub:
   prePuller:
     hook:
       enabled: false
-
+  ingress:
+    enabled: true
+    hosts: [fadi.minikube]
+    pathSuffix: ''
 
 nifi:
   enabled: true
+  service:
+    loadBalancer:
+      enabled: true
+      type: ClusterIP
+  ingress:
+    enabled: true
+    annotations: {}
+    tls: []
+    hosts: 
+      - fadi.minikube
+    path: /nifi
 
 pgadmin:
   enabled: true
+  service:
+    type: ClusterIP
+  ingress:
+    enabled: true
+    annotations: {traefik.ingress.kubernetes.io/rule-type: Path}
+    path: /pgadmin
+    hosts:
+      - fadi.minikube
+  pgadmin:
+    scriptname: /pgadmin