In this example we will show how you can use containerdbg in order to debug petclinic, an application which allows you to manage a clinic for pets.
The application contains:
- A tomcat server with the web application written in Java.
- A posgres DB which contains the information about the pets.
In addition the normal petclinic Java web app, the app has been modified to throw exception by trying to open non existant files on specific actions, an issue which we will debug.
The code for the petclinic application can be found in examples/spring-framework-petclinic/ and is a modified version of the opensource project
In order to run this tutorial you need to have the following resources:
- a kubernetes cluster
kind
cluster can be used for this. - have
skaffold
installed on your machine https://skaffold.dev/docs/install/
Next we will have to build and prepare our yamls for the tutorial, this can be done in the following steps:
- Set up your image repository (this is not needed for kind clusters)
export REGISTRY=<my repository>
- Run the following command to generate the tomcat yaml
cd tomcat
skaffold build -d $REGISTRY --file-output artifacts.json
skaffold render -d $REGISTRY -a artifacts.json > ../tomcat.yaml
cd ..
- Run the following command to generate the db yaml
cd db
skaffold build -d $REGISTRY --file-output artifacts.json
skaffold render -d $REGISTRY -a artifacts.json > ../db.yaml
cd ..
First we will deploy this application and show you where it fails.
In order to start the app, please run the following while being connected to a cluster:
kubectl apply -f ./db.yaml
kubectl apply -f ./tomcat.yaml
Now retrieve the external IP address of the tomcat site by running:
export IP=$(kubectl get services petclinic-tomcat -ojsonpath='{.status.loadBalancer.ingress[0].ip}')
Now open your browser and go to http://$IP:8080/petclinic
try the following:
- Open find owners page.
- Add new owner.
- Add new pet for the owner.
At this point you should see an error that something is missing.
In the next section we will see how to use containerdbg in order to discover what is missing.
In this example we will demonstrate a situation where one container was moved to another location but the DB wasn't made to be accesible in the new location.
To demonstrate this, we will delete the previously created db pod by running the following command:
kubectl delete statefulsets.apps petclinic-postgres
Now by traversing the site (for example go to the find owners page) you will see errors are starting to occur.
Now in order to test what is broken in the tomcat container we will remove the currently installed tomcat container:
kubectl delete -f tomcat.yaml
Next we will run tomcat using containerdbg:
containerdbg debug -f tomcat.yaml -o tomcat.pb
Then open your browser and traverse to the petclinic application.
export IP=$(kubectl get services petclinic-tomcat -ojsonpath='{.status.loadBalancer.ingress[0].ip}')
open your browser and go to http://$IP:8080/petclinic
.
You should see an error when entering the site, after seeing the error press Ctrl-C in the terminal from which you ran containerdbg.
Now run the analysis tool:
containerdbg analyze -f tomcat.pb
this should produce the following output:
While executing the container the following files were missing:
===============================================================
/usr/local/openjdk-11/conf/jndi.properties is missing
/usr/local/tomcat/work/Catalina/localhost/petclinic/SESSIONS.ser is missing
While executing the container the library type files were missing:
==================================================================
While executing the container the following files where attempted to be moved but failed to docker limitation:
==============================================================================================================
While executing the container the following connections failed:
==============================================================================================================
10.108.12.9:5432
The last connection using port 5432 is the postgres DB cluster IP which can be viewed from kubectl get services
.
Now restore the db by running kubectl apply -f db.yaml
and proceed to the next example.
In this step we will once again containerdbg to run tomcat
containerdbg debug -f tomcat.yaml -o tomcat2.pb
And repeat the steps to reproduce the error as described in Deploying a working application.
After seeing the error stop the debugging session by pressing Ctrl-C in the same terminal where you ran containerdbg.
By running containerdbg analyze -f tomcat2.pb
you will see the following output:
While executing the container the following files were missing:
===============================================================
is missing
/etc/group is missing
/etc/passwd is missing
/etc/selinux/config is missing
/etc/selinux/contexts/lxc_contexts is missing
/usr/lib/locale/locale-archive is missing
/usr/local/openjdk-11/conf/jndi.properties is missing
/usr/local/tomcat/work/Catalina/localhost/petclinic/SESSIONS.ser is missing
/usr/share/containers/selinux/contexts is missing
file.txt is missing
While executing the container the library type files were missing:
==================================================================
While executing the container the following files where attempted to be moved but failed to docker limitation:
==============================================================================================================
While executing the container the following connections failed:
==============================================================================================================
By comparing to the old output or by seeing that most files are not relevant (selinux related or /etc/group /etc/passwd files) we can see that the relevant error is caused by file.txt .
This should lead you to edit the tomcat Dockerfile and add the missing file, luckily we can see that there is a commented out line in tomcat/Dockerfile adding this file so simply uncomment it.
Now repeat the steps in Deploying a working application and the application should run without issues.
In this example we saw how, by running containerdbg multiple times we could detect issues in the application and fix them this is an iterative approach that should be repeated multiple times until issues are fixed.