PersistentVolume for mysql-data

[naveensrinivasan]

I was trying to run the zipkin-starter-0.1.5-kubernetes.yml in minikube and ran into an issue where the PVC was stuck in the pending mode.

That is when I realized the yml file didn't have an entry for the PersistentVolume which was causing the PVC to be stuck forever.

When I created an entry for the PersistentVolume things have started working.

[jimmidyson]

Glad you got it sorted.

Most Kubernetes deployments now use dynamic provisioning and this post to provision PVs. If the YAML file had a PV in it, it becomes less portable so it is preferred to leave that out and either statically create the PV or use dynamic provisioning to create it.

[naveensrinivasan]

@jimmidyson Thanks.

IMHO either it would have to be documented in the README.md or fix the yml with the annotation like this

- apiVersion: "v1"
  kind: "PersistentVolumeClaim"
  metadata:
    labels:
      provider: "fabric8"
      project: "zipkin-mysql"
      version: "0.1.5"
      group: "io.fabric8.zipkin"
    name: "mysql-data"
    annotations:
     volume.alpha.kubernetes.io/storage-class: default
  spec:
    accessModes:
    - "ReadWriteOnce"
    resources:
      requests:
        storage: "100Mi"

Thoughts?

[eschaef1]

I'm having a similar issue with OpenShift, however because the PVC is still pending, it doesn't seem as though there is a YAML available for me to edit, or any option to create one. I'm very new to OpenShift, so does anyone have any suggestions? Thanks!

[simongui]

I'm having the same issue on minikube. Even after adding the annotation @naveensrinivasan mentioned.

Events:
  FirstSeen	LastSeen	Count	From			SubobjectPath	Type		Reason		Message
  ---------	--------	-----	----			-------------	--------	------		-------
  4m		4m		1	{default-scheduler }			Normal		Scheduled	Successfully assigned zipkin-mysql-285018599-1frrk to minikube
  2m		25s		2	{kubelet minikube}			Warning		FailedMount	Unable to mount volumes for pod "zipkin-mysql-285018599-1frrk_default(cb28ad3c-0837-11e7-b949-96d251eab0c6)": timeout expired waiting for volumes to attach/mount for pod "default"/"zipkin-mysql-285018599-1frrk". list of unattached/unmounted volumes=[mysql-data]
  2m		25s		2	{kubelet minikube}			Warning		FailedSync	Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"zipkin-mysql-285018599-1frrk". list of unattached/unmounted volumes=[mysql-data]

[naveensrinivasan]

@simongui What does kubectl get pv looks like?

[simongui]

I got it to work. I had to create a persistent volume before deploying Zipkin.

mysql-data.yml

kind: PersistentVolume
apiVersion: v1
metadata:
  name: mysql-data
  labels:
    type: local
spec:
  capacity:
    storage: 100Mi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/tmp/data"

kubectl create -f mysql-data.yml

I'm new to Kubernetes but I'm assuming this isn't a good volume to create because it's pointed to /tmp/data. Suggestions?

[rafabene]

Following as I'm having the same problem to deploy in OpenShift/CDK

[naveensrinivasan]

I was able to get it working

 ~/temp  k8s-minikube/zipkin  k create namespace zipkintest
namespace "zipkintest" created
 ~/temp  k8s-minikube/zipkin  kubectl config set-context minikube --namespace=zipkintest
Context "minikube" set.
 ~/temp  k8s-minikube/zipkintest 
 ~/temp  k8s-minikube/zipkintest  k apply -f zipkin-starter-0.1.5-kubernetes.yml
service "kafka" created
service "zipkin" created
service "zipkin-mysql" created
service "zookeeper" created
persistentvolumeclaim "mysql-data" created
configmap "zipkin-mysql-cnf" created
configmap "zipkin-mysql-initdb" created
deployment "kafka" created
deployment "zipkin" created
deployment "zipkin-mysql" created
deployment "zookeeper" created
 ~/temp  k8s-minikube/zipkintest  k get pv
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                   REASON    AGE
pvc-4b5e428d-10fa-11e7-b903-080027cac7d7   100Mi      RWO           Delete          Bound     zipkintest/mysql-data             16s
 ~/temp  k8s-minikube/zipkintest  k get po
NAME                            READY     STATUS    RESTARTS   AGE
kafka-2002085703-3jhw8          1/1       Running   0          27s
zipkin-358939735-cg4r3          0/1       Running   0          27s
zipkin-mysql-3963011279-zw8m0   1/1       Running   0          27s
zookeeper-650666963-x3jcm       1/1       Running   0          27s
 ~/temp  k8s-minikube/zipkintest  kubectl get po
NAME                            READY     STATUS    RESTARTS   AGE
kafka-2002085703-3jhw8          1/1       Running   0          40s
zipkin-358939735-cg4r3          1/1       Running   0          40s
zipkin-mysql-3963011279-zw8m0   1/1       Running   0          40s
zookeeper-650666963-x3jcm       1/1       Running   0          40s    

Here is the YAML https://gist.github.com/naveensrinivasan/58ef741faa69f18fafb2c3e0df95d62f

[simongui]

What did you change to make this work @naveensrinivasan? I tried to add this to the minimal yaml file but it still fails.

- apiVersion: "v1"
  kind: "PersistentVolumeClaim"
  metadata:
    labels:
      provider: "fabric8"
      project: "zipkin-mysql"
      version: "0.1.5"
      group: "io.fabric8.zipkin"
    name: "mysql-data"
    annotations:
      volume.beta.kubernetes.io/storage-class: default

[naveensrinivasan]

Did you have a look at the Yaml in the gist?

[simongui]

Yeah, the main thing I see different is what I pasted. Was there anything more? I'm also having difficulty deploying this version of the YAML from your gist. It seems sometimes all the pods come up except the Zipkin one. Almost like it's trying to connect to dependent services and fails before they come up.

[naveensrinivasan]

What is the error from the pod logs?

…

On Sun, Apr 2, 2017 at 4:27 PM Simon Guindon ***@***.***> wrote: Yeah, the main thing I see different is what I pasted. Was there anything more? I'm also having difficulty deploying this version of the YAML from your gist. It seems sometimes all the pods come up except the Zipkin one. Almost like it's trying to connect to dependent services and fails before they come up. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#92 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKimaLthDQt4yekbgmLA5yeYbZgQZ0iks5rsAS5gaJpZM4MUoJS> .

[simongui]

2017-04-02T20:56:38.289555358Z 2017-04-02 20:56:38.289 ERROR 6 --- [io-9411-exec-10] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.jooq.exception.DataAccessException: SQL [select distinct `zipkin_annotations`.`endpoint_service_name` from `zipkin_annotations` where (`zipkin_annotations`.`endpoint_service_name` is not null and `zipkin_annotations`.`endpoint_service_name` <> ?)]; Table 'zipkin.zipkin_annotations' doesn't exist
2017-04-02T20:56:38.289597982Z Query is : select distinct `zipkin_annotations`.`endpoint_service_name` from `zipkin_annotations` where (`zipkin_annotations`.`endpoint_service_name` is not null and `zipkin_annotations`.`endpoint_service_name` <> '')] with root cause
2017-04-02T20:56:38.289603334Z 
2017-04-02T20:56:38.289617753Z org.mariadb.jdbc.internal.util.dao.QueryException: Table 'zipkin.zipkin_annotations' doesn't exist
2017-04-02T20:56:38.289621817Z Query is : select distinct `zipkin_annotations`.`endpoint_service_name` from `zipkin_annotations` where (`zipkin_annotations`.`endpoint_service_name` is not null and `zipkin_annotations`.`endpoint_service_name` <> '')
2017-04-02T20:56:38.289625676Z 	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:939) ~[mariadb-java-client-1.4.6.jar!/:na]
2017-04-02T20:56:38.289629298Z 	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQueries(AbstractQueryProtocol.java:775) ~[mariadb-java-client-1.4.6.jar!/:na]
2017-04-02T20:56:38.289632460Z 	at org.mariadb.jdbc.MariaDbClientPreparedStatement.executeInternal(MariaDbClientPreparedStatement.java:201) ~[mariadb-java-client-1.4.6.jar!/:na]

[simongui]

Oh weird, after 30-60mins I refreshed and it finally shows up as running. That's a really long time.

[simongui]

Scratch that, it's still not running.

kubectl get pods
NAME                           READY     STATUS    RESTARTS   AGE
kafka-1764556007-hk3f0         1/1       Running   0          1h
zipkin-1572713903-vp7sn        0/1       Running   0          1h
zipkin-mysql-285018599-pk6bk   1/1       Running   1          1h
zookeeper-3120290255-nv64q     1/1       Running   0          1h

[cmoulliard]

Here is what I did to delete/create a new pvc

Versions
- Minishift : 1.0.1
- OpenShift Origin : 1.5

minishift start
oc new-project zipkin
oc create -f http://repo1.maven.org/maven2/io/fabric8/zipkin/zipkin-starter-minimal/0.1.9/zipkin-starter-minimal-0.1.9-openshift.yml
oc delete pvc/mysql-data
cat << EOF | oc create -f - 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-data
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

[r4j4h]

To clarify, the problem issues for me were that spec.volumeName was defined so automatic provisioners failed as it tried to look up the existing volume, which did not exist.

Manually creating a volume with that name, or removing the volumeName before creating the resource should work, depending on usage of automatic PV provisioners.

[adrianchifor]

Hey guys, I have followed your suggestions but with no luck...the zipkin-mysql Pod keeps erroring:

Initializing database
2017-07-31T19:52:40.460188Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-07-31T19:52:40.460229Z 0 [Warning] 'NO_AUTO_CREATE_USER' sql mode was not set.
2017-07-31T19:52:40.462439Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
2017-07-31T19:52:40.462527Z 0 [ERROR] Aborting

I am creating a fresh PVC with AWS EBS right after deploying everything.

Any ideas?

[adrianchifor]

Ok, apparently on k8s there is this active issue with running mysql docker-library/mysql#186. The solution is to add the following args to the zipkin-mysql Pod:

image: mysql:5.7
name: zipkin-mysql
readinessProbe:
  initialDelaySeconds: 5
  tcpSocket:
    port: 3306
args:
  - "--ignore-db-dir=lost+found"

[codefromthecrypt]

Ok, apparently on k8s there is this active issue with running mysql docker-library/mysql#186 <docker-library/mysql#186>. The solution is to add the following args to the zipkin-mysql Pod:

Thanks for following up

[davgordo]

I think the small change proposed in #118 would save a lot of confusion. For example, a minishift user will automatically have PVs that are appropriate for the template, but they won't bind because they're not named mysql-data. It seems overly prescriptive to me to name the volume.

[2ZZ]

I had to specify the storageClassName to get the PVC to bind to the PV

kind: PersistentVolume
metadata:
  name: mysql-data
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 100Mi
  hostPath:
    path: /data/mysql-data/