-
Notifications
You must be signed in to change notification settings - Fork 112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Solr operator going in CrashLoopBackOff state #41
Comments
Can you share a kubectl describe pod and describe the deploy as well. Also, same behavior if you delete the solr-operator pod and let it come back? |
The Solr Operator version it prints out is wrong, and is an issue that I need to fix. What version of the container are you using? |
Might be solr-operator is getting OOMKilled. Happened in our case. |
Yes, operator pod is getting OOMKilled, but we've enough resources on nodes. Also, its same behavior if you delete the solr-operator pod.
|
After increasing the memory in resources section, it started working. Thank you for your support. |
haha I was about to paste this - https://github.com/bloomberg/solr-operator/blob/master/config/operators/solr_operator.yaml#L453 Cool. You figured it. |
Did you face the problem in next step?
|
I don't think it's able to create the Zookeeper instance. You have to have a Persistent Volume that your Zookeeper Persistent Volume Claim Spec can map to. The Zookeeper Operator does not support non-persistent storage yet. pravega/zookeeper-operator#64 |
I tried to give the storageclass name to provision dynamic volumes. In which section we can pass the volume detail for zookeeper and solr itself in test_solrlcloud.yaml file.
|
your Zookeeper Ref:
Provided:
Zookeeper:
Persistent Volume Claim Spec:
storageClassName: "openebs-cstor"
Access Modes:
ReadWriteOnce
Resources:
Requests:
Storage: 100Mi |
Even that's(Zookeeper level added) not helping. Also I tried to pass the Persistent Volume Claim Spec after resource SolrCloud deployed. Zookeeper is lunching with default specification, its not picking custom values. Anything I'm missing. |
Im pretty sure the yaml input is case sensitive. Also, there are spaces in your spec field names. Let me show you something that works... one sec |
Example YAML that requests persistentVolumeClaimSpec of Zookeeper. (Note, case and spacing matter in the yaml spec). apiVersion: solr.bloomberg.com/v1beta1
kind: SolrCloud
metadata:
name: example
spec:
replicas: 3
solrImage:
tag: 8.2.0
solrJavaMem: "-Xms1g -Xmx3g"
solrPodPolicy:
resources:
limits:
memory: "1G"
requests:
cpu: "65m"
memory: "156Mi"
zookeeperRef:
provided:
zookeeper:
persistentVolumeClaimSpec:
storageClassName: "hostpath"
resources:
requests:
storage: "5Gi"
replicas: 3
zookeeperPodPolicy:
resources:
limits:
memory: "1G"
requests:
cpu: "65m"
memory: "156Mi"
solrOpts: "-Dsolr.autoSoftCommit.maxTime=10000"
solrGCTune: "-XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8" Verified statefulset for ZK applied the 5Gi storage request. |
Thanks for providing the correct yaml. It moved on but its now failing in Zookeeper cluster recouncilations.
|
Looks like an issue with your openebs storage class setup
Get Outlook for iOS
On Sat, Nov 2, 2019 at 1:01 PM -0700, "deepak71" <notifications@github.com> wrote:
Thanks for providing the correct yaml. It moved on but its now failing in Zookeeper cluster recouncilations.
k8 get pods
NAME READY STATUS RESTARTS AGE
example-solrcloud-0 0/1 CrashLoopBackOff 8 17m
example-solrcloud-zookeeper-0 1/1 Running 0 17m
example-solrcloud-zookeeper-1 0/1 CrashLoopBackOff 8 16m
k8 describe pod example-solrcloud-zookeeper-1
Name: example-solrcloud-zookeeper-1
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: k8sstp-kube-worker-lx04/10.190.67.166
Start Time: Sun, 03 Nov 2019 01:11:02 +0530
Labels: app=example-solrcloud-zookeeper
controller-revision-hash=example-solrcloud-zookeeper-6f9f446cc8
kind=ZookeeperMember
statefulset.kubernetes.io/pod-name=example-solrcloud-zookeeper-1
Annotations: kubernetes.io/psp: default
seccomp.security.alpha.kubernetes.io/pod: docker/default
Status: Running
IP: 10.244.8.139
Controlled By: StatefulSet/example-solrcloud-zookeeper
Containers:
zookeeper:
Container ID: docker://db56e467f2d8c9ed296022a2a75dde7f80cb202c388f295a0689fce96110dec2
Image: emccorp/zookeeper:3.5.4-beta-operator
Image ID: docker-pullable://emccorp/zookeeper@sha256:c4656ca1e0103b1660a978fa8b2bfecfae5baf6746bb1839360821abd209082f
Ports: 2181/TCP, 2888/TCP, 3888/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/usr/local/bin/zookeeperStart.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sun, 03 Nov 2019 01:27:02 +0530
Finished: Sun, 03 Nov 2019 01:27:02 +0530
Ready: False
Restart Count: 8
Liveness: exec [zookeeperLive.sh] delay=10s timeout=10s period=10s #success=1 #failure=3
Readiness: exec [zookeeperReady.sh] delay=10s timeout=10s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/conf from conf (rw)
/data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xrqn6 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-example-solrcloud-zookeeper-1
ReadOnly: false
conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: example-solrcloud-zookeeper-configmap
Optional: false
default-token-xrqn6:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-xrqn6
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 17m default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 4 times)
Normal Scheduled 17m default-scheduler Successfully assigned default/example-solrcloud-zookeeper-1 to k8sstp-kube-worker-lx04
Normal SuccessfulAttachVolume 17m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-b2e20492-fda8-11e9-a7f3-0050569b2667"
Normal Pulling 16m (x4 over 16m) kubelet, k8sstp-kube-worker-lx04 Pulling image "emccorp/zookeeper:3.5.4-beta-operator"
Normal Pulled 16m (x4 over 16m) kubelet, k8sstp-kube-worker-lx04 Successfully pulled image "emccorp/zookeeper:3.5.4-beta-operator"
Normal Created 16m (x4 over 16m) kubelet, k8sstp-kube-worker-lx04 Created container zookeeper
Normal Started 16m (x4 over 16m) kubelet, k8sstp-kube-worker-lx04 Started container zookeeper
Warning BackOff 102s (x78 over 16m) kubelet, k8sstp-kube-worker-lx04 Back-off restarting failed container
k8 logs example-solrcloud-zookeeper-1
+ source /conf/env.sh
++ DOMAIN=example-solrcloud-zookeeper-headless.default.svc.cluster.local
++ QUORUM_PORT=2888
++ LEADER_PORT=3888
++ CLIENT_HOST=example-solrcloud-zookeeper-client
++ CLIENT_PORT=2181
+ source /usr/local/bin/zookeeperFunctions.sh
++ set -ex
++ hostname -s
+ HOST=example-solrcloud-zookeeper-1
+ DATA_DIR=/data
+ MYID_FILE=/data/myid
+ LOG4J_CONF=/conf/log4j-quiet.properties
+ DYNCONFIG=/data/zoo.cfg.dynamic
+ [[ example-solrcloud-zookeeper-1 =~ (.*)-([0-9]+)$ ]]
+ NAME=example-solrcloud-zookeeper
+ ORD=1
+ MYID=2
+ WRITE_CONFIGURATION=true
+ REGISTER_NODE=true
+ '[' -f /data/myid ']'
+ set +e
+ nslookup example-solrcloud-zookeeper-headless.default.svc.cluster.local
nslookup: can't resolve '(null)': Name does not resolve
Name: example-solrcloud-zookeeper-headless.default.svc.cluster.local
Address 1: 10.244.5.214 10-244-5-214.example-solrcloud-zookeeper-client.default.svc.cluster.local
+ [[ 0 -eq 1 ]]
+ set -e
+ set +e
++ zkConnectionString
++ set +e
++ nslookup example-solrcloud-zookeeper-client
++ [[ 0 -eq 1 ]]
++ set -e
++ echo example-solrcloud-zookeeper-client:2181
+ ZKURL=example-solrcloud-zookeeper-client:2181
+ set -e
++ java -Dlog4j.configuration=file:/conf/log4j-quiet.properties -jar /root/zu.jar get-all example-solrcloud-zookeeper-client:2181
Error: Unable to access jarfile /root/zu.jar
+ CONFIG=
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
Please re-open if you are still having issues. |
solr 15:53:01.84 my conf is below Please see all available overrides at https://github.com/bitnami/charts/tree/master/bitnami/solr/#installing-the-chartsolr.enabled -- Flag to control whether to deploy SOLRenabled: true solr.auth.enabled -- Enable or disable auth (if auth is disabled solr-init cant upload the configset/schema.xml for ckan)
solr.auth.adminUser -- The name of the solr admin user
solr.auth.adminPassword -- The password of the solr admin user
solr.collection -- the name of the collection created by solrsince we are creating one with solr-init this needs to be blankcollection: solr.collectionShards -- Number of shards for the SOLR collectioncollectionShards: solr.collectionReplicas -- Number of replicas for each SOLR shardcollectionReplicas: solr.fullnameOverride -- Name override for the SOLR deploymentfullnameOverride: *SolrName solr.replicaCount -- Number of SOLR instances in the clusterreplicaCount: 1 |
This is a long closed issue, and had to do with the solr operator crashing. You are using the bitnami Solr helm chart, completely unrelated to this project and an unofficial (and unsupported) way of running Solr on kubernetes. Please ask bitnami for help or start using the Solr Operator. |
Hi Apache/Solr-Operator
I am new on this please suggest which repository should I use and what
would be the tag version
…On Tue, 28 Nov 2023 at 11:10 AM, Houston Putman ***@***.***> wrote:
This is a long closed issue, and had to do with the *solr operator*
crashing. You are using the bitnami Solr helm chart, completely unrelated
to this project and an unofficial (and unsupported) way of running Solr on
kubernetes. Please ask bitnami for help or start using the Solr Operator.
—
Reply to this email directly, view it on GitHub
<#41 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AF4FNKMLDVGMPDP2VP4DFQ3YGYEHLAVCNFSM4JGUHYM2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBTGAYTOMBZG44Q>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Hi Team,
I tried with below
repository: apache/solr-operator
# solr.image.tag -- Tag for the SOLR image
tag: v0.8.0
and getting error
exec /scripts/setup.sh: no such file or directory
…On Tue, Nov 28, 2023 at 11:10 AM Houston Putman ***@***.***> wrote:
This is a long closed issue, and had to do with the *solr operator*
crashing. You are using the bitnami Solr helm chart, completely unrelated
to this project and an unofficial (and unsupported) way of running Solr on
kubernetes. Please ask bitnami for help or start using the Solr Operator.
—
Reply to this email directly, view it on GitHub
<#41 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AF4FNKMLDVGMPDP2VP4DFQ3YGYEHLAVCNFSM4JGUHYM2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBTGAYTOMBZG44Q>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
kaushal kumar chaman
|
Please go to https://solr.apache.org/operator/resources.html for more information. I don't know where that error is coming from, but it's not ours. We have other discussion channels, this is not the place to ask this, especially on an unrelated, long-closed issue. |
Describe the bug
While deploying the Solr operator on our K8s cluster environment, operator pod is failing with CrashLoopBackOff error.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
As per getting started steps, solr operator should be running once we deploy Zookeeper operator(optional) but solr operator is failing to start.
Environment (please complete the following information):
The text was updated successfully, but these errors were encountered: