CrashLoopBackOff on the my-otel-demo-cartservice running on OpenShift

[Tiagodge]

Bug Report

I am using the version 730:
#730

Symptom

The cart service is not working, I can access the website, but cannot add items to the cart

I cleaned up everything in the cluster and apply the new version, but now, with the new version I can navigate on the application, open Jaeger and everything, but is failing when I try to add items to the cart.

the error:
Connecting to Redis: my-otel-demo-redis:6379,ssl=false,allowAdmin=true,abortConnect=false
2 Wasn't able to connect to redis
3 Unhandled exception. System.ApplicationException: Wasn't able to connect to redis
4 at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 89
5 at cartservice.cartstore.RedisCartStore.InitializeAsync() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 62
6 at Program.

$(String[] args) in /usr/src/app/src/Program.cs:line 40

But as you can see here, Redis is accepting connection:

1:C 07 Feb 2023 18:34:44.280 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2 1:C 07 Feb 2023 18:34:44.280 # Redis version=7.0.8, bits=64, commit=00000000, modified=0, pid=1, just started
3 1:C 07 Feb 2023 18:34:44.280 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
4 1:M 07 Feb 2023 18:34:44.286 * monotonic clock: POSIX clock_gettime
5 1:M 07 Feb 2023 18:34:44.291 * Running mode=standalone, port=6379.
6 1:M 07 Feb 2023 18:34:44.292 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
7 1:M 07 Feb 2023 18:34:44.292 # Server initialized
8 1:M 07 Feb 2023 18:34:44.292 # WARNING Your system is configured to use the 'xen' clocksource which might lead to degraded performance. Check the result of the [slow-clocksource] system check: run 'redis-server --check-system' to check if the system's clocksource isn't degrading performance.
9 1:M 07 Feb 2023 18:34:44.295 * Ready to accept connections

[julianocosta89]

@Tiagodge thanks for reporting this.
How are you deploying the sample? (Docker or K8s)
Is the redis container running?
Which version of the demo are you using? (you have referred to 730, but this is from the docs, and not from the demo)

[Tiagodge]

I deployed via helm chart using exactly this command:

helm install my-otel-demo open-telemetry/opentelemetry-demo

Yes, Redis pod is running and accepting connections as we can see on the pod log:

9 1:M 07 Feb 2023 18:34:44.295 * Ready to accept connections

This is the version I am using:

apiVersion: v2
appVersion: 1.3.0
dependencies:

• condition: observability.otelcol.enabled
  name: opentelemetry-collector
  repository: https://open-telemetry.github.io/opentelemetry-helm-charts
  version: 0.45.0
• condition: observability.jaeger.enabled
  name: jaeger
  repository: https://jaegertracing.github.io/helm-charts
  version: 0.66.1
• condition: observability.prometheus.enabled
  name: prometheus
  repository: https://prometheus-community.github.io/helm-charts
  version: 19.3.1
• condition: observability.grafana.enabled
  name: grafana
  repository: https://grafana.github.io/helm-charts
  version: 6.50.0
  description: opentelemetry demo helm chart
  home: https://opentelemetry.io/
  icon: https://opentelemetry.io/img/logos/opentelemetry-logo-nav.png
  maintainers:
• name: dmitryax
• name: puckpuck
• name: tylerhelmuth
  name: opentelemetry-demo
  sources:
• https://github.com/open-telemetry/opentelemetry-demo
  type: application
  version: 0.18.0

[julianocosta89]

@puckpuck and @TylerHelmuth any ideas here?

The Cart pod is in CrashLoopBackOff and can't seem to be able to connect to Redis, even though Redis is up and running.

[puckpuck]

These two warnings are intriguing. I don't see them on my side in any of the deployments I have running (also using Redis 7.0.10 now instead of 7.0.8)

6 1:M 07 Feb 2023 18:34:44.292 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
7 1:M 07 Feb 2023 18:34:44.292 # Server initialized
8 1:M 07 Feb 2023 18:34:44.292 # WARNING Your system is configured to use the 'xen' clocksource which might lead to degraded performance. Check the result of the [slow-clocksource] system check: run 'redis-server --check-system' to

Can you try reinstalling again? Redis is up to 7.0.10 now, and though I doubt that would be the cause, I want to ensure we are all using the same version.

[puckpuck]

The Helm chart has been updated to work with OpenShift. There are a few special instructions you need to follow in the Helm chart's readme for OpenShift installation.