Upgrade from 0.15.0 or fresh install dynatrace-operator 1.0.0: oneagent unable to start container process

[dhorner71]

dynatrace support ticket #306247

Describe the bug
upgrading from 0.15.0 to 1.0.0 with helm or fresh install produces oneagent daemonset pod warnings:

pod event:
OCI runtime exec failed: exec failed: unable to start container process: exec: "/usr/bin/watchdog-healthcheck64": stat /usr/bin/watchdog-healthcheck64: no such file or directory: unknown

pod status:
type: Ready
status: 'False'
reason: ContainersNotReady
message: 'containers with unready status: [dynatrace-oneagent]'

To Reproduce
Steps to reproduce the behavior:

1. either fresh install or upgrade from 0.15.0 helm chart

helm sets:
apiUrl = <our_customer_url>
apiToken =
dataIngestToken =
installDRD = true
webhook.hostNetwork = true
image = <private_repo>
customPullSecret = <private_repo_secret>
csidriver.enabled = false

dynakube 1.0.0:
apiVersion: dynatrace.com/v1beta1
kind: DynaKube
metadata:
annotations:
feature.dynatrace.com/automatic-kubernetes-api-monitoring: 'true'
name:
namespace: dynatrace
spec:
activeGate:
capabilities:
- routing
- kubernetes-monitoring
- dynatrace-api
- metrics-ingest
group: aws-eks
image: <private_repo>/docker/dynatrace/linux/activegate:latest
resources:
limits:
cpu: 1000m
memory: 1.5Gi
requests:
cpu: 500m
memory: 512Mi
apiUrl: https://<customer_id>.live.dynatrace.com/api
customPullSecret: <private_repo_secret_name>
networkZone: <custom_zone_name>
oneAgent:
classicFullStack:
args:
- '--set-host-group=<custom_host_group>'
env:
- name: ONEAGENT_ENABLE_VOLUME_STORAGE
value: 'false'
image: <private_repo>/docker/dynatrace/linux/oneagent:latest
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
skipCertCheck: true

dynakube 0.15.0:
apiVersion: dynatrace.com/v1beta1
kind: DynaKube
metadata:
annotations:
feature.dynatrace.com/automatic-kubernetes-api-monitoring: 'true'
name:
namespace: dynatrace
spec:
activeGate:
capabilities:
- routing
- kubernetes-monitoring
- dynatrace-api
- metrics-ingest
group: <custom_group>
image: <private_repo>/docker/dynatrace/linux/activegate:latest
resources:
limits:
cpu: 1000m
memory: 1.5Gi
requests:
cpu: 500m
memory: 512Mi
apiUrl: https://<customer_id>.live.dynatrace.com/api
customPullSecret: <private_repo_secret_name>
networkZone: <custom_zone>
oneAgent:
classicFullStack:
args:
- '--set-host-group=<custom_host_group>'
env:
- name: ONEAGENT_INSTALLER_SCRIPT_URL
value: https://<customer_id>.live.dynatrace.com/api/v1/deployment/installer/agent/unix/default/latest?arch=x86
- name: ONEAGENT_INSTALLER_DOWNLOAD_TOKEN
value:
- name: ONEAGENT_INSTALLER_SKIP_CERT_CHECK
value: 'true'
image: <private_repo>/docker/dynatrace/linux/oneagent:latest
oneAgentResources:
limits:
cpu: 300m
memory: 1.5Gi
requests:
cpu: 100m
memory: 512Mi
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
skipCertCheck: true

Expected behavior
after helm install, either upgrade or fresh, healthy pods reported with no warnings reported just like 0.15.0

Screenshots
n/a

Environment (please complete the following information):

• EKS/Kubernetes 1.27
• terraform provider hashicorp/helm v2.13.0
• helm v3.9.4
• dynatrace-operator 1.0.0 helm chart
• oneagent imageID oneagent@sha256:35b90670a961e5fa932cbee14a87b60d6e583bb9883bb05dfbbe7e180f7131b2
• activegate imageID activegate@sha256:2a1277acacfc81dc7e1a39bf24577c900026def26b740e61013549a0d17ddc0e

Additional context
-probably unimportant but with EKS autoscaler, version 1.0.0 does require an additional node (4 total) from 0.15.0 (3 total)
-could be related but even using latest tag on images and helm chart still produces dashboard warning: "The ActiveGate monitoring this Kubernetes cluster is outdated. Please make sure that all ActiveGates have [a minimum version of 1.279] to get the latest enhancements in Kubernetes monitoring."

• used these links as supporint info:https://docs.dynatrace.com/docs/whats-new/release-notes/dynatrace-operator/dto-fix-1-0-0#upgrade-from-dynatrace-operator-version-0-15-0
  https://docs.dynatrace.com/docs/whats-new/release-notes/dynatrace-operator/dto-fix-1-0-0

[github-actions]

Thank you for opening a Dynatrace Operator Issue. We've identified and tagged the issue as a "Support request".

Dynatrace responds to requests like these via Dynatrace ONE support rather than Github. This helps our team respond as quickly as possible using the support team's tools and procedures.

Thanks for your help!