Unable to bootstrap OSS 3.68.0-04 (but 3.67.1 worked)

[sprior]

In my homelab setup I’m installing Nexus OSS using the following statefulset.yaml and using an NFS share (with userid squashing) for the data volume. What I’ve found is that I attempt a fresh install no data using sonatype/nexus3:latest (currently OSS 3.68.0-04) the install falls into an endless loop of errors like:

2024-05-11 19:44:02,579-0400 INFO [ForkJoinPool.commonPool-worker-3] *SYSTEM com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - Storage ‘plocal:/nexus-data/db/analytics’ is opened under OrientDB distribution : 2.2.37 (build a7541e7ceeabf592dd9a7b2928b6c023cbc73193, branch 2.2.x)
2024-05-11 19:44:02,615-0400 INFO [ForkJoinPool.commonPool-worker-0] *SYSTEM com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - Storage ‘plocal:/nexus-data/db/config’ is opened under OrientDB distribution : 2.2.37 (build a7541e7ceeabf592dd9a7b2928b6c023cbc73193, branch 2.2.x)
2024-05-11 19:44:02,660-0400 ERROR [ForkJoinPool.commonPool-worker-2] *SYSTEM com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - Exception 5F52F931 in storage plocal:/nexus-data/db/component: 2.2.37 (build a7541e7ceeabf592dd9a7b2928b6c023cbc73193, branch 2.2.x)
com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage ‘/nexus-data/db/component’ with mode=rw
DB name=“component”
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:323)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:259)
at org.sonatype.nexus.orient.DatabaseManagerSupport.connect(DatabaseManagerSupport.java:178)
at org.sonatype.nexus.orient.DatabaseManagerSupport.createInstance(DatabaseManagerSupport.java:312)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at org.sonatype.nexus.orient.DatabaseManagerSupport.instance(DatabaseManagerSupport.java:289)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: com.orientechnologies.orient.core.exception.OStorageException: File with name ‘ouser.pcl’ does not exist in storage ‘component’
DB name=“component”
at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.loadFile(OWOWCache.java:475)
at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.openFile(ODurableComponent.java:180)
at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.open(OPaginatedCluster.java:217)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.openClusters(OAbstractPaginatedStorage.java:387)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:290)
… 14 common frames omitted

But if I use the sonatype/nexus3:3.67.1 instead I can get through an install then shut down and switch to the latest image and the system works fine.

So it appears that the latest docker image has a regression error that it can’t bootstrap install itself. Has anyone else experienced anything similar?

Here’s the statefulset.yaml that I came up with for the install:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nexus
  namespace: services
  labels:
    app.kubernetes.io/name: nexus
spec:
  serviceName: "nexus"
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: nexus
  template:
    metadata:
      labels:
        app.kubernetes.io/name: nexus
    spec:
      securityContext:
        fsGroup: 200
        fsGroupChangePolicy: OnRootMismatch
      nodeSelector:
        kubernetes.io/arch: amd64
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              preference:
                matchExpressions:
                  - key: wired
                    operator: In
                    values:
                      - "true"
      containers:
        - name: nexus
          #image: sonatype/nexus3:3.67.1
          image: sonatype/nexus3:latest
          securityContext:
            runAsUser: 200
          imagePullPolicy: Always
          livenessProbe:
            httpGet:
              path: /
              port: 8081
              scheme: HTTP
            initialDelaySeconds: 180
            timeoutSeconds: 1
          readinessProbe:
            httpGet:
              path: /
              port: 8081
              scheme: HTTP
            initialDelaySeconds: 20
            timeoutSeconds: 1
          env:
            - name: TZ
              value: "America/New_York"
            - name: NEXUS_SECURITY_RANDOMPASSWORD
              value: "false"
          resources:
            limits:
              memory: "8Gi"
              cpu: "4"
            requests:
              memory: "4Gi"
              cpu: "4"
          volumeMounts:
            - name: nexus-data
              mountPath: /nexus-data
      volumes:
        - name: nexus-data
          nfs:
            server: my.nas.ip
            path: /volume1/kubernetes/nexus-data

[nblair]

Hi @sprior thanks for opening an issue. One of the flaws of the embedded OrientDB database is that it does not mix well with kubernetes deployments. The container orchestration may shut down the container in such a way that the database is not flushed completely to disk. Sometimes this works ok, but other times it results in corruption. That corruption may sit silently, or come to the surface during an upgrade.
There aren't any changes to the OrientDB dependency in the most recent release. I wonder if there is a task running during the first-boot-after-upgrade that is surfacing some previously silent corruption.

More of the log file prior to the error would be necessary to answer that question. Nexus Repository Pro customers can interact with our support team, where they can get more specific analysis. I'll add the help wanted label hear to see if the community can pitch in with ideas.

[sprior]

@nblair I was a bit dismayed that after seeing the OrientDB being referred to as legacy/frowned upon to see that the ability to use an external database is a Pro only feature. Can you confirm this is true and that OSS users cannot use a PostgreSQL database instead? That seemed like excessive crippling the OSS version to me.

[nblair]

Thanks for the feedback @sprior - I understand this change can be disruptive. The version of OrientDB we have been reliant on for a long time achieved final sunset late last year, and there is no path forward to similar technology by the vendor.

While we don't have a public facing roadmap, I will share early here that July's 3.70.0 release will provide an alternate embedded database that can be used with OSS deployments. Our provided DB migrator will help you move existing content, and new installs will start with the new embedded database by default.

Even with the new embedded database, it won't blend well with kubernetes deployments, as the same risk of corruption can occur from premature shutdown. You may want to consider provisioning a deployment outside of kubernetes, or consider upgrading to Pro to externalize the database.

[sprior]

@nblair As I mentioned this deployment is in a home lab situation for educational purposes, not really a candidate for a Pro license. I did switch to a Docker based deployment which seems to be doing better, but I had hoped to phase out new Docker deployments in favor of Kubernetes (actually k3s). I still think that making external DB support a Pro feature is a bit severe.

[github-actions]

This issue is stale because it has been open for 60 days with no activity.