Failed to start server: rename file encounter error: Resource busy

[kinoute]

Search before asking

• I had searched in the issues and found no similar issues.

Version

2.7.0

Minimal reproduce step

Kubernetes manifest:

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kvrocks
  namespace: ns-svc
spec:
  selector:
    matchLabels:
      run: kvrocks
  serviceName: "kvrocks"
  replicas: 1
  template:
    metadata:
      labels:
        run: kvrocks
    spec:
      containers:
        - name: kvrocks
          image: apache/kvrocks:2.7.0
          volumeMounts:
            - name: kvrocks
              mountPath: /data
            - name: kvrocks-config
              mountPath: /var/lib/kvrocks/kvrocks.conf
              subPath: kvrocks.conf
          ports:
            - containerPort: 6379
          resources:
            requests:
              cpu: 1
              memory: 2Gi
            limits:
              memory: 2Gi
      volumes:
        - name: kvrocks
          persistentVolumeClaim:
            claimName: kvrocks
        - name: kvrocks-config
          secret:
            secretName: kvrocks-config

What did you expect to see?

Something like this (here running 2.5.0):

WARNING: 'repl-namespace-enabled' at line 62 is not a valid configuration key.
WARNING: 'json-max-nesting-depth' at line 318 is not a valid configuration key.
WARNING: 'json-storage-format' at line 325 is not a valid configuration key.
WARNING: 'rocksdb.block_cache_size' at line 565 is not a valid configuration key.
I20240305 11:56:15.776150     1 main.cc:328] kvrocks version 2.5.0
I20240305 11:56:15.910107     1 storage.cc:322] [storage] Success to load the data from disk: 45 ms
I20240305 11:56:15.911934     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.911958     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.912247     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.912269     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.912484     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.912501     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.912691     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.912716     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.912907     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.912925     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.913108     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.913156     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.913347     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.913365     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.913568     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.913585     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:56:15.913951     1 worker.cc:519] [worker] Thread #139894731045680 started
I20240305 11:56:15.914000     1 worker.cc:519] [worker] Thread #139894730869552 started
I20240305 11:56:15.914044     1 worker.cc:519] [worker] Thread #139894730693424 started
I20240305 11:56:15.914089     1 worker.cc:519] [worker] Thread #139894730517296 started
I20240305 11:56:15.914139     1 worker.cc:519] [worker] Thread #139894730341168 started
I20240305 11:56:15.914178     1 worker.cc:519] [worker] Thread #139894730165040 started
I20240305 11:56:15.914217     1 worker.cc:519] [worker] Thread #139894729988912 started

What did you see instead?

I20240305 11:58:19.403860     1 main.cc:141] kvrocks version 2.7.0 (commit 33b49591)
I20240305 11:58:19.553138     1 storage.cc:335] [storage] Success to load the data from disk: 44 ms
I20240305 11:58:19.555069     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:58:19.555099     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:58:19.555413     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:58:19.555435     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:58:19.555653     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:58:19.555670     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:58:19.555874     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:58:19.555892     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:58:19.556083     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:58:19.556099     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:58:19.556288     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:58:19.556304     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
E20240305 11:58:19.558729     1 main.cc:180] Failed to start server: rename file encounter error: Resource busy
I20240305 11:58:19.556530     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:58:19.556561     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:58:19.556756     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
I20240305 11:58:19.556772     1 worker.cc:72] [worker] Listening on: 0.0.0.0:6379
Stream closed EOF for ns-svc/kvrocks-0 (kvrocks-collect)

Anything Else?

We are trying to update our KvRocks instances from 2.5.0 to 2.7.0. Unfortunately, we encounter the same error over and over again, even on fresh installs.

I found issues talking about this error where it seems to be about namespaces (we didn't change that in the conf) and/or wrong lock on the conf file and a restart could be fix the problem, it doesn't.

It runs perfectly on 2.5.0. I looked at the Dockerfile and I don't see any major changes that could lead to this error. I also tried to override the entrypoint/command of the container, put the conf file elsewhere and call it with "-c", no chances.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[git-hulk]

Hi @kinoute

Thanks for your feedback. The root cause is the single mount file kvrocks.conf which is unable to be written, you can have a try at: config rewrite in 2.5.0 and it should return the same error.

And this start error was introduced in #1776 which would try to rewrite the configuration after loading the namespaces. Maybe we should turn this error into a warning when starting the server.

You can workaround this issue by mounting the whole dir where kvrocks.conf exists to /var/lib/kvrocks without the subPath kvrocks.conf.

[kinoute]

Thanks for the quick answer @git-hulk ! I was able to fix it with your workaround. I needed to create an initContainer for this as mounting a secret as volume is now read-only since Kubernetes 1.9.

For those who are interested, here is a working manifest:

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kvrocks
  namespace: sahar-svc
spec:
  selector:
    matchLabels:
      run: kvrocks
  serviceName: "kvrocks"
  replicas: 1
  template:
    metadata:
      labels:
        app: kvrocks
        run: kvrocks
    spec:
      initContainers:
        - name: copy-kvrocks-conf
          image: busybox
          command: ["sh", "-c", "cp /tmp/kvrocks.conf /var/lib/kvrocks/"]
          volumeMounts:
            - name: kvrocks-config
              mountPath: /tmp/kvrocks.conf
              subPath: kvrocks.conf
            - name: kvrocks-folder
              mountPath: /var/lib/kvrocks
      containers:
        - name: kvrocks
          image: apache/kvrocks:2.7.0
          volumeMounts:
            - name: kvrocks-folder
              mountPath: /var/lib/kvrocks
            - name: kvrocks
              mountPath: /data
          ports:
            - containerPort: 6379
          resources:
            requests:
              cpu: 1
              memory: 2Gi
            limits:
              memory: 2Gi
      volumes:
        - name: kvrocks-folder
          emptyDir: {}
        - name: kvrocks
          persistentVolumeClaim:
            claimName: kvrocks
        - name: kvrocks-config
          secret:
            secretName: kvrocks-config

[git-hulk]

@kinoute Thanks for your sharing. I would refer them here if other folks also suffer from this issue.