[🐛 Bug]: Recording video, files with the session ID only in the name

[robertw-dev]

What happened?

We are running Selenium in AKS, with a PVC attached to the video container mounted in the record1 directory, where the uploader (internal) transfers the recordings. Most of the recordings in the record1 folder are valid, with correct names such as MultiTabNotEmptyTest_202409091415181_v4.mp4. However, from time to time, files are created that contain only the session ID in their names, e.g., 06f6fea74cc1b473b792956c2055d425.mp4. These files do not contain any recordings. We have set the variable SE_VIDEO_FILE_NAME_SUFFIX="false", so files with such names should not appear.

Command used to start Selenium Grid with Docker (or Kubernetes)

videoRecorder:
  enabled: true
  # Container name is set to resource specs
  name: video
  # imageRegistry: selenium
  # Image of video recorder
  imageName: video
  # Image of video recorder
  # imageTag: ffmpeg-7.0.2-20240830
  # Image pull policy (see https://kubernetes.io/docs/concepts/containers/images/#updating-images)
  imagePullPolicy: IfNotPresent
  # Image pull secret (see https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/)
  targetFolder: "/videos"
  uploader:
    enabled: true
    # Where to upload the video file e.g. remoteName://bucketName/path. Refer to destination syntax of rclone https://rclone.org/docs/
    destinationPrefix:
      /records1
    # What uploader to use. See .videRecorder.s3 for how to create a new one.
    name:
    configFileName: upload.conf
    entryPointFileName: upload.sh
    # For environment variables used in uploader which contains sensitive information, store in secret and refer envFrom
    # Set config for rclone via ENV var with format: RCLONE_CONFIG_ + name of remote + _ + name of config file option (make it all uppercase)
    secrets:
    #  RCLONE_CONFIG_S3_TYPE: "s3"
    #  RCLONE_CONFIG_S3_PROVIDER: "AWS"
    #  RCLONE_CONFIG_S3_ENV_AUTH: "true"
    #  RCLONE_CONFIG_S3_REGION: "ap-southeast-1"
    #  RCLONE_CONFIG_S3_LOCATION_CONSTRAINT: "ap-southeast-1"
    #  RCLONE_CONFIG_S3_ACL: "private"
    #  RCLONE_CONFIG_S3_ACCESS_KEY_ID: "xxx"
    #  RCLONE_CONFIG_S3_SECRET_ACCESS_KEY: "xxx"
    #  RCLONE_CONFIG_S3_NO_CHECK_BUCKET: "true"
    #  RCLONE_CONFIG_GS_TYPE: "s3"
    #  RCLONE_CONFIG_GS_PROVIDER: "GCS"
    #  RCLONE_CONFIG_GS_ENV_AUTH: "true"
    #  RCLONE_CONFIG_GS_REGION: "asia-southeast1"
    #  RCLONE_CONFIG_GS_LOCATION_CONSTRAINT: "asia-southeast1"
    #  RCLONE_CONFIG_GS_ACL: "private"
    #  RCLONE_CONFIG_GS_ACCESS_KEY_ID: "xxx"
    #  RCLONE_CONFIG_GS_SECRET_ACCESS_KEY: "xxx"
    #  RCLONE_CONFIG_GS_ENDPOINT: "https://storage.googleapis.com"
    #  RCLONE_CONFIG_GS_NO_CHECK_BUCKET: "true"  
  ports:
    - 9000
  resources:
    requests:
      memory: "100Mi"
      cpu: "50m"
    limits:
      memory: "800Mi"
      cpu: "100m"
  # SecurityContext for recorder container
  securityContext:
  extraEnvironmentVariables:
   - name: SE_VIDEO_FOLDER
     value: /videos
   - name: SE_VIDEO_FILE_NAME_SUFFIX
     value: "false"
   - name: SE_VIDEO_UPLOAD_ENABLED
     value: "true"
   - name: SE_VIDEO_INTERNAL_UPLOAD
     value: "true"
  # Custom environment variables by sourcing entire configMap, Secret, etc. for video recorder.
  extraEnvFrom:
  # - configMapRef:
  #   name: proxy-settings
  # - secretRef:
  #   name: mysecret
  # Wait for pod startup
  terminationGracePeriodSeconds: 30

  # Wait for pod startup
  startupProbe: {}
  #   httpGet:
  #     path: /
  #     port: 9000
  #   failureThreshold: 120
  # periodSeconds: 5

  # Liveness probe settings
  livenessProbe: {}
  # Define lifecycle events for video recorder
  lifecycle: {}

  # Custom video recorder back-end scripts (video.sh, video_ready.py, etc.) further by ConfigMap.
  # NOTE: For the mount point with the name "video", or "video-scripts", it will override the default. For other names, it will be appended.
  extraVolumeMounts:
  # - name: video-scripts
  #   mountPath: /opt/bin/video.sh
  #   subPath: custom_video.sh
  # - name: video-scripts
  #   mountPath: /opt/bin/video_ready.py
  #   subPath: video_ready.py
  - name: records1
    mountPath: /records1


  extraVolumes:
  # - name: video-scripts
  #   configMap:
  #     name: my-video-scripts-cm
  #     defaultMode: 0500
  # - name: video
  #   persistentVolumeClaim:
  #     claimName: video-pv-claim
  - name: records1
    persistentVolumeClaim:
      claimName: pvc-smb-vid1

Relevant log output

Defaulted container "selenium-grid1-selenium-chrome-node" out of: selenium-grid1-selenium-chrome-node, video
2024-09-09 15:52:26,244 INFO Included extra file "/etc/supervisor/conf.d/chrome-cleanup.conf" during parsing
2024-09-09 15:52:26,244 INFO Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing
2024-09-09 15:52:26,247 INFO RPC interface 'supervisor' initialized
2024-09-09 15:52:26,247 INFO supervisord started with pid 8
2024-09-09 15:52:27,248 INFO spawned: 'xvfb' with pid 9
2024-09-09 15:52:27,250 INFO spawned: 'vnc' with pid 10
2024-09-09 15:52:27,251 INFO spawned: 'novnc' with pid 11
2024-09-09 15:52:27,252 INFO spawned: 'selenium-node' with pid 12
2024-09-09 15:52:27,256 INFO success: selenium-node entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Appending Selenium option: --session-timeout 300
Appending Selenium option: --register-period 60
Appending Selenium option: --register-cycle 5
Appending Selenium option: --heartbeat-period 30
Appending Selenium option: --log-level INFO
Appending Selenium option: --http-logs false
Appending Selenium option: --structured-logs false
Generating Selenium Config
Setting up SE_NODE_HOST...
Appending Selenium option: --tracing false
Tracing is disabled
Selenium Grid Node configuration:
[events]
publish = "tcp://selenium-grid1-selenium-hub.default:4442"
subscribe = "tcp://selenium-grid1-selenium-hub.default:4443"

[server]
port = "5555"
[node]
grid-url = "http://selenium-grid1-selenium-hub.default:4444"
session-timeout = "300"
override-max-sessions = false
detect-drivers = false
drain-after-session-count = 1
max-sessions = 1

[[node.driver-configuration]]
display-name = "chrome"
stereotype = '{"browserName": "chrome", "browserVersion": "128.0", "platformName": "Linux", "goog:chromeOptions": {"binary": "/usr/bin/google-chrome"}, "se:containerName": "selenium-grid1-selenium-chrome-node-8l9mf-thrkm"}'
max-sessions = 1

Starting Selenium Grid Node...
2024-09-09 15:52:28,282 INFO success: xvfb entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-09 15:52:28,282 INFO success: vnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-09 15:52:28,282 INFO success: novnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
15:52:28.480 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
15:52:28.485 INFO [LoggingOptions.getTracer] - Using null tracer
15:52:28.523 INFO [UnboundZmqEventBus.<init>] - Connecting to tcp://selenium-grid1-selenium-hub.default:4442 and tcp://selenium-grid1-selenium-hub.default:4443
15:52:28.684 INFO [UnboundZmqEventBus.<init>] - Sockets created
15:52:29.686 INFO [UnboundZmqEventBus.<init>] - Event bus ready
15:52:29.799 INFO [NodeServer.createHandlers] - Reporting self as: http://10.244.124.85:5555
15:52:29.802 INFO [LoggingOptions.getTracer] - Using null tracer
15:52:29.812 INFO [NodeOptions.getSessionFactories] - Detected 1 available processors
15:52:29.902 INFO [NodeOptions.report] - Adding chrome for {"browserName": "chrome","browserVersion": "128.0","goog:chromeOptions": {"binary": "\u002fusr\u002fbin\u002fgoogle-chrome"},"platformName": "linux","se:containerName": "selenium-grid1-selenium-chrome-node-8l9mf-thrkm","se:noVncPort": 7900,"se:vncEnabled": true} 1 times
15:52:29.917 INFO [Node.<init>] - Binding additional locator mechanisms: relative
15:52:30.198 INFO [NodeServer$1.start] - Starting registration process for Node http://10.244.124.85:5555
15:52:30.199 INFO [NodeServer.execute] - Started Selenium node 4.24.0 (revision 748ffc9bc3): http://10.244.124.85:5555
15:52:30.210 INFO [NodeServer$1.lambda$start$1] - Sending registration event...
15:52:30.403 INFO [NodeServer.lambda$createHandlers$2] - Node has been added
2024-09-09T15:52:30CEST [Probe.Startup] - Node responds the ID: 963fac4c-dfcf-4747-b411-cd96bdb68638 with status: UP
2024-09-09T15:52:31CEST [Probe.Startup] - Grid responds a matched Node ID: 963fac4c-dfcf-4747-b411-cd96bdb68638
2024-09-09T15:52:31CEST [Probe.Startup] - Node ID: 963fac4c-dfcf-4747-b411-cd96bdb68638 is found in the Grid. Node is ready.
15:52:32.186 INFO [LocalNode.checkSessionCount] - Draining Node, configured sessions value (1) has been reached.
15:52:32.192 INFO [LocalNode.newSession] - Session created by the Node. Id: 87905f84629238fb646bf56b11b7080b, Caps: Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 128.0.6613.113, chrome: {chromedriverVersion: 128.0.6613.86 (3045ed680fda..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:42985}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://selenium-grid1-seleniu..., se:cdpVersion: 128.0.6613.113, se:containerName: selenium-grid1-selenium-chr..., se:name: CheckCompanyAfterDeleteMult..., se:noVncPort: 7900, se:recordVideo: true, se:screenResolution: 1920x1080, se:vnc: ws://selenium-grid1-seleniu..., se:vncEnabled: true, se:vncLocalAddress: ws://10.244.124.85:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: ignore, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
15:57:59.996 INFO [LocalNode.stopTimedOutSession] - Session id 87905f84629238fb646bf56b11b7080b timed out, stopping...
15:58:00.091 INFO [SessionSlot.stop] - Stopping session 87905f84629238fb646bf56b11b7080b
15:58:00.092 INFO [LocalNode.stopTimedOutSession] - Node draining complete!
15:58:01.115 INFO [NodeServer.lambda$createHandlers$3] - Shutting down
2024-09-09 15:58:01,504 INFO exited: selenium-node (exit status 0; expected)
2024-09-09 15:58:01,504 WARN received SIGINT indicating exit request
2024-09-09 15:58:01,505 INFO waiting for xvfb, vnc, novnc to die
2024-09-09 15:58:04,508 WARN stopped: novnc (terminated by SIGTERM)
2024-09-09 15:58:04,508 INFO waiting for xvfb, vnc to die
2024-09-09 15:58:05,509 WARN stopped: vnc (terminated by SIGTERM)
2024-09-09 15:58:06,510 WARN stopped: xvfb (terminated by SIGTERM)

Operating System

Azure Kubernetes Service

Docker Selenium version (image tag)

4.24.0-20240830

Selenium Grid chart version (chart version)

0.35.0

[github-actions]

@robertw-dev, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[VietND96]

So, from binding, your video file name is set to capabilities se:name or se:videoName?

[robertw-dev]

@VietND96 Thank you for the response.
Deeper analysis, we see that if se:recordVideo is set to true, the file name is displayed correctly, for example, MultiTabNotEmptyTest_202409091415181_v4.mp4. If it is set to false, a file with an ID in its name appears, such as 06f6fea74cc1b473b792956c2055d425.mp4. The se:name is always set."

[VietND96]

Ok sure, based on your info provided, I think I can find the problem, will give the fix if possible.

[VietND96]

@robertw-dev, if you have time, can you replace video image tag nightly to try and confirm?
I also add a test, without caps se:recordVideo from binding, the file name could be seen correctly as below

[robertw-dev]

@VietND96 Thank you, I will check it as soon as possible.

[robertw-dev]

@VietND96 Unfortunately, files with the session ID in the name still appear when the se:recordVideo is set to false

[VietND96]

As implementation, when se:recordVideo is true or not set, the recording will be started. However, when se:recordVideo is false it means recording will not be triggered.
Do you have any logs of the video container to refer to what was happening for a recording session (same as my above comment)?

[robertw-dev]

Sorry for the delay. I've attached a sample Chrome-node log and a part of the hub log. There's a time difference between them since our time zone is set in Chrome-node but not in the hub.
hub.txt
selenium-grid-selenium-chrome-node.txt

[VietND96]

I saw this caps in Node logs

12:56:23.572 INFO [LocalNode.newSession] - Session created by the Node. Id: 1af7c0eff98ba8e391c159468beae095, Caps: Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 128.0.6613.113, chrome: {chromedriverVersion: 128.0.6613.86 (3045ed680fda..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:42321}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://selenium-grid-selenium..., se:cdpVersion: 128.0.6613.113, se:containerName: selenium-grid-selenium-chro..., se:name: FillAndApplicationPPKAndRes..., se:noVncPort: 7900, se:recordVideo: false, se:screenResolution: 1920x1080, se:vnc: ws://selenium-grid-selenium..., se:vncEnabled: true, se:vncLocalAddress: ws://10.244.12.2:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: ignore, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}

Any session with this cap se:recordVideo: false, recorder will be aware that not trigger record for this session.

[robertw-dev]

OK, but this is the point of the problem, recording files are appearing. These files have names with session IDs and are several dozen kilobytes in size. They look like they are corrupted and cannot be played.