Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duplicate ffmpeg processes starting for sources #894

Closed
troykelly opened this issue Jan 21, 2024 · 5 comments
Closed

Duplicate ffmpeg processes starting for sources #894

troykelly opened this issue Jan 21, 2024 · 5 comments
Assignees
Labels
bug Something isn't working
Milestone

Comments

@troykelly
Copy link

troykelly commented Jan 21, 2024

Versions Impacted:

  • 1.8.4 as bundled with frigate ghcr.io/blakeblackshear/frigate
  • 1.8.5 downloaded from releases

For some reason, on restart of frigate (and thus restart of go2rtc), some cameras are getting multiple ffmpeg processes started for them. SIGHUP'ing the processes makes them go away, but for a little while after restarting, go2rtc will keep creating more and more processes. Typically 8 - but not always.

An example of relevant processes is below. In this example, the livingptz.security camera has multiple ffmpeg processes all publishing to 7c80f440f9ccc4ac8b1073e541a1e9d2

(I've changed all the publish tokens for security, but retained the uniqueness and grouping)

I'm using the script (linked below) to clean up the multiple processes, and after some time - everything seems to settle down. New processes stop getting created and everything continues as it should.
https://gist.github.com/troykelly/a23ca304be5cd52961544673ba67a1e0

During the multiple ffmpeg process phase after restart, everything does appear to be working correctly - The only reason I am aware is because the multiple instances for each camera is causing the encoder on the GPU to max out.

-hide_banner -v error -allowed_media_types video+audio -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -rtsp_flags prefer_tcp -i rtsp://[REDATED_USER]:[REDACTED_PASS]@front.intercom.example.invalid -vf hwdownload,format=nv12,drawtext=text='%{localtime}':x=2:y=20:fontsize=28:fontcolor=white:box=1:boxcolor=black:boxborderw=2:fontfile='/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf',hwupload -map 0:v:0? -c:v:0 hevc_nvenc -g 50 -no-scenecut 1 -bf 0 -profile:v:0 main -level:v:0 auto -preset:v:0 p2 -tune:v:0 ull -tag:v:0 hvc1 -map 0:a:0? -c:a:0 libopus -application:a:0 lowdelay -frame_duration 20 -min_comp 0 -map 0:a:0? -c:a:1 aac -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/bc866768c7d70929c30bd5323cec52b0	2898
-hide_banner -v error -allowed_media_types video+audio -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -rtsp_flags prefer_tcp -i rtsp://[REDATED_USER]:[REDACTED_PASS]@foyer.security.example.invalid -map 0:v:0? -c:v:0 hevc_nvenc -g 50 -no-scenecut 1 -bf 0 -profile:v:0 main -level:v:0 auto -preset:v:0 p2 -tune:v:0 ull -tag:v:0 hvc1 -map 0:a:0? -c:a:0 libopus -application:a:0 lowdelay -frame_duration 20 -min_comp 0 -map 0:a:0? -c:a:1 aac -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/47e8d411ff66b22ee9166aa7144bb670	2931
-hide_banner -v error -allowed_media_types video+audio -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -rtsp_flags prefer_tcp -i rtsp://[REDATED_USER]:[REDACTED_PASS]@l2stairs.security.example.invalid -map 0:v:0? -c:v:0 hevc_nvenc -g 50 -no-scenecut 1 -bf 0 -profile:v:0 main -level:v:0 auto -preset:v:0 p2 -tune:v:0 ull -tag:v:0 hvc1 -map 0:a:0? -c:a:0 libopus -application:a:0 lowdelay -frame_duration 20 -min_comp 0 -map 0:a:0? -c:a:1 aac -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/94835a161463e41816704dc35cc515c5	3469
-hide_banner -v error -allowed_media_types video+audio -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -rtsp_flags prefer_tcp -i rtsp://[REDATED_USER]:[REDACTED_PASS]@livingptz.security.example.invalid:80/videoMain -map 0:v:0? -c:v:0 hevc_nvenc -g 50 -no-scenecut 1 -bf 0 -profile:v:0 main -level:v:0 auto -preset:v:0 p2 -tune:v:0 ull -tag:v:0 hvc1 -map 0:a:0? -c:a:0 libopus -application:a:0 lowdelay -frame_duration 20 -min_comp 0 -map 0:a:0? -c:a:1 aac -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/7c80f440f9ccc4ac8b1073e541a1e9d2	14798
-hide_banner -v error -allowed_media_types video+audio -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -rtsp_flags prefer_tcp -i rtsp://[REDATED_USER]:[REDACTED_PASS]@livingptz.security.example.invalid:80/videoMain -map 0:v:0? -c:v:0 hevc_nvenc -g 50 -no-scenecut 1 -bf 0 -profile:v:0 main -level:v:0 auto -preset:v:0 p2 -tune:v:0 ull -tag:v:0 hvc1 -map 0:a:0? -c:a:0 libopus -application:a:0 lowdelay -frame_duration 20 -min_comp 0 -map 0:a:0? -c:a:1 aac -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/7c80f440f9ccc4ac8b1073e541a1e9d2	15079
-hide_banner -v error -allowed_media_types video+audio -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -rtsp_flags prefer_tcp -i rtsp://[REDATED_USER]:[REDACTED_PASS]@livingptz.security.example.invalid:80/videoMain -map 0:v:0? -c:v:0 hevc_nvenc -g 50 -no-scenecut 1 -bf 0 -profile:v:0 main -level:v:0 auto -preset:v:0 p2 -tune:v:0 ull -tag:v:0 hvc1 -map 0:a:0? -c:a:0 libopus -application:a:0 lowdelay -frame_duration 20 -min_comp 0 -map 0:a:0? -c:a:1 aac -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/7c80f440f9ccc4ac8b1073e541a1e9d2	15367
-hide_banner -v error -allowed_media_types video+audio -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags low_delay -timeout 5000000 -rtsp_flags prefer_tcp -i rtsp://[REDATED_USER]:[REDACTED_PASS]@storesptz.security.example.invalid:80/videoMain -map 0:v:0? -c:v:0 hevc_nvenc -g 50 -no-scenecut 1 -bf 0 -profile:v:0 main -level:v:0 auto -preset:v:0 p2 -tune:v:0 ull -tag:v:0 hvc1 -map 0:a:0? -c:a:0 libopus -application:a:0 lowdelay -frame_duration 20 -min_comp 0 -map 0:a:0? -c:a:1 aac -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/879767282d0a2966bf0d6369ff97d285	10686

Config

api:
  origin: '*'
ffmpeg:
  cudah264video: -map 0:v:0? -c:v:0 h264_nvenc -g 50 -bf 0 -profile:v:0 high -level:v:0
    auto -preset:v:0 p2 -tune:v:1 ull
  cudahevcvideo: -map 0:v:0? -c:v:0 hevc_nvenc -g 50 -no-scenecut 1 -bf 0 -profile:v:0
    main -level:v:0 auto -preset:v:0 p2 -tune:v:0 ull -tag:v:0 hvc1
  cudartspinput: -hwaccel cuda -hwaccel_output_format cuda -fflags nobuffer -flags
    low_delay -timeout 5000000 -rtsp_flags prefer_tcp -i {input}
  h264/rk: -c:v h264_rkmpp_encoder -g 50 -bf 0
  h265/rk: -c:v hevc_rkmpp_encoder -g 50 -bf 0
  hevcboxcaminput: -hwaccel cuda -hwaccel_output_format cuda -r 20 -rtsp_transport
    tcp -i {input}
  hevccamout: -c:v hevc_nvenc
  hevcinput: -loglevel info -hwaccel cuda -hwaccel_output_format cuda -rtsp_transport
    tcp -i {input}
  hevcoutput: -map_metadata -1 -c:v hevc_nvenc -g 50 -profile:v main -level:v auto
    -preset:v p2 -tune:v ull -multipass disabled -movflags +faststart -zerolatency
    1 -tag:v hvc1
  hlshevcoutput: -map_metadata -1 -c:v hevc_nvenc -g 50 -profile:v main -level:v auto
    -preset:v p2 -tune:v ull -multipass disabled -movflags +faststart -zerolatency
    1 -tag:v hvc1
  hlsrealtime: -hwaccel cuda -hwaccel_output_format cuda -extra_hw_frames 2 -fflags
    nobuffer -flags low_delay -rtbufsize 1M -fflags +igndts -fflags +nofillin -fflags
    +genpts -avoid_negative_ts +make_zero -re -i {input}
hass:
  config: /config
log:
  api: info
  exec: info
  format: text
  level: info
  ngrok: info
  rtsp: info
  streams: info
  webrtc: info
rtsp:
  default_query: mp4
streams:
  abc:
  - ffmpeg:https://[URL_TO_ABC_NEWS24]#audio=opus#audio=aac#input=hlsrealtime#video=cudahevcvideo
  abch264:
  - ffmpeg:abc#audio=opus#audio=aac#video=h264#hardware
  birdseye: exec:ffmpeg -hide_banner -f rawvideo -pix_fmt yuv420p -video_size 1920x1080
    -r 10 -i /tmp/cache/birdseye -c:v h264_nvenc -g 50 -profile:v high -level:v auto
    -preset:v p2 -tune:v ll -rtsp_transport tcp -f rtsp {output}
  building_basement:
  - ffmpeg:rtsp://[REDATED_USER]:[REDACTED_PASS]@nvr001.building.example.invalid/unicast/c4/s0/live#video=copy
  building_ground:
  - ffmpeg:rtsp://[REDATED_USER]:[REDACTED_PASS]@nvr001.building.example.invalid/unicast/c2/s0/live#video=copy
  building_intercom:
  - ffmpeg:rtsp://[REDATED_USER]:[REDACTED_PASS]@front.intercom.example.invalid#audio=opus#audio=aac#input=cudartspinput#video=cudahevcvideo#raw=-vf
    hwdownload,format=nv12,drawtext=text='%{localtime}':x=2:y=20:fontsize=28:fontcolor=white:box=1:boxcolor=black:boxborderw=2:fontfile='/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf',hwupload
  building_intercomh264:
  - ffmpeg:building_intercom#audio=opus#audio=aac#video=h264#hardware
  building_level_1:
  - ffmpeg:rtsp://[REDATED_USER]:[REDACTED_PASS]@nvr001.building.example.invalid/unicast/c1/s0/live#video=copy
  building_street:
  - ffmpeg:rtsp://[REDATED_USER]:[REDACTED_PASS]@nvr001.building.example.invalid/unicast/c3/s0/live#video=copy
  cnn:
  - ffmpeg:https://[URL_TO_SOURCE_M3U8]#audio=opus#audio=aac#input=hlsrealtime#video=cudahevcvideo
  cnnh264:
  - ffmpeg:cnn#audio=opus#audio=aac#video=h264#hardware
  foyer:
  - ffmpeg:rtsp://[REDATED_USER]:[REDACTED_PASS]@foyer.security.example.invalid#input=cudartspinput#video=cudahevcvideo#audio=opus#audio=aac
  l2stairs:
  - ffmpeg:rtsp://[REDATED_USER]:[REDACTED_PASS]@l2stairs.security.example.invalid#input=cudartspinput#video=cudahevcvideo#audio=opus#audio=aac
  livingptz:
  - ffmpeg:rtsp://[REDATED_USER]:[REDACTED_PASS]@livingptz.security.example.invalid:80/videoMain#input=cudartspinput#video=cudahevcvideo#audio=opus#audio=aac
  masterbed:
  - ffmpeg:rtsp://[REDATED_USER]:[REDACTED_PASS]@masterbed.security.example.invalid/#video=copy#audio=opus#audio=aac
  officedesks:
  - ffmpeg:rtsp://[REDATED_USER]:[REDACTED_PASS]@officedesks.security.example.invalid/0#video=copy
  officeentry:
  - ffmpeg:rtsp://[REDATED_USER]:[REDACTED_PASS]@officeentry.security.example.invalid/0#video=copy
  storesptz:
  - ffmpeg:rtsp://[REDATED_USER]:[REDACTED_PASS]@storesptz.security.example.invalid:80/videoMain#input=cudartspinput#video=cudahevcvideo#audio=opus#audio=aac
  webcam_antsoutworld:
  - ffmpeg:rtsp://[REDATED_USER]:[REDACTED_PASS]@cam003.security.example.invalid/live/main#video=copy
  webcam_antsoutworldh264:
  - ffmpeg:webcam_antsoutworld#video=h264#hardware
  webcam_iss:
  - echo:yt-dlp --print "ffmpeg:%(urls)s#input=hlsrealtime#video=cudahevcvideo#audio=opus#hardware"
    https://www.youtube.com/watch?v=jPTD2gnZFUw
  webcam_iss2:
  - echo:yt-dlp --print "ffmpeg:%(urls)s#input=hlsrealtime#video=cudahevcvideo#audio=opus#hardware"
    https://www.youtube.com/watch?v=P9C25Un7xaM
  webcam_iss2h264:
  - ffmpeg:webcam_iss2#audio=opus#audio=aac#video=h264#hardware
  webcam_issh264:
  - ffmpeg:webcam_iss#audio=opus#audio=aac#video=h264#hardware
  webcam_sydney:
  - ffmpeg:rtsp://[REDATED_USER]:[REDACTED_PASS]@cam005.security.example.invalid/h264_stream#video=copy
  webcam_sydney2:
  - ffmpeg:rtsp://[REDATED_USER]:[REDACTED_PASS]@cam004.security.example.invalid/live/main#video=copy
  webcam_sydney2h264:
  - ffmpeg:webcam_sydney2#video=h264#hardware
  webcam_sydney3:
  - echo:yt-dlp --print "ffmpeg:%(urls)s#input=hlsrealtime#video=cudahevcvideo#audio=opus#hardware"
    https://www.youtube.com/watch?v=KPrrWB1eo1I
  webcam_sydney3h264:
  - ffmpeg:webcam_sydney3#audio=opus#audio=aac#video=h264#hardware
  webcam_sydney4:
  - echo:yt-dlp --print "ffmpeg:%(urls)s#input=hlsrealtime#video=cudahevcvideo#audio=opus#hardware"
    https://www.youtube.com/watch?v=sWWCy2WCjRA
  webcam_sydney4h264:
  - ffmpeg:webcam_sydney4#audio=opus#audio=aac#video=h264#hardware
  webcam_sydneyh264:
  - ffmpeg:webcam_sydney#video=h264#hardware
webrtc:
  candidates:
  - stun:8555
  listen: :8555
@AlexxIT AlexxIT added the bug Something isn't working label Jan 22, 2024
@AlexxIT AlexxIT self-assigned this Jan 22, 2024
@troykelly
Copy link
Author

Tested and the issue remains in Version: 1.8.5, updating issue.

@AlexxIT
Copy link
Owner

AlexxIT commented May 28, 2024

That's a powerful config, by the way.

@troykelly
Copy link
Author

That's a powerful config, by the way.

Testament to the power of go2rtc. It all "just works", and doesn't cripple the server doing it.
I know the YouTube stuff is bloat, but, I like the ISS stream on the video wall :-D

This is how I get yt-dlp (inside frigate... but same for pure go2rtc)...

  frigate:
    <<: *default-opts
    privileged: true
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.13.2
    # sysctls:
    #   - net.ipv6.conf.all.disable_ipv6=0
    shm_size: "4096mb"
    cap_add:
      - CAP_PERFMON
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
        limits:
          cpus: "6.00"
          memory: 12G
      restart_policy:
        condition: unless-stopped
        delay: 5s
    mem_limit: 12G
    ulimits:
      memlock:
        soft: -1
        hard: -1
    entrypoint: /bin/sh -c "if ! [ -x /usr/local/bin/yt-dlp ]; then curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp && chmod a+rx /usr/local/bin/yt-dlp || echo 'yt-dlp download failed'; fi && exec /init"
    environment:
      - TZ=Australia/Sydney
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=all
      - NVIDIA_REQUIRE_CUDA=cuda>=12.0
      - FRIGATE_MQTT_HOST
      - FRIGATE_MQTT_USER
      - FRIGATE_MQTT_PASSWORD
      - FRIGATE_RTSP_USER
      - FRIGATE_RTSP_PASSWORD
      - CAMERA_RTSP_USER
      - CAMERA_RTSP_PASSWORD
      - INTERCOM_RTSP_USER
      - INTERCOM_RTSP_PASSWORD
      - BUILDINGNVR_RTSP_USER
      - BUILDINGNVR_RTSP_PASSWORD
      - PLUS_API_KEY
    devices:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
      - /dev/dri/renderD129 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - "/etc/timezone:/etc/timezone:ro"
      - "/etc/localtime:/etc/localtime:ro"
      - "/data/frigate/config:/config:rw"
      - "camera-security:/media/frigate-dropbox"
      - "camera-webcam:/media/webcam"
      - "/data/frigate/media:/media/frigate:rw"
      - type: tmpfs
        target: /tmp/cache
        tmpfs:
          size: 2GB
    network_mode: host

@AlexxIT AlexxIT added this to the v1.9.3 milestone Jun 6, 2024
@AlexxIT
Copy link
Owner

AlexxIT commented Jun 6, 2024

I hope this will be fixed with killing any exec/rtsp process on stream stop.

@AlexxIT
Copy link
Owner

AlexxIT commented Jun 6, 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants