Multiple audio streams don't seem to work as expected

[tsightler]

Hi @AlexxIT,

I've been working on replacing rtsp-simple-server with go2rtc in my project and, overall, it is working really well. My project uses RTSP and dynmically starts streams from Ring cameras using an external script so I am using the exec source which starts a publisher that sends data using FFmpeg as an RTSP publisher. This works just fine, however, there's one issue I haven't been able to figure out.

For maximum compatibility I have ffmpeg configured to provide two audio channels, one encoded as AAC and another encoded as Opus (the video is H264). This way clients that requires AAC (such as HLS in Home Assistant) can use the AAC stream, while clients that need Opus, such as WebRTC, can use the Opus stream. With rtsp-simple-server this works perfectly. If I open the RTSP stream in a tool like VLC, I can see both the AAC and Opus channels listed in the codecs section.

However, with go2rtc, it seems that, even though I am publishing two channels of audio as part of the stream, only the first one is forwarded to the client. So, for example, if I make AAC the first one, the WebRTC in Home Assistant will have no audio because it picks the AAC audio instead of the available Opus track. If I open the RTSP stream in VLC, it will only show whichever audio channel I place first.

I thought perhaps this is a configuration issue on my side, I really just want go2rtc to copy and make both audio streams available to the client, just like rtsp-simple-server does, but I can't figure out any way to do that with the exec source. I assumed that the exec source would simply forward all channels that are being sent.

I need to go back and test -rc5, I somehow feel like the problem didn't exist then, because I only just noticed it, but I've only just started really testing all the various scenarios that users might have so perhaps I missed the behavior previously.

[AlexxIT]

By default go2rtc outputs only 1 video and only 1 audio to RTSP. You can choose which codec you want with the filters:

• rtsp://192.168.1.123:8554/camera1?video&audio=aac
• rtsp://192.168.1.123:8554/camera1?video&audio=aac&audio=opus

The problem is that with an RTSP stream, you don't know which codec the client can play.

With WebRTC or MSE you know which codec the client can play. Autoselect should work there.

Of course I can output all the tracks in RTSP. I just don't know if that would be the right thing to do.

[tsightler]

The problem is that with an RTSP stream, you don't know which codec the client can play.

Exactly, which is why I provide both tracks and let the client choose. In my case it's even the same RTSP client because in HA there's only a single RTSP URL, but they may use HLS or WebRTC, and you need a different audio codec based on which one is used. Providing both allowed both streaming protocols to support audio from the very same RTSP URL.

Of course I can output all the tracks in RTSP. I just don't know if that would be the right thing to do.

I'm not 100% sure if it's the right thing to do either, as I'm certainly not an expert here, but I've used a number of RTSP servers over the years and they've all always offered the client all tracks available from the publisher. Based on my limited understanding of the RTSP protocol, it basically uses SDP to describe the available streams to the client, so I think that it is generally expected that all streams are offered, and the client chooses which streams it would like to decode based on what it supports and its preference.

It sounds like I could get similar behavior with go2rtc, but only if I have users change their RTSP URL to explicitly request both audio tracks. I'll play with this, but it definitely makes the transition more difficult.

Thanks for your consideration.

[AlexxIT]

Problem will be with multisource streams. Because go2rtc will start all of them before RTSP response.

I'll think about this concept

[tsightler]

Ah, makes sense. Maybe a server-side configuration option that enables this behavior or perhaps enable only if there is a single source. I'll live with whatever choice you make. Thanks again for this great project, I can see a ton of potential here.

[tsightler]

rtsp://192.168.1.123:8554/camera1?video&audio=aac&audio=opus

I did test this and it works just fine, so I guess worst case I can tell users that have audio issues to update the RTSP URL, not the end of the world. However, I wonder if perhaps you could allow defining a default set of query parameters for each stream to override the default behavior. I took a quick glance at the code and, while I'm far from a go expert, it looks like it would be pretty easy to implement and wouldn't change any existing default behavior.

[AlexxIT]

I think new default_query option may be useful for you
https://github.com/AlexxIT/go2rtc#module-rtsp

[tsightler]

Thanks a lot @AlexxIT!