Record Desktop Audio

[narickmann]

This PR allows Studio to record desktop audio and resolves #551

Unfortunately, not all browsers are supported yet, as you can see here and as Lukas mentioned here: #551

The users get a hint after selecting display as source, with which browsers they can record desktop audio.

Output of the recordings for download:

Only camera recorded: Nothing changed, the finished camera video will contain microphone audio (if recorded)

Only display recorded: If the user recorded both, desktop audio and microphone audio, the finished video will have both audio tracks.

Display and camera recorded: The finished desktop video will contain desktop audio and the finished camera video will contain the microphone audio

One thing/problem:
In the "Review & trim" step (also after uploading to Opencast), if both the display AND the camera were recorded: you only hear the microphone audio or nothing (depending on whether microphone audio was recorded or not). I'm not entirely sure why.
The audio is correct and audible for each of the videos that can be downloaded in the last step in Studio.

[lkiesow]

Only display recorded: If the user recorded both, desktop audio and microphone audio, the finished video will have both audio tracks.

Does this mean the resulting media container will have two audio streams? Or is it one stream with a mix of both audio sources?

Depending on how exactly this works, I think this could cause serious problems.

[narickmann]

Does this mean the resulting media container will have two audio streams? Or is it one stream with a mix of both audio sources?

The resulting file contains one audio stream with a mix of both audio sources.

[lkiesow]

Okay, that's definitely better than the alternative having two audio streams in one container.

Still, while this might be a bit more “professional”, I think it may cause some trouble, and it may be easier to just have a mix of all audio streams in all video containers. The current patch failing to play back audio correctly right now is a good example. For further processing, you now also need to make sure in some cases to mix audio afterward. It would be so much easier to just have the same audio everywhere.

[LukasKalbertodt]

One thing/problem:
In the "Review & trim" step (also after uploading to Opencast), if both the display AND the camera were recorded: you only hear the microphone audio or nothing (depending on whether microphone audio was recorded or not). I'm not entirely sure why.
The audio is correct and audible for each of the videos that can be downloaded in the last step in Studio.

opencast-studio/src/steps/review/preview.tsx

Lines 128 to 132 in 3621089

	// If we have two recordings, both will have audio. But the user doesn't
	// want to hear audio twice, so we mute one video element. Particularly,
	// we mute the desktop video, as there the audio/video synchronization is
	// not as critical.
	notNullish(videoRefs[desktopIndex].current).volume = 0;

So: you can just remove that (or make it conditional).

---

Currently, CI fails and there is no test deployment. Can you fix the issues and push again? There are also style lints that are visible here on GitHub.

---

It would be so much easier to just have the same audio everywhere.

I'm unsure about that. I do have remind myself of the target audience of Studio, yes, you are right. I have one specific situation where separate audio streams are useful: if a user recorded a long session and only notices in the end that the microphone is way louder (or quieter) than the desktop audio. To make a useful video out of that, one would have to boost on of these tracks. With both tracks already combined that's basically impossible. Yes, this boosting would have to happen outside of Studio but at least it's possible.

One idea that was floating around in our chat: add both. The first audio track of both files is the "all sources combined" audio. And then we still add a second audio track that only has desktop audio or microphone. Players default to playing the first audio stream, so normal users wouldn't notice it. Sure, the file size would be slightly larger and the recording would require slightly more resources.

Dunno, I guess I don't have a strong opinion 🤷

[JulianKniephoff]

Rebased and fixed conflicts.

[JulianKniephoff]

There is a test deployment now, but the code is subject to change, still. FWIW: https://test.studio.opencast.org/2023-11-16_12-41-36-pr1079-6890854908/

In the "Review & trim" step (also after uploading to Opencast), if both the display AND the camera were recorded: you only hear the microphone audio or nothing (depending on whether microphone audio was recorded or not). I'm not entirely sure why.

For the record: This I can't confirm. Is this a platform issue? Can others reproduce this? EDIT: Ah, nevermind. The all-caps "AND" was important. Who would've thunk. 🙈

[JulianKniephoff]

OK that bug should be fixed now, and I had to change a few other things, but this should work: All the audio tracks are now mixed and added as single audio track to all the resulting videos.

[LukasKalbertodt]

Only small stuff. Otherwise this looks good. Will test the deployment now.

[LukasKalbertodt]

Everything looks good! I think I now tested every possible combination of video-source and microphone and seems like the output files are exactly as expected.

Namely, as we recently decided: all available audio sources (display audio, microphone) are combined together into one track and every output video file gets that exact audio track. That will likely be the option leading to the least user confusion. All videos file will be useful. Only small disadvantage is that it's not possible to separate the two sources again, e.g. to make only the microphone input louder. We can still allow users to download the individual audio tracks in the future.

We both noticed that the audio indicator UI can be fairly hidden, depending on what was shared. But that's the same for the settings button, so we will fix that later.