Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Voice Channel support #49

Open
bitfl0wer opened this issue May 12, 2023 · 13 comments
Open

Voice Channel support #49

bitfl0wer opened this issue May 12, 2023 · 13 comments
Assignees
Labels
Difficulty: Complex Priority: Low Status: Reserved Neither in progress, nor still available: This issue has been reserved by someone. Use sparingly. Type: Enhancement Enhances an existing feature or behaviour.
Milestone

Comments

@bitfl0wer
Copy link
Member

Add support for sending and receiving live streams of WebRTC Video and Audio.

@bitfl0wer bitfl0wer added this to the Alpha 0.1.0 milestone May 12, 2023
@kozabrada123
Copy link
Member

At least this is well documented

Some relevant info:

  • Encryption is done using xsalsa (specifically xsalsa20_poly1305, xsalsa20_poly1305_suffix or xsalsa20_poly1305_lite), I've found the easiest way to implement this is probably the RustCrypto secretbox crate (which already combines salsa20 with poly1305)
    (Also, Discord says they themselves use libsodium and secretbox is part of NaCl-compat, which is a rust implementation of libsodium's API)

  • Audio is encoded with the Opus open audio codec (stereo, @ 48kHz). For opus in rust, we can use opus-rs, which provides high level bindings for libopus, so we can just consult the official opus documentation

(for encryption, see this part of discord's docs)

The encryption and encoding seems to be the hardest part of implementing this, the actual communication is very similar to the gateway

@kozabrada123
Copy link
Member

kozabrada123 commented Jun 20, 2023

Oh and we also have no info about video, so we should probably implement voice first

@kozabrada123
Copy link
Member

Doing some research, video uses VP8 and VP9, maybe env-libvpx-sys in rust?

{
	"codecs": [
		{
			"name": "opus",
			"type": "audio",
			"priority": 1000,
			"payload_type": 109,
			"rtx_payload_type": null
		},
		{
			"name": "VP8",
			"type": "video",
			"priority": 2000,
			"payload_type": 120,
			"rtx_payload_type": 124
		},
		{
			"name": "VP9",
			"type": "video",
			"priority": 3000,
			"payload_type": 121,
			"rtx_payload_type": 125
		}
	]
}

@kozabrada123
Copy link
Member

Webrtc-rs may also be worth looking into, since it seems to support all the needed codecs

@kozabrada123
Copy link
Member

Potentially maybe perhaps looking into this on feature/webrtc
I probably won't manage to implement it though lol

@bitfl0wer
Copy link
Member Author

Good luck :)

@kozabrada123
Copy link
Member

Also looking at how Serenity implemented this, they seem to have developed their own rtp parser and even their own opus bindings

@bitfl0wer
Copy link
Member Author

I wonder if we could... yoink this :D

@kozabrada123
Copy link
Member

yup :D

@kozabrada123
Copy link
Member

kozabrada123 commented Jun 21, 2023

Slight change of plans, raw UDP first then webrtc
because webrtc is really complex and raw udp seems to be way easier to implement for now

@kozabrada123 kozabrada123 self-assigned this Dec 8, 2023
@kozabrada123 kozabrada123 linked a pull request Dec 16, 2023 that will close this issue
14 tasks
@kozabrada123
Copy link
Member

wasm cannot support udp.

@kozabrada123
Copy link
Member

@kozabrada123 kozabrada123 removed a link to a pull request Mar 9, 2024
14 tasks
@kozabrada123 kozabrada123 added Status: Reserved Neither in progress, nor still available: This issue has been reserved by someone. Use sparingly. and removed Status: Pending labels Apr 16, 2024
@kozabrada123
Copy link
Member

kozabrada123 commented Sep 11, 2024

A few notes:

  • First, it seems that UDP is not strictly the older and Webrtc the newer version; Discord seems to use UDP in their native clients and Webrtc on web - we could very likely do the same

  • Second, the official udp voice docs have been expanded a bit, notably in the encryption modes. Once I pick this back up again we should take a closer look at that

  • Third, I feel that Primitive voice implementation (feature/voice) #457 was about 90% of the way there. I just need to find a code snippet for how to decode opus from rtp packets in rust (or generally, how to use any of the opus bindings / libraries to decode opus packets)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Difficulty: Complex Priority: Low Status: Reserved Neither in progress, nor still available: This issue has been reserved by someone. Use sparingly. Type: Enhancement Enhances an existing feature or behaviour.
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

2 participants