Use livekit's Rust SDK instead of their swift SDK #13343

maxbrunsfeld · 2024-06-21T00:55:18Z

See livekit/rust-sdks#355

Todo:

make call / live_kit_client crates use the livekit rust sdk
create a fake version of livekit rust API for integration tests
capture local audio
play remote audio
capture local video tracks
play remote video tracks
tests passing
bugs
- deafening does not work (Can't stop a remote audio track from playing with RtcAudioTrack::set_enabled(false) livekit/rust-sdks#359)
- mute and speaking status are not replicated properly: (Not receiving TrackMuted or ActiveSpeakersChanged events livekit/rust-sdks#358)
- linux - crash due to symbol conflict between WebRTC's BoringSSL and libcurl's openssl (libwebrtc-sys conflicting with openssl-sys livekit/rust-sdks#89)
- linux - libwebrtc-sys adds undesired dependencies on libGL and libXext
- windows - linker error, maybe related to the C++ stdlib (windows: libwebrtc-sys conflicts with other crates that use C/C++ livekit/rust-sdks#364)
```
libwebrtc_sys-54978c6ad5066a35.rlib(video_frame.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MT_StaticRelease' doesn't match value 'MD_DynamicRelease' in libtree_sitter_yaml-df6b0adf8f009e8f.rlib(2e40c9e35e9506f4-scanner.o)
```
- audio problems

Release Notes:

Switch from Swift to Rust LiveKit SDK 🦀

co-authored-by: mikayla <mikayla@zed.dev>

Co-authored-by: Conrad <conrad@zed.dev>

Co-authored-by: Mikayla <mikayla@zed.dev>

co-authored-by: Mikayla Maki <mikayla@zed.dev>

SomeoneToIgnore · 2024-11-15T21:12:18Z

Small heads-up for the participants interested: we're about to merge this PR and test it internally.
This is NOT ready for Linux yet, but seems to fully replace the old macOS Swift SDK without any issues found so far.

Next rough steps look like this:

supporting Linux audio
fixing audio issues with wireless earphones and other dynamic frequency devices (same issue is in the old SDK, but now we can control things and fix it, finally)
work on Linux screenshare and fixing whatever odd segfaults certain Linux distros experience

Issues found: * audio does not work well with various set-ups using USB * switching audio during initial join may leave the client with no audio at all * audio streaming is done on the main thread, beachballing certain set-ups * worse screenshare quality (seems that there's no dynamic scaling anymore, compared to the Swift SDK) This reverts commit 1235d08.

SomeoneToIgnore · 2024-11-18T09:24:02Z

Not that fast, alas, we have to revert it and try harder: #20809

…20809) Issues found: * audio does not work well with various set-ups using USB * switching audio during initial join may leave the client with no audio at all * audio streaming is done on the main thread, beachballing certain set-ups * worse screenshare quality (seems that there's no dynamic scaling anymore, compared to the Swift SDK) This reverts commit 1235d08. Release Notes: - N/A

…20809) This reverts commit d92166f.

…20809) This reverts commit d92166f. # Conflicts: # Cargo.lock

cla-bot bot added the cla-signed The user has signed the Contributor License Agreement label Jun 21, 2024

SomeoneToIgnore assigned maxbrunsfeld Jun 21, 2024

maxbrunsfeld force-pushed the livekit-rust-sdk branch from 56c53a8 to e0b873b Compare June 21, 2024 15:55

maxbrunsfeld and others added 4 commits June 24, 2024 11:13

Start work on using livekit's Rust SDK instead of their swift SDK

aa6dd68

Initialize dispatcher for livekit

5ed5c90

Start implementing playing of remote audio tracks

33c7ead

co-authored-by: mikayla <mikayla@zed.dev>

🎨 Rename fns

971d754

maxbrunsfeld force-pushed the livekit-rust-sdk branch from 1cc0594 to 971d754 Compare June 24, 2024 18:14

maxbrunsfeld added 4 commits June 24, 2024 12:27

Configure audio output stream to match webrtc source stream

9de1cb4

Wire up screen sharing in livekit test app

2532740

Fix some warnings

233eb92

Tweak livekit example app

8dfe3b0

maxbrunsfeld force-pushed the livekit-rust-sdk branch from 7ca3e2a to 8dfe3b0 Compare June 24, 2024 21:14

maxbrunsfeld and others added 5 commits June 24, 2024 14:22

Retain screen capture sources

3ab56f7

Start making screen capture work

84d3a81

Co-authored-by: Conrad <conrad@zed.dev>

Fix obj-c selector in MacScreenCaptureStream::drop

5a97a95

Co-authored-by: Mikayla <mikayla@zed.dev>

Wire up more logic for screen sharing

8d739e0

Co-authored-by: Mikayla <mikayla@zed.dev>

Use H264 codec for screen sharing

0ac44ab

maxbrunsfeld force-pushed the livekit-rust-sdk branch from 218cb1a to 0ac44ab Compare June 26, 2024 00:44

maxbrunsfeld added 2 commits June 25, 2024 17:54

Use RemoteVideoTrackView in SharedScreen view

48bdd23

Play audio tracks that are published

9348a4f

maxbrunsfeld force-pushed the livekit-rust-sdk branch 3 times, most recently from 609b5ae to 635f1fa Compare June 26, 2024 15:16

Bump livekit sha

bc3bb5c

maxbrunsfeld force-pushed the livekit-rust-sdk branch from 635f1fa to bc3bb5c Compare June 26, 2024 15:18

Fix issues with test livekit client

641749e

maxbrunsfeld force-pushed the livekit-rust-sdk branch from a31856f to 641749e Compare June 26, 2024 15:41

maxbrunsfeld added 2 commits June 26, 2024 08:43

Update live_kit_client crate description

4c2aa8e

Remove todos in livekit test client

c388dd6

SomeoneToIgnore and others added 13 commits November 1, 2024 18:11

Fix the formatting

0ee75ca

Properly exclude Windows

a296553

Merge branch 'main' into livekit-rust-sdk

f64512a

Use latest livekit-rust-sdk

82f4f89

co-authored-by: Mikayla Maki <mikayla@zed.dev>

Small clippy fix

76936fa

Provide HttpClient impl to our fork of LiveKit

72939bc

Merge branch 'main' into livekit-rust-sdk

2d6800c

Make Windows build compile

2d443c4

More fixes for the binaries to compile

ae7d7b8

Fix other platforms

7b827e7

Test lib fixes

1f3407e

Enable the microphone in the UI

8d271bc

Move cpal override into the patch section

d099ac4

SomeoneToIgnore force-pushed the livekit-rust-sdk branch from eaac4a5 to d099ac4 Compare November 15, 2024 07:24

SomeoneToIgnore and others added 5 commits November 15, 2024 12:08

Merge branch 'main' into livekit-rust-sdk

01ea28c

Feature-gate Linux audio for now

e914eae

Small style fix

9aa1468

Final fixes before macOS release

41a10fe

co-authored-by: Mikayla Maki <mikayla@zed.dev>

Linux: Fixed the tests, hid the screenshare

80c6fc9

co-authored-by: Mikayla Maki <mikayla@zed.dev>

SomeoneToIgnore merged commit 1235d08 into main Nov 15, 2024
12 checks passed

SomeoneToIgnore deleted the livekit-rust-sdk branch November 15, 2024 21:18

niklaskorz mentioned this pull request Nov 17, 2024

zed-editor: 0.165.4 -> 0.166.1 NixOS/nixpkgs#356757

Merged

13 tasks

SomeoneToIgnore added a commit that referenced this pull request Nov 18, 2024

Reapply "Use livekit's Rust SDK instead of their swift SDK (#13343)" (#…

72854a6

…20809) This reverts commit d92166f.

SomeoneToIgnore mentioned this pull request Nov 18, 2024

Use livekit's Rust SDK instead of their swift SDK, take 2 #20826

Closed

3 tasks

SomeoneToIgnore added a commit that referenced this pull request Nov 19, 2024

Reapply "Use livekit's Rust SDK instead of their swift SDK (#13343)" (#…

be5f0b4

…20809) This reverts commit d92166f.

SomeoneToIgnore added a commit that referenced this pull request Nov 19, 2024

Reapply "Use livekit's Rust SDK instead of their swift SDK (#13343)" (#…

46329d6

…20809) This reverts commit d92166f. # Conflicts: # Cargo.lock

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use livekit's Rust SDK instead of their swift SDK #13343

Use livekit's Rust SDK instead of their swift SDK #13343

maxbrunsfeld commented Jun 21, 2024 •

edited by SomeoneToIgnore

Loading

SomeoneToIgnore commented Nov 15, 2024

SomeoneToIgnore commented Nov 18, 2024

Use livekit's Rust SDK instead of their swift SDK #13343

Use livekit's Rust SDK instead of their swift SDK #13343

Conversation

maxbrunsfeld commented Jun 21, 2024 • edited by SomeoneToIgnore Loading

SomeoneToIgnore commented Nov 15, 2024

SomeoneToIgnore commented Nov 18, 2024

maxbrunsfeld commented Jun 21, 2024 •

edited by SomeoneToIgnore

Loading