Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use LiveKit's Rust SDK on Linux while continue using Swift SDK on Mac #21550

Merged
merged 57 commits into from
Dec 5, 2024
Merged
Changes from 1 commit
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
46329d6
Reapply "Use livekit's Rust SDK instead of their swift SDK (#13343)" …
SomeoneToIgnore Nov 18, 2024
0077602
Move all cpal-related code into a background thread
SomeoneToIgnore Nov 19, 2024
8b20ce6
Stop requiring a WebRTC.framework dir with Info.plist
SomeoneToIgnore Nov 19, 2024
22d10c2
An attempt to fix the sound issues
SomeoneToIgnore Nov 19, 2024
88d48f4
Log errors in more places in live_kit_client
mgsloan Nov 21, 2024
5e192f7
WIP improvement to audio quality by doing some buffering during playback
mgsloan Nov 26, 2024
a487f77
Merge branch 'main' into livekit-rust-sdk
mgsloan Nov 26, 2024
f6bfaab
Switch to using livekit's internal re-sampler
mikayla-maki Nov 26, 2024
b35e235
Rework audio playback to use the local device's sample rate
mikayla-maki Nov 26, 2024
a1a35a9
Fix some threading in capture_local_audio_track
mikayla-maki Nov 27, 2024
8b6408f
Fix clippy?
mikayla-maki Nov 27, 2024
98eddef
Don't query the host when running tests
mikayla-maki Nov 27, 2024
085325e
remove accidental debug
mikayla-maki Nov 27, 2024
4b54773
Bump the buffer size to 100ms
SomeoneToIgnore Nov 27, 2024
25f2cf9
Fix microphone not working,,, maybe?
mikayla-maki Nov 27, 2024
218a311
Log more info on microphones used
mikayla-maki Nov 27, 2024
b7b0a66
Typo
SomeoneToIgnore Nov 27, 2024
0a10a1c
WIP: rewrite livekit integration to use a single output stream
mikayla-maki Nov 27, 2024
c68729f
Revert "WIP: rewrite livekit integration to use a single output stream"
SomeoneToIgnore Nov 28, 2024
9f90dab
Exclude Alsa static libraries from Zed Linux bundle
SomeoneToIgnore Nov 28, 2024
4507a0d
Merge branch 'main' into livekit-rust-sdk
SomeoneToIgnore Nov 28, 2024
454f380
Update LiveKit repo reference to include patched webrtc
mgsloan Nov 29, 2024
993f2b3
Initial implementation of rendering screen shares on Linux and Window…
mgsloan Nov 30, 2024
6f1f579
Fix lints + enable `play_remote_video_track` on windows
mgsloan Dec 1, 2024
73e343c
Improvements to screen share rendering code
mgsloan Dec 1, 2024
458ae9a
Merge branch 'main' into livekit-rust-sdk
mgsloan Dec 2, 2024
7050288
Merge branch 'main' into livekit-rust-sdk
mikayla-maki Dec 2, 2024
c6f1ab9
bump livekit version
mikayla-maki Dec 2, 2024
0f5bd9a
Upgrade to latest livekit
mikayla-maki Dec 2, 2024
be967b7
Merge branch 'main' into linux-render-screenshare
mgsloan Dec 2, 2024
6174f50
Merge branch 'main' into livekit-rust-sdk
SomeoneToIgnore Dec 3, 2024
ddef94a
Deallocate images used for rendering screenshares
mgsloan Dec 3, 2024
93d8677
Merge branch 'linux-render-screenshare' into livekit-rust-sdk
mgsloan Dec 3, 2024
3861d07
Fix build on Mac
mgsloan Dec 3, 2024
1f1c274
fix livekit naming
mikayla-maki Dec 3, 2024
d9b5af2
Revert over eager livekit renames
mikayla-maki Dec 3, 2024
6578604
Fix build on windows
mgsloan Dec 3, 2024
6367b2d
fix more rename issues
mikayla-maki Dec 3, 2024
9c9e1d5
Revert protobuf name migrations
mikayla-maki Dec 3, 2024
39069ad
Fix colors in linux screenshare
mgsloan Dec 3, 2024
83d6625
Fix resize for linux screenshare rendering
mgsloan Dec 4, 2024
825c630
Cargo fmt
mgsloan Dec 4, 2024
afa01a2
Fix Mac clippy
mgsloan Dec 4, 2024
7cc9a76
Rename `live_kit_client crate` to `livekit_client_macos`
mgsloan Dec 4, 2024
cd71d7c
Merge branch 'mv-to-livekit-client-macos' into livekit-rust-sdk
mgsloan Dec 4, 2024
c0a0887
WIP progress towards having both LiveKit swift and rust sdk
mgsloan Dec 4, 2024
d9fc146
Make the swift sdk work on macOS
mikayla-maki Dec 4, 2024
23abf14
Fix build on Linux
mgsloan Dec 4, 2024
b06fd35
Undo some unintended changes
mgsloan Dec 4, 2024
8bec001
Fixes
mgsloan Dec 4, 2024
7aaf8d0
fix macos tests
mikayla-maki Dec 4, 2024
421c94c
Document why audio output is filled with 0s when samples are too few
mgsloan Dec 3, 2024
fbc286b
Fix import errors on macos
mikayla-maki Dec 5, 2024
5d9fdf1
Revert "Stop requiring a WebRTC.framework dir with Info.plist"
SomeoneToIgnore Dec 5, 2024
f739e45
make it possible to build livekit's rust SDK on macOS with a feature …
mikayla-maki Dec 5, 2024
7fb7170
Fix clippy
mikayla-maki Dec 5, 2024
1d63ff2
Merge branch 'main' into livekit-both
mgsloan Dec 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
WIP improvement to audio quality by doing some buffering during playback
  • Loading branch information
mgsloan committed Nov 26, 2024
commit 5e192f756d29c1bbd250b6ef5d123c9d51322875
45 changes: 29 additions & 16 deletions crates/live_kit_client/src/live_kit_client.rs
Original file line number Diff line number Diff line change
@@ -279,44 +279,55 @@ pub fn play_remote_audio_track(
track: &track::RemoteAudioTrack,
background_executor: &BackgroundExecutor,
) -> AudioStream {
let buffer = Arc::new(Mutex::new(Vec::<i16>::new()));
// TODO(mgsloan): use a concurrent queue that references the Cow slices the source gives us.
//
// TODO(mgsloan): put this on the channel? rationale is to not mix up samples that came from a
// different configuration.
let buffer_mutex = Arc::new(Mutex::new(Vec::<i16>::new()));
let (stream_config_tx, stream_config_rx) = std::sync::mpsc::channel();
// TODO livekit: Pull these out of a proto later
let mut stream = NativeAudioStream::new(track.rtc_track(), 48000, 1);

let receive_task = background_executor.spawn({
let buffer = buffer.clone();
let buffer_mutex = buffer_mutex.clone();
async move {
let mut stream_config = None;
while let Some(frame) = stream.next().await {
let mut buffer = buffer.lock();
let buffer_size = frame.samples_per_channel * frame.num_channels;
debug_assert!(frame.data.len() == buffer_size as usize);

let frame_config = StreamConfig {
channels: frame.num_channels as u16,
sample_rate: cpal::SampleRate(frame.sample_rate),
buffer_size: cpal::BufferSize::Fixed(buffer_size),
// TODO(mgsloan): this multiplier was arbitrarily chosen, but a larger buffer
// size seems to help my audio quality.
buffer_size: cpal::BufferSize::Fixed(buffer_size * 2),
};

if stream_config.as_ref().map_or(true, |c| *c != frame_config) {
buffer.resize(buffer_size as usize, 0);
stream_config = Some(frame_config.clone());
stream_config_tx.send(frame_config).log_err();
}

if frame.data.len() == buffer.len() {
buffer.copy_from_slice(&frame.data);
} else {
buffer.iter_mut().for_each(|x| *x = 0);
let mut buffer = buffer_mutex.lock();
// TODO(mgsloan): max_size multiplier was arbitrarily chosen.
let max_size = (buffer_size * 3) as usize;
let new_size = buffer.len() + frame.data.len();
if new_size > max_size {
let drain_ix = new_size - max_size;
if drain_ix > buffer.len() {
buffer.clear();
} else {
buffer.drain(..new_size - max_size);
}
}
buffer.extend_from_slice(&frame.data);
}
Ok(Some(()))
}
});

let play_task = background_executor.spawn({
let buffer = buffer.clone();
async move {
if cfg!(any(test, feature = "test-support")) {
anyhow::bail!("can't play audio in tests");
@@ -331,14 +342,16 @@ pub fn play_remote_audio_track(
_output_stream = Some(device.build_output_stream(
&config,
{
let buffer = buffer.clone();
let buffer_mutex = buffer_mutex.clone();
move |data, _info| {
let buffer = buffer.lock();
if data.len() == buffer.len() {
data.copy_from_slice(&buffer);
} else {
data.iter_mut().for_each(|x| *x = 0);
let mut buffer = buffer_mutex.lock();
while data.len() > buffer.len() {
drop(buffer);
std::hint::spin_loop();
buffer = buffer_mutex.lock();
}
data.copy_from_slice(&buffer[..data.len()]);
buffer.drain(..data.len());
}
},
|error| log::error!("error playing audio track: {:?}", error),
Loading