Rust UI + Android update

[wiiznokes]

Hi,

I came accros a bug in the windows app, and tried to fix in C# without success. This motivated me to implement a UI on the rust code.

There is still a tons of thing to add to get feature parity with the C# one, but it's a start.

Currently, wifi & adb should work.

I plan to improve the rust UI someday so i will keep this a draft.

I also updated the Android code

• update deps
• use version catalog
• use .kts gradle extension
• use the datastore api instead of preference
• rework how the state where stored (basically, separate config from ui state, store uistate in compose. Make things a bit more type safe)

[teamclouday]

Hey thanks for making this! I like the idea of transitioning the desktop app to rust. It's faster and cross platform.
Btw I tried the rust app and the window is a bit laggy on start and when I drag around. A quick guess is that the audio/network related code are running on the UI thread. It's best to create two main threads, one for UI only, and another for the rest, like how Android does it. Not sure how Iced handles it.
And let me know how I can help

[teamclouday]

If web UI is an option, we can try this library too https://github.com/tauri-apps/tauri
So the Rust side service handles the internal logic, and web code for displaying the UI and interactivity

[wiiznokes]

Btw I tried the rust app and the window is a bit laggy on start and when I drag around. A quick guess is that the audio/network related code are running on the UI thread. It's best to create two main threads, one for UI only, and another for the rest, like how Android does it. Not sure how Iced handles it.

It shouldn't block the UI.
For cpal (audio) https://docs.rs/cpal/latest/cpal/

Creating and running a stream will not block the thread. On modern platforms, the given callback is called by a dedicated, high-priority thread responsible for delivering audio data to the system’s audio device in a timely manner. On older platforms that only provide a blocking API (e.g. ALSA), CPAL will create a thread in order to consistently provide non-blocking behaviour (currently this is a thread per stream, but this may change to use a single thread for all streams). If this is an issue for your platform or design, please share your issue and use-case with the CPAL team on the GitHub issue tracker for consideration.

For the streamer, it run in a tokio runtime (the one iced provide)

If the UI is a bit laggy at launch, my guess is that it's due to the first audio initialisation. This could probably be improved. But on my PC it's fine.

Btw, i was using libcosmic https://github.com/pop-os/libcosmic, which is based on Iced, with some improvment. It's the library used to build the cosmic DE from pop-os (which i contribute to). It's still in early state and have a lot of bugs but it's backed by pop-os so it's evolving quickly.

I'm not against trying other option tho. I like iced because it's full rust. There is also https://github.com/linebender/xilem but i've never tried it. I've never tried tauri either, nor javascript for ui.

And let me know how I can help

Not sure. Personally, i want to works on this

• match the same layout of the C# app
• advanced windows with some audio options
• make a widget for audio wave
• fix most bugs
• package flatpak & .msi

[teamclouday]

Sounds good! That makes sense. I can help with some of the network and audio processing stuff. I've never done it in rust so I'm interested

[wiiznokes]

Alright, i finished implementing all i wanted. This include a working adb streamer, which use a tcp streamer under the hood.
I was able to create a wave widget, but the CPU cost was to high, and i had no more idea for optimization so i gave up. + there is no tray icon rn so no way to deactivate the feature in the background.

It is possible to create a nsis package with cargo packager --release --formats nsis --verbose (must install cargo packager before, cargo install cargo-packager)

Also, the adb binaries should be added in res/linux/adb & res/windows/adb. I think they are portable. I already added the lfs paths.

Let me know if you want to change something in the app

[wiiznokes]

Also, i was thinking of making another pull request to remove the record audio option on Android.
We could automatically stream audio when we are connected. This could be useful to not have to click on the notification to stop recording audio for nothing. Last time, i drained my battery because i unplugged my phone without turning one the screen to stop the recording. Wdyt ?

[wiiznokes]

Also, concerning #33, i personally don't care about the mono and i16 format (even tho someone do care about having stereo #25 (comment)), but i think sampling is an usefull option. When i started using the RustApp recently, switching from 16000 to 48000, my friend on Valorant asked me if I had finally invested in a microphone lol

[teamclouday]

Awesome thanks for the updates! I'll be developing with a macbook and linux laptop in the next month. So we don't have to worry about the tray icon or packaging yet. We can add that once the app is good to go.

I was able to create a wave widget, but the CPU cost was to high

I can help with the wave widget thing and use opengl to render it. I assume libcosmic or iced allows that.

Also, i was thinking of making another pull request to remove the record audio option on Android.

Yes we can remove the record audio option for sure. The original purpose of it was that I've implemented the local buffering and being able to restart recording can help clear the buffer and make audio stream faster. I'll explore some better options of streaming data so that we don't need local buffer anymore.

However we still need the notification to appear. Because it runs as a background service, without notification we can't tell if it's still running when the screen is closed or changed to another app. The notification can just stay there and do nothing.

Also, concerning #33, i personally don't care about the mono and i16 format (even tho someone do care about having stereo #25 (comment)), but i think sampling is an usefull option

That was an experiment long ago. I'll close it. mono and 16k sample rate are there to make streaming audio faster. However I think tcp/udp can handle it easily in local network connection. It'll be a little challenging with bluetooth, but we can keep the 16k as an option

This goes back to buffering but I'm thinking of some ways to send multiple types of data packets in one socket connection, so we can send the current number of channels, audio format, sample rate to the rust app. The rust app then use the info to resample the audio stream

switching from 16000 to 48000, my friend on Valorant asked me if I had finally invested in a microphone lol

lol that's sounds like a good improvement!

[teamclouday]

Merging the pr now

[wiiznokes]

I'll be developing with a macbook and linux laptop in the next month. So we don't have to worry about the tray icon or packaging yet. We can add that once the app is good to go.

Nice! I've just added it to start using it on my PC, but i didn't expect a release just now.

I can help with the wave widget thing and use opengl to render it. I assume libcosmic or iced allows that.

My current solution use a canvas. It should be possible to write a custom shader for it. I opened a discussion in the Iced discord server if you want to see https://discord.com/channels/628993209984614400/1313542762901012480.

Another challenge is how we get the data. Because rn, i was using https://crates.io/crates/rtrb for passing data from the streamer to the audio thread, but using it in the UI is not an option. I think the most optimized way is creating a stream with tokio time, and adding it to the streamer subscription. This way, we can set a state in the streamer so it know when to send a message to the UI with a value it just read.

Yes we can remove the record audio option for sure. The original purpose of it was that I've implemented the local buffering and being able to restart recording can help clear the buffer and make audio stream faster. I'll explore some better options of streaming data so that we don't need local buffer anymore.

However we still need the notification to appear. Because it runs as a background service, without notification we can't tell if it's still running when the screen is closed or changed to another app. The notification can just stay there and do nothing.

Alright, i will wait before making a PR then. The notification will indeed be required, but it should be possible to remove it when the foreground service die ig.

That was an experiment long ago. I'll close it. mono and 16k sample rate are there to make streaming audio faster. However I think tcp/udp can handle it easily in local network connection. It'll be a little challenging with bluetooth, but we can keep the 16k as an option

This goes back to buffering but I'm thinking of some ways to send multiple types of data packets in one socket connection, so we can send the current number of channels, audio format, sample rate to the rust app. The rust app then use the info to resample the audio stream

We could do that, or just set the same configuration on both side, and let the user change it in the option. Keep it mind that on my PC, and only for the RustApp, 16000 sample rate was not an accepted stream configuration, and just didn't worked.

[teamclouday]

Sounds good! I'll explore and do some experiments. Btw I might remove oboe from Android side, because it's essentially doing the recording and then copying data to kotlin side. It might be overhead compared to the builtin APIs in java/kotlin