Duplex Stream Support

[mitchmindtree]

This is a tracking issue for supporting duplex streams. Duplex streams are streams that have device-synchronised input and output, an essential requirement for many real-time and pro-audio applications.

This is a follow-up to #116 but focused specifically on duplex support.

[sletz]

CoreAudio has a notion of "aggregate" device that allows to combine several inputs/outputs in a single device, that will 1) deal with audio clock synchronization 2) provide a single full duplex callback to deal with.

This aggregate device can be built with an API (not only the system Audio Midi Setup tool).

Possible code to look at, Rust here at : https://github.com/padenot/cubeb-coreaudio-rs or C++ here from JACK project here :
https://github.com/jackaudio/jack2/blob/develop/macosx/coreaudio/JackCoreAudioDriver.h and https://github.com/jackaudio/jack2/blob/develop/macosx/coreaudio/JackCoreAudioDriver.mm

[xasopheno]

+1 for this feature. It would allow me to move from portaudio and to cpal.

[ErikNatanael]

In JACK duplex streams are the generally the default. Or rather, all ports opened by a single client will be handled by a single callback regardless of their type:

• A client is created
• Input and output ports are opened for the client
• The ports are transferred to a ProcessHandler containing the single audio callback. The ports give direct access to read/write their buffers in the callback.
• When activating the client it is consumed together with the ProcessHandler yielding an AsyncClient.

Hope we can find an abstraction that works well for all Hosts that support duplex streams!

[baadc0de]

Hey guys, I'm considering the complexity of this feature and whether it's something I could feasibly tackle. Is it possible for someone more experienced with the code to make a rundown of what would need to be coded/refactored for this to land? Anyone started to work on it that I could join / support?

[baadc0de]

I've made a stab at it for ASIO. It compiles and runs without crashing, but I'm remotely controlling my test windows machine so I don't know if it actually works (reads and outputs samples).

The code is in this branch:
https://github.com/audiocloud/cpal/tree/baadc0de/duplex-streams

There is a new method DeviceTrait::build_full_duplex_stream.

[ErikNatanael]

Thanks for making progress on this @baadc0de! If there is agreement on the interface for creating/using duplex streams I can do the JACK implementation (very small changes needed). Are we following the structure in #116?

Maybe a first implementation could just have all hosts that don't support duplex streams (yet) not return any default device/valid formats for duplex and we can then implement them one by one instead of one huge PR for every host to keep it from being too great of an undertaking?

[baadc0de]

Oops I did not check #116 for the proposed API. It seems good, much better than what I added. I'll see if I can rework my end to conform. It also seems better to return empty iterators for duplex formats / devices when duplex is not supported. My current implementation has a blanket unimplemented! in the trait instead, which is not ideal.

[ollpu]

I took a stab at an end-to-end design for the API: see #553.

@baadc0de I'd be happy to incorporate your work on ASIO into this once there is some consensus on the details.

[kawogi]

Does a duplex device always have one input and one output or can there be more of them (like merger, mixer, splitter, etc.)?
How flexible are those in- and outputs configuration wise? I assume they all have the same sample rate. But what about sample format (i32, f32, …), endianness, number of channels or interleaving? What about buffer sizes? Are they guaranteed to be the same length across all in- and outputs per call? Do all those mixtures exist in the wild or are they more coupled?

Maybe I can help to develop and implement a proper API if I learn more about the use cases.

[x37v]

I've never experienced a duplex stream with buffer sizes that are different for input vs output.
It is very common to have differing channel counts for inputs vs outputs.. in fact, my setup usually gives me 1 input (mic) and 2 outputs (stereo speakers/headphones).

[kawogi]

Thanks. What about the remaining aspects:

• mixed types (i32, f32 , …)
• mixed endianness
• channel interleaving
• number of in-/outputs

[ErikNatanael]

Only speaking for JACK, don't know about the other backends: * f32 only * I'm not aware of any manual endianness handling, and this doc page suggests all platforms have the sam endianness for f32: https://doc.rust-lang.org/std/primitive.f32.html * channels are separate buffers accessed through port handles, not interleaved * number of in-/outputs is completely user controlled, you can open any number of input or output ports from a single JACK client Kai Giebeler ***@***.***> skrev: (20 september 2022 22:10:30 CEST)

…

Thanks. What about the remaining aspects: * mixed types (`i32, `f32`, …`) * mixed endianness * channel interleaving * number of in-/outputs -- > Reply to this email directly or view it on GitHub: #349 (comment) You are receiving this because you commented. Message ID: ***@***.***>

[kawogi]

Thanks for the explanation. Just to get my vocabulary right: is a single JACK-port mono or can it be multi-channel?

[ErikNatanael]

A JACK port is always mono. A JACK client is a node within the JACK audio graph with any configuration of ports. There is no distinction between duplex/non duplex within JACK except the ports we choose to open. Kai Giebeler ***@***.***> skrev: (21 september 2022 00:37:32 CEST)

…

Thanks for the explanation. Just to get my vocabulary right: is a single JACK-port mono or can it be multi-channel? -- Reply to this email directly or view it on GitHub: #349 (comment) You are receiving this because you commented. Message ID: ***@***.***>

[x37v]

Thanks. What about the remaining aspects:

* mixed types (`i32`, `f32` , …)

* mixed endianness

* channel interleaving

* number of in-/outputs

I've never experienced mixed types or endianness.
but yes, often different numbers of I/O
Jack for instance has unique access to port audio vectors (non interleaved) but, port audio and alsa either use or maybe just optionally use interleaved read/writes..

[sainteckes]

I would love this features as well, I can't get the latency low enough with the strategy from the feedback example.

[j-n-f]

I think trying to support multi-device full duplex would be a longer-term effort (maybe even impossible in some cases) but I'd like this to work in the single device with single driver case at a minimum.

In some cases you can have multiple devices with clock inputs that will keep everything sync'd. There would need to be a mechanism to discover that or explicitly indicate it.

[sainteckes]

yeah it would already be worth a lot when it only works for the same device.
There are many applications where I need low latency and I am sure that input and output will be on the same audio interface.