Coordinate various audio and DSP packages

[ssfrr]

Note: This is a pretty broad issue and I imagine that as conversation develops it may get split into sub-issues, so feel free to go ahead and open up separate issues for different parts of the discussion off the bat.

After a bit of a hiatus I'm getting back into AudioIO development, and it looks like in the mean time WAV.jl from @dancasimiro made a bunch of progress integrating with various audio backends like PulseAudio and AudioQueue, and now there's FLAC.jl from @dmbates. There's also been a bunch of cool things happening in DSP.jl, and a flurry of exciting speech analysis/synthesis packages from @r9y9. I'm thinking it would be great to get the Julia audio and DSP package ecosystem working together more seamlessly.

I think it would be useful to have a common type used for audio signals and audio streams. It seems like there have been a number of explorations into this design space (@mbauman's Signals.jl and AxisArrays.jl, NamedArrays.jl, Images.jl (if we think of signals more broadly)

I'm going to be prototyping some of these ideas, but I wanted to try to get some buy-in from others in the community and try to get some advantage from your experience.

I also am interested in input from other audio processing folks (@JayKickliter, @simonster, @jfsantos, @gummif, @davidavdav). Also @timholy has thought a lot about uniformly-sampled signals and might have some insight.

Adding support for loading/saving audio formats using FileIO.jl seems like an obvious step. FileIO also distinguishes between Files and Streams, so perhaps that separation is useful here as well, if we represent audio devices as Streams.

It might be that the audio signal type is just a simple wrapper that adds some indexing convenience and keeps track of the sample rate.

Advantages of keeping the samplerate in the type:

• we can auto-resample if the user wants to play the sound out of a stream running at a different rate
• more easily plot on real time axis
• specify filter parameters in real-world Hz instead of rad/sample
• keep track of real-world frequencies when we switch into frequency domain, so plotting spectrograms can have the right axes without the user thinking about it

One thing I've thought of is to keep track of the sample rate as a type parameter so that we can have methods that decide at compile time whether they need to auto-resample or not. The frequency stuff makes me think maybe it would make sense to have AudioTimeSignal and AudioFrequencySignal. We could make it a single type and keep both domain representations, but then it's not obvious how indexing works and also tricky to invalidate one representation when the other changes.

Use Cases

Read in a wav file, save to ogg vorbis with a bitrate of 256kbps. Using the FileIO infrastructure this should look like

x = load("myfile.wav")
save("myfile.ogg", x; bitrate=256)

Read file, write first 1024 samples to new file

x = load("myfile.wav")
save("myfile_short.wav", x[1:1024])

Read file, write the first second to a new file (using SIUnits)

x = load("myfile.wav")
save("myfile_short.wav", x[0.0s:1.0s])

Read stereo file, write mono mix

x = load("myfile.wav")
save("myfile_mono.wav", x[:, 1] + x[:, 2])

Play a stereo wav file on the default audio stream

x = load("myfile.wav")
play(x)

Plot an audio signal

x = load("myfile.wav")
plot(x[:, 1]) # plots with samples on the x axis
plot(timerange(x), x[:, 1]) # plots with time on the x axis (there's probably a better name than timerange)

[dmbates]

I am happy to move my unregistered FLAC package to a JuliaAudio group. I am by no means expert in audio encoding, I simply wanted to automate conversion of music and audiobook CD's to FLAC and/or Ogg/Vorbis. The FLAC package provides an interface to enough of the libflac API to get me there, but nothing more sophisticated than that.

One thing I did was to use memory-mapping on .wav files which I think should be more efficient than reading the file into memory as a Julia array. I haven't benchmarked it, however.

I will need to add tests such as checking that a round-trip wav -> flac -> wav produces the same .wav file. It seems like it would be wasteful to incorporate .wav files in the package just for testing. Are there standard files for testing codec implementations?

I think that adding support for FileIO is a great idea.

[davidavdav]

Hi,

It would be great to have a really good design for audio data structures, more than "audio is just a matrix of Nsamples x Nchannels" which is what most languages have. Audio data is often processed in (overlapping) frames, and in speech research often labelled with time-aligned (or not yet aligned) labels. We would want an interface that can deal with streams just as well as with files, and handles frame-by-frame processing nicely. Also, we seem to be always converting back and forth between signed shorts and floats.

I have MFCC which computes fairly standard speech features. It is not yet a streaming version. Simple things like the state of an IIR filter (used in standard MFCC computation) makes it slightly non-trivial to move to frame/stream based processing.

I also have an implementation of WSOLA (not yet on github I think), an algorithm that speeds up or slows down audio. That could go into JuliaAudio.

Cheers,

---david

[gummif]

A few months back I made a naive https://github.com/gummif/Musical.jl unregistered packed for manipulating audio for musical purposes. The samplerate was built into the audio type so I think that is a natural way to do it, a type parameter is probably even better. The number of channels type parameter is also something to think about. Similar to Array maybe:

typealias MonoAudio{F} Audio{F, 1}
typealias StereoAudio{F} Audio{F, 2}

[ssfrr]

Thanks for the feedback and the pointer to your package, @gummif. I'm currently working on a new package with types defined for multichannel buffers, streams, and DSP nodes that should be suitable for audio, radio, anything with a multi-channel time-series of scalar values. I'll keep folks posted once I have enough to look at. Eventually I think I'd like it to live in JuliaDSP, but that's a conversation for later.

[seebk]

I am currently trying to write a VST plugin wrapper to execute Julia code in a DAW. Therefore, I need a set of types and functions with a standard interface that could be called from C/C++. The best option IMO would be to take a module which is widely used in other applications and that is just extended with a C/C++ wrapper. Unfortunately AudioIO.jl is not suitable for that purpose, so my hope is on the next generation :)

Are there any news or updates about this?

For testing I have started to implement an own AudioProcessor Julia module, but a collaboration would make more sense I think...

[ssfrr]

Sounds really interesting, do you have a repo I can follow? I'd definitely be happy to talk more about how to integrate a VST-wrapped julia instance, probably in the Issues for that package.

The current status is:

1. The SampleTypes package is a "glue" package that provides a set of buffer and stream types that can be used by other packages
2. I've ported over the libsndfile functionality from AudioIO to use the new types into a new LibSndFile package
3. I've got a package JACKAudio working that interfaces to the JACK system for getting audio in and out of Julia to your sound card or other JACK-aware programs. Right now it just supports read, read! and write. It's also serving as a test for what functionality should be built into SampleTypes and what should be in the individual audio interface packages.
4. I think my next step is going to be porting over the portaudio functionality from AudioIO to the new system, so we'll have cross-platform audio for people who don't use JACK. At that point I think we'll be ready for a bigger announcement on julia-users.
5. There's now a "JuliaAudio" organization to hold these packages

[seebk]

Thanks for the info, I was not aware of the JuliaAudio group. I also did not yet setup a repo, but it will hopefully be ready in the next couple of days.

For another project I have also extracted and extended your portaudio wrapper to live in an own module. Did you already start working on the portaudio stuff? Otherwise, my basic module my be a good starting point...

Anyway, I will inform you when the repos are ready...

[ssfrr]

I haven't started rolling on the PortAudio extraction, so I'd definitely be interested in seeing your work. Then there's also all the work that @wherrera10 has done that I'd like to make sure makes it into the new stuff (particularly improving windows support).

[seebk]

Finally, I had time to clean up the portaudio wrapper and created a repository
https://github.com/seebk/PortAudio.jl

The basics work well, however performance is still an issue and drop outs occur with small buffers. Probably, the types are not strict enough or I do something wrong when passing array. "@time" indicates that a lot of allocations happen all the time during read and write....

Another repo for the VST plugin wrapper will follow in the next weeks. I will try to refactor it and directly base it on your SampleTypes package as this seems to be far more mature than the basic package I have created for that purpose.

[ssfrr]

@seebk thanks for the link, definitely could be useful.