macOS / iOS sound player. AVAudioEngine - WIP

[ofTheo]

Now re-opened as #7293

Addressing #167

Features:

• Multiplay
• Pitch/Speed
• Handles switching audio outputs mid play
• Global FFT
• Streaming from disk vs loading to memory?

ToDo:

• Add in code in ofConstants.h to allow easy switching from fmod to this player.
• Add in iOS interrupt handling ( when a call comes in etc )
• Test on iOS
• Look at Global FFT( see @arturoc 's comment here: replace fmod #167 (comment) )

Questions:

• Multiplay works by creating new internal players for each play request. ofAVEngineSoundPlayer uses an ofEvents().update listener to cleanup the extra multiplay players. This seems more reliable than using a thread ( had some unexpected crashes when using a thread ), but curious if people see any downside to an update event on the main thread, vs spawning a new thread for each player.

[danoli3]

Looking good!!

[roymacdonald]

Hi, I have several of this features, and more, working on ofxSoundObjects. it only relies on dr_wav, dr_mp3, dr_flac and stb_vorbis, to load the files.
Then it uses ofSoundStream to send whatever it is playing to the sound device. The good thing of this is that the player and its data are not "black boxed" on its way to the sound device. It is already fully cross platform. Having an global FFT object so it can behave exactly as ofSoundPlayer is no problem. I could add such. It also comes with the extra feature of writing audio files to disk.

I think that having an approach as this would allow to have a more unified sound engine, where you could be able to manipulate audio data from a sound file in the same way as you can with a sound input.

The current audio resampling of ofSoundBuffer is not good. I have also implemented using libsamplerate and works really well.

I could add a bunch of optimizations as using SIMD as ofxPDSP does.

I think that this implementation would be a lot cleaner than having a different one for each platform.

[ofTheo]

@roymacdonald couple of questions about ofxSoundObjects

• Does it support m4a audio files?
• Does it support multi play? ( ie: playing the same sound multiple times )
• Have you checked performance vs the fmod audio player?

I have to update this PR but I basically have everything except getSpectrum / fft working.
I might have Obj C peeps like @2bbb and @danoli3 take a look once its ready to make sure I am not leaking memory or doing anything terrible from an Obj C perspective 🙂 but I am curious if ofxSoundObjects or ofxAudiFile could be a good solution too ( as we'll need to make a windows based sound player too to switch away from fmod ).

[roymacdonald]

Hi @ofTheo

Does it support m4a audio files?

ofxSoundObjects use ofxAudioFile as the backend for loading files. So far it supports Wav, mp3, FLAC and Ogg Vorbis. But adding a library for m4a does not sound like much of a problem.

Does it support multi play? ( ie: playing the same sound multiple times )
Yes it does. I have 2 branches, one that is just like the regular ofSoundPLayer and another one which gives you granular control over each playing instance, ie: you can set different speeds or change the playback position of a particular play instance, yet still it defaults to mimic ofSoundPLayer.
Have you checked performance vs the fmod audio player?
Nope, but I have used it on some projects being able to play 2000+ sounds at the same time with no problem.

I think that it would be reasonable for me to make a special branch before adding into OF because there is useful things than can be extracted from the different branches.

ofxSoundObjects has an example that runs an fft, so adding the getSpectum part to the player is not much of a problem.

And running benchmarks to compare with "native" players should be also done.

I would be super happy if we merged ofxSoundObjects into the OF core. Let me know what you think so I can work on it to make it mergeable.

BTW, I also have ofxSoundObjects working with ofxAudioUnits and ofxNDI.

[ofTheo]

@roymacdonald nice that it doesn't need any libs compiled!
I imagine aiff and m4a might be fairly important if it was the core audio player.

Yes it does. I have 2 branches, one that is just like the regular ofSoundPLayer and another one which gives you granular control over each playing instance, ie: you can set different speeds or change the playback position of a particular play instance, yet still it defaults to mimic ofSoundPLayer.

I'll give it a spin and try it with the soundPlayer example. Which branch would you recommend?

[roymacdonald]

I imagine aiff and m4a might be fairly important if it was the core audio player.

Certainly. I will look at such.

Which branch would you recommend?

go with the master branch. experimental has more features, but the core of it is pretty much the same.

Currently you need to set up the player along with the sound stream, etc. but It is quite straight forwards to implement something that behaves exactly as the current ofSoundPLayer. I mean, no need to setup the sound stream, connect to it, etc.

The one thing I think would be very important to update/fix in order to make this all work nicely is to properly implement the ofSoundBuffer resampling, which can be quite tricky and currently does not work nicely. I found out that the best approach is to use a library for such, like libsamplerate.

[dimitre]

this is great news @ofTheo and @roymacdonald
I would love to have global FFT, being able to write audio files to disk and not having fmod

[dimitre]

Just wondering here @ofTheo if this could be merged as it is, and improved by PRs later.
I can update Xcode project to list the files.

[ofTheo]

@dimitre I think I have a few local updates I need to push, but then might ask @2bbb @danoli3 and @roymacdonald to take a look. The FFT stuff should be doable natively, but after a couple of attempts I might ask people with better knowledge to take a look.

[roymacdonald]

Hi @ofTheo i have been extra busy recently but I can take a look at this during the week. I will let you know.

[ofTheo]

Just updated this - it still needs FFT.
If you want to test quickly you can grab the two new ofAvEngineSoundPlayer files and then at the top of soundPlayerExample add this:

	synth.setPlayer(make_shared<ofAVEngineSoundPlayer>());
	beats.setPlayer(make_shared<ofAVEngineSoundPlayer>());
	vocals.setPlayer(make_shared<ofAVEngineSoundPlayer>());

Would be great if someone who has a better idea of Objective C and or Sound like @roymacdonald @2bbb or @admsyn could take a look. I feel like its close but needs some tweaks.

One nice thing it does handle well is switching audio outputs while its playing.
It pauses the engine, saves the info of all the playing sounds and then restarts everything with the new output hardware / rate etc as it was when the audio was switched ( a lot of other apps just crash when this happens ).

[dimitre]

Hey @ofTheo I'll test this one today and made some changes to make it work with ARC, not sure if they are correct.
https://github.com/dimitre/openFrameworks/tree/avengine
Cheers

[ofTheo]

Thanks @dimitre !!
Definitely in need of an objective-c expert here 🙂

[2bbb]

this is a small part, getter/setter method in basic obj-c style is

- (void)setVolume:(float)value;
- (float)volume;

then we can use property style access like

obj.volume = 0.5f; // equivalent to [obj setVolume:0.5f];
float v = obj.volume; // equivalent to float v = [obj volume];

[2bbb]

basically, no need to explicit @synthesize if we use timer as self.timer (or _timer on internally. _timer will be defined automatically with @prorpety declaration).
if we want to use simply as timer then @synthesize timer is needed.

[ofTheo]

@2bbb thanks for these suggestions.
To be honest I know very little about Obj-C. If you wanted to fork this branch and do a PR with Obj-C fixes would be more than happy to close this one :)

[dimitre]

One thing I really miss compared to default ofSoundPlayer is the capability of playing audio backwards setting a negative speed. I've noticed the values are clamped but probably varispeed doesn't allow negative speed or maybe it is other property. I've noticed a while ago the same thing in @admsyn ofxAudioUnit.
https://github.com/admsyn/ofxAudioUnit/

Maybe looking at ofxAudioUnit can be helpful for fft too

[dimitre]

Heyy I think this should be merged soon, so it is easier for all to make PRs on the top of the existing code.
And there are other changes in the project that should be made to acommodate this:

• Adding the files to osx XCode template
• Adding the files to exclusions in certain platforms (if needed)
• Adding the files to ios XCode template
  I can take care of this changes in parallel

[danoli3]

Will need Objective-C++ filename of .mm instead of .m
Giving it a spin

[dimitre]

Yeah @danoli3 I've did that in my fork
and some other ARC changes.
PS: I don't know much about obj-c
https://github.com/dimitre/openFrameworks/tree/avengine

[danoli3]

Awesome yeah I've been testing it out tonight! Looks and works great! I'm testing in the ARC branch however just with "-fno-objc-arc" currently on this .mm! I think this is production ready

[dimitre]

I would love to see this merged, even if incomplete. I don't think it will interfere with any other OF functionality.
I am available this week to work on template updates to include the new files.

[ofTheo]

Thanks! There are definitely some things in there that I’m not sure I quite understand correctly. :)

[danoli3]

Working pretty well right now, solves most issues / resume / audio interruption events (restore audio engine / hardware connection fixed) / allows sound to play with phone silence button on by default now with new AVAudioEngine (apple control this).

Some minor changes to push

[ofTheo]

Awesome @danoli3!!
Thanks for working on this! Would be great to get it merged in and then we can try and figure out the FFT stuff separately.

[dimitre]

Heyy let's merge this one!
I'll be using / testing a lot.
Future work can be done in other PRs :)

[ofTheo]

okay sounds good @dimitre !
@danoli3 if you have any small fixes you could PR to master.

[ofTheo]

reverting this as there is too much to fix in master to get this working.
the ARC requirements are quite intsensive and there is still a lot of iOS code which needs ifdefs