Sample Track Recording Stage One

[Reflexe]

recording.mov

This is the first stage of the Sample Track Recording feature and was intended to only work on one backend (SDL) with no fancy features like looping and visualization.
It has been tested very quickly on my setup with a monitor interface.

Note: the changes are from #4994.

[LmmsBot]

🤖 Hey, I'm @LmmsBot from github.com/lmms/bot and I made downloads for this pull request, click me to make them magically appear! 🎩

Linux

• Linux (AppImage): lmms-1.2.0-rc6.1231+gb2f3a1f86-linux-x86_64.AppImage (build link)

Windows

• Windows 64-bit: lmms-1.2.0-rc6.1231+gb2f3a1f86-mingw-win64.exe (build link)
• Windows 32-bit: lmms-1.2.0-rc6.1231+gb2f3a1f86-mingw-win32.exe (build link)
• Windows 32-bit: lmms-1.3.0-alpha-msvc2017-win32.exe (build link)
• Windows 64-bit: lmms-1.3.0-alpha-msvc2017-win64.exe (build link)

macOS

• macOS : lmms-1.2.0-rc6.1231+gb2f3a1f86-mac10.14.dmg (build link)

🤖

{"platform_name_to_artifacts": {"Linux": [{"artifact": {"title": {"title": "(AppImage)", "platform_name": "Linux"}, "link": {"link": "https://output.circle-artifacts.com/output/job/a63b2a2d-f02b-4b83-8235-cdb92f29af95/artifacts/0/lmms-1.2.0-rc6.1231+gb2f3a1f86-linux-x86_64.AppImage"}}, "build_link": "https://circleci.com/gh/Reflexe/lmms/1626?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link"}], "Windows": [{"artifact": {"title": {"title": "64-bit", "platform_name": "Windows"}, "link": {"link": "https://output.circle-artifacts.com/output/job/0f86efe9-e3ad-4d7a-9f38-5831d542b9d2/artifacts/0/lmms-1.2.0-rc6.1231+gb2f3a1f86-mingw-win64.exe"}}, "build_link": "https://circleci.com/gh/Reflexe/lmms/1624?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link"}, {"artifact": {"title": {"title": "32-bit", "platform_name": "Windows"}, "link": {"link": "https://output.circle-artifacts.com/output/job/c724e479-cedb-48eb-affc-ba13fa6ce6fc/artifacts/0/lmms-1.2.0-rc6.1231+gb2f3a1f86-mingw-win32.exe"}}, "build_link": "https://circleci.com/gh/Reflexe/lmms/1628?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link"}, {"artifact": {"title": {"title": "32-bit", "platform_name": "Windows"}, "link": {"link": "https://ci.appveyor.com/api/buildjobs/m58xebqefwmpqok0/artifacts/build/lmms-1.3.0-alpha-msvc2017-win32.exe"}}, "build_link": "https://ci.appveyor.com/project/Lukas-W/lmms/builds/44344781"}, {"artifact": {"title": {"title": "64-bit", "platform_name": "Windows"}, "link": {"link": "https://ci.appveyor.com/api/buildjobs/2b35vlj7jhr3ov4a/artifacts/build/lmms-1.3.0-alpha-msvc2017-win64.exe"}}, "build_link": "https://ci.appveyor.com/project/Lukas-W/lmms/builds/44344781"}], "macOS": [{"artifact": {"title": {"title": "", "platform_name": "macOS"}, "link": {"link": "https://output.circle-artifacts.com/output/job/b8b29d25-eaf5-4b78-a156-eaec12fed67a/artifacts/0/lmms-1.2.0-rc6.1231+gb2f3a1f86-mac10.14.dmg"}}, "build_link": "https://circleci.com/gh/Reflexe/lmms/1627?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link"}]}, "commit_sha": "b2f3a1f86e95abb4f1565b7be86e69187028d028"}

[IanCaio]

I'm still going to review this with more care when I get home, but decided to take a peak at it from my phone and did a first pass review just for fixing code style (stuff like explicit blocks for one line ifs, spacing, camelCase variable names). Added all as suggestions if you'd like to just batch commit them.

Is there any reason you decided to use a braced-init-list instead of the expression-init-list for those new members?

[PhysSong]

My initial comments:

• If I remember correctly, we concluded to remove shouldApplyMasterGain from pushInputFrames before.
• I think it's okay to add more backends in this PR.

[Reflexe]

@PhysSong You are right. I forgot to remove it before pushing.
Hopefully will do it after work (in about 12h).
About other backends: I think we can first merge this, and then add pulseaudio and jack one by one. That will make testing more focused and wilp make review much easier.

[enp2s0]

Fully agree with doing backends 1-by-1. SDL is a good start since it's available everywhere and is the default, so recording will work ootb for most users.

[sebageek]

I compiled and ran this PR today on Linux and it seemed to work pretty well for me. Used the SDL backend (with pipewire), recorded a bit of Guitar music.
I had to search a bit for "how to actually record something", but after I found out that I could arm painted tracks for being recorded everything became clear. Maybe the o rec symbol could be accompanied by painting the track background orange/red so I know I'll record over it on the next run.

Things that worked for me:

• saving and loading (without saving resources) - sample tracks were there on load
• recording, cutting, moving, duplicating sampletracks

Things that did not work for me:

1. Using the "Record samples from audio-device" button - It does nothing for me.
   No recording or anything will happen. I can only record audio using the "record while playing song / BB-Track" button. Am I holding it wrong?

2. saving "with resources" (click in save dialog) breaks the saving process
   Directories will be created, but save will be aborted. The error seen on the console is as follows:

QFile::copy: Empty or null file name
ERROR: Failed to copy resource

I think lmms tries to save the sample track, but it has no associated file name. See DataFile::copyResources() in ./src/core/DataFile.cpp.

Another feature that would be really nice is if I had a way to save the sample track I recorded with lmms (maybe I decide I need it as an extra wav/mp3/something or want to post-process it somewhere else).

All in all: A big plus from my side! Looking forward to seeing this feature integrated in lmms once it's done!

[Reflexe]

@sebageek Thanks for the feedback. I will look into both. In any case. feel free to open an issue about being able to save individual Samples (if there isn't one open already).

• The second issue was very simple, I will open another PR for the fix.

• The first issue, however, IDK really know how to solve because I don't really know what is the meaning of the button (Maybe we better just remove it?) I have no idea what could it mean. The only thing I could think about is to play the song like every other track has been muted (e.g. only record w/o playing anything). But that will require some heavy modifications of the recording code I am not sure are in scope for this PR. CC: @PhysSong . @Umcaruje ?

[sebageek]

Hey @Reflexe, nice to hear that it was helpful && that the second issue is easy to fix! I also opened #6022 for the feature request.

Regarding the first issue: The record button apparently dates back to a stub implementation in 2008 and is only now appearing as it checks if the current audio backend is implementing capturing. So, how about using this button for its original intention? It could just allow the user to record samples without any other track playing (essentially the same as the two record buttons in the piano roll). On the other hand while I could imagine this functionality useful.. if this is not needed I'd opt for removing the button.

[PhysSong]

I think you should remove applyMasterGainToInputBuffer function as well, it's not used anymore after the removal of shouldApplyMasterGain.

[firewall1110]

I have compiled this branch ...
Is real recording implemented or stage 1 means some preparations for this?

I am about to PR that fix Issue #4668 , but also changes some things how capture works in goal to enable input capture callback only if needed. Also see my questions on Discord ...
Let's collaborate!

[firewall1110]

Now recording is working if merge this branch over my capture branch:

RecordingStageOne.mp4

Is this correct?

[Reflexe]

Rebased on master, remove apply master gain related changes.
Left to apply styling issues. Will do it shortly.

[JGHFunRun]

I think at some point we should switch from set/clear record to a proper record button but that's for the future IG

[JGHFunRun]

@LmmsBot

[Mayravixx]

Tried doing this today, sadly it isn't working for me on Arch. When I place down a sample section then right click it, I get no "set/clear record" option at all. Would be really simple if I could grab the exe's in some way but none of the .exe's can be downloaded, same story with the appimage file.

2021-09-21.17-07-36.mp4

[DomClark]

@Mayravixx That version doesn't contain the code from this PR - it's instead the latest master version (see the commit hash in the title bar - 8a9a2fa).

[BluesM18A1]

Not only would there need to be upmixing for mono recording devices, but for audio input devices that have more than two channels, I'd appreciate the ability to pick which two channels are recorded for a stereo track as well as downmixing if more than two are selected.

[JohannesLorenz]

We now have #7567 , which seems to extend this PR to jack, which is really cool. I tested it, and with Jack, you get 0 latency (at least for me, I have setup jack with realtime prio). It might be a good idea to merge that PR into this PR here.

From my tests, and consolidating that PR with this PR here, I see only 4 major issues:

1. Base64 encoding (mentioned above: Sample Track Recording Stage One #5990 (comment))
2. The record button seems to have no effect (mentioned above: Sample Track Recording Stage One #5990 (comment))
3. I am not sure if the DataFile changes could break anything else.
4. AudioEngine::pushInputFrames is very likely not realtime-safe, since it calls requestChangesInModel, which uses a mutex. IMO not a blocker, since LMMS itself is not (yet), and being able to record non-realtime safe is better than not being able to record at all. From my tests, there was no notable latency.

Lastly, about SDL vs Jack: While I acknowledge that some like to use it for simplicity or for testing, SDL is usually not known as a professional backend, and I would not expect it to be realtime safe - I would expect latency issues. I am fine to merge the whole thing with Jack-Only, or with Jack and latency-or-broken-SDL.

[JohannesLorenz]

Also, another note about Jack mostly(?) being Linux (as discussed above): It's not a disadvantage for other back-ends, but rather an advantage IMO. Right now, if you want to add support for, let's say PortAudio, you would first need to read a 100 pages large PR, and in the end you would not be sure how many issues are in this PR and how soon it will get merged. If we consolidate a common framework for sample recording and merge it to master, all you would need to do is implement some virtuals in PortAudio.h and PortAudio.cpp - that is much easier than consolidating this WIP PR.

[michaelgregorius]

I agree that it would make sense to merge #7567 in here. In that case there would already be two implementations that could be used to flesh out some general interface/framework regarding audio inputs. IMO such an interface should for example enable the users to select from all potential inputs in the application instead of letting them only select one in the settings dialog.

Example: it should be possible to arm two tracks for recording, set one of the to record from "Input 1" and the other from "Input 2" and then record everything at once.

[JohannesLorenz]

So, what is the state about saving the record? IMO, this should not differ from the way we save non-recorded samples:

1. If saved as a resource bundle, the sample goes in there -> Can be applied for recorded samples, too
2. If saved normally, only the file path to the sample is being saved -> Can be applied for recorded samples, if the sample is being saved on disk at the time of saving the song (that matches what michealgregorius says about REAPER: Sample Track Recording Stage One #5990 (comment))

So all that is undefined for case 2 is where exactly we save such files (what directory, what naming)? Which also seems to have an answer:

REAPER stores recordings in a global, configurable folder if there is no saved project yet. The file names contain date and time and other things to make them unique.

As a default folder, I would propose "records/" in the LMMS main folder (next to "samples/").

[michaelgregorius]

So, what is the state about saving the record? IMO, this should not differ from the way we save non-recorded samples:

1. If saved as a resource bundle, the sample goes in there -> Can be applied for recorded samples, too

2. If saved normally, only the file path to the sample is being saved -> Can be applied for recorded samples, if the sample is being saved on disk at the time of saving the song (that matches what michealgregorius says about REAPER: [Sample Track Recording Stage One #5990 (comment)](https://github.com/LMMS/lmms/pull/5990#issuecomment-2241545523))

Please note that all of the above only makes sense if the data is stored immediately into a file while being recorded. Implementing it like this is a large undertaking as it would result in a reimplementation of how LMMS handles samples and audio data.

Currently LMMS simply holds all data in main memory. With this alternative implementation it would stream data from a pool of files into memory for playback and would also directly stream the data into files during recording.

However, IMO the latter is the only "sane" implementation. Imagine recording some tracks, LMMS crashes (or the whole machine goes down for some reason) and you would not even have the raw recordings.

So all that is undefined for case 2 is where exactly we save such files (what directory, what naming)? Which also seems to have an answer:

REAPER stores recordings in a global, configurable folder if there is no saved project yet. The file names contain date and time and other things to make them unique.

As a default folder, I would propose "records/" in the LMMS main folder (next to "samples/").

I propose a folder called "audio" in the LMMS main folder. To me "records" sounds like vinyl records. 😉

The naming pattern of REAPER is "Track Number-TrackName-Date_Time", e.g. "01-Guitar Left-250101_2114.wav". If a track does not have a name yet then it is omitted, e.g. "01-250101_2114.wav".

[JohannesLorenz]

However, IMO the latter is the only "sane" implementation. Imagine recording some tracks, LMMS crashes (or the whole machine goes down for some reason) and you would not even have the raw recordings.

I would not store it "immediately". The danger of LMMS crashing is usually solved by auto-saves. Losing your record is bad, but losing your project file is equally bad. So you keep it in memory and save it at the next autosave (and of course at the normal save procedure). Shouldn't this be an easy solution to the whole problem?

I propose a folder called "audio" in the LMMS main folder. To me "records" sounds like vinyl records. 😉

Then we have "audio/" and "samples/" in one folder. To me, this does not tell me the difference. Both is audio and both are samples 😄

[michaelgregorius]

However, IMO the latter is the only "sane" implementation. Imagine recording some tracks, LMMS crashes (or the whole machine goes down for some reason) and you would not even have the raw recordings.

I would not store it "immediately". The danger of LMMS crashing is usually solved by auto-saves. Losing your record is bad, but losing your project file is equally bad. So you keep it in memory and save it at the next autosave (and of course at the normal save procedure). Shouldn't this be an easy solution to the whole problem?

Well, there's a difference between being able to at least quickly import the recorded audio files and recreate the project or to lose everything.

I have just checked and REAPER seems to store the data in 512 KB chunks while recording them. IMO it makes sense to check how other similar software behaves because usually there's a reason. And if almost every DAW does something in a certain way it is very often not a good idea to invent something completely different. However, things can of course also be implemented incrementally.

I propose a folder called "audio" in the LMMS main folder. To me "records" sounds like vinyl records. 😉

Then we have "audio/" and "samples/" in one folder. To me, this does not tell me the difference. Both is audio and both are samples 😄

To me "samples" has the connotation of rather short audio files that can be used in a sampler. And in the context of the LMMS structure of something that is shipped for that purpose. Into "audio" everything would go that's created by the users in the context of audio instead of MIDI, etc.

Technically the borders are blurry of course. In many DAWs you can record some audio, process it, e.g. by trimming it, slicing it up, etc. and then drag the result into a sampler for sampler-like playback. Or you just copy them across the track, etc.

[firewall1110]

My opinion: (Sample Track Recording) Stage One should not change:

• project file (.mmp , .mmpz file) format;
• LMMS folder structure;
  But should simply make record (audio sample from audio-devive) button working.
  If it will be implemented with good recorded audio placement to sample track - it will be excellent.

P.S.
This PR should be splinted (with share-pick) in order:
[1] Audio recording API: start, stop, change format (may be is in Master already).

[2] Record button enable stage one
(May be needed to be implemented - this PR using different conception)

[3] Recorded audio placement in Sample Track
(How I understand - is implemented here, so may be implemented after [1] )

[4] Audio recording format setting (dialog, configuration, ...)

[firewall1110]

I have just checked and REAPER seems to store the data in 512 KB chunks while recording them.

We should go LMMS way , and in "stage one" use shortest (less code lines - easy readable code) implementation variant .
Chunk audio representation used also in Audacity, but it make project not manually manageable .

Then we have "audio/" and "samples/" in one folder. To me, this does not tell me the difference.

My opinion:

• project should have custom folder names : names, configurable by track (Anna/BackVocal_xxx);
• recording project should not be in "factory folder system".

But all this should be separated from this PR in improvement issue "Audio mixing project in external folder".
So sorry, but my opinion is: such discussion here is off-topic.
But it is only my opinion.

[qnebra]

I want to point out that #7627 is a workaround / solution for recordings saving dilemma presented here.

[firewall1110]

is a workaround / solution for recordings saving dilemma presented here.

How I understand , in this case audio should be recorded in memory and, after recording stopped, sample export dialog should be opened ...
But is implement recording to memory simpler than implement direct recording to file ?

Anyway it is good variant for stage one.

[michaelgregorius]

My opinion: (Sample Track Recording) Stage One should not change:

* **project file** (.mmp , .mmpz file) **format**;

* LMMS **folder structure**;
  But should simply **make record** (audio sample from audio-devive) **button working**.
  If it will be implemented with _good recorded audio placement_ to sample track - it will be **excellent**.

Why not? It is a relatively large new feature which needs to somehow organize data that is created on-the-fly. So it would be a bit surprising to me if it would not have an impact on the project file or the folder structure.

* project should have custom folder names : names, configurable by track (Anna/BackVocal_xxx);

I am not sure if is a good idea to have sub folders which are named like the tracks. It's just one more layer that might go out of sync. Imagine the following scenario:

• The users record something on a track called "Track A" and the audio file is stored in a sub folder of the same name.
• The users rename the track to "Track B" and record again.

What should now happen? Should the folder also be renamed? If no, then it might quickly become unwieldy for the users due to the differences. If yes, what is really gained by this? To me it just seems like some additional unnecessary complexity.

* recording project should not be in "factory folder system".

It would go to a folder where we already store other user created/related data.

[...] such discussion here is off-topic. [...]

Where should it be discussed if not here?

[...] But is implement recording to memory simpler than implement direct recording to file ?

Both implementation will have their challenges if implemented correctly. The first thing to notice is that the audio data comes in via the real-time audio thread. However, both implementation will trigger actions that must not be done on the audio thread:

• Recording to memory will at one point have to allocate new memory, especially for long recordings. This must not be done on the real-time thread.
• Recording to file(s) will need to write the data to the file(s) from time to time. Obviously this is not real-time safe either.

A "classic" text on the topic: "Real-time audio programming 101: time waits for nothing"

[firewall1110]

It is a relatively large new feature

For final variant - yes. For stage one - no: only one thing be added - samples to sample track not only imported, but may be recorded too.

My opinion: this PR problem is that stage one is treated as last one. Why it is problem? LMMS device drivers are not ready to final variant - but device drivers can not be improved without testing possible: are waiting for this stage one be in master.

Both implementation will have their challenges if implemented correctly. The first thing to notice is that the audio data comes in via the real-time audio thread.

So in this case there are no difference in complexity (only variant):

• we have to use some kind of ring buffer for real time thread;
• we should copy (from this ring buffer) to clip-memory or save to file in another thread.
  Is this implemented here? How I understand - yes. What variant is implemented?

(All after I treat as off-topic)

The users rename the track to "Track B" and record again.

I mean folder is in track configuration, so changes the file name pattern.
(Track name usage as audio recording pattern is Cubase style. But Cubase use Audio folder in project folder.)

I am not sure if is a good idea to have sub folders which are named like the tracks.

OK, but
How I understand, LMMS have no project folder in this (Cubase, Ardour, ...) meaning,
so simpler variant may be
folder with the same name as project but in samples (or even in projects).
But for file name use pattern based on track name.

Where should it be discussed if not here?

In PR "Sample Track Recording Stage Five - Final" .
Or in Issue "Audio mixing project in external folder: adding feature - save to external folder as independent project"

P.S.
Can You somehow restart workflow - or I should compile this PR myself?
I want to test this PR, but job details - artifacts are not here anymore ...

[qnebra]

I mean, there is a way to have audio recording working in most common audio backend used in LMMS, SDL. There is also proposed solution for saving recordings, without writing base64 blob into project file. Rudimentary, but it is.

Other stuff, like recording in other audio API (JACK is already in another PR), autosaving, project folders, can be done later.

[firewall1110]

OK, I compiled this PR (but only for my Linux Debian Stable 64-bit).

It seems, that audio is recorded to memory and saved in project file as huge data="...." line .

In this case #7627 is a workaround , but it should be tested.

Now I will check, how Audio Recording is implemented ...

P.S.
To restart workflow it seems some conflicts must be resolved (why I don't see here, that PR has conflicts?)

[michaelgregorius]

@JohannesLorenz, I have just checked with Bitwig. In the project folder there is a sub folder called "recordings" where the files go that have been recorded with Bitwig. There's also a sub folder called "samples" where the samples go that come from somewhere else, e.g. samples that have been loaded from disk and which are used in a drum machine.

Perhaps this is some nice compromise so that we still do not have any vinyls in our folders. 😉

[michaelgregorius]

In this case #7627 is a workaround , but it should be tested.

It seems that #7627 only exports the content of sample clips into files, i.e. it will not replace the recorded data with references to the exported files. This means that the data would still be stored in unwieldy data blocks in the file.

Saving the data as data blocks in files has the disadvantage that it wastes lots of space on disk if there are several versions of a project and that it likely also has a bad performance when these project files are loaded.

Example for several versions:

• "My big hit 01.mmpz"
• "My big hit 02.mmpz"
• ... [several more iterations] ...
• "My big hit - Final.mmpz"
• "My big hit - Final for real.mmpz" 😉

[JohannesLorenz]

Well, there's a difference between being able to at least quickly import the recorded audio files and recreate the project or to lose everything.

Sorry, I don't understand what you're trying to say. Can you please elaborate?

[michaelgregorius]

Well, there's a difference between being able to at least quickly import the recorded audio files and recreate the project or to lose everything.

Sorry, I don't understand what you're trying to say. Can you please elaborate?

I meant to say that relying on the auto-save feature might lead to losing lot of work if the auto-save is not "quick" enough, e.g. if you have done a lot of recordings between the last auto-save point and the next one that is not reached anymore due to a crash. If the recordings are streamed to disk then you will have at least all your recordings and might only lose part of some recordings if LMMS crashes during the recording.

The auto-save files are also another example where storing everything in memory and then in data blocks in the file would waste a lot of disk space because every auto-save would contain all the data instead of just references.

[michaelgregorius]

I have just also checked how Bitwig handles projects which have not been saved yet. The are assigned a GUID as the "project name" and their data is stored in a temporary folder, e.g. the recordings are stored under $HOME/.BitwigStudio/temp-projects/21a3ed0e-00aa-465c-8053-e6b36e56bf06/recordings/.

I guess if you then save a project for the first time then all that stuff is moved into the actual project folder together with a newly created save file.

[JohannesLorenz]

I meant to say that relying on the auto-save feature might lead to losing lot of work if the auto-save is not "quick" enough, e.g. if you have done a lot of recordings between the last auto-save point and the next one that is not reached anymore due to a crash. If the recordings are streamed to disk then you will have at least all your recordings and might only lose part of some recordings if LMMS crashes during the recording.

The same accounts for the rest of the project file, too. I assume you say that it's more easy to create the notes and the instrument/effect tunings of the last 1 hour than the records of the last 1 hour? In this case, we could allow 2 autosave options (How often should records be saved? How often should the rest be saved).

However, I am not against the proposal of saving it while recording - it's just more overhead to code that 😬 Possibly the audio thread would smash everything into a ringbuffer, and another thread fetches it and writes it on disk. I imagine it's more ugly to split the data into chunks and find a good naming for those chunks. At this point though, I am also not sure why we should split it...

[szeli1]

I would like to mention that I have made 3 PRs about audio exporting:

1. Exporting with AudioEngine: Remove audio engine from exporting code #7452
2. Exporting to sample folder: adding Sample Folder #7538
3. Exporting without sample folder: adding sample exporter #7627

I would suggest moving the discussions about project folders to #7538 (since that pull request is a project folder implementation)

[firewall1110]

IMO recording to files should be implemented as fast as possible instead of some workarounds.

It is 1-st thing, that should be implemented and it is surprise for me that it is not done yet ...

P.S.

But about question: "Where and how place recorded files?"
Any reasonable variant is good if user can find just recorded files, can find recorded files associated with project.