Issue in running ffmpeg using CmdExecuteService

[Natkeeran]

This is a ticket to follow up on issue related to running ffmpeg commands using Crayfish-Commons CmdExecuteService. In trying to create a micro service to return ffmpeg result as stream (Example: https://github.com/Natkeeran/Homarus/blob/master/src/Controller/HomarusController.php#L54), we are running into a hang situation. It seems to be hanging here: https://github.com/Islandora-CLAW/Crayfish-Commons/blob/master/src/CmdExecuteService.php#L83.

Additional Notes from the Chat:

<dhlamb> Natkeeran, yeah ok.  we're doing video over http, so i wouldn't expect it to be the fastest.  but if it's an unreasonable wait for the file size, it's gotta be issues with the precise ffmpeg command to run
13:44 <Natkeeran> dhlamb: "ffmpeg -f mp4 -i - -f avi -
13:44 <Natkeeran> This is the command, hard coded for testing
13:44 <dhlamb> Natkeeran, let me see if i can emulate the streaming
13:44 <Natkeeran> dhlamb: basically I am saying mp4 file input format, -i - take input as stream avi output format - output to stream
13:45 <dhlamb> Natkeeran, ok, seems similar to the one for imagemagick
13:45 <dhlamb> Natkeeran, let me see if i can reconstruct it with bash and make it hang
13:47 <Natkeeran> dhlamb: ffmpeg -f mp4 -i test.mp4 -f avi test.avi works, and I  substituted - for filenames 
13:48 <Natkeeran> dhlamb:  https://batchloaf.wordpress.com/2017/02/12/a-simple-way-to-read-and-write-audio-and-video-files-in-c-using-ffmpeg-part-2-video/
13:49 <dhlamb> yeah, i'm gonna try something like cat test.mp4 | ffmpeg -f mp4 -i - -f avi -
13:49 <dhlamb> and see if it fills the terminal with binary, or just hangs
13:51 -!- yamil [~yamil@clients1.berklee.edu] has joined #islandora
13:52 <Natkeeran> dhlamb: interesting ffmpeg -f mp4 -i test.mp4 -f avi -
13:52 <Natkeeran> dhlamb: goes and waits with this prompt Enter command: <target>|all <time>|-1 <command>[ <argument>]
13:52 <Natkeeran> dhlamb: https://unix.stackexchange.com/questions/36310/strange-errors-when-using-ffmpeg-in-a-loop
13:52 <dhlamb> Natkeeran, ah
13:54 <dhlamb> Natkeeran, i can reproduce
13:55 <Natkeeran> dhlamb++
13:55 <Natkeeran> dhlamb: does it hang like above noted?
13:55 <dhlamb> Natkeeran, yes, it does
13:56 <dhlamb> Natkeeran, oddly, ffmpeg -f mp4 -i SampleVideo_1280x720_5mb.mp4 -f avi - > SampleVideo.avi runs successfully
13:56 <dhlamb> if you pipe the output somewhere, it goes.  leaving it as STDOUT seems to hang
13:59 <dhlamb> Natkeeran, it definitely has something to do with how it still prints out information on the command line while streaming the converted output
13:59 <Natkeeran> dhlamb: that is what I am seeing from the forums - https://lists.ffmpeg.org/pipermail/ffmpeg-user/2014-May/021215.html
14:00 <Natkeeran> dhlamb: but, their workarounds are not working
14:01 <dhlamb> Natkeeran, have you tried this?  seems to work
14:02 <dhlamb> < /dev/null ffmpeg -f mp4 -i SampleVideo_1280x720_5mb.mp4 -f avi -
14:02 <dhlamb> Natkeeran, it spits out binary to the terminal and terminates
14:02 <Natkeeran> < /dev/nul at the end?
14:02 <dhlamb> Natkeeran, at the beginning
14:03 <dhlamb> Natkeeran, only thing you should have to do is change the filename
14:03 <Natkeeran> dhlamb: hmm, would it not interpret < /dev/null as the command?
14:04 <dhlamb> Natkeeran, i don't know 100% what that does in bash, just pulled it from the stack overflow you posted
14:04 <dhlamb> seems to run
14:05 <Natkeeran> dhlamb: may work in bash
14:05 <dhlamb> Natkeeran, http://mywiki.wooledge.org/BashFAQ/089
14:06 <dhlamb> Natkeeran, they do it at the end in that post
14:06 <dhlamb> ffmpeg -i "$file" -vcodec libxvid -acodec libfaac -ar 32000 "${file%.avi}".mkv </dev/null
14:07 <dhlamb> Natkeeran, ffmpeg -f mp4 -i SampleVideo_1280x720_5mb.mp4 -f avi - </dev/null
14:07 -!- dwilcox [~dwilcox@134.41.63.122] has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
14:07 <dhlamb> also works...  try it at the end in the microservice and see what happens
14:11 <Natkeeran> dhlamb: nope, that does not help
14:12 <Natkeeran> dhlamb: in the microservice ffmpeg -f mp4 -i - -f avi </dev/nul
14:12 <dhlamb> :(
14:13 <dhlamb> Natkeeran, ok, let me get back to seeing if i can recreate the scenario with pipes or tee or something
14:22 <Natkeeran> dhlamb: sure
14:34 <dhlamb> Natkeeran, everything i'm doing is blowing up for other reasons
14:34 <dhlamb> Natkeeran, i'm having a hard time getting it to read from stdin and write to stdout
14:35 <dhlamb> one or the other is fine, but both I can't seem to finesse
14:39 -!- dhlamb [~dhlamb@xplr-96-44-94-111.xplornet.com] has quit [Quit: Leaving]
14:40 <Natkeeran> https://ffmpeg.org/ffmpeg-protocols.html#pipe

This issue is related to #929

[jonathangreen]

I dunno if this is the case, I'm really just speculating here, but this ticket sounds like a pipe race condition to me.

If FFMPEG fills the STDERR pipe it could be waiting for that pipe to empty before continuing to process data out to STDOUT. And in our code we completely read STDOUT before we read STDERR at all.

We might want to make sure we are emptying STDERR as we drain STDOUT to make sure a race like this doesn't happen. I've seen a similar bug before in the imagemagick module (Patch here).

We might be able to replace this code with some nifty PHP7 library code to do this for us, since working with pipes in a non-blocking way is always an error filled thing, offload the work to someone else, but I don't have any libraries to suggest.

[dannylamb]

@jonathangreen FFMPEG does fill STDERR with error messages, so that is very likely the issue. We probably also want to do something with it in the event of an error. I found this useful when digging around for some answers: https://trac.ffmpeg.org/wiki/PHP

We also use - as a placeholder in the command to execute, and we should use /dev/stdin and /dev/stdout as well. When toying with the command @Natkeeran is struggling with, that was the only way I could get it to emulate what PHP was attempting to do.

[Natkeeran]

@dannylamb @jonathangreen

It is actually hanging here: https://github.com/Islandora-CLAW/Crayfish-Commons/blob/master/src/CmdExecuteService.php#L65, not on L83 as I originally noted.

Jonathan's fix does not seem to help.

[seth-shaw-unlv]

Well, according to http://php.net/manual/en/function.feof.php there are a few reasons why it might hang there, all dependent on how that passed $data variable is set up. Although, I haven't looked at how execute gets called to see if they apply.

…

On Thu, Sep 27, 2018 at 2:36 PM Natkeeran ***@***.***> wrote: @dannylamb <https://github.com/dannylamb> @jonathangreen <https://github.com/jonathangreen> It is actually hanging here: https://github.com/Islandora-CLAW/Crayfish-Commons/blob/master/src/CmdExecuteService.php#L65, not on L83 as I originally noted. Jonathan's fix does not seem to help. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#943 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AcfHpF93BoohWDCRjaee454biN0Nild1ks5ufUTIgaJpZM4W4_kP> .

[dannylamb]

@Natkeeran Any updates on this? We keep missing each other on IRC. Is there anything I can help you with?

[dannylamb]

@Natkeeran https://github.com/PHP-FFMpeg/PHP-FFMpeg/#using-with-silex-microframework Looks like there's some OO wrappers that might be worth checking out. They've even got a service provider for silex.

[dannylamb]

@Natkeeran Yeah... so... looks like you'd have to be very explicit with PHP-FFMpeg. And I've played around with streams for a few hours now and have had absolutely no luck. I cannot seem to specify the format of the input stream for the life of me.

Let's just save the request body to a temp file and pass that to ffmpeg. That's what PHP does anyway when uploading files 🤷‍♂️ It ruins my ideal world of just passing streams around all the time, but pragmatically... let's just write the temp file.

[Natkeeran]

This issue is related to how ffmpeg handles input stream. We are working around this issue by directly providing the http url of the resource.

[fhernandezl]

I recently installed CLAW using the claw-playbook. I noticed that when I added videos to a repository item, the derivatives were not generated. I saw on homarus.log that the command was being executed, but it remained at that stage. I tried running the command on my command line to see what was going on, and noticed that at some point I was asked for input. After some research, I added the flag -nostdin to this line and that fixed it. I'm not sure if anyone else has experienced this behaviour or if this is the right solution for it.