EncodeBox Transcoding Watchfolder

About

EncodeBox is aimed by the goal of providing a watch-folder that transcode the input media files to various output formats (adaptive streaming). The outputs are sent to a streaming server for streaming from Wowza.

EncodeBox is, at time of writing this, composed of two services controlled by supervisor:

1. The watch-folder daemon.py: Launch a new transcoding task for each new file in the input folder.
2. The workers tasks.py: Detect the quality (SD/HD) of the input file and transcode to multiple output files.

The settings file /etc/encodebox/config.yaml of EncodeBox permit to configure the following options:

filenames_seed:	(filenames_seed) The seed for the function generating unguessable filenames.
local_directory:	(/var/www/data) All the files managed by EncodeBox must be transfered there.
remote_directory:	(username@host_ip:/usr/local/WowzaStreamingEngine/content) The outputs are copied to this directory with rsync. Can be remote or local.
remote_url:	(http://host_ip:1935/vod/content/{publisher_id}/{product_id}/smil:{name}.smil/Manifest) The Wowza player use this kind of URL, so here is the template.
completed_cleanup_delay:	Completed files are removed if older than this delay in seconds, default means 7 days.
api_servers:	([url: http://127.0.0.1:5000/encoding/report, auth: null]) The progress reports of the transcoding tasks will be POST (API) to the given servers. Set to null to disable this feature.
email_body:	(/etc/encodebox/email_body.j2) The template used to generate the body of the error report.
email_host:	(smtp.gmail.com) The SMTP host to send the e-mail messages.
email_username:	(encodebox.test@gmail.com) The username of the mailbox used to send the e-mail messages.
email_password:	(1**************e) The password of the mailbox used to send the e-mail messages.
email_recipients:	(['aaron@sigalamedia.com', 'david.fischer.ch@gmail.com']) Recipients for the error reports. Set to null to disable this feature.
hd_smil_template:	(/etc/encodebox/hd.smil) The absolute path to the template SMIL file for the HD content.
sd_smil_template:	(/etc/encodebox/sd.smil) The absolute path to the template SMIL file for the SD content.
hd_transcode_passes:	(a long list) The worker follows this list of passes (calls to encoders) to transcode the HD content.
sd_transcode_passes:	(a long list) The worker follows this list of passes (calls to encoders) to transcode the SD content.

The transcoding workers will handle any input file into local_directory/user_id/product_id/uploaded/ by using the following structure:

• The intermediate files are temporarily stored into the sub-directory <local_directory>/user_id/product_id/temporary/.
• The outputs are saved into the sub-directory <local_directory>/user_id/product_id/outputs/.
• The input file is moved into <local_directory>/user_id/product_id/completed/ in case of success.
• The input file is moved into <local_directory>/user_id/product_id/failed/ in case of failure.

The transcoding workers will (try) to POST the following informations at the URL api_url:

{

"publisher_id": "5", "product_id": "7", "filename": "Abyss.mkv", "original_size": 473005964, "state": "ENCODING", "progress": 0.67316, "elapsed": 415.329475, "eta": 135, "url": null

}

Remark: The url key is set to the URL remote_url only if the task succeeded, when the media asset is available for streaming.

Following this state-machine:

The state-machine of a media asset handled by EncodeBox.

State of the project

This project is currently a beta release that implement the complex encoding workflow trigerred for each new file in the inputs directory. The transcoding worker is now able to parse both the output of FFmpeg and x264 to retrieve the statistics like ETA, fps, progress %, ... It means that we can report a PROGRESS status with both the current pass (e.g. 4 of 14) and the statistics about the current pass.

The "core" available features are:

• The watch-folder daemon react to IN_CLOSE_WRITE and launch a new transcoding task

• The transcoding worker is able to :

  • Detect the resolution of the input media file
  • Select the appropriate transcoding steps (described in the settings file /etc/encodebox/config.yaml)
  • Execute the transcoding steps and update the status of the task to ENCODING + statistics
  • Generate the SMIL file according to a template
  • Remove the temporary files
  • Remove the output files in case of error
  • Update the status of the task to SUCCESS or FAILURE + console output of the encoder
  • POST the progress reports to the API during the whole transcoding process
  • Copy the output files to the remote streaming server

• The periodic cleanup task remove completed files older than 7 days

Some "extra" are also available to help developers:

• The test API server to collect progress of the transcoding tasks for debugging purposes.
• The test API client reporting the progress of the transcoding tasks by calling the test API server.

Some "core" features are also not yet implemented:

• The watch-folder does not revoke/relaunch tasks if the input files are removed or updated during transcoding (issue #13)

What the release number stands for ?

I do my best to follow this interesting recommendation : Semantic Versioning 2.0.0

How to install it ?

Add the following PPA if you want to install the real FFmpeg:

sudo apt-add-repository ppa:jon-severinsson/ffmpeg
sudo apt-get update

Make sure that pip is up-to-date (PIPception):

sudo pip install --upgrade pip

Then, you only need to run setup.py:

python setup.py test
sudo python setup.py install

You may also install the optional Celery web interface (Flower):

sudo pip install flower

How to configure it ?

• The main configuration file is /etc/encodebox/config.yaml.
• The template SMIL files are /etc/encodebox/{sd.smil,hd.smil}.
• The workers configuration file is celeryconfig.py.
• The services are registered in /etc/supervisor/encodebox.conf.

How to use it ?

Manage the services:

sudo service supervisor {start|stop|restart|force-reload|status|force-stop}
sudo supervisorctl
> status
> restart
> ...

Follow the logs:

tail -f /var/log/encodebox-*.log

Watch the watch-folder directories:

watch ls -lh ~/EncodeBox/*/*/*

Start the optional Celery web interface (Flower):

celery flower &
xdg-open http://localhost:5555

Start the optional test API server:

screen -dmS api_server python -m encodebox.api_server

Use the test API client to get progress of the transcoding tasks:

watch python -m encodebox.api_client

A typical testing scenario:

1. Install, start EncodeBox and open two terminals, one to follow the logs, the other to monitor directories.
2. [optional] Install, start Flower and open a browser to monitor transcoding tasks and workers.
3. Start the test API server.
4. Copy some media files into the inputs directory ~/EncodeBox/inputs to trigger some new transcoding tasks.
5. Call the test API client few times or use watch to call it in a regular basis.