media-nommer is a Python-based, distributed media encoding system. It aims to implement much of the same services offered by commercial encoding services. The primary advantage afforded by media-nommer is that one only pays for what they use on Amazon AWS, with no additional price mark-up. It may also be completely customized to meet your needs.
There are any number of encoding services around the web, some of which are cheap and very easy to use. We (DUO Interactive) were searching for a suitable fit for two different projects that an extreme amount of encoding scalability. However, the more we looked into it, the more we struggled with picking a service that had a good balance of affordability and permanence. We also were looking for a little more customizability in the encoding process.
Many encoding services use software such as FFmpeg, and encode media to codecs that they have not paid licensing fees for. If the service has any degree of success, it is only a matter of time before MPEG-LA and others come knocking to get their licensing fees. This often spells end-of-game for the smaller (often cheaper) services. On the other end of the spectrum, some 100% legal services have to raise their prices up to cover the large licensing fees, putting it outside the realm of what we’d like to pay. A good read on this subject can be found on the FFmpeg legal page.
For those of us who don’t need to encode to royalty-encumbered formats, a cheaper, more customizable, and more permanent solution was to just write our own. The result of all of this was the creation of media-nommer, a Python-based media encoding system with scalability and flexibility in mind.
Distributed encoding requires a director or orchestrator to keep everything running as it should. This is where the feederd daemon comes into play. Here are some of the things it does:
feederd may be ran on your current infrastructure, or on EC2. Your EC2 encoding nodes are never in direct contact with feederd, and instead communicate through Amazon AWS. This means no firewall holes need to be opened.
Where feederd is the manager, ec2nommerd is the worker. Your basic unit of work is an EC2 instance that runs ec2nommerd. This daemon simple checks a SQS queue for jobs that need to be encoded, and does so until the queue is empty. After a period of inactivity, your EC2 instances will terminate themselves, saving you money.