Distributed encoding is a complex task, with lots of moving parts and components. To create, monitor, and “motivate” our EC2 encoding instances during the process, we have feederd.
feederd is a Twisted plugin that runs as a daemon on the machine of your choice. This can be on your own bare metal server, or on an EC2 instance. The daemon keeps track of encoding jobs through Amazon SimpleDB, tells your EC2 encoder instances to get to work via Amazon SQS, and exposes a lightweight JSON web API for your own custom applications to schedule and manage jobs through.
If feederd is the brains of the operation, ec2nommerd is the brawn. feederd simply saves job state data to SimpleDB and creates SQS queue entries to let your EC2 instances know that there’s work to be done. Each of your EC2 instances run an ec2nommerd daemon, which pulls jobs from the queue (or terminates the instance if there isn’t any work to be done).
Along with providing an entry point to start, manage, and track the encoding process, feederd also handles scaling your encoding cloud up as needed. Based on your configuration, feederd will look at the current list of jobs that need to be encoded in comparison to the number of encoding instances you currently have running on EC2. It will then decide whether it should spawn additional instances to handle the load.
EC2 instances are spawned from a public AMI that we maintain as part of the project. If you wish to create your own custom AMI, you may easily specify your own or clone and modify ours.
Automated scaling is an optional feature, and can be configured and restricted in a number of different ways. For example, perhaps you don’t want any more than one or two EC2 instances running at any given time.
To actually get jobs scheduled and work done, you will need to communicate with feederd through its JSON API.
One of the requirements for media-nommer was that at no time would feederd and one of its ec2nommerd instances communicate directly. As noted earlier, job state information is saved and retrieved from SimpleDB by both feederd and ec2nommerd instances. SQS is used to delegate work to the ec2nommerd instances. This does a few things for us:
There is a lot of code that feederd and ec2nommerd shares, which can all be found in the media_nommer.core module. This is best thought of as a lower level API that these two components use to save and retrieve job state information.
Source for feederd can be found in the media_nommer.feederd module.