The feederd module contains Twisted daemon that monitors queues, creates EC2 instances, schedules encoding jobs, handles response pings from external encoding services, and notifies your external applications when encoding has completed.
Note
Only more general management logic should live in this module. Anything specific like encoding commands or EC2 management should live elsewhere. feederd is just the orchestrator.
Nommers are classes that EC2 instances use with ec2nommerd to consume and produce media in the desired format. These usually involve wrapping external commands, setting job state details as the process continues, and uploading the results.
Classes in this module serve as a basis for Nommers. This should be thought of as a protocol or a foundation to assist in maintaining a consistent API between Nommers.
This is a base class that can be sub-classed by each Nommer to serve as a foundation. Required methods raise a NotImplementedError exception by default, unless overridden by child classes.
| Variables: | job (EncodingJob) – The encoding job this nommer is handling. |
|---|
Start nomming. If you’re going to override this, make sure to follow the same basic job state updating flow if possible.
Contains a class used for nomming media on EC2 via FFmpeg. This is used in conjunction with the ec2nommerd Twisted plugin.
This Nommer is used to encode media with the excellent FFmpeg utility.
Example API request
Below is an example API request. The job_options dict that is passed to the media_nommer.feederd JSON API is the important part that is specific to this nommer.:
{
'source_path': 'some_video.mp4',
'dest_path': 'some_video_hqual.mp4',
'notify_url': 'http://somewhere.com:8000/job_state_pingback',
'job_options': {
'nommer': 'media_nommer.ec2nommerd.nommers.ffmpeg.FFmpegNommer',
# This options key has ffmpeg command line arguments for a 2-pass
# encoding. If you're doing single pass, you'd only have one
# dict in this list.
'options': [
# First pass command specification.
{
# Just documenting this here so its existence is known.
'infile_options': [],
# Fed to ffmpeg as command line flags. A None key means
# that just the flag name is provided with no arg.
'outfile_options': [
('threads', 0),
('vcodec', 'libx264'),
('preset', 'medium'),
('profile', 'baseline'),
('b', '400k'),
('vf', 'yadif,scale=640:-1'),
# This denotes the first pass in ffmpeg.
('pass', '1'),
('f', 'mp4'),
('an', None),
],
},
# Second pass command specification.
{
'outfile_options': [
('threads', 0),
('vcodec', 'libx264'),
('preset', 'medium'),
('profile', 'baseline'),
('b', '400k'),
('vf', 'yadif,scale=640:-1'),
# Notice that this is now 2, for the second pass.
('pass', '2'),
('acodec', 'libfaac'),
('ab', '128k'),
('ar', '48000'),
('ac', '2'),
('f', 'mp4'),
],
},
], # end options list, max of 2 passes.
}, # end job_options
}
To show how this would be put together, here is the command that would be ran for the first pass:
ffmpeg -y -i some_video.mp4 -threads 0 -vcodec libx264 -preset medium -profile baseline -b 400k -vf yadif,scale=640:-1 -pass 1 -f mp4 -an /dev/null
Note that the an key in the outfile_options list of the first pass above has a None value. You’ll need to do this for flags or options that don’t require a value.
This module contains tasks that are executed at intervals, and is imported at the time the server is started. The intervals at which the tasks run are configurable via media_nommer.conf.settings.
All functions prefixed with task_ are task functions that are registered with the Twisted reactor. All functions prefixed with threaded_ are the interesting bits that actually do things.
Registers all tasks. Called by the ec2nommerd Twisted plugin.
Looks at the number of currently active threads and compares it against the MAX_ENCODING_JOBS_PER_EC2_INSTANCE setting. If we are under the max, fire up another thread for encoding additional job(s).
The interval at which ec2nommerd checks for new jobs is determined by the NOMMERD_NEW_JOB_CHECK_INTERVAL setting.
Calls threaded_encode_job() for any jobs to encode.
Checks in with feederd in a non-blocking manner via threaded_heartbeat().
Calls threaded_heartbeat().
Given a job, run it through its encoding workflow in a non-blocking manner.
Fires off a threaded task to check in with feederd via SimpleDB. There is a domain that contains all of the running EC2 instances and their unique IDs, along with some state data.
The interval at which heartbeats occur is determined by the NOMMERD_HEARTBEAT_INTERVAL <media_nommer.conf.settings.NOMMERD_HEARTBEAT_INTERVAL setting.
Contains the NodeStateManager class, which is an abstraction layer for storing and communicating the status of EC2 nodes.
Tracks this node’s state, reports it to feederd, and terminates itself if certain conditions of inactivity are met.
Looks at how long it’s been since this worker has done something, and decides whether to self-terminate.
| Parameters: | thread_count_mod (int) – Add this to the amount returned by the call to get_num_active_threads(). This is useful when calling this method from a non-encoder thread. |
|---|---|
| Return type: | bool |
| Returns: | True if this instance terminated itself, False if not. |
Determine this EC2 instance’s unique instance ID. Lazy load this, and avoid further re-queries after the first one.
| Parameters: | is_local (bool) – When True, don’t try to hit EC2’s meta data server, When False, just make up a unique ID. |
|---|---|
| Return type: | str |
| Returns: | The EC2 instance’s ID. |
Checks the reactor’s threadpool to see how many threads are currently working. This can be used to determine how busy this node is.
| Return type: | int |
|---|---|
| Returns: | The number of active threads. |
Pat ourselves on the back each time we do something.
Used for determining whether this node’s continued existence is necessary anymore in contemplate_termination().
Determine whether this is an EC2 instance or not.
| Return type: | bool |
|---|---|
| Returns: | True if this is an EC2 instance, False if otherwise. |
Sends a status update to feederd through SimpleDB. Lets the daemon know how many jobs this instance is crunching right now. Also updates a timestamp field to let feederd know how long it has been since the instance’s last check-in.
| Parameters: | state (str) – If this EC2 instance is anything but ACTIVE, pass the state here. This is useful during node termination. |
|---|