media_nommer.core

The core module is home to code that is important to both the media_nommer.ec2nommerd and media_nommer.feederd top-level modules. This is best thought of as a lower level API that all of the pieces of the encoding system use to communicate and interact with one another.

Note

There should be nothing that is specific to just ec2nommerd or just feederd here if at all possible.

storage_backends

Storage backends are used to handle download and uploading content to a number of different protocols in an abstracted manner.

media-nommer uses URI strings to represent media locations, whether it be the file to download and encode, or where the output should be uploaded to. A reference to the correct backend for a URI can be found using the get_backend_for_uri() function.

media_nommer.core.storage_backends.get_backend_for_protocol(protocol)

Given a protocol string, return the storage backend that has been tasked with serving said protocol.

Parameters:protocol (str) – A protocol string like ‘http’, ‘ftp’, or ‘s3’. Returns:A storage backend for the specified protocol.

media_nommer.core.storage_backends.get_backend_for_uri(uri)

Given a URI string , return a reference to the storage backend class capable of interacting with the protocol seen in the URI.

Parameters:uri (str) – The URI to find the appropriate storage backend for. Return type:StorageBackend Returns:A reference to the backend class to use with this URI.

s3

This module contains an S3Backend class for working with URIs that have an s3:// protocol specified.

class media_nommer.core.storage_backends.s3.S3Backend

Abstracts access to S3 via the common set of file storage backend methods.

classmethod download_file(uri, fobj)

Given a URI, download the file to the fobj file-like object.

Parameters:

• uri (str) – The URI of a file to download.
• fobj (file) – A file-like object to download the file to.

Return type:

file

Returns:

A file handle to the downloaded file.

classmethod upload_file(uri, fobj)

Given a file-like object, upload it to the specified URI.

Parameters:

• uri (str) – The URI to upload the file to.
• fobj (file) – The file-like object to populate the S3 key from.

Return type:

boto.s3.key.Key

Returns:

The newly set boto key.

job_state_backend

The job state backend provides a simple API for feederd and ec2nommerd to store and retrieve job state information. This can be everything from the list of currently un-finished jobs, to the down-to-the-minute status of individual jobs.

There are two pieces to the job state backend:

• The EncodingJob class is used to interact with and manipulate individual jobs and their state.
• The JobStateBackend class is used to operate as the topmost manager. It can track all of the currently running jobs, get new jobs from the SQS queue, and process state change notifications from SQS.

class media_nommer.core.job_state_backend.EncodingJob(source_path, dest_path, nommer, job_options, unique_id=None, job_state='PENDING', job_state_details=None, notify_url=None, creation_dtime=None, last_modified_dtime=None)

Represents a single encoding job. This class handles the serialization and de-serialization involved when saving and loading encoding jobs to and from SimpleDB.

Tip

You generally won’t be instantiating these objects yourself. To retrieve an existing job, you may use JobStateBackend.get_job_object_from_id().

Parameters:

• source_path (str) – The URI to the source media to encode.
• dest_path (str) – The URI to upload the encoded media to.
• job_options (dict) – The job options to pass to the Nommer in key/value format. See each Nommer’s documentation for details on accepted options.
• unique_id (str) – The unique hash ID for this job. If save() is called and this value is None, an ID will be generated for this job.
• job_state (str) – The state that the job is in.
• job_state_details (str) – Any details to go along with whatever job state this job is in. For example, if job_state is ERROR, this keyword might contain an error message.
• notify_url (str) – The URL to hit when this job has finished.
• creation_dtime (datetime.datetime) – The time when this job was created.
• last_modified_dtime (datetime.datetime) – The time when this job was last modified.

is_finished()

Returns True if this job is in a finished state.

Return type:bool Returns:True if the job is in a finished state, or False if not.

save()

Serializes and saves the job to SimpleDB. In the case of a newly instantiated job, also handles queueing the job up into the new job queue.

Return type:str Returns:The unique ID of the job.

set_job_state(job_state, details=None)

Sets the job’s state and saves it to the backend. Sends a notification to feederd to re-load the job’s data from SimpleDB via SQS.

Parameters:

• job_state (str) – The state to set the job to.
• job_state_details (str) – Any details to go along with whatever job state this job is in. For example, if job_state is ERROR, this keyword might contain an error message.

class media_nommer.core.job_state_backend.JobStateBackend

Abstracts storing and retrieving job state information to and from SimpleDB. Jobs are represented through the EncodingJob class, which are instantiated and returned as needed.

FINISHED_STATES = ['FINISHED', 'ERROR', 'ABANDONED']

Any jobs in the following states are considered “finished” in that we won’t do anything else with them. This is a list of strings.

JOB_STATES = ['PENDING', 'DOWNLOADING', 'ENCODING', 'UPLOADING', 'FINISHED', 'ERROR', 'ABANDONED']

All possible job states as a list of strings.

classmethod get_job_object_from_id(unique_id)

Given a job’s unique ID, return an EncodingJob instance.

TODO: Make this raise a more specific exception.

Parameters:unique_id (str) – An EncodingJob‘s unique ID.

classmethod get_unfinished_jobs()

Queries SimpleDB for a list of pending jobs that have not yet been finished.

Return type:list Returns:A list of unfinished EncodingJob objects.

classmethod pop_new_jobs_from_queue(num_to_pop)

Pops any new jobs from the job queue.

Warning

Once jobs are popped from a queue and delete() is ran on the message, they are gone for good. Be careful to handle errors in the methods higher on the call stack that use this method.

Parameters:num_to_pop (int) – Pop up to this many jobs from the queue at once. This can be up to 10, as per SimpleDB limitations. Return type:list Returns:A list of EncodingJob objects.

classmethod pop_state_changes_from_queue(num_to_pop)

Pops any recent state changes from the queue.

Warning

Once jobs are popped from a queue and delete() is ran on the message, they are gone for good. Be careful to handle errors in the methods higher on the call stack that use this method.

Parameters:num_to_pop (int) – Pop up to this many jobs from the queue at once. This can be up to 10, as per SimpleDB limitations. Return type:list Returns:A list of EncodingJob objects.

classmethod wipe_all_job_data()

Deletes the SimpleDB domain and empties the SQS queue. These are both used to store and communicate job state data.

Return type:bool Returns:True if successful. False if not.