Greg Taylor Discusses Media Nommer with Ubuntu's Ahmed Kamal
Ahmed Kamal, the Ubuntu cloud community liaison interviews Greg Taylor to discuss “media-nommer” an open-source cloud based distributed media encoding system that runs over the Ubuntu server platform over Amazon EC2.
Ahmed Kamal: Hi Greg, obviously I'm very interested in cloud based technologies and scalable apps. I know you have done some great work building a distributed media encoder cloud application based on Ubuntu server running over the Amazon EC2 cloud. Would you please give us some background on yourself and media-nommer?
Greg Taylor: I am a semi-recent of graduate of Clemson University, now working for DUO Interactive, LLC in Greenville, South Carolina. Python has been my weapon of choice for the last five years or so, and Linux and I have been going steady for a bit over a decade now. My employer shares my love for open source software, and has put me in the position to be able to maintain the vast majority of our development stack in the open. media-nommer is just one of a growing number of projects that we maintain openly.
As far as media-nommer, it is a Python-based, distributed media encoding system that uses Amazon Web Services as its backbone. The aim is to implement many of the same capabilities offered by commercial encoding services, meanwhile offering massive scalability. The primary advantage afforded by media-nommer is that one only pays for what they use on Amazon Web Services, with no additional price mark-up. It may also be completely customized to meet your needs.
For those of us who don’t need to encode to royalty-encumbered formats, a cheaper, more customizable, and more permanent solution was to write our own. The result of all of this was the creation of media-nommer.
AK: When you say media encoding, do you mean video only? Can it do audio as well. Also I'm interested if media-nommer can be twisted to perform general purpose cloud based “crunching” say run an ocr application over thousands of pictures, or convert thousands of files to PDFs, etc. Is such a thing possible. Also, how fault tolerant is the system?
GT: media-nommer is still in heavy development, so we had to work on processing the kinds of media that our clients need first. For our purposes, we’ve been working on video encoding, but it would be easy enough to consume audio, text, images, or almost any other kind of media.
We consume (nom) media via "Nommers", which are simple classes that facilitate the encoding process. One could write a simple OCR or file conversion Nommer to handle any of the tasks you have mentioned. In fact writing nommers is quite easy!
Since the project is open source, you can check out the source code for the FFmpeg nommer that we (DUO) use for video encoding.
As you may have noticed, the bulk of the lines in this Nommer are for assembling command line options. It has very little front-facing complexity there.
Regarding fault tolerance, no single point of failure should immediately paralyze the system. If the master overseer daemon (feederd) goes down, any current jobs that are in the Amazon SQS Queue are still encoded. Work will continue until the job queue is emptied. After that, the EC2 instances will terminate themselves automatically after a period of inactivity. If an EC2 instance locks up, bugs out, or becomes otherwise uncooperative, it is terminated if it doesn’t check in as being “healthy”.
AK: Ubuntu is becoming an extremely popular cloud focused operating system. Either for running Ubuntu server over a commercial cloud such as EC2, or for building private clouds using technologies such as UEC (Ubuntu Enterprise Cloud). Can you let me know why you chose to go with Ubuntu and what are you impressions so far?
GT: Because of a few admittedly subjective factors (organization, community, and the presence of apt), I had always been more of a Debian lean for servers. I had been using Ubuntu personally since 5.04 as my Desktop, and really enjoyed it for my development work. Moving on towards Ubuntu 8.04 and 8.10, the platform matured to the point where I started running production servers on it. The last few companies I worked with adopted Ubuntu in their server rooms, and a track record of reliability and ease of administration was established.
Fast-forward to today, and Ubuntu is on the bleeding edge of the cloud movement, which gave us the tools we needed to create an easy way to deploy media-nommer encoding instances on Amazon EC2.
The “killer” feature that ultimately sold us on Ubuntu for media-nommer was cloud-init. Cloud-init is an innovative Ubuntu technology for customizing an Ubuntu server instance as it boots in the cloud. The beauty of this approach is that you keep one clean and small Ubuntu server image, which can shape-shift into whatever you need (web-server, DB-server, encoder node ...etc). We didn't want any users of media-nommer to have to set up their own AMI, unless they wanted to customize.
The end result of all of this is that a user can install our package, set three or four config options, start a daemon, and get encoding.
AK: Obviously Ubuntu strongly believes in open-source. In fact, not only the code is open, but the whole design and coding cycles are done in the open. However for many this model is still not familiar. Could you let us know why you chose to open source media-nommer?
GT: The primary reason is that I work for an open source-reliant company with some great clients who allow us to publish and maintain components openly. As a company, our philosophy is that if we're working on something that we don't plan on directly profiting from, it might as well be open sourced so others may use/improve it. This can be almost anything that doesn’t contain any business logic specific to our clients.
As mentioned before, our reliance on open source software is huge. It helps us get things done quickly, and it allows us to collaborate with others to continuously improve our development stack. Since the many open projects out there have helped us so much, it is only “right” that we in turn make some of the cool things we write, using said projects, open too.
While the term “Open Source” can be seen as a dirty word by some in the business sector, the best way to change that perception is to continue to produce and maintain excellent tools and utilities in an open manner. Ubuntu is an excellent example of a banner-holder for open source, in that it has a great balance of support, open development, and an excellent community.
We are thankful for all that have invested time in Linux, Debian, Ubuntu, and open source software in general. You have all have enabled us to create media-nommer, and other projects. We now hope to share it with all who need it, as did those who wrote the tools that we use every day at DUO Interactive.
AK: Thanks Greg! All the best of luck with media-nommer and other projects. Users can learn how to interact with the Ubuntu Cloud community on the Ubuntu Cloud Webpages, the cloud portal and by following the cloud planet RSS stream.
Comments