Introduction

We live in the content economy today. From every corner, content is generated & distributed for our consumption. And video has taken the center stage of all content strategies. Be it advertising & marketing or online learning, video content is the primary format being consumed. Naturally, this has increasingly seen rise in the number of applications which need to support videos, and eventually video processing.
Video processing can encompass a variety of activities. From compression to quality and from content editing to accessibility, there are multiple options available that can be used to convert a raw video into a desirable output.

Undoubtedly, there are a lot of tools and libraries available out there which can be used when it comes to video processing. Today, we are going to focus on utilizing some of them to write our own video processing code. For this use case, we have considered FFMPEG, NodeJs and Gearman.

The first thing we need to do while considering video processing is to analyse our requirements. Usual Video Processing Requirements include factors such as Scalability, Time to process, response time, Size, load of video and amount of processing to be done.
Our Requirements:

• Scalability – high
• Time to process and response time – low
• Size and load of video – medium
• Amount of processing to be done – scalable
• Should be robust and smooth while the application is available to the user
• Client Side or Server Side?

We decided to take our processing on the server side considering various factors (we even experimented with both on a small piece of code). The considerations were :
Cost
Server side rendering and computation is going to require CPU cycles but client side rendering is going to take a greater response time which can have a negative impact on our application.
Load and capability
Potential load or capability issues on the client. Based on the particular target demographic and their phone usage, our application was already loading content from the server and thus the rendering time on client will be longer.
Storage and visibility
Our processed video needs to be shared on various platforms and should be available to an administrator or client. Given this requirement, the applicationuser will have to upload the video after processing in background which would have a negative impact as well.
If the video needs to stored and be available locally, client-side processing is going to handle that seamlessly.
Files required in processing
Audio, image, text files etc need to be available to the client end for adding extra content. If client-side downloads multiple files or stores them on the application itself, then our application will become bulky?

Server Side Video Processing

Based on our requirements, we decided to process our videos on the server for the following reasons:
To avoid application downtime
To keep our application relatively lightweight
To allow additions/deletions in video processing to exist as an isolated entity thereby avoiding many changes and builds in the application.
To be platform independent which will allow us to easily plug in video processing with applications that we may develop in the future on other platforms

We choose to use the following libraries:
NodeJS
Gearman
FFmpeg
NodeJS

NodeJS

NodeJS is an open-source JavaScript runtime which is lightweight and efficient. It has a very active and vast community and thus, it offers many libraries that can be utilized to accomplish multiple number of tasks like, downloading files, uploading files, deleting temporary files. Since NodeJS is asynchronous, promises or callbacks can be used to ensure processing is done linearly for a given video (if required).

Gearman

Gearman is an open-source application framework or tool for load and task distribution. It is important to handle multiple processes running at the same time ensuring maximum utilization of the processing units in the server or system. It also saves time as tasks are taken up parallely.

Gearman in video processing can be used to process multiple videos together based on the number of CPUs present on the server.
For Gearman as well, wrapper libraries written in multiple languages are in plenty. We can also use Gearman directly based on the command line interface.
It watches the service thread for new commands and assigns to CPU with the least amount of work.

FFmpeg

FFmpeg: We were looking for a tool that can accomplish video processing tasks in the background. We chose FFmpeg as it is mature and reliable tool that can be used in the background to accomplish video processing giving high flexibility and control over the video to the user. It is also faster and can deal with complex filters and video capabilities easily.

FFmpeg is a multimedia framework able to decode, encode, transcode, mux, demux, stream, filter and play almost any type of content. FFmpeg compiles and runs in a wide variety of environments such as Linux, Mac OS X, Microsoft Windows under different machine architectures, and configurations.

FFmpeg has several open source libraries and wrappers in languages like python, javascript. These libraries may not offer all the capabilities of FFmpeg but they offer the basic to moderate functionality which can be used by less complex video processing requirements.

Other libraries: avconv can also be used. It is a fork of ffmpeg, maintained by former FFmpeg developers. FFmpeg has added aliases so that users can use either the ffmpeg API or that of avconv/LibAv.

Installing FFMpeg

Node package: @ffmpeg-installer/ffmpeg can be used to automatically install the proper ffmpeg executable for your system, right in your project’s node_modules folder.

Starting with FFmpeg

FFmpeg has a great documentation so one can easily understand the basics. It has extensive number of options available ranging between videos, images, audio and filtering. The command line code that needs to be used finally may look confusing and can be a learning curve for anybody who is new to FFmpeg. It is a recommended practice to start FFmpeg with a basic command and then add parameters one by one accordingly.

FFmpeg using command line (sample):

ffmpeg -i /path/to/input.avi -codec:a:1 libmp3lame -codec:v libx264 /path/to/output.mp4

Using Node Libraries

Gearmanode

We can start with a node package: gearmanode package which is wrapper for Gearman on nodeJS. We need to setup gearman on our system and start the process in the background on a particular port. The nodeJS library connects with the running process and allows us to setup gearman in our project directly. This makes change management fast and easy to understand.

Fluent

We can start with a node package: fluent-ffmpeg package which is a wrapper to FFmpeg. Fluent APIs are interfaces where every method call returns the object that is the owner of that method. This lets you chain together a series of method calls in a single statement like a pipeline, rather than write as many separate statements, each making a method call on the object. This approach is well-suited for tasks such as configuring multiple options on an ffmpeg command. Although Fluent sufficiently fulfils most of the requirements of FFmpeg, it may miss out on certain parameters that need to be managed by creating the FFmpeg command directly. For that, we can use NodeJS child process to run the command on an command line interface that will run in the background and return the output and error back to the Node function.

Get an ffmpeg command upon which to set all these options:

var ffmpegPath = require('@ffmpeg-installer/ffmpeg').path;
var ffmpeg = require('fluent-ffmpeg');
ffmpeg.setFfmpegPath(ffmpegPath);
var command = ffmpeg();

Command Example:

ffmpeg('/path/to/input.avi')
.audioCodec('libmp3lame')
.videoCodec('libx264')
.size('320x240')
.on('error', function(err) {
console.log('An error occurred: ' + err.message);
})
.on('end', function() {
console.log('Processing finished !');
})
.save('/path/to/output.mp4');

Implementation

Setup for Gearman, NodeJS and FFMPEG is available along with their documentation. Once we have installed the required software, we can go ahead.

Start gearman and FFMPEG as a background job using the following command.

gearman -d
ffmpeg start

Once both have been started in the background, we can proceed with setting up our project and node libraries for Gearman and FFmpeg.
For our particular case, we needed greater control over our video processing and so used FFMPEG via command line and Gearman node library for our particular implementation.

Gearman

Add client and worker for gearman following along the lines of the given code snippet. The worker will keep waiting for a job to be processed while the client will send it applicable jobs when available.
Client is responsible for assigning the appropriate job to the given worker. Worker is responsible for taking the video and apply the given function to it.

Client: video.processing.client.js:

var gearmanNode = require('gearmanode');
var videoProcessingClient = gearmanNode.client();
var job;

job.on('workData', function(data) {
console.log(' Gearman worker started processing of video :' + data);
});
};

Worker: video.processing.worker.js:

var gearmanNode = require('gearmanode');
var videoProcessingWrapper = require('./video.processing.wrapper');
var videoProcessingWorker = gearmanNode.worker();
videoProcessingWorker.addFunction('processVideo',function(job){
videoProcessingWrapper.processVideo(job);
});

Note:
A machine can have number of workers equal to the number of cores in the system. If we use clustering in Gearman it helps us to use our resources efficiently and assign jobs to each worker in a core depending on it availability. Finally gearman will handle traffic and errors accordingly and thus keep your project up and running.

Video processing and FFMPEG

Now, write your video processing code under the file ‘video.processing.wrapper.js’ under the same directory as the gearman client and worker.

video.processing.wrapper.js:

exports.processVideo = function (videoId)
{
// code goes here using FFMPEG
};

Running the project

Run the worker first. It will allow worker to be ready if a job is sent by the client.
Run the client. We had defined a cron job that queried for unprocessed videos and called the client with the video details.

videoProcessingClient.submitVideoForProcessing(videosToProcess[i]._id);

Ending Note

Our Observations

• FFmpeg is CPU consuming.
• Video output more than usually requires multiple tests on variety of videos ranging from size, quality and type to get the most appropriate or desirable output.
• Having a test video suite where changes are tested in an automated script can reduce test time drastically.