HandbrakeCLI vs FFMPEG on GEARMAN setup

Finally we completed 4 out 5 dedicated video encoder using Mac Mini i5 4GB ram each. For encoder #2, we use FFMPEG and the rests are using HandBrakeCLI. All machine connected to the storage server via LAN and only processing power that we can test at this moment of truth.

The test :

1. Sample.mp4 copied to storage server
2. Gearman choose which Worker  to process Sample.mp4
3. Worker download Sample.mp4
4. Worker started to encode in PHP loop to generate 3 profiles ( 400kbps,700kbps,1100kbps )
5. Worker upload back all the videos to Storage server

The results :

• FFMPEG use more processing power
• FFMPEG encode much slower

So from a very rough test, we decided to use HandBrakeCLI for our production servers.

Joomla tuning on high traffic website

Our client use JOOMLA extensively and hosted on cloud VPS, and there are times where the sites received up to 5000 users at once due to special event like local football match.

So we need to tune the server to accomodate the high concurrent users. We use Varnish-Cache to cache the dynamic page and relocate static contents like CSS,IMAGE,JS to another server. A clever use of RSYNC and .htaccess made this possible. So the load for static contents will be thrown to our NGINX server.

For database, we use MySQL and to optimize the performance, we use MySQL-CACHE. Monitored using MYTOP during the event, MYSQL-CACHE HITS did save us.

And for JOOMLA itself, we only change the SESSION HANDLER from DATABASE to NONE ( file based ).

With more budget, we would relocate everything to AWS and design a more reliable and resilience architecture over there like AWS LoadBalancer, RDS,Route53 and lots more.

Encoding on different server managed by Gearman

Available workers and its status

4 videos being processed in parallel

Encoding duration for each file

We finally send the encoding job to external server in the network. No longer burdening the storage server for encoding job ( although the storage server did very well with its 8 cores CPU )

Gearman will queue the job and choose which server. On the server parth, upon receiving job, they will mark the VODCMS database  and user can see which server their files being sent.

MyKLIK Architecture Design

High Level Design

MyKLIK originally was a small scale web application. User just record the TV program, encode manually and upload to server and playback using HTTP Pseudo Streaming using FLV.

But those were early days of MyKLIK. Now with increasing and more engaging customers ( thanks to Google Analytics data ), MyKLIK need to evolve.

Currently we serving 4 departments, with 5 uploaders/editors per department with average 500mb to 1GB per RAW video. And each video need to be transcoded into 400,700 and 1100kbps for multibitrate delivery.

We solved the video queue by adding more encoders. We put 2 workers per encoder. Now our encoding capacity is processing 8 videos in parallel.

Gearman Workers in action.

Understanding Gearman :: gearadmin

gearadmin --status & htop -d 5

Now we have workers running and gearman working great. But we want to know the current status of the jobs sent.

1) We sent 5 jobs using PHP Gearman Client and using runTasks(); function
2) We started 3 workers from Terminal
3) All 3 workers start picking the jobs and left 2 in queue

gearadmin -- status will show as following :

• Function Name
• Number in queue
• Number of jobs running
• Number of capable workers

Distributed encoding using Gearman,PHP,HandBrakeCLI tests

The Gearman Client on Storage Server

The Gearman Worker on MacMini

We wrote a simple script that will download the master video from storage server and encode locally, after that push them all back to storage.

We have 4 unused Mac Mini i5 with 4 CPU Cores each.

Gearman Job Server will be on Storage Server and will listening to Job query and Job load.

Our plan is :

1) User upload raw video file to /incoming folder
2) InCron detects and invoke PHP script
3) PHP script send the job to Gearman Job Server
4) Gearman Job Server caling out the workers ( 4 mac minis )
5) Worker download the master file and encode to 3 different bitrates
6) Each Worker have SupervisorD daemon on each machine and set max to 1 worker per machine
7) when completed, Storage server will receive on /encoded folder and trigger Amazon S3 command to sync with our AWS S3 folder
8) Our Wowza Server in EC2 will packetized to VOD into multi format ( HLS,HDS,MPEG-DASH)
9) Serve the video using Amazon Cloudfront

At the end, we can increase the encoding capabilities by adding more Mac Minis.

to tests :
- Gearman Round Robin
- Mac OSX long running PHP scripts ( no Upstart / SupervisorD )

Increasing the number of Gearman workers using SupervisorD

Currently our online encoder have 8 Cores CPU and each Worker will output 3 video qualities. Means, that FFMPEG will work with all available CPU core.

Today we want to test what if we run the 5 workers at once. After reading this google search result, we adjusted our /etc/supervisord.conf and increaces the numprocs value from 1 to 5. Restarted the SupervisorD service and we can see all 5 workers working in parallel.