High usage on our online video encoder

top -d1 on our encoder server

We hit the roadblock, eventually. Storm of users uploaded massive video size to our server and the server CPU usage skyrocketed to 600% :D More than poor server can handle.

Our encoder design quite pretty forward,

User Upload Raw Video > FFMPEG encode to 3 bitrates > Store metadata in DB > Upload to CDN

Our mistakes

1) Design failure. We should use queue system like Gearman ( which is what we looking forward  now)
2) Use Cloud Service.We can the raw video to AWS Elastic Encoder and store the resulted video to S3
3) Not putting nice command infront of our encoder software ( ffmpeg & HandBrake CLI )
4) Unprepared for scaling. Classic design mistake, initially the server was designed for 5 uploaders with no expected growth during the contract period. Now we have 25 active uploaders :D
5) FFMPEG tuned for normal H264 conversion instead of FAST. Well we need to retain the original quality.

Now we are planning the new design.  2 dedicated encoder servers ( hardware ) and 1 storage to leverage the load.