![Page 1: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/1.jpg)
Distributed Video Transcoding
Rischan, Sunni
December, 18, 13
![Page 2: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/2.jpg)
What we’re doing?
Comparing output time when we transcode the video files in different mode:
● standalone mode● distributed but using program that we’re
made● distributed using twitter storm
![Page 3: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/3.jpg)
Apologize
We didn’t use OpenStack because …..
1. Server just has 2 GB RAM 2. Real Machine more faster than Virtual Machine.
![Page 4: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/4.jpg)
Standalone mode
1. Transcode video files using ffmpeg library in one PC.
2. No Split and concat.3. Calculate the time processes.
![Page 5: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/5.jpg)
Distributed using program that we made
1. The program developed by python.2. Video Splitter.3. Send the video to the worker.4. Call the Ffmpeg to transcode. (problem?)5. Send back video from worker to server.6. Video Concatter.7. Show time processes.
![Page 6: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/6.jpg)
distributed using twitter storm
Something that must we know for understanding storm:1. Setting storm environment for cluster.2. Components of a storm cluster.
![Page 7: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/7.jpg)
Setting Storm env1. We need more than one PC (we can use VM).
PC Server(x.x.x.160) :Ubuntu, Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz, RAM 2 GBPC Worker 1(x.x.x.160):Ubuntu, Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, RAM 8 GiBPC Worker 2(X.X.40.44) : Macbook, Intel Core 2 Duo 2.4 GHz, RAM 2 GiB
![Page 8: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/8.jpg)
Setting Storm env (2)2. Every PC must have Ffmpeg & storm3. Storm Server, we need :
● ZeroMQ● JZMQ● Java● Python● Zookeeper
![Page 9: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/9.jpg)
Storm Arch
Nimbus = JobTracker
![Page 10: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/10.jpg)
Installing Storm for ClusterDont forget to add id_rsa.pub workers in the server .ssh directory
wget https://raw.github.com/maltoe/storm-install/master/storm_install.sh
chmod +x storm_install.sh
storm_install.sh all <nimbus> <installdir>
![Page 11: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/11.jpg)
Running Storm
We must create topology and then submit the topology for running storm video transcoding & kill the topology for stop it.
![Page 12: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/12.jpg)
Result & Code Explanation
![Page 13: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/13.jpg)
File Informations
file : storm-video.mp4duration : 00:07:57.84resolution : 576x360size : 9.728 MB
![Page 14: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/14.jpg)
Standalone modeffmpeg -i storm-video.mp4 storm-video.mkv
Transcode from mp4 to mkv in PC Server :
0m36.936s
![Page 15: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/15.jpg)
distributed using program that we made
Code Explanation:
![Page 16: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/16.jpg)
Transcode Code
![Page 17: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/17.jpg)
Transcode Code (1)
![Page 18: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/18.jpg)
Transcode Code (2)
![Page 19: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/19.jpg)
Transcode Code (3)
![Page 20: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/20.jpg)
Transcode Code (4)
![Page 21: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/21.jpg)
Code for Distribution
![Page 22: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/22.jpg)
Code forDistribution1)
![Page 23: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/23.jpg)
Result:file size before split in server send to worker
transcode in worker send to server concat in server file size after note
9.728 MB 0m12.473s 0m0.456s 4.152 MB PC Server
0:00:43 0m6.484s 0:00:00 PC worker 1 (Lab)
0:00:44 1m4.308s 0:00:04PC Worker 2 (Dorm) Macbook
Total : 124 s / 2m4s
![Page 24: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/24.jpg)
Using Storm
![Page 25: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/25.jpg)
SplitVideo
![Page 26: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/26.jpg)
Transcode and Concat Video
![Page 27: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/27.jpg)
Main Topology
![Page 28: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/28.jpg)
VideoSpout
![Page 29: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/29.jpg)
Transcode Video Bolt [python]
![Page 30: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/30.jpg)
Split Video Bolt [python]
![Page 31: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/31.jpg)
Result:● Error, because we use subprocess in python for calling ffmpeg.● If we use python libraries for ffmpeg, it’s deprecated, for concat function.
The libraries is too old compared to latest version of ffmpeg● We have tried to use java library (xuggler) for calling ffmpeg, but this library
is discontinued. Most java libraries for ffmpeg is discontinued or too old.
![Page 32: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/32.jpg)
Conclusion
● Storm is not suitable for video transcoding○ Storm is for stream data, so we can’t use static video○ If we use stream video, it can’t be distributed○ It’s better to use Hadoop, although based on a
paper, it doesn’t give a good performance● Until now, we haven’t found reference for the
same research
![Page 33: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/33.jpg)
Twitter Topology
![Page 34: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/34.jpg)
Twitter Spout
![Page 35: Transcoding Distributed Video - …docshare04.docshare.tips/files/21365/213652105.pdf · Transcode video files using ffmpeg library in ... Distributed using program that we made 1](https://reader031.vdocuments.mx/reader031/viewer/2022021803/5b891e677f8b9abf5c8d01d5/html5/thumbnails/35.jpg)
Twitter Result