snickers: open source http api for media encoding
TRANSCRIPT
![Page 1: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/1.jpg)
Open Source HTTP API for Media Encoding
Snickers
GoLab 2017. Florence, Italy
![Page 2: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/2.jpg)
/flavioribeiro
/flavioribeiro
flavioribeiro.com
senior engineer @ the new york times
![Page 3: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/3.jpg)
context
![Page 4: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/4.jpg)
context motivation
![Page 5: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/5.jpg)
context motivation
how it works?
![Page 6: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/6.jpg)
context motivation
how it works? future
![Page 7: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/7.jpg)
context
![Page 8: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/8.jpg)
context
media factory client
acquistion API
media factory API
transcoding API
distribution API
CDN
Storage
database
http://nyt.ms/mediafactory
![Page 9: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/9.jpg)
transcoding api
transcoding API
http://github.com/nytimes/video-transcoding-api
![Page 10: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/10.jpg)
{ "providers": ["elastictranscoder", "elementalconductor", "encodingcom", "zencoder"], "preset": { "name": "sample_preset", "description": "This is an example preset", "container": "mp4", "rateControl": "VBR", "video": { "profile": "Main", "profileLevel": "3.1", "height": "720", "width": "1080", "codec": "h264", "bitrate": "1000000", "gopSize": "90", "gopMode": "fixed", "interlaceMode": "progressive" }, "audio": { "codec": "aac", "bitrate": "64000" }}
transcoding api - presets
![Page 11: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/11.jpg)
transcoding api - jobs{ "provider": "elastictranscoder", "source": “ftp://nytimes:[email protected]/folder/my_video_source.mov”, “destination”: “ftp://nytimes:[email protected]/outputs”, "outputs": [ {"preset": "720p_mp4", "fileName": "my_video_720p.mp4"}, {"preset": "1080p_mp4", "fileName": "my_video_1080p.mp4"}, {"preset": "256p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "480p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "720p_hls", "fileName": "hls/my_video_720p.m3u8"}, {"preset": "1080p_hls", "fileName": "hls/my_video_1080p.m3u8"}, {"preset": "2160p_hls", "fileName": "hls/my_video_2160p.m3u8"} ], "streamingParams": { "segmentDuration": 5, "protocol": "hls" }}
![Page 12: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/12.jpg)
• old system worked well for ~10 years, not anymore
• new system is a set of microservices in Go
• fast encoding, scalability, reliability
• transcoding API is a wrapper for encoding services
• presets, encoding jobs
context recap
![Page 13: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/13.jpg)
motivation
![Page 14: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/14.jpg)
motivation
• team fluent in Go — except me
![Page 15: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/15.jpg)
motivation
• team fluent in Go — except me
• I wanted to test different approaches
![Page 16: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/16.jpg)
motivation
• team fluent in Go — except me
• I wanted to test different approaches
• What if I create another encoding service in Go?
![Page 17: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/17.jpg)
• open source alternative to encoding providers
• deploy & run everywhere
• add features any time
• compatible with transcoding api jobs and presets
What if I create another encoding service in Go?
![Page 18: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/18.jpg)
![Page 19: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/19.jpg)
how it works?
![Page 20: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/20.jpg)
RESTful API
how it works?
![Page 21: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/21.jpg)
var Routes = map[Route]RouterArguments{ //Job routes CreateJob: RouterArguments{Path: "/jobs", Method: http.MethodPost}, ListJobs: RouterArguments{Path: "/jobs", Method: http.MethodGet}, GetJobDetails: RouterArguments{Path: "/jobs/{jobID}", Method: http.MethodGet}, StartJob: RouterArguments{Path: "/jobs/{jobID}/start", Method: http.MethodPost},
//Preset routes CreatePreset: RouterArguments{Path: "/presets", Method: http.MethodPost}, UpdatePreset: RouterArguments{Path: "/presets", Method: http.MethodPut}, ListPresets: RouterArguments{Path: "/presets", Method: http.MethodGet}, GetPresetDetails: RouterArguments{Path: "/presets/{presetName}", Method: http.MethodGet}, DeletePreset: RouterArguments{Path: "/presets/{presetName}", Method: http.MethodDelete}, }
how it works?
server/routes.go
![Page 22: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/22.jpg)
RESTful API
database
how it works?
![Page 23: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/23.jpg)
RESTful API
database
pipeline
downloaders uploaders encoding engines
how it works?
![Page 24: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/24.jpg)
func StartJob(cfg gonfig.Gonfig, dbInstance db.Storage, job types.Job) { newJob, err := SetupJob(job.ID, dbInstance, cfg) job = *newJob if err != nil { updateJobWithError(dbInstance, job, err.Error()) return } downloadFunc := downloaders.GetDownloadFunc(job.Source) if err := downloadFunc(log, cfg, dbInstance, job.ID); err != nil { updateJobWithError(dbInstance, job, err.Error()) return } encodeFunc := encoders.GetEncodeFunc(job) if err := encodeFunc(logger, dbInstance, job.ID); err != nil { updateJobWithError(dbInstance, job, err.Error()) return } uploadFunc := uploaders.GetUploadFunc(job.Destination) if err := uploadFunc(logger, dbInstance, job.ID); err != nil { updateJobWithError(dbInstance, job, err.Error()) return } CleanSwap(dbInstance, job.ID); err != nil
job.Status = types.JobFinished dbInstance.UpdateJob(job.ID, job) }
how it works?
pipeline/pipeline.go
![Page 25: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/25.jpg)
RESTful API
database
pipeline
downloaders uploaders encoding engines
ffmpeg binding
how it works?
![Page 26: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/26.jpg)
how it works?
• encoding engines
• Cgo wrapper for FFmpeg functions
• https://github.com/3d0c/gmf
![Page 27: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/27.jpg)
how it works?package segmenter
/* #include <stdio.h> #include "libavformat/avformat.h" #include <libavdevice/avdevice.h> #include "c/segmenter.h" #include "c/util.h"
#cgo LDFLAGS: -‐L${SRCDIR}/../build -‐lsegmenter -‐lavcodec -‐lavformat -‐lavutil */ import "C"
import ( "fmt" "os" "unsafe"
"github.com/3d0c/gmf" )
![Page 28: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/28.jpg)
DEMO!
![Page 29: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/29.jpg)
![Page 30: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/30.jpg)
future
![Page 31: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/31.jpg)
future
transcoding API
![Page 32: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/32.jpg)
future
• multibitrate HLS
• go client (WIP)
• add another encoding engine using GStreamer
![Page 33: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/33.jpg)
We’re hiringnyti.ms/technology
@NYTDevs | developers.nytimes.com
![Page 34: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/34.jpg)
Stay updatedopen.blogs.nytimes.com
@NYTDevs | developers.nytimes.com
![Page 35: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/35.jpg)
Connect with us on Slack!http://video-dev.org
@NYTDevs | developers.nytimes.com
![Page 36: Snickers: Open Source HTTP API for Media Encoding](https://reader034.vdocuments.mx/reader034/viewer/2022051101/588828211a28ab733a8b570b/html5/thumbnails/36.jpg)
thank you!{ , } /flavioribeiro