media processing in the network wei tsang ooi. research area u how to build multimedia network...
Post on 28-Dec-2015
214 Views
Preview:
TRANSCRIPT
Current Solutions
black box library code easy to write but too high level need finer optimization, resource
control write from scratch
too time consuming
What is Dalí
A small set of simple, fast, reusable primitives and abstractions for image, audio and video processing
Example: Black Box Code gd GIF programming library 1.3
gdImagePtr im;FILE *in;in = fopen("mygif.gif", "rb");im = gdImageCreateFromGif(in);fclose(in);gdImageRectangle(im,0,0,8,8,red);
Example: Dalí code
f = fopen(argv[1], "r");BitStreamFileRead(bs,f,0);BitParserWrap(bp, bs);GifSeqHdrParse(bp, sh);rmap = ImageMapNew();gmap = ImageMapNew();bmap = ImageMapNew();n = GifSeqHdrGetCtSize(sh);GifCtParse(bp,n,rmap,gmap,
bmap); ih = GifImgHdrNew();ph = PnmHdrNew();status = GifImgFind(bp);GifImgHdrParse(bp, ih);
h = GifImgHdrGetHeight(ih);w = GifImgHdrGetWidth(ih);r = ByteNew(w, h);g = ByteNew(w, h);b = ByteNew(w, h);i = ByteNew(w, h);GifImgParse(bp,sh,ih,i);ImageMapApply(rmap,i,r);ImageMapApply(gmap,i,g);ImageMapApply(bmap,i,b);
Dalí : Abstraction
Byte Image Audio Buffer Bitstream MPEG Headers JPEG Headers Lookup Table
Bit Image DCT Image Vector Image Convolution
Kernel
Dalí : Abstraction
Byte Image Audio Buffer Bitstream MPEG
Headers JPEG Headers Lookup Table
Bit Image DCT Image Vector Image Convolution
Kernel
Byte Image
Two dimensional array of bytes Can be either physical or
virtual
physicalimage
virtualimage
Composable Abstraction Byte images can be used to
represent: Gray scale images RGB images YUV images Alpha channels
target = byte_clip(image,0,0,dx,2*dy);byte_set(target,0);target = byte_clip(image,3*dx,0,dx,2*dy);byte_set(target,0);
image
smaller
neww
newh
dx
2dy
target
image
smaller
neww
newh
dx
2dy
target
target = byte_clip(image,dx,0,neww,newh);byte_copy(smaller,target);
image
smaller
neww
newhtarget = byte_clip(image,2*dx,2*dy,neww,newh);byte_copy(smaller,target);target = byte_clip(image,0,2*dy,neww,newh);byte_copy(smaller,target);
Dalí strategies
specialized primitives byte_shrink_2x2
memory sharing byte_clip
explicit memory allocation byte_new
Example
Skip to the n-th framefor (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs);
}
Novel Features of Dalí
explicit I/O explicit memory management predictable performance programs are highly reconfigurable push semantics to programmer composable primitives,
abstractions
Other Features of Dalí
fast easily extensible support wide ranges of formats
(GIF, JPEG, MPEG-1, WAV, AVI, ..)
Dalí : Status
Currently in alpha release C, Tcl and Java binding Better performance than other
tools (e.g. Rivl, mpeg_play)
Breaking it up ..
capture process compress store
capture process compress
displayde-
compress
source receiver
on onemachine
on twomachines
Optimization
capture process compress store
capture process compress
displayde-
compress
8 bit colors, 10 frame/s
source receiver
on onemachine
on twomachines
Optimization
capture process compress store
capture process compress
displayde-
compress
8 bit colors, 10 frame/s
8 bit colors, 10 frame/s
on onemachine
on twomachines
Suggested Solution
Layered Multicast streams are hierarchically
encoded each layer is multicast into a
group incrementally combine the layers
to refine the quality of the streams
Layered Multicast
Receivers find out if they can join the next higher layer by experiments
Join the layer, if packet drops/low throughput detected, leave it
Problems
Needs layered encoding More groups means more
states to maintained Uses a lot of multicast
addresses
Alternative Approach
Move computation into the network where to run ? when to run ? how to specify the computation ?
Filters
transform one or more input streams into an output stream
Examples : drop frames decode frames mix input audio transcode to lower quality
Answers
QOS Filters where to run ?
filter servers when to run ?
when resource(out) < resource(in) how to specify the
computation ? pick a pre-determine filter
Related Work
Active Service (Amir et.al 98) where to run ? when to run ? how to specify the computation ?
Servlets : MeGa
An application level video gateway
Transcodes MJPEG to H.261 Written using Mash
next generation multimedia network toolkit
built on tcl
Active Service
Where to run ? active server at bottleneck
When to run ? when receiver requested it
How to specify a computation ? request for pre-determine servent
Problems
How to specify computation ? Not easily extensible Significant extension requires
manual installation of servlets/filter Where to run ?
AS : statically located
The answers should be
How to specify a computation ? receivers can inject customized
programs into network Where to run ?
can be specified by users or dynamically place itself
Thesis Proposal
active media network (Medan) allow customizable media
processing code to be injected into the network
application: self-adaptive transcoder (SAT)
Language Requirements Safe & portable Support common format/ops Fine grain operations
distributed processing possible Explicit resource usage
resource management and admission control
Dalí : How does it fit ?
Features of Java Dalí: safe and portable
Features of Dalí : thin primitives predictable explicit memory allocation explicit I/O
Medan Components
media processing language Dalí
active network kernel JKernel, SANE, etc.
user interface ???
Medan Components
media processing language Dalí
active network kernel JKernel, SANE, etc.
user interface Mash
Medan:Problems to be solved Resource usage determination Load balancing Deployment Security*
Resource management*
Medan Application:
Self-adaptive Transcoder (SAT) Injected into the network when
a session starts Perform bandwidth adaptation
by transcoding session’s packets
Basic algorithm
each host who joined a group will receive at an initial rate
each node will report it’s state (congested, under-utilized) to it’s parent
parent will adapt sending rate accordingly
Problems to be solved
Detail algorithm Scalability Reflect receiver’s interest Interaction between programs Deployment
Other Medan Applications Remote Sensing Multimedia Database Video Conference Auto-Editing Receiver Heterogeneity
Contributions
Dalí : a media processsing library
Medan : a framework for active media processing in the network
SAT : apply Medan to solve the network heterogeneity problems
Summaries
How to build multimedia apps Adding the word “network” Current solutions and problems Proposal
Where Medan is going..
Computation can organized itself (divided, pipelined, factored) into sub-computations
Packets can carry computation Capsules Mobile Code
top related