on-demand sharing of a high-resolution panorama video from networked robotic cameras
DESCRIPTION
I. I. I. I. I. I. I. I. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. Client i. Vide frames. Server side. Live video. Client i. - PowerPoint PPT PresentationTRANSCRIPT
On-Demand Sharing of a High-Resolution Panorama Video from Networked Robotic
Cameras
Supported in part by
CPSC 643
Dezhen Song
Texas A&M University
Vide framesSphere
wrappingImage
alignment
1
i1
im N
Server side
IBBBBBBIBBBBBBIBBBBBB1
IBBB
i
1
IBBBBBB IBBBBBBN
……
Client i
IBBBBBBIBBBBBBIBBBBBB……
i
m
Client iClient iClient i
Decoding and composing panorama
Rendering, Display
User request [area i, at time t1~tk]
… …
Live video
Time k
BBB
2
Panosonic HCM 280
– PTZ Robotic Camera: • 350° Pan, 120° Tilt, 42x Zoom
• Maximum spatial resolution: 500 Megapixel per
steradian
• 3 Gigapixels panorama
– Network Video Camera:• Built-in streaming server
• 640x480 pixels video
• >30 frames per second
Network PTZ Robotic Camera for Nature Observation
5
Giga-pixel Motion Panorama VS. Fixed Lens Camera
• Fixed lens with mirror• 10M Pixel CCD• $ 20.0 K• 2M Pixel / Steradian
• Pan, Tilt, Zoom (21x)• 0.37M Pixel CCD• $ 1.2 K• 500M Pixel / Steradian
6
Existing Panoramic Video Systems
SystemCamer
a
Bandwidt
hVideo Output
Sample System
s
Wide angle lens/mirrors
Single fixed
Low Low quality live stream
[Baker 1999], [Nayar 1997], [Xiong
1997], [Ng 2005]
Multiple camera
panorama video
Multiple fixed
High Live panoramic video[Foote2001], [Swaminath
an 2000]
Panoramic video texture
Single pan
HighPseudo-live panorama
video by changing video temporal display
[Agarwala 2005]
Dynamosaics Single pan
HighPseudo-live panorama
video by changing space-time volume
[Rav2005]
Motion panorama
Single LowStatic panorama overlaid with living moving objects
trajectory
[Irani 1996], [Bartoli 2004]
Our systemPTZ
Cameras
Low Partial live panorama
7
Evolving Panorama: High Resolution Live Panoramic Video Using PTZ
Camera
Tilt
Pan
Frame sequence
Panorama
Tilt
Time
Panorama
Live frame sequence
Updated Part in
Panorama
8
robotic video cameras
Collaborative Observatories for Natural Environments (www.c-o-n-e.org)
motion sensors
timed checks
sensor networks
humans: amateurs and profs.
2005-2008
9
On-demand Panoramic Video
SharingVide frames
Sphere wrapping
Image alignment
1
i1
im N
Server side
I B B B B B B I B B B B B B I B B B B B B1
I B B B
i1
I B B B B B B I B B B B B BN
……
Client i
I B B B B B B I B B B B B B I B B B B B B
……
im
Client iClient iClient i
Decoding and composing panorama
Rendering and Display
User request [area i, at time t1~tk]
… …
Live video
Time k
B B B
10
Challenges:
– Dynamic video coverage
– High resolution panorama coverage
– Multiple different spatial-temporal client requests.
On-demand Panoramic Video
Sharing
11
User Request
Live Live
Timek-1 k
… ……
Live video
…
Client i
User i request: ri=[u, v, w, h, ts, te]
12
Camera Coverage
Camera Coverage
pan-180o -180o
60o
tilt
N
1 pjk
Patch-based Panorama Video
Snapshot at time k
Live patch Static patch
Camera Coverage
13
Patch-based Panorama Video
Live video
Live
Timek-1 k
… ……k-2
…
Camera coverage at time kPatch j at time k
14
On-demand Patch-based Panorama Video Sharing
Vide frames
Sphere wrapping
Image alignment
1
i1
im N
Server side
I B B B B B B I B B B B B B I B B B B B B1
I B B B
i1
I B B B B B B I B B B B B BN
……
Client i
I B B B B B B I B B B B B B I B B B B B B
……
im
Client iClient iClient i
Decoding and composing panorama
Rendering and Display
User request [area i, at time t1~tk]
… …
Live video
Time k
B B B
15
Frame Insertion Algorithm
Input: Ft
Output: Updated evolving panorama video
Wrap Ft onto the spherical surface;
Estimate Ft’s registration parameters by aligning it with previous frames;
Project Ft onto the sphere panorama surface;
for each pj and pj ∩ Ft ≠ Ø do
Insert pjt into pj’s GOP buffer;
for each pj, j=1, …,N do
if pj’s GOP buffer is full then
Encode patch video segment;
Store patch segment start position and time data into lookup table;
Reset GOP buffer for incoming data;
16
On-demand Patch-based Panorama Video Sharing
For User i request:
Send patch data:
ri=[u, v, w, h, ts, te]
ri ∩ Pt = { pjk | j Є {1,…,N}, k Є [ts, te], pjk ∩ ri ≠Ø , pjk ≠Ø }
17
User Query Algorithm
Input: ri
Output: ri ∩ P in MPEG-2 format
Identify patch set S= { pj | j Є { 1,…,N }, pj ∩ ri ≠Ø };
for each pj Є S do
Find the nearest I frame pjb earlier or equal to ts;
Find the nearest I frame pjc later or equal to te;
Transmit the patch segments between pjb and pjc;
18
Experiments and Results
Hardware configuration:
•Dell Dimension DX, 3.2Ghz Pentium dual-core processor, 2GB RAM
•Panasonic HCM 280A video camera
Software configuration:
•Visual C++ in Microsoft Visual Studio 2003 .NET
•MPEG-2 encoder/decoder from MPEG Software Simulation Group
Input data set: •Frame number: 609
•Frame resolution: 640x480 pixels
•Frame rate: 25 fps
•Raw RGB data size; 536 MB
•Panorama resolution: 2742x909 pixels