![Page 1: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/1.jpg)
Delivering Audio and Video
Over the Web
Norman WhiteStern School of Business
![Page 2: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/2.jpg)
Agenda
• Audio/video formats• Rates of transmission• Hardware & software advances• Network requirements• Streaming audio & video
![Page 3: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/3.jpg)
Business Case
• Multimedia is a powerful way to deliver a message
• As a listener/viewer uses more senses to get a message, retention rate goes up.
• Much information is better understood visually
• Question? What are the problems of delivering audio and video over the internet?
![Page 4: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/4.jpg)
Audio Formats
• WAV – direct digitization of audio– Size and quality depend on
“sampling” rate– The higher the sample rate, the larger
the file size for the same length, but the better the quality
• Au• MPEG• Many others
![Page 5: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/5.jpg)
Video Formats
• AVI – Windows video format
• Supports variety of CODECs– CODEC – Compression-Decompression
• Quicktime (MOV)– Apple supported format
• MPEG – Motion Pictures Experts Group– Hard to manipulate, high quality
![Page 6: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/6.jpg)
Digital Audio – ISDN Integrated Service, Digital
Network
• To digitize speech, the telephone standard is to take 8000 samples a second, each sample having a value (0-255, I.e. 8bits)
• Hence 1/2 of an ISDN line is 64kb (64000 bits per second) (allows simultaneous voice and data transmission)
• Note, this is uncompressed. We will talk about compression later)
![Page 7: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/7.jpg)
Bit rates for different audio streams
• Telephone– G.711 PCM– G.721 ADCMP– G.722– G.728
• CD – CD - DA– MPEG consumer– MPEG studio
• Bit rate Kbps– 64– 32– 48,56,64– 16
– 1411– 192– 384
![Page 8: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/8.jpg)
Motion Video Bit Rates (Mbps)
• HDTV– Compressed
• Digital TV (studio)– MPEG2
• Broadcast TV (mpeg2)
• VCR (MPEG-1)• Videoconferencin
g
• 2,000 – 34
• 166– 3-6
• 2-4
• 1.2• .1 – 1.5
![Page 9: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/9.jpg)
Some typical network rates
• 56k modem - 56K bits per second• ISDN - 64kbps (2 channels 128)• Wireless - 2MB - 11Mb/s, 54Mbs• Cable Modem - 10 Mb/s• ethernet/token ring 10mb/s, now
100mbs, 1GB• T3 - 45 mb/s• OC3 - 154MB/s
![Page 10: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/10.jpg)
Video Tips
• Very important to capture video with the highest quality you can
• Consumer Camcorders lose quality quickly as light level drops
• Remember, audio quality is just as important, consumer camcorders have poor microphones, need separate high quality microphone for good results.
![Page 11: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/11.jpg)
Digital Camcorders
• The new digital Prosumer cameras have much higher quality, but still some problems
• Recording time limits– Usually 1 hour for consumer cameras, OK
for some applications, but unacceptable if there is no break to change media
• Blair Witch Project – First major movie to be shot with Digital Camera
![Page 12: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/12.jpg)
Cost Dropping quicklyApplications expand
• Until recently, needed a > $10K Sony beta max for decent video
• Now $1,000 digital cameras give comparable quality (and MUCH smaller and lighter)
• Inexpensive disk drives now offer capacity to deal with “raw” video (100MB per minute, 6 gigabytes for a one hour avi file)
• Need fast processors to encode, decode, transfer data
![Page 13: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/13.jpg)
More hardware advances
• IEEE 1394 – “Firewire” standard– Allows direct connection of digital
devices without a separate card– 400 Mbit/s– Also supplies power to camera
• USB – Universal Serial Bus– USB 2 will be 480 Mbit
![Page 14: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/14.jpg)
Networking Advances
• Gigabit ethernet – Allows streaming of MPEG Video
• “Switched” Ethernet – allows a separate stream to each individual PC without collisions
• Multicasting router capabilities allow listeners to share a stream
• Cable Modems, ADSL, Satellite Dishes much higher bandwidth to home, small business
• TCP/IP v6 – Quality of Service support• Internet 2- 600Mbit connections• Gigabit switches to handle traffic
![Page 15: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/15.jpg)
Software Advances
• Rapidly improving streaming and compression techniques to take advantage of more processing power, new computer capabilities (special instructions for A/D)
• Inexpensive, powerful video manipulation software
![Page 16: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/16.jpg)
Demand Explosion
• E-Commerce and B2B now can use on-line video training, sales clips etc.
• Person to Person IP Video Conferencing
• On-line TV? Movies? Not yet…
![Page 17: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/17.jpg)
Editing Video
• Adobe Premiere– Powerful Mac and PC based editor
• Supports “non-linear” editing– Used to have to edit by splicing tape (linear)
• Copy tape from one machine to another as you edit
– Now can hold digital video content on disk and edit frame by frame
• AVID – Commercial quality edit system
![Page 18: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/18.jpg)
Adobe Premiere Functionality
• Cut / Splice Video clips• Dissolve between clips• Multiple audio tracks• Supports many CODECS• Background colors• Titles• Filters (blur, pan, zoom,sharpen distort
etc.)
![Page 19: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/19.jpg)
Premiere Basics
• Video clips are shot• Clips are brought into Premiere• Clips are dropped on one of the video
tracks in constructor window• Audio clips added, sound adjusted• Titles, transitions, filters added• “Make Movie” produces a new video clip
in AVI , QT, Real Media etc format.• AVI format can be edited again
![Page 20: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/20.jpg)
Conclusion
• Video is now capable of being edited on consumer class computers
• Need LOTS of disk space– 100MBS / minute of video– Need 3 times as much space– Hard drive defragmented– Lot’s of memory– Fast processor (Pentium minimum)– Digital capture card, Firewire or USB to
bring video in
![Page 21: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/21.jpg)
Conclusion
• All indications are that Streaming Media applications will continue to have rapid growth
• Recent Dot-com problems have shut down some of the early entrants in the consumer market.
![Page 22: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/22.jpg)
BUT!!!
• Audio and video file sizes very large and need transfer rates much faster than available bandwidths.
• Many difficult problems need to be solved in order to use audio and video over the internet.
![Page 23: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/23.jpg)
Network Requirements of audio and motion video
![Page 24: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/24.jpg)
Problems for live usage
• Error rate– Network may not deliver bits reliably– Shouldn’t happen that often in a
digital network due to error control
• Transit delay– Data is not delivered at a constant
speed, or at the right speed
![Page 25: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/25.jpg)
Audio Characteristics
• Ear is a differentiator– Has high powers of differentiation– Means that small audio problems are
easily recognized
![Page 26: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/26.jpg)
Video Characteristics
• Eye is an integrator– It puts signals together– Hard to recognize multiple intermixed
drawings and interpret them– Missing information is less noticeable
• Means video can tolerate more errors and higher delays
![Page 27: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/27.jpg)
Implications for multimedia
• In a transmission of mixed audio and video, audio should have a higher priority– Audio more sensitive to delays
• Note - this is a problem on the internet– Internet currently has no priority
mechanism
![Page 28: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/28.jpg)
Compression
• Can we reduce the amount of information we need to send?
• We can compress it• two types of compression
– Lossy - Can’t exactly recreate original– Lossless - Can exactly recreate
original file, bit for bit.
![Page 29: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/29.jpg)
Compressing different types of info
• Images/ Graphics – Gif (lossless) – JPEG - Can be lossy
• Sound (first digitize at some sampling rate)– Then compress
• Video (usually lossy)– Intraframe compression– interframe compression
![Page 30: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/30.jpg)
Video compression techniques
• Intraframe compression– Compress the pixels within one frame
• Interframe compression– Since video frames often change very little
between frames, we can increase compression by only recording the changed bits from one frame to the next.
– This works extremely well for many situations
• Video - 1 frame = 640*480*8 bits = 300K bytes ( 10MB/sec 30 fps)
![Page 31: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/31.jpg)
Compression Problems
• Sending compressed data means that problems in one block (packet) of data may impact subsequent data blocks that have related information which is based on the content of an earlier data block (I.e. keyframes)
![Page 32: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/32.jpg)
Partial solution to network delay
• Add an offset delay to reception of message
• The receiving side will collect N seconds of information before it starts playing the audio or video clip
• How big should ‘N’ be?– Depends on effective speed of
communications channel– higher variance means larger N
![Page 33: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/33.jpg)
Other Problems
• Intermedia Synchronization– Need to synchronize audio and video
streams• Compression and burstiness
– Compression can introduce burstiness into a non-bursty transmission
– Example. Audio uncompressed has a fixed number of bits per second, compressed it can go down to zero (periods of silence)
![Page 34: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/34.jpg)
Conclusion
• Network multimedia demands many complex tradeoffs between bandwidth, errors, delays and cost
• Compression makes the problem even more complex– Error rates have to be lower to use
compression
• What are some reasonable levels for audio and video?
![Page 35: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/35.jpg)
Streaming Audio and Video
![Page 36: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/36.jpg)
Problems with audio/video over the web
• File sizes large, perhaps infinite for continuous live webcasts– Can’t wait for file to completely download
• WEB bandwidth unreliable - high variance• Bandwidth varies from minute to minute• Need to have complex compression and
streaming protocols to compensate– Not supported by HTTP protocol
• http is basically a download protocol, not sophisticated
– Demands much more complex protocol to try to compensate for problems
![Page 37: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/37.jpg)
Solution
• Move audio and video files to a separate server which uses a different (more sophisticated) communications protocol
• Have client application communicate directly with the media server.
![Page 38: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/38.jpg)
Current Solutions
• Real Networks– RTSP - Real Time Streaming Protocol– Needs a separate server– Protocol supports multiple, simultaneous
streams (audio, video, animation, text)– Helper App to play the streams (REAL PLAYER )– SMIL language to describe relationship between
streams– Automatic, dynamic bandwidth negotiation
• Server and client continue negotiating transfer rate in real time.
![Page 39: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/39.jpg)
RTSP features
• Determines average bandwidth between video server and client and variance
• Preloads part of file before it starts playing• Dynamically adjusts needed bandwidth by
varying quality of stream as necessary– Audio switches codecs when bandwidth drops
• CD -> HiFi-> mono – Voice
– Video increases compression and drops frames
• Objective is to keep a “reasonable” output at all times.
![Page 40: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/40.jpg)
Real Player Encoding
• User specifies bit rates to encode in the file at encoding time.
• Real media file (.rm) is generated that contains data necessary to generate any of the supported bit streams
• More bit streams larger files on server
• More bit streams better possibility of playback
![Page 41: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/41.jpg)
Example
• User is connected to server with a connection that is varying between 30 and 45 K bits per second
• User plays a 60 second video clip that was encoded at 30, 40, 50, 100 K bits per second
• Server and Player will negotiate a 40 K bit per second stream
• Player will buffer (predownload part of file) enough to ensure the stream will play continuosly even if throughput is only 30K bits per second
![Page 42: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/42.jpg)
How long does Real Player wait before starting to
play?
• Stream is 60*40K bits/second = 2,400,000 bits• At 30K bits per second it will take
2,400,000/30,000 seconds = 80 seconds• Need to wait 20 seconds before starting to play
stream, i.e load up 20 seconds of material (20*40,000 = 800,000 bits or 100,000 characters
• Note the many design limitations here.– Is there enough memory– How long can user wait before giving up– What happens if bandwidth changes and they
need to renegotiate …
![Page 43: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/43.jpg)
Other Solutions
• Microsoft ASF - Advanced Streaming Format– Only supports one stream, need 3rd party
tools to combine multiple streams into ASF format
– Works with Microsoft NETSHOW – Supports multiple bitrates in one stream– includes high bandwidth intranet option
![Page 44: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/44.jpg)
Publishing Streaming Media with Real Networks
• Need 3 components– WEB Server
• Create links to .RAM, or .SMI files• RAM files point to REAL Networks server
– Real Networks Server• Files are housed here• Special “Streaming” support• Multicast support• Handles Live Streams
– Browser Helper APP• Real Player G2 - can run standalone
![Page 45: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/45.jpg)
Example ..
• First, need to capture and encode Video/audio in Real Media (.rm) Format
• Run Encoder software– Specify audio and video “codecs” to use
• Each bitrate has a different codec• Can use multiple bitrates
– Move Encoded file to Real Networks Server– Create Links on WEB server to point to file
![Page 46: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/46.jpg)
Video file TEST.RM
• create link to .ram file on WEB server– <a href=“TEST.ram”> Click here to
see video</a>
• create TEST.ram file– rtsp://videos.stern.nyu.edu:554/
TEST.rm
![Page 47: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/47.jpg)
What Happens• User clicks on link
– Link is a .ram file, so helper app is invoked to process the file
– Helper App (REAL PLAYER ) starts and is passed the URL of the 1 line ram file by the web server
– REAL PLAYER reads (HTTP GET request) ram file and then issues RTSP connection to video server.
• Video Server and Real Player now have a separate TCP/IP connection (port 554) to communicate over using the Real Time Streaming Protocol.
• Why so complicated?– Because web servers only support http protocol
![Page 48: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/48.jpg)
Synchronized Media
• What happens if I want to synchronize several different streams. Need to use SMIL– Synchronized Media Interpretation Language?
• The link needs to point to a .SMI file instead of a RAM file.
• Simple example, put an intro in front of the video
• I.e. play intro file followed by TEST.rm
![Page 49: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/49.jpg)
SMIL Example
• <smil>• <root> </root>• <body>• <seq>• <rtsp://videos.stern.nyu.edu:554/file1.rm
….> first file to play• <rtsp://videos.stern.nyu.edu:554/file2.rm
…> second file to play• </seq>• </body>• </smil>
![Page 50: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/50.jpg)
More Advanced SMIL
• Smil can also play material in parallel in different windows
• Example would be a REAL TEXT presentation in one window, with synchronized Real pics in another window, and synchronized audio playing as the slides (and text) changes.
• Real Presenter – Real Networks add-on ($29.95) to Powerpoint to allow streaming Powerpoint narration.
![Page 51: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/51.jpg)
Some links
• WWW.REAL.COM• WWW.ABC.COM
![Page 52: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/52.jpg)
Real Networks Tools Beta versions available for free
• Real Player – Helper App to play Real player audio and
video
• Real Publisher – Set of tools to encode audio/video to support
a variety of speeds (network and computer)– Supports SURESTREAM ™
• Automatically switches speeds
• REAL SERVER– Host content
![Page 53: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/53.jpg)
Other Features
• REAL TOOLS– REAL PIX - Stream images– REAL TEXT - Stream Text– REAL FLASH - Stream Macromedia
Flash– SMIL - Synchronize all of the above
![Page 54: Delivering Audio and Video Over the Web Norman White Stern School of Business](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649e0e5503460f94af7d91/html5/thumbnails/54.jpg)
Final Project Ideas
• Stern TV Station - Multiple Channels– Live events– Archived events– News?
• Video classes with power point slides coordinated with talk.