study the effects of video frames lost over wireless networks
TRANSCRIPT
Master Thesis
Computer Science
Thesis no: MCS-2010-14
January 2010
School of Computing
Blekinge Institute of Technology
Box 520
SE – 372 25 Ronneby
Sweden
Study the Effects of Video Frames Lost over
Wireless Networks – Simulator Development
Anbarasan Thamizharasan
Dotun Ogunkanmi
School of Computing
Blekinge Institute of Technology
Box 520
SE – 372 25 Ronneby
Sweden
1
Contact Information:
Author(s):
Anbarasan Thamizharasan
E-mail: [email protected]
Dotun Ogunkanmi
E-mail: [email protected]
University advisor:
Hussein Aziz Email: [email protected]
School of Computing
Internet : www.bth.se/tek
Phone : +46 457 38 50 00
Fax : + 46 457 102 45
This thesis is submitted to the School of Computing at Blekinge Institute of Technology in
partial fulfillment of the requirements for the degree of Master of Science in Computer Science.
The thesis is equivalent to 20 weeks of full time studies.
School of Computing
Blekinge Institute of Technology
Box 520
SE – 372 25 Ronneby
Sweden
2
ABSTRACT
Mobile networks are highly unpredictable and
might be unreliable for real-time mobile video
streaming due to various wireless network
conditions. Video frames could be delayed or lost
and could affect the perceived quality of a video
stream. The time-sensitive nature of real-time
video streaming is to deliver the video frames to
the mobile devices and to provide smooth
playback. This study examines the concept of
streaming the video frames over two wireless
channels from the server to the mobile device in
order to play the complete video frame sequence to
the mobile users. The simulator is been developed
to provide a reliable video streaming until if there
is a missing frame(s). A switching mechanism is
been implemented in the client side to switch
between the video streams to replace the missing
frames. This will provide a smooth playback of the
video stream and perceived video quality to the
mobile user.
Keywords: Streaming Video, Frame Missing,
Switching Mechanism, Simulator.
3
ACKNOWLEDGEMENTS
First, we would like to express our sincere gratitude to our parents for blessing
us with the potential and ability to work on this master thesis. We would also
like to thank our university advisor, Mr. Hussein Aziz without him this work
could not be possible and our thesis supervisor, Dr. Guohua Bai for giving us
an opportunity to work on this interesting as well as challenging topic under
their keen guidance and support through the course of this thesis work.
We would also like to thank Olasunkanmi Awosanya, Deepika Kumar and our
families and friends for their support and encouragement throughout the course
of this Master’s Degree program.
Finally, we thank everyone who has encouraged us directly and indirectly with
this thesis work.
4
TABLE OF CONTENTS
Abstract 2
Acknowledgements 3
Table of contents 4
Table of Figures 6
List of Tables 6
List of Acronyms 7
Chapter 1: Introduction 8
1.1. Problem Definition 8
1.2. Objectives 8
1.3. Expected Outcome 9
1.4. Research Questions 9
1.5. Research Methodology 9
1.6. Thesis Outline 10
Chapter 2: Video Streaming over Wireless Networks 11
2.1. Video Distribution Channels 11
2.1.1. Non-Real Time Video Streaming 11
2.1.2. Real Time Video Streaming 12
2.2. Video Display Resolutions 12
2.3. TCP/IP Protocol 13
2.3.1. Video Streaming Protocols 13
2.3.2. Transmission Control Protocol 14
2.3.3. User Datagram Protocol 16
2.3.4. Real-time Transport Protocol 16
2.3.5. Real-time Streaming Protocol 17
2.4. Real Time Video Streaming over User Datagram Protocol 17
Chapter 3: Streaming Architecture 18
3.1. Streaming Over Two Channels 19
3.2. Streaming Servers 19 3.2.1. Bandwidth Consideration Settings 20
3.2.2. Duplication, Queue and Delay 21
3.3. Frame Dropping 21
3.4. Switching in Mobile Device 21
Chapter 4: Streaming Simulator 22
4.1. The Streaming Simulator 22
4.2. Streaming Server 23
5
4.3. Traversing Network 24
4.4. Receiving Client 25
Chapter 5: Results and Analysis 27
Chapter 6: Conclusion 29
References 30
6
TABLE OF FIGURES
Figure 2.1 TCP/IP Protocol 14
Figure 2.2 TCP Three-way Handshake 15
Figure 3.1 Streaming Video Environment 18
Figure 3.2 Flow diagram of Video Streaming over two wireless channels 19
Figure 4.1 A Snapshot of the Streaming Simulator 22
Figure 4.2 Streaming Server 23
Figure 4.3 Snapshot of Colour Frame Number 151 24
Figure 4.4 Snapshot of Grayscale Frame Number 151 24
Figure 4.5 Frame Dropping Mechanism 25
Figure 4.6 Receiving Client 26
LIST OF TABLES
Table 2.1 Standard Video Resolution Formats 13
Table 2.2 Transmission Control Protocol Header Format 15
Table 2.3 User Datagram Protocol Header Format 16
Table 3.1 File sizes of Colour and Grayscale Frames 21
Table 5.1 Process Time for the Test Video 27
7
LIST OF ACRONYMS
3G Third Generation Mobile
CIF Common Intermediate Format
HSDPA High-Speed Packet Downlink Access
IEC International Electro-technical Commission
IETF Internet Engineering Task Force
IP Internet Protocol
ISO International Organization for Standardization
ITU-T International Telecommunications Union – Telecommunication
Standardization Sector
JVT Joint Video Team
MGCP Media Gateway Control Protocol
MPEG Moving Picture Experts Group
QCIF Quarter Common Intermediate Format
QoS Quality of Services
RFC IETF Request for Comments
RTCP Real-time Transport Control Protocol
RTP Real-time Transport Protocol
RTSP Real Time Streaming Protocol
RTSPT Real Time Streaming Protocol using TCP
RTSPU Real Time Streaming Protocol using UDP
SIP Session Initiation Protocol
TCP Transport Control Protocol
UDP User Datagram Protocol
UMTS Universal Mobile Telecommunications Systems
VCEG Video Coding Experts Group
8
CHAPTER 1
INTRODUCTION
Sharing of video files has taken a prominent role in recent years and this has been
followed by improved methods for distributing video content [1]. A typical method
of distributing video to multiple users is to use the concept of download-save-play.
This method is known as Video On-Demand which requires the server to send out a
complete video file to one or more users, and the users will download and save the
video file before playback can be started [2]. Another method of distributing the
video content is by streaming the video to the target users over wired or wireless
networks [3,4]. Streaming video involves the transfer of the video and immediate
playback of the video stream without having to save the video before or after
playback. This method of video sharing is particularly useful in wireless networks
with mobile devices since the mobile devices have limited resources for data storage
[1,5]. However, there is a growing need to share video in real time as the use of
mobile devices increases, mobile devices can receive real time video streams over
wireless mobile networks [4,6].These would be useful in video conferencing, mobile
TV, and the broadcasting of live events. A lot of research is ongoing in the
adaptation of present 3G mobile networks to support real-time video streaming
services [3,4,7].
These advancements in terms of bandwidth available in Universal Mobile
Telecommunications Systems (UMTS) networks have made it easier to provide a
wider variety of multimedia applications that require robust bandwidth and stable
networks [8].
1.1 Problem Definition
Mobile networks are highly unpredictable and might be unreliable [9]. These
unreliable network conditions lead to the video frames being dropped or delayed
while traversing through the network [10,11]. Due to the time-sensitive nature of
real-time video streaming, dropped or delayed packets have an undesired effect of
the perception of the quality of the streamed video. This paper aims to study the
effects of missing video frames on streamed video on the mobile device and to
examine the quality of the received video stream on the user’s perception.
1.2 Objectives
The objective of this study is to develop a simulator that implements the concept of
sending two video streams in order to reduce the effects of missing frames on each
stream on the mobile device [12,13,14]. The duplication of video on two streaming
streams is used to replace missing frames on the mobile device by switching between
both streams, the simulator is developed according to [13] to study the effects of
frame dropping mechanism that illustrates the instability of wireless networks effects
9
on real time video streaming. A delay mechanism is considered in the study to avoid
the interruption on the same video frames of both streams. A two-second delay on
the server side is applied on the first stream to prevent the dropping of same video
frames on both streams. Another two-second delay is applied on the mobile device
on the second stream to synchronize both received streams. Once the frames are
received at the mobile device, it buffers the video to get sufficient amount of frames
to be played. If there is a missing video frame from one of the video streams, the
simulator will switches to the other video stream to replace the missing video frames.
The two video streams will enhance the smoothness of the playback of the streamed
video in the mobile devices.
1.3 Expected Outcome
A simulator being developed for streaming duplicate video frames over two channels
to avoid the frozen picture on the mobile device. While the mobile users will
received a complete video frame sequences until if there is a lost frame from any
streams. Therefore a switching between streams is highly needed to replace the
missing frame and to provide smoothness on the mobile devices.
1.4 Research Questions The research questions that we address in our study are formulated as follows:
What are the effects of low-bandwidth wireless networks on real-time video
streaming and how can they be resolved?
How can we reduce the effects of missing video frames and improve the quality
of video on mobile devices?
1.5 Research Methodology
The proposed thesis work uses a mixture of both the qualitative and quantitative
research methodologies in order to investigate the aforementioned research questions
by means of effective literature review and also by developing a simulator to study
the performance of the video quality after the frame lost. The literature review forms
the qualitative research methodology and the development of the simulator forms the
quantitative part of research methodology. At the initial stage, a detailed literature
study is made to understand the concept of video streaming over wireless networks
and the issues concerning the mobile video streaming. A simulator is developed to
study the performance smoothness of the video frames on the mobile device and to
address the key issues with the streaming and switching mechanism. The creation of
the simulator is to validate our initial assumption of streaming duplicate frames over
two channels that could improve the smoothness of the video frames on the mobile
device. Tests are carried out with the aid of different test videos by using different
data transmission rates and streaming the test videos in both colour and grayscale
formats.
10
1.6 Thesis Outline
In Chapter One, the problem is identified and the research questions are formulated
to achieve the desired goals. Chapter Two lays the background on the video
streaming over wireless networks, video distribution channels, and video streaming
protocols used in the streaming environment. The reasons why the TCP protocol is
not suitable for real-time video streaming are explained and possible suggestions for
making the UDP protocol better for real-time video streaming are introduced. In
Chapter Three introduces the streaming architecture that tells the concept of
streaming video over two channels, design steps, low bandwidth considerations and
the duplication, delay and switching mechanisms. This gives the necessary
background to the simulator. The development of our simulator; the design steps and
implementation procedures are described in Chapter Four. In Chapter Five, the
analysis of our findings and observations are carried out from multiple test cases.
Chapter Six concludes by delivering the highlights of the study and possible
improvements for the future.
11
CHAPTER 2
VIDEO STREAMING OVER WIRELESS NETWORKS
Video streaming is the transfer of videos over the internet from one source to one or
more destinations. Video is usually streamed from the server to clients. Various
methods are used for streaming video over the internet, they include, broadcasts,
multicasts, and unicasts. On-Demand video streaming is usually done with unicasts
since the video is requested by a user at a time [2]. Real-time video streaming is done
with multicasts and broadcasts since there are multiple requests for the video stream
at a time [2,15,16].
Video files typically contain a lot of information about the video frames, which
requires huge storage resources. In order to stream contents such as video, it has to
be compressed packaged in a way that is suitable for transmission over networks
[17,18]. The H.264/MPEG-4 AVC is a popular standard for video streaming. It is
developed by the Joint Video Team (JVT); a partnership effort between the ITU-T
Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts
Group (MPEG) [19]. The intent of the H.264/AVC project was to create a standard
capable of providing good video quality at substantially lower bit rates than previous
standards without increasing the complexity of design so much that it would be
impractical or excessively expensive to implement [17,18,19]. An additional goal
was to provide enough flexibility to allow the standard to be applied to a wide variety
of applications on a different networks and systems, including low and high bit rates,
low and high resolution video and RTP/IP packet networks [19,20]. The support for
low bit rates makes the H.264/MPEG-4 AVC very suitable for video streaming on
mobile networks.
2.1 Video Distribution Channels
Due to limited bandwidth of UMTS networks [8], video distribution to mobile
devices is somewhat limited. Video cannot be distributed in the same formats as is
done on larger computer networks. Mobile video content has to be manipulated to
decrease the size and quality requirements in order to fit the transmission network.
There are two main methods for delivering video content to mobile devices. These
are:
i. Non-real time video streaming
ii. Real time video streaming
2.1.1 Non-Real Time Video Streaming
Non-real time video streaming involves the playback of a video stream after a
complete or partial download of the video stream [21]. Usually, playback can be
started while the video clip is still downloading [21,22]. The player on the mobile
device makes use of meta-data information to check for completion and it creates a
12
buffer on the mobile device to save the downloading file temporarily. When there is
sufficient data in the buffer, the playback begins [22]. In some cases when the speed
for playback is faster than the rate at which the file is being downloaded, the video
pauses until there is sufficient data in the buffer to continue playback. This concept
of download-save-play enhances the smoothness of playback of the video since all of
the video frames are received on the mobile device. This method of video streaming
is subject to several mobile device and network constraints [23]. A mobile device
requires adequate storage to save the video stream before playback. This method of
streaming also contributes to congestion on the wireless network because all of the
video streams have to delivered to the mobile device [24,25].
2.1.2 Real Time Video Streaming
The streaming server accepts video requests from the mobile device, sets the
streaming format and data transmission rate and communicates with the mobile
device to monitor the streaming and playback. The video is streamed to the mobile
device, which plays the video stream without having to store it [26,27]. In real time
video streaming, the mobile device plays the received video stream immediately after
a short buffering period, as soon as a video frame is played, it is discarded from the
mobile device [28]. The streaming server sends the video content in a manner that is
compliant with the transmission mechanism. It takes note of the transmission speed
of the network and adjusts the quality of the video content to match the speed of the
network [29].
2.2 Video Display Resolutions
Mobile phones make use of various display resolutions. Initially, the ITU-T H.261
standard proposed the Quarter Common Intermediate Format (QCIF) and Common
Intermediate Format (CIF) video display resolutions for video telephony and video
conferencing applications over ISDN telephone lines [30,31,32]. The CIF and QCIF
display formats were designed to be used in low speed transmission networks. The
improved H.263 video codec standard later introduced, supports the previous display
resolutions as well as the Sub-Quarter Common Intermediate Format (SQCIF), Four
times Common Intermediate Format (4CIF) and the Sixteen times Common
Intermediate Format (16CIF) [33]. The H.263 requires lesser bandwidth to achieve
the same results as found in the previous H.261 video codec. Table 2.1 contains the
display resolutions of some available video formats. Since most mobile phones
support the SQCIF and QCIF video display resolutions, and the H.263 video codec
makes it mandatory to include the SQCIF and QCIF display resolutions in H.263
implementations [30], we will be conducting the simulations using a QCIF-sized test
video.
13
Format Video Resolution
SQCIF 128 * 96
QCIF 176 * 144
CIF 352 * 288
4CIF 704 * 576
16CIF 1408 * 1152
Table 2.1: Standard video resolution formats
2.3 TCP/IP Protocol
The TCP/IP Protocol was developed by the Defense Advanced Projects Research
Agency (DARPA) [34,35] for communication across any set of interconnected
networks, especially on the internet. The TCP/IP protocol is usually known for its
two initial protocols; the Transmission Control Protocol (TCP) and the Internet
Protocol (IP). According to RFC 1122, [36] the TCP/IP protocol suite is divided into
a set of five layers, these are the Physical Layer, Data Link Layer, the Network
Layer, the Transport Layer and the Application Layer in order of ascension [37]. The
transport layer provides Host-to-Host communication between two applications. It
also provides flow control and reliable transportation of data in sequence. On the
transport layer, the TCP and the User Datagram Protocol (UDP) are the two
protocols used for the transport of data segments. IP is the basic Internet layer
protocol and is responsible for the transmission of TCP and UDP packets to the end
users [38]. The relationship of the TCP, UDP and IP protocols are illustrated in
Figure 2.1.
Figure 2.1: TCP/IP protocol.
TCP/IP Protocol
Internet Protocol
TCP
UDP
14
By sending data across the network medium, the transport layer splits the data into
network buffer-sized segments, selects a transmission protocol and sends the
segments over the network. In flow control, it ensures that the receiving end gets just
the amount of data it can process at a time. It also ensures that the destination
receives the entire data [38].
2.3.1 Video Streaming Protocols
Streaming Servers make use of streaming protocols that are suitable for video
streaming. It makes use of TCP, UDP, Real-time Transport Protocol (RTP) and Real-
Time Streaming Protocol (RTSP). TCP is more reliable and ensures the delivery of
the video stream to the mobile devices [39]. UDP is a simple datagram protocol that
sends the video stream as a series of packets to the mobile device; however it doesn’t
guarantee the correct delivery of packets [38]. RTSP includes default control
mechanisms for interaction between the server and the mobile device [40].
The TCP and UDP are designed for end-to-end transmission and are carried over the
Internet protocol (IP). The RTP and the RTSP are specially designed for the
streaming of video over wireless networks.
2.3.2 Transmission Control Protocol (TCP)
The TCP is a connection oriented and reliable transport layer protocol [39,41]. It
makes use of a three-way handshake procedure to establish and maintain a session
between the sending and receiving hosts before sending of data. TCP is usually used
for error-free delivery [42]. The three-way handshake illustrated in Figure 2.2
consists of three steps and they are as follows:
The client initiates a connection by sending a synchronization packet (SYN) to
the server. This SYN packet is a request for the server to synchronize its
sequence numbers with that of the client.
The server responds to the client’s request by sending an acknowledgement of
the client’s request (ACK) with a SYN. The SYN is a request for the client to
synchronize its own sequence numbers with that of the server.
Finally the client sends another ACK to the server to acknowledge the server’s
synchronization request back to the server.
15
Figure 2.2: TCP Three-way handshake
Due to its use of acknowledgements, it has to retransmit missing or broken data
segments. It also requires additional processing of data since it splits the data into
segments before transmission and after they have been received by the mobile
device. This increases the latency when streaming video. This is not suitable for real-
time applications such as real-time video streaming because users are very sensitive
to delays that may take between 150 – 200 milliseconds [43].
0 15 16 31
Source Port (16 bits) Destination port (16 bits)
Sequence Number(32 bits)
Acknowledgement Number
Header
Length
(4 Bits)
Reserved
(4 Bits)
Code Bits
(8 Bits) Window (16 Bits)
Checksum (16 bits) Urgent Pointer
Options + Padding
Data (Variable Length)
Table 2.2: Transmission control protocol header format
16
2.3.3 User Datagram Protocol (UDP) The User Datagram Protocol is a simple connection-less transmission protocol that
assumes error checking, error correction and flow control are not required or are
handled by other layers of the TCP/IP Protocol Suite [44]. It focuses on timely
delivery of datagrams to the destination, which makes it suitable for broadcasts and
multicasts [38]. The UDP protocol is almost the exact opposite of the TCP protocol
since it does not include the transmission management capabilities that TCP
implements. It just transfers the data without any sequential segmentation and error
checks.
0 15 16 31
Source Port (16 bits) Destination port (16 bits)
Length (16 bits) Checksum (16 bits)
Data (Variable Length)
Table 2.3: User datagram protocol header format
Real-time video streaming requires timely delivery of the video frames from the
streaming server to the mobile devices. This makes the UDP a better alternative since
it does not make use of acknowledgements unlike the TCP which ensures the
delivery of datagrams through acknowledgements [45].
2.3.4 Real-time Transport Protocol
The Real-time Transport Protocol (RTP) defines a standardized packet format for
delivering audio and video over the Internet. It was developed by the Audio-Video
Transport Working Group of the Internet Engineering Task Force and first published
in 1996 as RFC 1889, and later upgraded to RFC 3550 in 2003[46].
RTP is used extensively in communication and entertainment systems that involve
streaming media, such as telephony, video teleconference applications and web-
based push to talk features. For these it carries media streams controlled by H.323,
Media Gateway Control Protocol (MGCP) or Session Initiation Protocol (SIP)
signaling protocols [46]. RTP is usually used in conjunction with the Real Time
Control Protocol (RTCP), which serves as a remote control protocol from the mobile
device to the server. While RTP carries the media streams (e.g., audio and video),
RTCP is used to monitor transmission statistics and quality of service (QoS)
information [46].
17
2.3.5 Real Time Streaming Protocol (RTSP)
The Real Time Streaming Protocol (RTSP) was developed by the Internet
Engineering Task Force (IETF) and published in 1998 as RFC 2326 [40]. It is a
protocol used in streaming media environments. It allows a mobile device to control
a streaming media server remotely, issuing commands that are similar to start and
pause commands on video players. It also allows time-based access to files on
streaming servers. Some RTSP servers use RTP as the transport protocol for the
actual audio/video data. RTSP can be transported using the UDP as RTSPU or by the
TCP protocol (RTSPT) [40].
2.4 Real Time Video Streaming Over UDP
High quality real-time video streaming requires a huge bandwidth and a highly
reliable transmission medium. However, wireless mobile networks still have
difficulties in achieving the required reliability. Streaming with the UDP protocol has
less overhead than TCP packets, which have a greater overhead due to packet
acknowledgements. This makes UDP a suitable choice for real time video streaming.
18
CHAPTER 3
STREAMING ARCHITECTURE
Streaming is a method of making video and other multimedia available in real-time,
over the Internet [47]. Through streaming, a user can begin to watch the requested
video stream after only a short delay [48]. The short delay is to enable the mobile
device receive sufficient video frames in its buffers before playback can start. The
buffers enable smoothness in the playback even if there are variations in the
transmission rate of received video stream. To stream the video, the video is broken
into small packets, which are sent over the wireless network from the server to the
mobile device [47]. It is then possible to begin viewing the video stream from the
beginning as the rest of the video frames are transferred to the mobile device while
playing.
A streaming environment typically consists of a server, a network and devices that
receive the video stream [49]. Figure 3.1 illustrates the Streaming video environment
where the server streams video to the mobile devices through a wireless network.
Figure 3.1: Streaming video environment
Video streaming is a two-step process comprised of the following:
1. The streaming server retrieves the video from live sources or from the storage
device connected to the server.
2. The streaming server processes the video and streams it to the mobile devices
through the wireless network.
In this chapter, the steps required to implement the methods of streaming over two
wireless channels are described to enhance smoothness during playback on a mobile
device [13].
19
3.1 Streaming Over Two Channels
According to Hussein and Lars [13], enhanced smoothness in playback can be
achieved if the original video and a duplicated copy of the video are streamed to the
mobile device. The server duplicates the original video, places them on two buffers
and streams them simultaneously to the mobile devices through the wireless network.
The mobile device receives both video streams and starts playback. When there are
missing video frames on any of the received video streams, the mobile device
switches between both video streams to replace the missing frames, thus enhancing
the smoothness in playback of the video. This method is an improvement over
similar suggestions for delivery of video streams to multiple clients [50,51]. Figure
3.2 below illustrates real time video streaming over two wireless channels to a
mobile device.
Figure 3.2: Flow diagram of video streaming over two wireless channels based on [13]
SELECT THE VIDEO FILE CONVERT THE VIDEO TO
GRAYSCALE SEND VIDEO TO THE TWO
STREAMING BUFFERS
STREAM BOTH VIDEO STREAMS OVER THE WIRELESS NETWORK
(Delay Stream One for Two Seconds)
DROP PACKETS ON THE NETWORK
(It is assumed each data packet contains exactly one video
frame)
RECEIVE BOTH VIDEO STREAMS ON TWO BUFFERS ON THE
MOBILE DEVICE
(Delay Stream Two for Two Seconds to Synchronize the
Video Streams)
START PLAYBACK OF VIDEO STREAM AFTER SUFFICIENT
VIDEO FRAMES HAVE ARRIVED
SWITCH BETWEEN THE VIDEO STREAMS TO REPLACE LOST
VIDEO FRAMES
20
3.2 Streaming Servers
Streaming server communicates with a web server connected to a storage database
from where it fetches the video to be streamed over the wireless network [52]. The
server observes the video formats and properties, sets the transmission rate for the
video streams to traverse through the network to the mobile device [29]. In real time
video streaming, the servers use broadcasts transmission to transmit the video stream
to multiple mobile devices at the same time. This conserves bandwidth and requires
lesser resources on the streaming server [53,54].
The streaming server is developed to perform only a fraction of the tasks that a real
time streaming server does. The tasks performed on the server side according to the
proposed case study are as followed.
1. Bandwidth Consideration Settings
2. Duplication, Queue and Delay
3. Frame Dropping
4. Switching in Mobile Device
3.2.1 Bandwidth Consideration Settings
Due to bandwidth limitations of wireless networks identified in Chapter 2, two steps
are taken on the streaming server to improve video streaming over low-bandwidth
network conditions. In the first step, the transmission rate of the video is adjusted to
suit the available bandwidth on the wireless network. This will ensure that the mobile
device receives all the video frames sent from the streaming server without any
frames being dropped by the wireless network.
The second approach is to convert the colour video to grayscale in order to reduce
the size of the video to be streamed. This is done by analyzing the video and
converting each frame of the video stream into grayscale. To convert each frame, the
pixels of the original video frame are analyzed for the red, green and blue (RGB)
values of its luminescence. A mathematical operation is carried out on the red, green
and blue values of the luminescence to obtain equivalent grayscale frame [55,56].
The formula for obtaining the grayscale values of the luminescence of each pixel is:
( )
After the conversion of each pixel, they are combined to form a corresponding
grayscale frame. The conversion of the colour frames to grayscale reduces the frame
size by 34%. Table 3.1 gives information on the file sizes of the test QCIF video for
both colour and grayscale according to the corresponding transmission rates used to
stream the video over the network. The file size per second can be obtained with the
use of the frame size formula indicated below [57]. This helps to determine the
required bandwidth for streaming the video to the mobile device [58].
( )
21
Transmission
Rate
(frames/sec)
File Sizes
Colour Grayscale
15 1.10MB 742.5KB
20 1.49MB 990KB
25 1.86MB 1.24MB
30 2.23MB 1.49MB
Table 3.1: File sizes of colour and grayscale frames
3.2.2 Duplication, Queue and Delay
A video clip is selected from the input devices or the attached storage devices and it
will be duplicated and queued into two streaming buffers on the server [13]. Once the
streaming command is executed, the first video stream is allowed to stream
immediately while the second video stream is delayed for two seconds. The delay
mechanism is applied to the second stream in order to prevent the dropping of similar
frames of both video streams due to the network interferences and traffic load [13].
3.2.3 Frame Dropping
A frame dropping mechanism is introduced in the simulator to portray the
inconsistencies on the wireless network [59,60,61]. In order to drop the video frames
as they pass through the wireless network, a range of video frames are manually
selected on the first and/or second video stream before the streaming starts. Once the
streaming is initiated, the selected range of video frames is dropped manually as they
are streamed over the wireless network to the mobile devices.
3.2.4 Switching in Mobile Device
The mobile device receives the video frames from both the streams on the two
wireless channels and queues them into two separate buffers. Due to the delay added
on the second video stream on the streaming server, the mobile device synchronizes
both the video streams by implementing a two second delay on the first video stream.
The mobile device starts playing the video from one of the video streams, once
sufficient video frames have been received in both buffers. The mobile device
fetches the frames sequentially from the buffer and displays on the screen. The next
set of video frames must be buffered while playing the current ones.
If there is a missing video frame on the current stream, the mobile device switches to
the other buffer to replace the missing video frame. If the missed video frame is
available, the mobile device continues to play the video from the second stream.
22
CHAPTER 4
STREAMING SIMULATOR
4.1 The Streaming Simulator
To study the effects of video frames lost on streamed video, it is important to provide
smoothness to the end users. The study requires an observation of the video
streaming process in a real time environment. Since it is time consuming and not
easy to implement the streaming application in a real time environment, this study
was carried out by developing a simulator that depicts a real time video streaming
environment. The programming tool used in the development of the simulator is the
C# language on the Microsoft Visual Studio.NET development environment
In the design of the simulator, we avoided the use of third-party .NET components in
order to make the simulator easy to modify. The use of third-party components like
the VideoLabs.NET would have restricted the development and manipulation of
simulator. To overcome this issue, the conversion to grayscale is done at runtime
with each video frame being extracted and converted to grayscale and then merged
back to form a new grayscale video stream. This significantly increased the process
time for extracting the video frame and streaming it to the mobile device. Due to the
use of two continuous loops, time and system resources were used when performing
the grayscale streaming.
Figure 4.1: A snapshot of the streaming simulator
The simulator is split into three main parts as shown in Figure 4.1. It consists of a
streaming section, a frame dropping section and a client section. The streaming
section models the tasks carried out on a streaming server in a real life situation
while the frame dropping section implements a frame dropping mechanism that
23
resembles the dropping of video frames over a wireless network. The client section
portrays a mobile device that receives the video stream and plays it.
4.2 Streaming Server
In this section of the simulator, a streaming server that implements the streaming of
two video streams is described as shown in Figure 4.2. The server is used to select
the video clip to be streamed to the mobile device. Due to different bandwidth
conditions that exist between the streaming server and mobile clients in a real life
situation, the server section includes the ability to change the transmission rate for
the video stream and to convert the video to be streamed into grayscale before the
streaming process starts.
Figure 4.2: Streaming server
Two streaming buffers are used for buffering the two video streams before
streaming. The two buffers indicated in Figure 4.2 below are used to illustrate the
current video frames that are currently being processed by both video streams on the
streaming server. The “Frame Sequence” status bars display the current video frames
that are being processed by the respective streaming buffers. The total number of
24
video frames to be streamed by the server is indicated in the “Number of Frames
Sent” status bars. The “Process Time” status bar indicates the amount of time used
for extracting the entire video clip converting it to grayscale and streaming it.
To stream the video by using the simulator, the video clip to be streamed is fetched
from the network or attached storage devices. The transmission rate of the video to
be streamed over the network is then set due to varying bandwidth on wireless
networks. The transmission rate can be done by selecting from the preset values in
the system. The transmission of color videos would further increase the network
traffic and time consuming. In order to reduce this, the videos can also be transmitted
in the form in grayscale format. In order to transmit the color videos in grayscale
format, the video is converted into grayscale before being streamed. A snapshot of
colour and grayscale image for video frame number 151 is shown in Figures 4.3 and
4.4 respectively.
Figure 4.3: Snapshot of colour frame number 151 Figure 4.4: Snapshot of grayscale frame number 151
The second stream is delayed for two seconds after the first stream in order to avoid
or limit the event of loss of same video frames from both video streams as they pass
through turbulent activity in the wireless network. The video is also accompanied by
the information about the current frame being transmitted, the total number of frames
sent from the server and the total process time since the server has started streaming
the video.
4.3 Traversing Network
In this section, a frame dropping mechanism is used to select ranges of video frames
to be dropped as they are sent to the mobile device. During video streaming, frames
may be dropped or lost due to congestion or other factors in the wireless network.
This frame dropping is the main cause of unsatisfactory smoothness of video
playback on the mobile device.
25
Figure 4.5: Frame dropping mechanism
In real life cases, a video frame is sent in several packets due to the size of the video
frame, however for the purpose of this study we assume each packet is carrying one
single frame to make our understanding easier. This makes dropping of the packets
easier while traversing through the network.
In order to simulate the instability in the wireless network, the simulator incorporates
a manual method for specifying which video frames to be dropped as the video is
being streamed over the wireless network. In the method used on the simulator, only
a single range of video frames are dropped from each video stream on the network.
Starting and end values are selected for each of the video streams as shown in Figure
4.3. When selecting the range of video frames to be dropped over the network, only
positive ranges are acceptable on both video streams on the simulator.
4.4 Receiving Client
This section of the simulator simulates the tasks carried out by a mobile device
during video streaming. It receives the video frames from the server and buffers it
before displaying it on the screen. In the simulator that has been developed, the client
part of the simulator receives both video frames and uses a switching mechanism to
replace missing video frames during the playback of the video stream.
26
Figure 4.6: Receiving client
The client part consists of two streaming buffers which indicate the video frames that
are being received and processed by the mobile device and is as shown in Figure 4.4.
The status bars used on the client part have the same functionality with those on the
streaming server. The “Frame Sequence” indicates the sequence of video frames that
have been received, the “Number of Frames” status bars indicates the total number of
video frames that have been received on the mobile device, while the “Process Time”
indicates the amount of time taken to process the received video frames. The Final
output represents the video seen by the end user on the mobile device.
The mobile device receives both video streams simultaneously with a two-second
delay. It investigates which of the video frames have been lost from the stream and
displays the final output by switching between the video streams whenever a video
frame is missing. The missed video frames can be obtained from the second video
stream and the resultant video is displayed without any affecting the smoothness of
the playback.
27
CHAPTER 5
RESULTS AND ANALYSIS
The results and observations of simulation tests were obtained while testing the
simulator. Simulation tests carried out made use of various input parameters
including changing the transmission rate of the streamed video and using colour and
grayscale options in different test videos, and different ranges for dropping the video
frames over the wireless network as they are transmitted from the server to the
mobile device.
The simulator was useful in proving the effectiveness of using two video streams
instead of one video stream and in the ways that the method improved the video
quality of the video stream on the mobile device. The two-second delay which was
introduced when sending the second stream and when receiving the first stream
serves two major purposes: the first was to prevent a case of losing identical video
frames since they were being transmitted together, this would defeat the purpose of
this streaming method. The other reason was to give the mobile device sufficient
time to fill its buffers with sufficient video frames before starting playback of the
video stream. Due to congestion, limited wireless network bandwidth and other
factors that make wireless networks unstable, some of the video frames may be
dropped or lost as they are transmitted to the mobile device. This dropping was
implemented with the help of video frame dropping mechanism and the effects were
observed on the streamed video received by the mobile device.
While performing the tests with the aforementioned input parameters, the following
observation was made. Although the grayscale videos are used to preserve the
bandwidth of the network by reducing the size of the colour video frame by 34%
[13], it was observed that the conversion process from colour video frames to
grayscale takes more time, thus resulting in the increase of overall process time. The
process time is calculated from the moment the video starts streaming over the
network until the playback is completed on the mobile device, as shown in Table 5.1.
Number
of
Frames
Server Client
Colour Grayscale Colour Grayscale
15 22 56 21 56
20 18 55 18 54
25 14 49 13 47
30 12 47 11 46
Table 5.1: Process time for the test video (seconds)
The time taken to process the colour video stream is 1/3rd
of the time taken for
processing the grayscale video. This is due to the conversion of the original colour
28
video frames to grayscale at the runtime. The colour video streamed is just
transmitted through the network directly. In the case of grayscale streaming, the
video frames have to be converted frame by frame before being streamed to the
client section in the simulator.
29
CHAPTER 6
CONCLUSION
The simulator was designed with some of the major functionalities that are used to
study the effects of the lost frames on the streamed video.
The simulator has been developed on a stand-alone computer to study the effects of
missing frames on the mobile device and how it affects the smoothness of playback
of the video on the mobile device. The use of the two video streams and the
switching mechanism on the client part of the simulator ensured that missing frames
were replaced on the mobile device during playback of the video. The delay
mechanism used both on the server and client parts of the simulator reduces the risk
of dropping similar video frames on both video streams if there is an interruption
during transmission over the wireless network. The simulator also considers low
bandwidth conditions by implementing varying transmission rates and colour-
grayscale conversion. It is evident that sending two video streams with a delay
mechanism to the mobile device improves the smoothness of the playback of the
video by reducing the effect of dropped or lost video frames on the wireless network.
There are less visible disruptions to the playback except in scenarios where the
network dropped many video frames.
The limitation of the simulator lies in the conversion process for capturing the colour
video frames and transferring it into grayscale. This can be observed through the
speed of the video process on the mobile section of the simulator. It would be
interesting to see the improvement of the video conversion process by using a more
efficient algorithm. Other areas of interest in the simulator development could be
based on based on a Client-Server architecture which runs the server and client parts
on different computers, while the network effect on real time situation could be
observed. Tests could be conducted in real time for different traffic load.
30
REFERENCES
[1] K. O'Hara, A.S. Mitchell, and A. Vorbau, “Consuming video on mobile
devices,” Proceedings of the SIGCHI conference on Human factors in
computing systems, San Jose, California, USA: ACM, 2007, pp. 857-866.
[2] S.G. Chan, “Distributed servers architecture for networked video services,”
IEEE/ACM Trans. Netw., vol. 9, 2001, pp. 125-136.
[3] M. Ries, O. Nemethova, and M. Rupp, “On the Willingness to Pay in Relation
to Delivered Quality of Mobile Video Streaming,” Consumer Electronics, 2008.
ICCE 2008. Digest of Technical Papers. International Conference on, 2008, pp.
1-2.
[4] Heejung Lee, Yonghee Lee, Jonghun Lee, Dongeun Lee, and Heonshik Shin,
“Design of a mobile video streaming system using adaptive spatial resolution
control,” Consumer Electronics, IEEE Transactions on, vol. 55, 2009, pp.
1682-1689.
[5] M. Stiemerling and S. Kiesel, “A system for peer-to-peer video streaming in
resource constrained mobile environments,” Proceedings of the 1st ACM
workshop on User-provided networking: challenges and opportunities, Rome,
Italy: ACM, 2009, pp. 25-30.
[6] A. Kyriakidou, N. Karelos, and A. Delis, “Video-streaming for fast moving
users in 3G mobile networks,” Proceedings of the 4th ACM international
workshop on Data engineering for wireless and mobile access, Baltimore, MD,
USA: ACM, 2005, pp. 65-72.
[7] Jen-Wen Ding, Chin-Tsai Lin, and Kai-Hsiang Huang, “ARS: an adaptive
reception scheme for handheld devices supporting mobile video streaming
services,” Consumer Electronics, 2006. ICCE '06. 2006 Digest of Technical
Papers. International Conference on, 2006, pp. 141-142.
[8] S. Chandra and S. Dey, “Addressing computational and networking constraints
to enable video streaming from wireless appliances,” Embedded Systems for
Real-Time Multimedia, 2005. 3rd Workshop on, 2005, pp. 27-32.
[9] Bo Jiang, Haixiang Zhang, Chun Chen, and Jianxv Yang, “Enable collaborative
graphics editing in mobile environment,” Computer Supported Cooperative
Work in Design, 2005. Proceedings of the Ninth International Conference on,
2005, pp. 313-316 Vol. 1.
[10] Guanfeng Liang and Ben Liang, “Effect of Delay and Buffering on Jitter-Free
Streaming Over Random VBR Channels,” Multimedia, IEEE Transactions on,
vol. 10, 2008, pp. 1128-1141.
[11] M. Claypool and J. Tanner, “The effects of jitter on the peceptual quality of
video,” Proceedings of the seventh ACM international conference on
Multimedia (Part 2), Orlando, Florida, United States: ACM, 1999, pp. 115-
118.
[12] W. Xu and S.S. Hemami, “Spectrally Efficient Partitioning of MPEG Video
Streams for Robust Transmission Over Multiple Channels,” IN
INTERNATIONAL WORKSHOP ON PACKET VIDEO, 2002.
[13] Hussein Aziz and Lars Lundberg, “Graceful degradation of mobile video
quality over wireless network,” Algarve, Portugal: International Association
for Development of the Information Society (IADIS), 2009, pp. 175-180.
[14] X. Zhu and B. Girod, “Video streaming over wireless networks,” Proceedings
of the European Signal Processing Conference, EUSIPCO-07, Poznan, Poland,
31
2007, pp. 1462 - 1466.
[15] E. Setton, J. Noh, and B. Girod, “Rate-distortion optimized video peer-to-peer
multicast streaming,” Proceedings of the ACM workshop on Advances in peer-
to-peer multimedia streaming, Hilton, Singapore: ACM, 2005, pp. 39-48.
[16] C. Neumann, V. Roca, and R. Walsh, “Large scale content distribution
protocols,” SIGCOMM Comput. Commun. Rev., vol. 35, 2005, pp. 85-92.
[17] D.L. Gall, “MPEG: a video compression standard for multimedia applications,”
Commun. ACM, vol. 34, 1991, pp. 46-58.
[18] J.L. Mitchell, C. Fogg, W.B. Pennebaker, and D.J. LeGall, MPEG video
compression standard, Kluwer Academic Publishers, 1996.
[19] R. de Queiroz, R. Ortis, A. Zaghetto, and T. Fonseca, “Fringe benefits of the
H.264/AVC,” Telecommunications Symposium, 2006 International, 2006, pp.
166-170.
[20] I.E. Richardson, H. 264 and MPEG-4 video compression: video coding for
next-generation multimedia, John Wiley & Sons Inc, 2003.
[21] S. Chattopadhyay, L. Ramaswamy, and S.M. Bhandarkar, “A framework for
encoding and caching of video for quality adaptive progressive download,”
Proceedings of the 15th international conference on Multimedia, Augsburg,
Germany: ACM, 2007, pp. 775-778.
[22] Z. Yetgin and G. Seckin, “Progressive Download for 3G Wireless
Multicasting,” Future Generation Communication and Networking (FGCN
2007), 2007, pp. 289-295.
[23] J. Brandt and L. Wolf, “Adaptive video streaming for mobile clients,”
Proceedings of the 18th International Workshop on Network and Operating
Systems Support for Digital Audio and Video, Braunschweig, Germany: ACM,
2008, pp. 113-114.
[24] R. Sharman, S.S. Ramanna, R. Ramesh, and R. Gopal, “Cache architecture for
on-demand streaming on the Web,” ACM Trans. Web, vol. 1, 2007, p. 13.
[25] Dihong Tian, Xiaohuan Li, G. Al-Regib, Y. Altunbasak, and J. Jackson,
“Optimal packet scheduling for wireless video streaming with error-prone
feedback,” 2004, pp. 1287-1292 Vol.2.
[26] M.A. Qadeer, R. Ahmad, M.S. Khan, and T. Ahmad, “Real Time Video
Streaming over Heterogeneous Networks.”
[27] M. Fraz, Y.A. Malkani, and M.A. Elahi, “Design and Implementation of Real
Time Video Streaming and ROI Transmission System using RTP on an
Embedded Digital Signal Processing (DSP) Platform.”
[28] Dapeng Wu, Y. Hou, Wenwu Zhu, Ya-Qin Zhang, and J. Peha, “Streaming
video over the Internet: approaches and directions,” Circuits and Systems for
Video Technology, IEEE Transactions on, vol. 11, 2001, pp. 282-300.
[29] M. Covell, S. Roy, and B. Seo, “Predictive modeling of streaming servers,”
SIGMETRICS Perform. Eval. Rev., vol. 33, 2005, pp. 33-35.
[30] International Telegraph and Telephone Consultative Committee CCITT
Recommendation H. 261: Video Codec for Audiovisual Services at $p \\times
64$ kbits/sec, WPXV/1, July 1990.
[31] M. Liou, “Overview of the px64 kbit/s video coding standard,” Commun. ACM,
vol. 34, 1991, pp. 59-63.
[32] R. Nicol and N. Mukawa, “Motion video coding in CCITT SG XV-the coded
picture format,” Global Telecommunications Conference, 1988, and Exhibition.
'Communications for the Information Age.' Conference Record, GLOBECOM
'88., IEEE, 1988, pp. 992-996 vol.2.
32
[33] B. Girod, E. Steinbach, and N. Färber, “Performance of the H.263 Video
Compression Standard,” J. VLSI Signal Process. Syst., vol. 17, 1997, pp. 101-
111.
[34] D. Comer, Internetworking with TCP/IP: principles, protocols, and
architecture, Prentice Hall, 2006.
[35] S.M. Bellovin, “Security problems in the TCP/IP protocol suite,” SIGCOMM
Comput. Commun. Rev., vol. 19, 1989, pp. 32-48.
[36] R. Braden, ed., “Requirements for Internet Hosts - Communication Layers,”
1989.
[37] B.A. Forouzan and S.C. Fegan, TCP/IP protocol suite, McGraw-Hill Science,
Engineering & Mathematics, 2002.
[38] W.R. Stevens, TCP/IP illustrated (vol. 1): the protocols, Addison-Wesley
Longman Publishing Co., Inc., 1993.
[39] L.R. Soles, D. Teodosiu, J.C. Pistritto, and X. Boyen, Transmission control
protocol, Google Patents, 2006.
[40] H. Schulzrinne, A. Rao, and R. Lanphier, “Real Time Streaming Protocol
(RTSP),” 1998.
[41] J. Postel, “DoD standard transmission control protocol,” ACM SIGCOMM
Computer Communication Review, vol. 10, 1980, pp. 52–132.
[42] S. McCreary and K.C. Claffy, Trends in wide area IP traffic patterns, Citeseer,
2000.
[43] C. Krasic, K. Li, and J. Walpole, “The Case for Streaming Multimedia with
TCP,” Proceedings of the 8th International Workshop on Interactive
Distributed Multimedia Systems, Springer-Verlag, 2001, pp. 213-218.
[44] R.J. Postel, “User Datagram Protocol,” 1980.
[45] D. Wetteroth, OSI Reference Model for Telecommunications, McGraw-Hill
Professional, 2001.
[46] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, “RTP: A Transport
Protocol for Real-Time Applications,” 2003.
[47] Y. Wang, I. Bouazizi, M.M. Hannuksela, and I.D.D. Curcio, “Mobile video
applications and standards,” Proceedings of the international workshop on
Workshop on mobile video, Augsburg, Bavaria, Germany: ACM, 2007, pp. 1-6.
[48] H. Kimiyama, K. Shimizu, T. Kawano, T. Ogura, and M. Maruyama, “Real-
time Processing Method for an Ultra High-Speed Streaming Server Running PC
Linux,” Proceedings of the 18th International Conference on Advanced
Information Networking and Applications - Volume 2, IEEE Computer Society,
2004, p. 441.
[49] “HP-UX Multimedia Streaming Protocols (MSP) Programmer's Guide: HP-UX
11i v1, HP-UX 11i v2, HP-UX 11i v3,” Sep. 2007.
[50] B. Zheng, X. Wu, X. Jin, and D.L. Lee, “TOSA: a near-optimal scheduling
algorithm for multi-channel data broadcast,” Proceedings of the 6th
international conference on Mobile data management, 2005, p. 37.
[51] C. Chereddi, P. Kyasanur, and N.H. Vaidya, “Design and implementation of a
multi-channel multi-interface network,” Proceedings of the 2nd international
workshop on Multi-hop ad hoc networks: from theory to reality, Florence,
Italy: ACM, 2006, pp. 23-30.
[52] D.L. Moal, T. Takeuchi, and T. Bandoh, “Cost-effective streaming server
implementation using Hi-tactix,” Proceedings of the tenth ACM international
conference on Multimedia, Juan-les-Pins, France: ACM, 2002, pp. 382-391.
[53] L. Silva and A. Correia, “HSDPA delivering MBMS video streaming,”
33
Proceedings of the 2006 ACM CoNEXT conference, Lisboa, Portugal: ACM,
2006, pp. 1-2.
[54] C. Hsu and M. Hefeeda, “Video communication systems with heterogeneous
clients,” Proceeding of the 16th ACM international conference on Multimedia,
Vancouver, British Columbia, Canada: ACM, 2008, pp. 1043-1046.
[55] K. Jack, Video demystified: a handbook for the digital engineer, Newnes, 2005.
[56] M. Grundland and N.A. Dodgson, “Decolorize: Fast, contrast enhancing, color
to grayscale conversion,” Pattern Recognition, vol. 40, Nov. 2007, pp. 2891-
2896.
[57] Adobe Technote Support “Calculate file sizes and disk space for digital video”,
http://kb2.adobe.com/cps/312/312640.html. 2009
[58] S. Johnson, Stephen Johnson on digital photography, O'Reilly Media, Inc.,
2006.
[59] K. Brown and S. Singh, “M-TCP: TCP for mobile cellular networks,”
SIGCOMM Comput. Commun. Rev., vol. 27, 1997, pp. 19-43.
[60] H. Balakrishnan, S. Seshan, and R.H. Katz, “Improving reliable transport and
handoff performance in cellular wireless networks,” Wireless Networks, vol. 1,
Dec. 1995, pp. 469-481.
[61] B.S. Bakshi, P. Krishna, N.H. Vaidya, and D.K. Pradhan, “Improving
Performance of TCP over Wireless Networks,” Distributed Computing Systems,
International Conference on, Los Alamitos, CA, USA: IEEE Computer
Society, 1997, p. 365.