Video Coding

Lecture: Mo., 13-14:30, K2003B

Seminar: Tue, 15-16:30 (G), K2003A

Seminar: Homework assignments for next seminar, homework counts 30% towards thefinal grade. Homework can be done in groups of max. 3 people.

We will also use Moodle2 quizes, wich will beabout weekly and done individually. They count as 25% for the homework.

To pass the course you need to pass the finalexam.

For homework assignments we will use Python:

https://www.python.org/

It is OpenSource, works for all operating systems, and you need to install it on your computer (if it is not already there). You should install Python 2.7x

We recommend to use Linux as operating system, for instance Ubuntu, because that's where Python is easiest. There, Python is already installed, and the installation of all

packages can be done easily from the software manager.

Parallels between Python and Matlab are described here:

http://wiki.scipy.org/NumPy_for_Matlab_Users

We will also need the OpenCV Library for Python. In Linux, this is the software package “python-opencv”. This already contains a few Python examples. And this even works on a Raspberry Pi!

A good development environment is for instance “Spyder”, but it is not necessary (print statements can e.g. be used for debugging)

Slides will be on our webpage (www.tu-ilmenau.de/mt ->Lehrveranstaltungen Master -> Videocoding), or in Moodle 2 (see below)

The course has 3 LP, each LP is about 2-3 hours per week, which means we have 6-9 hours total per week or 12-18 hours for 2 weeks (the homework cycle), which is about

6-8 hours for the bi-weekly homework and the weekly quizes).

We will use the system Moodle for information exchange, the most up-to-date slides, and for the quizes, at:

https://moodle2.tu-ilmenau.de/login/index.php

You need to register in Moodle for this course.

The lecture slides are intended as a guide for self study, not as a complete source. It is best to read the slides before the lecture. I will then read a paragraph, and then ask you for questions about it. Answers will be included in the slides. That is why I will keep them in editing mode.

-Goal: To be able to solve problems in the area of Video Coding, like how to design sampling rate converters or video codecs.

-Approach: Memorize the most basic facts or properties, like the definition of the z-transform, then know how to use them (likehow to derive the z-transform of a delayed signal). This is a skill (skills are obtained by practice, this is also why the homework is so

important).

As an Engineer, we need to get something towork or to function in the end. Engineering is also like applied physics, we need the theory and then the experimental verification.

Books:

Al Bovik: “The Essential Guide to Video Processing”

Khalid Sayood: “Introduction to Data Compression”

Jae S. Lim: “Two Dimensional Signal and Image Processing”

Brief Introduction to Python withNumpy, Scipy, Pylab

Python is a powerful, object-oriented programming language. At the same time, it does not need to be compiled, but can be interpreted at runtime. It also has a practicalCommand Line Interface (interactive mode) where a few lines of code can be written, and the result be seen immediately. In addition, there is an enormous variety of software packages or libraries for it, and everything in Open Source! Due to the command line interpreter "iPython" and the packages "numpy", "Scipy" and "matplotlib" (all together in the library Pylab), the operation is very similar to Matlab. However,MATLAB is a commercial and paid software. Furthermore, Python is very powerful as it can be, for example, used for real time audioand video streams.

Python Numpy and Scipy are packages, which give Python functions similar to Matlab.

They are open source, and can be installed in Linux, e.g. the Raspberry Pi, directly from the Software Center, as python-numpy and python-scipy. Since they run under Python, they also have the advanced capabilities of Python, such as if we want to use video streaming for recording and playback.

We can start Python by simply opening a terminal window and type „python“. This opens the command line interpreter. A somewhat more convenient command line interpreter is „iPython“, which can be installed with the (Linux) command:

sudo apt-get install ipython

The start it from the terminal or console window with:

ipython --pylab

The "--pylab" option above is equivalent to calling

ipython

and then within iPython, libraries are imported.

from numpy import *

from matplot.pyplot import *

or shorter:

from pylab import *

The most interesting mathematical

functions are in the Library "numpy". We canalso import them in python with

import numpy as np

The "as np" means that we can reference the Library "numpy" as "np" in this abbreviated form. This referencing of the library has the advantage that we can

differentiate among functions of the same name from different libraries.

If we want to access all the functions of Numpy without referencing (if no duplicates

from other libraries are there), we can use instead

from numpy import *

Interesting commands are:

help(function name): Help function for thecommand

Example:

help(sin)

In Python, an image is represented as an array (or Tensor), a pixel consists of 3 values(Each 1 value per color).

Example for the entry of an array (a

Black / White image) in Python:

C=array([[1,2],[3,4]])

or because it is part of the "numpy" library :

C=numpy.array([[1,2],[3,4]])

For a color image, each pixel appears, not only as a brightness value, but an array of 3 intensity values, one for each color.

Example:

C=numpy.array([[[1,2,3],[2,3,4]],[[3,4,5], [4,5,6]]])

We obtain the intensities of the primary colors of the pixel at position [0,0] (top left), with:

C[0,0]

array([1, 2, 3])

For pixel at position [0,0], we obtain the

intensity of the primary color red with:

C[0,0,2]

Note: Python distinguishes between arrays and Matrices.

Example for the entry of a 2x2 matrix in python:

A=matrix([[1, 2],[3, 4]])

or because it is part of the "numpy" Library:

A=numpy.matrix([[1, 2],[3, 4]])

A

matrix([[1, 2],

[3, 4]])

B=matrix([[5, 6],[7, 8]])

B

matrix([[5, 6],

[7, 8]])

Addition of two matrices:

A+B

matrix([[ 6, 8],

[10, 12]])

Multiplication:

A*B

matrix([[19, 22],

[43, 50]])

Element by element multiplication:

multiply(A,B)

matrix([[ 5, 12],

[21, 32]])

Inverse of a matrix:

inv(A)

matrix([[-2. , 1. ],

[ 1.5, -0.5]])

Alternative notation for the matrix inversion:

A.I

We can display the dimension of an array or a matrix with the function "shape":

shape(A)

or

A.shape

Example of a simple Python script:

The function y = 2 * a * b + 3 will be programmed as a script. Create a file with the extension “.py”. e.g testscript.py

For this purpose, a Text Editor is required. A commonly recommended editor, which also runs on Raspberry Pi, is “gedit”.

If necessary, it can be installed with:

sudo apt-get install gedit

and create the following content:

#Comment: This is a test script

a=4

b=5

y=2*a+3*b;

print y

We then store this(with the "save as") under the name "testscript.py".

In the launched/popped/displayed console orterminal window (or "shell"), we then type:

python testscript.py

The output is then:

>>23

Alternatively, we can also define a function within Python.

For this purpose, we write a file with the name "testfunction.py" which contains our functions (as it can be several functions), with the following content:

def myfunction(a,b):

"""This is the help text for the test function ..."""

y=2*a+3*b;

return y

Note the constant indentation below the function declaration: It serves the function ofbrackets in python.

Then we start iPython (or if we have already started it):

ipython -pylab

In iPython, we can import our function (s) with:

import testfunction

Now, our function is part of our own Library "Test function". We then call our function in iPython with (a = 7 and b = 8):

testfunction.myfunction(7,8)

Out[6]: 38

Note the pre-pending of "testfunction" This has the advantage that we can differentiate among more identical function names of several files (the "Libraries").

We obtain the help text of our function with:

help(testfunction.myfunction)

Help on function myfunction in the module

test function:

myfunction(a, b)

This is a test function …

By prefixing the library name, "testfunction", it helps us to make sure that there are no naming conflicts. We can then import with:

from testfunction import *

Then we can easily access our function with:

myfunction(7,8)

38

Introduction Video:

Video Coding Standards: H.264, MPEG4, Flash, AVC, Motion JPEG (digital cinema), uncompressed, AVI, DV, Quicktime, WMV...

Applications: digital cinema, internet streaming (HTML5...), DVD, Blu-Ray, DVB-T/C/S/H/IP, digital cameras, video conferencing, iPod, iTunes, Mediatheken,

DVB-T Example: Uses about a factor 4 less bandwidth than analog TV transmission.

(Digital compression depends on the representation and resolution of the original).

How do we obtain compression?

Two basic principles:

Lossless (redundancy reduction)

Lossy (redundancy and irrelevance reduction)

Lossless compression takes advantage of statistical dependencies in the signal (correlations, predictability). Hence the compression ratio depends on the signal. Often a factor of about 2 is obtained (but observe: even an increase of bit rate for certain types of data is possible, like randomdata). An example is predictive coding.

Lossy compression takes advantage of the properties of the receiver, here the human eye. This already starts with the capturing and recording process, by having only a limited amount of pixels (sampling the image at the location of the pixels), and by only having a limited number of pictures (frames) per second (sampling moving images at the time instances of the frames or images). This uses the fact that the eye has only a limited spatial resolution (as shown by its Contrast Sensitivity Function), and a limited temporal resolution (hence the25 pictures per second).

To obtain further compression using properties of the eye, like the CSF, often subband coding (for instance using the DCT)is used.

How is a video signal constructed?

The camera delivers a sequence of images, each image can be seen as a matrix of intensity values. The intensity values usually consist of the values for the 3 primary colors (red, green, blue). Observe that this can also be something else, for instance for earth surveillance cameras fromair planes or satellites, which often scan the earth in the infra-red or ultra-violet wavelength range. In this case, the usual irrelevance reduction techniques make no sense (like the luminance/chrominance transform), because they are made for the eye as the receiver. Here the receiver is a machine to obtain certain information about the earth, and hence the irrelevance reduction should be made for this receiver and this purpose. Redundancy reduction is

still possible, it is independent of the receiver.

Also if the captured image or video is to be further processed in a studio, or investigatedlike an x-ray image, it is best to apply lossless compression, since the assumptionsof lossy compression (human viewer with the naked eye at a certain distance) are not true here.

How is the usual representation in a computer?

For each primary color we have an array or matrix of intensity values, hence we have 3 matrices. For a video we have a sequence ofthese matrices:

Each matrix is of the size of the image. In comparison to image processing, we get the additional dimension of time, which also offers additional possibilities for redundancy and irrelevance reduction.

Python examples

The following is a python script which takes a photo from a connected webcam:

#Program to capture an image from a camera and display it on the screen 

#Gerald Schuller, October 2014 

import cv2 

cap = cv2.VideoCapture(0) 

 # Capture frame 

[ret, frame] = cap.read() 

# Display the resulting frame 

cv2.imshow('Photo',frame) 

#write photo to jpg file: 

cv2.imwrite('pycolorphoto.jpg', frame) 

#Keep photo window open until key 'q' is pressed: 

while(True): 

   if cv2.waitKey(1) & 0xFF == ord('q'): 

        break 

# When everything done, release the capture 

cap.release() 

cv2.destroyAllWindows() 

You start it with the command:

python imagerecdisp.py

Then we read the photo in python with:

photo=cv2.imread('pycolorphoto.jpg');

For instance the following program reads and displays an image from a file named in the argument:

#Program to display an image from a file given in the argument and display it on the screen 

#example command line: python imagedisp.py picture.jpg 

#Gerald Schuller, March 2015 

import cv2 

import sys 

photo=cv2.imread(str(sys.argv[1])); 

# Display the resulting frame 

cv2.imshow('Photo',photo) 

#Keep photo window open until key 'q' is pressed: 

while(True): 

   if cv2.waitKey(1) & 0xFF == ord('q'): 

        break 

# When everything done, release the capture 

cap.release() 

cv2.destroyAllWindows()

We start the program with the command line

python imagedisp.py  pycolorphoto.jpg

We then look at the size of the image with "shape" in the python console:

print("Photo dimensions:",photo.shape)

('Photo dimensions:', (480, 640, 3))

Here,480 and 640, is the number of pixels in the image vertically and horizontally. The “3” has come from the primary colors Red, Green, Blue (RGB). Note that Python stores them in the order BGR. Meaning, that the color components are the last index of a

pixel, and index 0 corresponds Blue, index 1 corresponds to green, and index 2 corresponds to red. Hence we get 480*640*3intensity values, usually represented as 8-bitIntegers.

We have access to the individual intensity levels of a pixels, for example the intensity values of the Pixel at position 0,0 of the primary color blue (index 0) is:

photo[0,0,0] 88

The "88" is an intensity value between 0 and255 for images (corresponding to 8 bits per color intensity value).We can display the image in the matrix "photo" with the command:cv2.imshow('Photo',photo)

The following shows only the blue component intensity values of the image as a grayscale image:cv2.imshow('Blue-Component',photo[:,:,0])