probability and random processes - ?· probability and random processes main aims i understand...

Download Probability and Random Processes - ?· Probability and Random Processes Main aims I Understand fundamental…

Post on 30-Aug-2018

213 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • Probability and Random Processes

    Probability and Random Processes

    Jinho ChoiGIST

    February 2017

    1 / 99

  • Probability and Random Processes

    What Albert Einstein said:

    I As I have said so many times, God doesnt play dice with theworld.

    I Two things are infinite: the universe and human stupidity.

    So, he does believe in infinity,but not in randomness.

    2 / 99

  • Probability and Random Processes

    Main aims

    I Understand fundamental issues of probability theory (pdf,mean, and variance)

    I Understand joint and conditional pdf, independence, andcorrelation

    I Learn properties of Gaussian random variables and be able toderive Chernoff bound

    I Understand random processes with key definitions

    I Be able to compute the mean and variance of samples fromrandom processes

    3 / 99

  • Probability and Random Processes

    Probability Theory

    Probability Theory

    The three axioms with a sample space, , a family of sets, F , forallowable events, and measure Pr():

    I The first axiom: The probability of an event, E, is anon-negative real number:

    Pr(E) 0.

    I The second axiom:Pr() = 1.

    I The third axiom: For any countable sequence of mutuallyexclusive events E1, E2, . . .,

    Pr(E1 E2 ) =

    i=1

    Pr(Ei).

    4 / 99

  • Probability and Random Processes

    Probability Theory

    Random variables: A random variable is a mapping from thesample space to the set of real numbers.

    Sample space (abstract space) Real number

    The main idea of random variables is to describe some randomevents by numbers.

    real numbers

    Event space

    event X( )

    X: random variable:

    X() (,)

    5 / 99

  • Probability and Random Processes

    Probability Theory

    Example of random variables: A game of dice

    I Before you draw a dice, the number of dots is unknown. This number can be considered as a random variable.I Once a dice is drawn, we have a particular number, which

    would be one of {1, . . . , 6}. This is called a realization.

    realizations of {2, 3, 4}6 / 99

  • Probability and Random Processes

    Probability Theory

    Cumulative distribution function (CDF):

    FX(x) = Pr(X x) ,

    where X is the random variable (r.v.) and x is a real number. Bydefinition, the CDF is a nondecreasing function.Example: a dice

    Cummulative distribution function (CDF):

    FX(x) = Pr(X x),where X is the random variable (r.v.) and x is a real number.

    Probability density function (PDF):

    fX(x) =d

    dxFX(x)

    Example: a dice

    1 4 5 62 3

    2/6

    3/6

    4/6

    5/6

    1

    1/6

    FX(x) = Pr(X x)

    x

    3

    7 / 99

  • Probability and Random Processes

    Probability Theory

    There are different types of r.v.s such as:

    I continuous r.v.: X has a continuous value

    I discrete r.v.: X has a discrete value

    Examples:

    I the phase of a sinusoid, : sin(ct+ ) continuous r.v.

    I the number of dots of a dice discrete r.v.

    8 / 99

  • Probability and Random Processes

    Probability Theory

    A discrete r.v. with binomial distribution

    I Consider a random experiement that has two possibleoutcomes. For example, the outcome of this experiment canbe expressed (1 to denote success; 0 to denote failure) as

    Y =

    {1, with probability p;0, with probability 1 p.

    I Consider a sum of n outcomes from independent experiments:

    X =

    n

    j=1

    Yj = {0, 1 . . . , n}.

    I Then, X is the binomial random variable with parameters nand p, X B(n, p):

    Pr(X = j) =

    (n

    j

    )pj(1 p)nj , j = 0, . . . , n.

    9 / 99

  • Probability and Random Processes

    Probability Theory

    Continuous r.v., X, has the probability density function (pdf) as

    fX(x) =d

    dxFX(x) .

    I As FX(x) is nondecreasing, fX(x) 0.

    I In general,

    t

    fX(x)dx = FX(t) .

    I Since limx FX(x) = 1, fX(x)dx = 1.

    For a discrete r.v., the pdf becomes the probability massfunction (pmf) which is actually probability.

    I Example of dice:

    fX(X = k) Pr(X = k) =1

    6, k = 1, 2, . . . , 6.

    10 / 99

  • Probability and Random Processes

    Probability Theory

    Mean and VarianceFor a r.v. X, the mean of X (or g(X), where g(x) is a function ofx) is given by

    I a continuous r.v.:

    E[X] =xfX(x)dx or E[g(X)] =

    g(x)fX(x)dx

    I a discrete r.v.:

    E[X] =

    k

    xk Pr(xk) or E[g(X)] =

    k

    g(xk) Pr(xk)

    The variance is given by

    V ar(X) = E[(X E[X])2]

    The mean and variance are used to characterize a random variable.11 / 99

  • Probability and Random Processes

    Probability Theory

    Mean of X B(n, p)

    E[X] =n

    j=0

    j

    (n

    j

    )pj(1 p)nj

    =

    n

    j=1

    jn!

    j!(n j)!pj(1 p)nj

    =

    n

    j=1

    n(n 1)!(j 1)!(n j)!

    ppj1(1 p)nj

    = np

    n

    j=1

    (n 1)!(j 1)!(n j)!

    pj1(1 p)nj

    = np(p+ (1 p))n1 = np.

    12 / 99

  • Probability and Random Processes

    Probability Theory

    Geometric random variable:

    I pmf: Pr(X = k) = (1 p)k1p, k 1Note that

    k=1

    (1 p)k1p = p

    k=0

    (1 p)k = p 11 (1 p)

    = 1

    I Example: The number of independent flips of a coin untilhead first apprears.

    I Mean: Letting q = 1 p, it can be shown that

    E[X] =

    k=1

    k(1 p)k1p = p ddq

    k=0

    qk

    = pd

    dq

    1

    1 q= p

    1

    (1 q)2=

    1

    p

    13 / 99

  • Probability and Random Processes

    Probability Theory

    A continuous r.v. with uniform distributionLet us consider a uniform r.v. X that has the pdf as

    fX(x) =

    {1A , 0 x A;0, otherwise

    The mean is

    E[X] = A

    0x

    1

    Adx =

    1

    A

    x2

    2

    A

    x=0

    =A

    2.

    The variance is

    E[(X E[X])2] = A

    0

    (x A

    2

    )2 1Adx

    =

    A/2

    A/2z2

    1

    Adz

    =1

    A 1

    3z3A/2

    A/2=A2

    1214 / 99

  • Probability and Random Processes

    Probability Theory

    More examples on expectationQ) Let X be a r.v. and a and c are constants. Show that

    E[aX] = aE[X] and E[X + c] = E[X] + c

    A)

    E[aX] =

    (a x)fX(x)dx = a

    xfX(x)dx = aE[X]

    E[X + c] =

    (x+ c)fX(x)dx =

    xfX(x)dx+

    cfX(x)dx = E[X] + c

    Q) Show that E[X2] = V ar(X) + (E[X])2A)

    E[(X E[X])2] = E[X2 2(E[X])X + (E[X])2]= E[X2] 2(E[X])E[X] + (E[X])2= E[X2] (E[X])2

    15 / 99

  • Probability and Random Processes

    Probability Theory

    Q) Show that V ar(aX + c) = a2V ar(X)

    Q) Suppose that X is a r.v. with mean 1 and variance 3. FindE[3X2 + 2X].A)

    E[3X2 + 2X] = 3E[X2] + 2E[X]= 3(V ar(X) + E2[X]) + 2E[X]

    = 3 (3 + 12) + 2 1 = 14.

    16 / 99

  • Probability and Random Processes

    Probability Theory

    Jensens inequality:For a convex function, g(x),

    E[g(X)] g(E[X]) .

    A convex function satisfiesg(x1) + (1 )g(x2) g(x1 + (1 )x2), [0, 1]

    17 / 99

  • Probability and Random Processes

    Probability Theory

    Gaussian or normal random variable:

    fX(x) = N (, 2) =12

    exp

    ((x )

    2

    22

    ),

    where the mean

    E[X] =

    xfX(x)dx =

    and the variance

    E[(X E[X])2] =

    (x )2fX(x)dx = 2

    18 / 99

  • Probability and Random Processes

    Probability Theory

    Normal or Gaussian pdfs:

    19 / 99

  • Probability and Random Processes

    Probability Theory

    Normal or Gaussian cdfs:

    20 / 99

  • Probability and Random Processes

    Probability Theory

    Q-function: a tail of the normal pdf.

    Pr(X x) = Q(x) 4=

    x

    12

    exp

    ( t

    2

    2

    )dt

    21 / 99

  • Probability and Random Processes

    Probability Theory

    Conditional probabilityThe conditional probability of an event A given B is denoted andgiven by

    Pr(A |B) = Pr(A,B)Pr(B)

    Q) Find the probability that the face of one dot occurs assumingodd dots are observed:A) We have

    Pr(odd) =1

    2and

    Pr(1, odd) = Pr(1) =1

    6.

    Hence, it follows

    Pr(1| odd) = Pr(1, odd)Pr(odd)

    =1

    3.

    22 / 99

  • Probability and Random Processes

    Probability Theory

    Multiple random variables: The joint pdf is written as

    fXY (x, y) =2

    xyFXY (x, y) =

    2

    xyPr(X x, Y y)

    Conditional pdf:

    fX|Y (x|y) =

    {fXY (x,y)fY (y)

    , if fY (y) 6= 0.0, otherwise.

    Marginalization:

    fX(x) =

    fXY (x, y)dy or fY (y) =

    fXY (x, y)dx

    23 / 99

  • Probability and Random Processes

    Probability Theory

    Expectation with two r.v.s:For continuous r.v.s, double-integral should take place:

    E[g(X,Y )] =

    g(x, y)fXY (x, y)dxdy (continuous)

    E[g(X,Y )] =

    x

    y

    g(x, y) Pr(X = x, Y = y) (discrete).

    The conditional expectation is defined by

    E[X|Y ] =xfX|Y (x|Y )dx (continuous)

    E[X|Y ] =

    x

    xPr(X = x |Y ) (discrete).

    Note that E[X|Y ] = g(Y ) is a function of Y , which is a randomvariable.

    24 / 99

  • Probability and Random Processes

    Probability Theory

    Q) Show thatE[XY ] = E[Y E[X|Y ]]

    A)

    E[XY ] =

    xyfXY (x, y)dxdy

    =

    xyfX|Y (x|y)fY (y)dxdy

    =

    y

    (xfX|Y (x|y)dx

    )fY (y)dy

    =

    yE[X|y]fY (y)dy = E[Y E[X|Y ]].

    25 / 99

  • Probability and Random Processes

Recommended

View more >