chapter 2 concepts of prob. theory

tch-prob 1

Chapter 2 Concepts of Prob. Theory • Random Experiment: an experiment in which outcome

varies in an unpredictable fashion when the experiment is repeated under the same condition

Specified by: 1. An experimental procedure 2. One or more measurements or observations

tch-prob 2

EXAMPLE 2.1 :

Experiment E1 : Select a ball form an urn containing balls numbered 1 to 50.

Note the number of the ball.

Experiment E2 : Select a ball form an urn containing balls numbered 1 to 4.

Suppose that balls 1 and 2 are black and that balls 3 and 4 are white. Note

number and color of the ball you select.

Experiment E3 : Toss a coin three times and note the sequence of heads/

tails.

Experiment E4 : Toss a coin three times and note the number of heads.

Experiment E6 : A block of information is transmitted repeatedly over a

noisy channel until an error-free block arrives at the receiver.

Experiment E7 : Pick a number at random between zero and one.

Experiment E12 : Pick two numbers at random between zero and one.

Experiment E13 : Pick a number X at random between zero and one, then

pick a number Y at random between zero and X.

tch-prob 3

Discussions

• Compare E3 and E4: same procedure, different observations

• A random experiment with multiple measurements or observations: E2, E3, E12, E13

• sequential experiment consists of multiple subexperiments: E3, E4, E6, E12, E13

• dependent subexperiments: E13

tch-prob 4

Sample Space• Outcome, sample point:

– one and only one per experiment– mutually exclusive, cannot occur simultaneously

• Sample space: set of all possible outcomes

• Example 2.2: sample spaces

• number of outcomes: – Finite– countably infinite S6

– unaccountably infinite S7

1 2 3 6 7 12 13, , , , ,,S S S S S S S

ContinuousSample space

Discrete sample space

tch-prob 5

1

2

3

4

6

7

12

13

1,2,......,50

1, , 2, , 3, , 4,

, , , , , , ,

0,1,2,3

1,2,3,

: 0 1 0,1

, : 0 1 0 1

, : 0 1

S

S b b w w

S HHH HHT HTH THH TTH THT HTT TTT

S

S

S x x

S x y x and y

S x y y x

EXAMPLE 2.2 :

The sample spaces corresponding to the experiments in Example

2.1 are given below using set notation :

tch-prob 6

Event: a subset of S, a collection of outcomes that satisfy certain conditions

Certain event: S

null event:

elementary event: event of a single outcome

Example 2.3:

A2: The ball is white and even-numbered

A4: The number of heads equals the number of tails

A7: The number selected is nonnegative

, ,72 4

E E E

tch-prob 7

Set Operations

1. Union

2. Intersection

mutually exclusive

3. Complement

imply

equal A=B

- Commutative properties of set operations

- Associative properties of set operations

- Distributive properties of set operations

- De Morgan’s Rules

Venn diagrams: Fig. 2.2

A B

A B

A B

cA S A

A B

A B B A A B B A

( ) ( )A B C A B C ( ) ( )A B C A B C

( ) ( ) ( )A B C A B A C

( )c c cA B A B ( )c c cA B A B

A B

tch-prob 8

Example 2.5: Configuration of a three-component system

a. series all three

b. parallel at least one of three

c. two-out-of-three

.....1 21

.....1 21

1

nA A A Ankk

nA A A Ankk

Akk

1

Akk

1 2 3A A AI I

1 2 3A A A

1 2 3 1 2 3( ) ( ) ...cA A A A A A

: event that component is functioningkA k

C1 C2 C3

C1

C2

C3

tch-prob 9

2.2 The Axioms of Probability

A probability law for the experiment E is a rule that assigns to each event a number P(A), called the probability of A, that satisfies the following axioms:

Axiom I: nonnegative

Axiom II: P(S)=1 total=1

Axiom III: If , then

Axiom III’: If for all , then

0 ( )P A

A B P A B P A P B

A Ai j i j11

P A P Ak kkk

A set of consistency rules that any valid probability assignment must satisfy.

tch-prob 10

Corollary 1.

pf:

Corollary 2. pf: from Cor.1, Corollary 3. pf: Let A=S, in Cor.1.

Corollary 4. are pairwise mutually exclusive, then for

1cP A P A

cA A

1 c cP S P A A P A P A

1P A

1 1cP A P A

0P

cA

, ,...., ,1 2

A A An

11

n nP A P A

k kkk

2n

tch-prob 11

Corollary 5.

pf:

Corollary 6.

Corollary 7. If , then pf.

P A B P A P B P A B

P A

P B

c cP A B P A B P B A P A BcP A B P A BcP B A P A B

A B

1.... 1 ....

111

n n nP A P A P A A P A Anj jk kj j kk

cP B P A P A B P A

A B P A P B

tch-prob 12

Axioms + corollaries provide rules for computing the probability of

certain events in terms of other events. However, we still need initial probability assignment

1. Discrete Sample Spaces

find the prob.of elementary events;

all distinct elementary events are mutually exclusive,

Example 2.6., 2.7

2. Continuous Sample Spaces

assign prob. to intervals of the real line or rectangular regions in the plane

Example 2.11

tch-prob 13

Discrete Sample Spaces

First, suppose that the sample space is finite, and

is given by 1 2, nS a a a

' ' '1 2, , , mP B P a a a

' ' '1 2 (2.9)mP a P a P a

If S is countably infinite, then Axiom ’ implies that the probabiliⅢty of an event such as is given by '

2'1,bbD

' '1 2 (2.10)P D P b P b

(by corollary ?)

tch-prob 14

If the sample space has n elements, , a probability

assignment of particular interest is the case of equally likely

outcomes. The probability of the elementary events is

The probability of any event that consists of k outcomes, say

, is

naaS ,1

(2.11) n

aPaPaP n

121

''1, kaaB

2.12 n

kaPaPBP k ''

1

Thus, if outcomes are equally likely, then the probability of

an event is equal to the number of outcomes in the event

divided by the total number of outcomes in the sample space.

(remember the classical definition?)

tch-prob 15

EXAMPLE 2.6

An urn contains 10 identical balls numbered 0,1,…, 9. A random

experiment involves selecting a ball from the urn and noting the

number of the ball. Find the probability of the following events :

A = “number of ball selected is odd,”

B = “number of ball selected is a multiple of 3,”

C = “number of ball selected is less than 5,”

and of and .BA CBA

tch-prob 16

The sample space is , so the sets of outcomes corre

sponding to the above events are

, , and .

If we assume that the outcomes are equally likely, then

.

.

From Corollary 5.

.

9,,1,0 S

9,7,5,3,1A 9,6,3B 4,3,2,1,0C

10

597531 PPPPPAP

10

3963 PPPBP

10

543210 PPPPPCP

10

6

10

2

10

3

10

5 BAPBPAPBAP

tch-prob 17

where we have used the fact that , so

. From Corollary 6,

9,3 BA 102 BAP

BAPCPBPAPCBAP

P A C P B C P A B C

10

1

10

1

10

2

10

2

10

5

10

3

10

5

10

9

tch-prob 18

EXAMPLE 2.7

Suppose that a coin is tossed three times. If we observe the sequenc

e of heads and tails, then there are eight possible outcomes

. If we assume t

hat the outcomes of S3 are equiprobable, then the probability of each

of the eight elementary events is 1/8. This probability assignment imp

lies that the probability of obtaining two heads in three tosses is, by C

orollary 3,

TTTHTTTHTTTHTHHHTHHHTHHHS ,,,,,,,

THHHTHHHTPtossesinheadsP ,,"32"

8

3 THHPHTHPHHTP

4

12"32" PtossesinheadsP

*If we count the number of heads in three tosses, then 0,1,2,3S

tch-prob 19

Continuous Sample Spaces

EXAMPLE 2.11

Consider Experiment E12, where we picked two number x and y at

random between zero and one. The sample space is then the unit

square shown in Fig. 2.8(a). If we suppose that all pairs of numbers

in the unit square are equally likely to be selected, then it is

reasonable to use a probability assignment in which the probability of

any region R inside the unit square is equal to the area of R. Find the

probability of the following events: , , and

.

Figures 2.8(b) through 2.8(c) show the regions

corresponding to the events A, B, and C. Clearly each of these

regions has area ½. Thus

5.0 xA 5.0 yB

yxC

2

1AP

2

1BP

2

1CP

tch-prob 20

FIGURE 2.8 a Two-dimensional sample space and three events.

(a) Sample space

x

y

0 1

S

y

x

(b) Event

0 1x

y

2

1x

2

1x

(c) Event

0 1 x

y

2

1y

2

1y (d) Event

0 1 x

y

yx

yx

tch-prob 21

2.3 Computing probabilities using counting methods

In many experiments with finite sample space, the outcomes can be

assumed to be equiprobable.

Prob. Counting the number of outcomes in an event

1. Sampling with Replacement and with Ordering

choose k objects from a set A that has n distinct objects, with replacement and note the order. The experiment produces an ordered k-tuple

where and

The number of distinct ordered k-tuples =

Ex.2.12 5 balls, select 2

Pr[2 balls with same number] =

x Ai 1,2,...,i k

kn

1,2,3,4,5 25 25525

1, 2,..., kx x x

tch-prob 22

2. Sampling without Replacement and with ordering

choose k objects from a set A that has n distinct objects, without replacement.

number of distinct ordered k-tuples = n (n-1)…(n-k+1)

Ex.2.13 5 balls, select 2

Pr [1st number >2nd number ]= 10/20

k n

5 4 20

tch-prob 23

Permutations of n Distinct Objects

Consider the case when k=n

number of permutations of n objects = n!

For large n, stirling’s formula is useful

Ex. 2.16 12 balls randomly placed into 12 cells, where more than 1 ball is

allowed to occupy a cell.

Pr [ all cells are occupied ]= ~

12! 2

nnn n e

1212!

1255.37 10

tch-prob 24

3. Sampling without Replacement and without ordering

~ ” combination of size k” =

each combination has k! possible orders

“binomial coefficient”, read “n choose k”

Ex. 2.18 The number of distinctive permutation of k white balls and n-k black balls

n

kC

! 1 .... 1

1 .... 1 !! ! !

n

k

n

k

C k n n n k

n n n k nnCk k n k k

nk

!

! !n

k n k

tch-prob 25

Ex. 2.19 A batch of 50 items contains 10 defective items. Select 10 items at random, Pr [ 5 out of 10 defective ]=?

Exercise prob. 47: Multinomial

Ex.2.20 Toss a die 12 times. Pr [ each number appears twice ]= ?

{1,1,2,2,3,3, …….6,6} permutation 12

12!2!2!2!2!2!2!

6

10 40

5 550

10

tch-prob 26

4. Sampling with Replacement and without ordering

Suppose k=5, n=4

object 1 2 3 4

XX X XX

The result can be summarized as XX // X / XX

= k: x

n-1: /

=

Ex. Place k balls into n cells.

1 1

1

n k n k

k n

88!3!5! 3

1 !! 1 !

k nk n

Pick k balls from an urn ofn balls. (k may be greater than n)

tch-prob 27

2.4 Conditional Probability

We are interested in knowing whether A and B are related, i.e. ,

if B occurs, does it alter the likelihood of A occurs.

Define the conditional probability as

for P[B]>0

- renormalize

|P A B

P A BP B

I

B

A

[ ]P A BI

tch-prob 28

Example 2.21. A ball is selected from an urn containing 2 black balls (1,2)

and 2 white balls (3,4).

A: Black ball selected ½

B: even-numbered ball ½

C: number > 2 ½

0.25 0.50.5

P A BP A B P A

P B

I

0 00.5

P A CP A C P A

P C

I

Not related

Related

tch-prob 29

is useful in finding prob. in sequential experiments

Ex.2.22 An urn contains 2 black, 3 white balls.

Two balls are selected without replacement.

Sequence of colors is noted.

Sol. 1. b b two subexperiments

b w

w b

w w

Tree diagram

P A B P B P A B

I

P A P B A

1 2 1 2 1] ] / ][ [ [B BP B P B P BI

B1 W1

2/5 3/5

W2 B2B2 W2

1/4 3/4 2/4 2/4

3/103/10 3/101/10

tch-prob 30

Let be mutually exclusive, whose union = S ,

partition of S.

“ Theorem on total probability”

Ex. 2.24 In prev. example (2.22) find P[W2]=Pr [second ball is white]

1 2

1 2

)(

...n

n

B BA A S A B

A B A B A B

U U UI I

I U I U U I

1 2...

nP A P A B P A B P A B

I I I

1 1 2 2...

n nP A P B P A B P B P A B P A B P B

2 2 1 1 2 1 13 2 1 3 3

5 5 54 2P W P W B P B P W W P W

1 2, , , nB B B

A

S

B1 B2 Bn

tch-prob 31

Suppose event A occurs, what is the prob. of event Bj?

Bayes’ Rule

Before experiment: P[ Bj ]= a priori probabilityAfter experiment: A occurs P[ Bj/A ]= a posteriori probability

1

j j jj

k k

P A B P A B P BP B A nP A P A B P B

k

I

tch-prob 32

Ex.2.26. Binary symmetric channel

Suppose

0 1

12

A AP P

1 1 0 0 1 1 11 1 112 2 2

P B P B A P A P B A A e e

/ 21 0 00 1 1/ 2

1

1 1

,

1

eP B A P AP A B e

P B

P A B e

0 0

Bj

11

Ai 1-e

e

e

1-e

1-p

p

tch-prob 33

2.5 Independence of Events

If knowledge of the occurrence of an event B does not alter the probability

of some other event A, then A is independent of B.

Two events A and B are independent if

P A BP A B P A

P B

I

P A B P A P B

I

tch-prob 34

Ex. 2.28. A ball is selected from an urn of 4 balls

{1,b},{2,b},{3,w},{4,w}

A: black ball is selected

B: even-numbered ball

C: number >2

P[A]=P[B]=P[C]=0.5

If two events have non zero prob., and are mutually exclusive, then they cannot be independent.

14

P A B P A P B

I

0P A C

I

P A B P A

P A C P A

0 0A B A P BP P I

tch-prob 35

Three events A, B, and C are independent if

Ex. 2.30.

Common application of the independence concept is in making the assumption that the event of separate experiments are independent.

Ex.2.31. A fair coin is tossed three times. P[HHH]=1/8, ….

P A B P A P B

P A C P A P C

P B C P B P C

P A B C P A P B P C

I

I

I

I I

141414

0

B P F

D P F

P B D P B P D

P B F P

P D F P

P B D F P B P D P F

I

I

I

I I

BD

F

F

tch-prob 36

2.6. Sequential Experiments

• Sequence of independent experiments

If sub experiments are independent, and event concerns outcomes of

the kth subexperiment, then are independent.

• Bernoulli trial:

Perform an experiment once, note whether an event A occurs.

success, if A occurs prob= p

failure, otherwise. prob= 1-p

Pn(k)= prob. of k successes in n independent repetitions of a Bernoulli trial = ?

1 2 1 2...

n nA AP A P A P A P A

I I I

kA

1 2, , , nA A A

tch-prob 37

Ex.2.34. toss a coin 3 times. P[ H ]= p

Binomial probability law

k=0,1,…,n

30 1P k p

21 3 1P k p p

22 3 1P k p p 33P k p

1 n kkp pn

k pn k

tch-prob 38

Binomial theorem

Let a=b=1 ,

Let

Ex.2.37. A binary channel with BER . 3 bits are transmitted.

majority decoding is used. What is prob. of error?

0

n nn k n ka b a bkk

20

n nnkk

, 1 ,a p b p 1 10 0

n kkn nn

p p kp nkk k

310

3 32 3 62 .001 .999 .001 3 102 3

P k

tch-prob 39

Multinomial Probability Law

Bj’s: partition of S; mutually exclusive

n independent repetitions of experiment

number of times Bj occurs.

Ex.2.38. dart 3 areas

throw dart 9 times, P[3 on each area]=?

B1 BMB2 S........

j jP B p

1 21

Mp p p

1 21 2 1 21 2

![( , ,..., )] ...! !... !

k k kMM MM

nP k k k p p pk k k

jk

0.2,0.3,0.5p

9! 3 3 33,3,3 .2 .3 .53!3!3!

P

tch-prob 40

Geometric Probability Law

Repeat independent Bernoulli trials until the occurrence of the first success.

P[ more than k trials are required before a success occurs ]

11 ... 1 1 , 1,2,...

1

mp m p p p p p m

m times

111 1

1 111

mp m p p ppmm

1[ ] 1 11

m kp m k p p pm k

P[The initial k trials are failures]

tch-prob 41

Sequence of Dependent Experiments

Ex.2.41. A sequential experiment involves repeatedly drawing a ball from one of two urns, noting the number on the ball, and replacing the ball in its urn.

Urn 0 : 1 ball #1, 2 balls #0

Urn 1 : 5 balls #1, 1 ball #0

Initially, flip a coin. Thereafter, the urn used in the subexperiment corresponds to the number selected in the previous subexperiment.

1 1 1

h

0

1 1

0

0 0

1 1

....

....

0 0 0

t

0S1

S2

S3

S

....

Trellis diagram

Urn 2

Urn 1

tch-prob 42

conditional prob.

Markov property

Ex.2.42 calculate

with

....

0 1 1 1 2 1 0 0

0 1 1 0 0

0 1 2 2 0 1 0 1

2 0 1 1 0 0

2 1 1 0 0,... ...n n n n n

P S S P S S P S

P S S S P S S S P S S

P S S S P S S P S

P S S P S S P S

P S S S P S S P S S P S S P S

0011 1 1 1 0 0 0 0

1 2 11 0 0 0 0 [1]

3 3 2151 1 0 166

P P P P P

P P P P

P P

tch-prob 43

Problem 95 on Page 83

95 Consider a well-shuffled deck of cards consisting of 52 distinct cards, of

which four are aces and four are kings.

a. Pr[Obtaining an ace in the first draw]=

b. Draw a card from the deck and look at it.

Pr[Obtaining an ace in the second draw]=

If does not know the previous outcome

4 152 13

3

51

4

51

A

A

31 12 413 51 13 51

1

13

tch-prob 44

c. Suppose we draw 7 cards from the deck.

Pr[7 cards include 3 aces]=

Pr[7 cards include 2 kings]=

Pr[7 cards includes 3 aces and 2 kings]=

d. Suppose the entire deck of cards is distributed equally among four

players. Pr[each player gets an ace]=

4 48

3 452

7

4 48

2 552

7

4 48 3 36 2 24 134!

1 12 1 12 1 12 1352 39 26 13

13 13 13 13

4!

4 4 44

3 2 252

7

tch-prob 45

Birthday problem.

p =No two people in a group of n people will have a common birthday.

Lotto Problem.

Six numbers and one special number are picked from a pool of 42 numbers. …

1. What is the probability of winning the grand prize?

2. What is the probability of winning 2nd prize?

3. What is the probability that number 37 is drawn next time?

1 2 11 1 1

365 365 365

123

20.01 56

np

p n

p n