lecture01 02 intro probability theory

7/23/2019 Lecture01 02 Intro Probability Theory

http://slidepdf.com/reader/full/lecture01-02-intro-probability-theory 1/62

Copyright © Syed Ali Khayam 2009

CSE801 Analysis of Stochastic

Systems

Welcome and Introduction

Dr. Muhammad Usman IlyasSchool of Electrical Engineering & Computer Science (SEECS)

National University of Sciences & Technology (NUST)




Course Information

Lecture Timings:

Tuesday: 5:30pm-7:20pm Thursday: 5:30pm-6:20pm

My Office:

Room # A-312

Office Hours

Thursdays, 5:00-5:30pm, or by appointment.

[email protected]

The course will be managed through LMS

www.lms.nust.edu.pk

Facebook group2

The lecture notes are designed and developed by Dr. Ali Khayam.

mailto:[email protected]

http://www.lms.nust.edu.pk/

http://www.lms.nust.edu.pk/

mailto:[email protected]




Timetable (MS EE-Telecom & Comp Networks)

3

Class Room # 5

TIME / DAYS Monday Tuesday Wednesday Thursday Friday Saturday

5:30pm-6:20pm Adv. Computer

NetworksStochastic Systems

Library/Make-up

Class/Seminar

Stochastic Systems Adv. Digital

Commnication

Library/Make-up

Class/Seminar

6:30pm-7:20pm Adv. Digital

Commnication Adv. Computer

Networks7:30pm-8:20pm

Library/Make-up

Class/Seminar

Library/Make-up

Class/Seminar

Library/Make-up

Class/Seminar

8:30pm-9:20pmLibrary/Make-up

Class/Seminar

course Code Subject Instructor Credit Hours

MS EE(Digital System and Signal Processing)-5

CSE 801 Stochastic Systems Dr. Shahzad Younis 3+0

EE 831 Advanced Digital

Signal ProcessingDr. Amir Ali Khan 3+0

EE 823 Advanced Digital

System DesignDr. Rehan Hafiz 3+0

MS EE(Telecommnication & Computer Networks)-5

CSE-801 Stochastic Systems Dr. Usman Illyas 3+0

CSE-820 Adv. Computer

NetworksDr. Junaid Qadir 3+0 Manager-PG

Academic Coord Branch

(Iftikhar Ahmed)

Sep 2013

EE-851 Adv. Digital

CommnicationDr. Rizwan Ahmed 3+0




Timetable (MS EE-Power & Control)

4

MS EE(Power & Control )-1First Semester (9 Sep 2013 - 10 Jan 2014)

Class Room # 7

TIME / DAYS Monday Tuesday Wednesday Thursday Friday Saturday

5:30pm-6:20pmPower Electronics and

Electric Drives

Stochastic Systems

CR #5

Power Electronics

and

Electric Drives

Stochastic

Systems

CR #5

Library/Make-up

Class/Seminar

Library/Make-up

Class/Seminar

6:30pm-7:20pmLinear Control

Systems

Linear ControlSystems

7:30pm-8:20pmLibrary/Make-up

Class/Seminar

Library/Make-up

Class/Seminar

Library/Make-up

Class/Seminar8:30pm-9:20pm

Library/Make-up

Class/Seminar

course Code Subject Instructor Credit Hours

MS-EE(RF & MW )-4

EE-901

Power Electronics and

Electric Drives

Dr. Syed Raza

Kazmi 3+0

EE-871

Linear Control

Systems

Dr. Ammar

Hassan 3+0

CSE-801 Stochastic Systems Dr. Usman ilyas 3+0




Textbooks

5

Probability & Random Processes

for Electrical Engineers, 2nd or 3rd ed.

Albert Leon-Garcia

Introduction to Probability Models,

9th ed.

Sheldon M. Ross

Elements of Information Theory Thomas M. Cover and Joy

A. Thomas

Chaos Theory Tamed Garnett P. Williams




Course Outline

Syllabus Introduction to Probability Theory

Random Variables

Limits and Inequalities

Central Limit Theorem

Application Area: Information Theory

Stochastic Processes

Prediction and Estimation

Markov Chain and Processes (time permitting)

Application Area: Chaos Theory

6


http://slidepdf.com/reader/full/lecture01-02-intro-probability-theory 7/62Copyright © Syed Ali Khayam 2008

Grading (subject to change)

Final Exam: 45%

Midterm Exam: 30%

Quizzes: 15%

Homework Assignments: 10%

7



Policies

Quizzes will be unannounced

Late homework submissions will be accepted for up to 24 hourswith a 50% penalty.

Strong disciplinary action will be taken in case of plagiarism orcheating in exams, homework or quizzes.

Attendance:

Will be taken at the beginning of the class.

The current rules of the school will be followed (75%minimum requirement).

8



What will we cover in this lecture?

This lecture is intended to be an introduction to elementary

probability theory

We will cover:

Random Experiments and Random Variables

Axioms of Probability Mutual Exclusivity

Conditional Probability

Independence

Law of Total Probability

Bayes’ Theorem

9



Definition of Probability

Probability:

1 : the quality or state of being possible

2 : something (as an event or circumstance) that is possible

3 : the ratio of the number of outcomes in an exhaustive set ofequally likely outcomes that produce a given event to thetotal number of possible outcomes, the chance that a given

event will occur

We will revisit these definitions in a little bit …

10



Definition of a Random Experiment

A random experiment comprises of:

A procedure

An outcome

Procedure(e.g., flipping a coin)

Outcome

(e.g., the value

observed [head, tail] afterflipping the coin)

Sample Space

(Set of All Possible

Outcomes)

11



Definition of a Random Experiment:Outcomes, Events and the Sample Space

An outcome cannot be further decomposed into other outcomes{s1 = the value 1}, …, {s6 = the value 6}

An event is a set of outcomes that are of interest to us

A = {s: such that s is an even number}

The set of all possible outcomes, S, is called the sample space

S = {s1, s2, s3, s4, s5, s6}

12



s 1

s 2

s 3

s 4

s 5

s 6

S

13




Example of a Random Experiment: Experiment: Roll a fair dice once and record the

number of dots on the top face

S = {1, 2, 3, 4, 5, 6}

A = “the outcome is even” = {2, 4, 6}

B = “the outcome is greater than 4” = {5, 6}

14




Axioms of Probability

Probability of any event A is non-negative:

Pr{ A} ≥ 0

The probability that an outcome belongs to the sample space is 1:

Pr{S} = 1

The probability of the union of mutually exclusive events is equalto the sum of their probabilities:

If A1 ∩ A

2=Ø,

=> Pr{ A1 U A2} = Pr{ A1} + Pr{ A2}

15



Mutual Exclusivity

Are A1 and A2 mututally exclusive?

For mutually exclusive events A1, A2 … AN, we have:

s 1

s 2

s 3

s 4

s 5

s 6

S

A 1

A 2

Find Pr{ A1 U A2}and Pr{ A1}+Pr{ A2}

in the fair dice

example

16



Mutual Exclusivity

Discarding the condition of exclusivity, in general, we have:

Pr{ A1 U A2} = ??

s 1

s 2

s 3

s 4

s 5

s 6

S

17



Mutual Exclusivity

Discarding the condition of exclusivity, in general, we have:

Pr{ A1 U A2} = Pr{ A1} + Pr{ A2} – Pr{ A1 ∩ A2}

s 1

s 2

s 3

s 4

s 5

s 6

S

18




Given that event B has already occurred, what is the probability

that event A will occur? Given that event B has already occurred, reduces the sample

space of A

s1

s2

s3

s4

s5

s6

S

s1

s2

s3

s4

s5

s6

Event B has

already occurred

=> s2, s4, s3

cannot occur

S

19




Given that event B has already occurred, we define a new

conditional sample space that only contains B’s outcomes The new event space for A is the intersection of A and B:

Event space -> E A|B = A ∩ B

s1

s2

s3

s4

s5

s6

S

s1

s2

s3

s4

s5

s6

Event B has

already

occurred

S

What’s missing here? S|B = {s1, s5, s6}

E A|B= A ∩ B = {s6}20




The probability of an event A in the conditional sample space is:

Pr = ∩

Pr ={}

{} = /

/ =

s1

s2

s3

s4

s5

s6

S

s1

s2

s3

s4

s5

s6

Event B has

already

occurred

S

S|B = {s1, s5, s6}

21



Independence

Two events are independent if they do not provide any

information about each other:

(|) = ()

In other words, the fact that B has already happened does notaffect the probability of A’s outcomes

Implications:

(|) = () ∩

() = ()

( ∩ ) = () ()

22




Independence: Example

Are events A and C independent?

Assume that all outcomes are equally likely

s4

s1

s2

s3

s6

s5

S

23






s4

s1

s2

s3

s6

s5

S

24






Pr{ A ∩ C } = Pr{s5} = 1/6

Pr{ A}Pr{C } = (3/6)x(2/6) = 1/6

Yes!

s4

s1

s2

s3

s6

s5

S

25





Are events A and B independent?

Assume that all outcomes are equally likely

s4

s1

s2

s3

s6

s5

S

26





Are events A and B independent?

Pr{ A ∩ B} = Pr{s5} = 1/6

Pr{ A}Pr{B} = (3/6)x(3/6) = ¼

No!

s4

s1

s2

s3

s6

s5

S

27




Mutual Exclusivity and Independence

Experiment:

Roll a fair dice twice and record the dots on the top face:

= {(1,1), (1,2), (1,3), (1,4), (1,5), (1,6),

(2,1), (2,2), (2,3), (2,4), (2,5), (2,6),

(3,1), (3,2), (3,3), (3,4), (3,5), (3,6),

(4,1), (4,2), (4,3), (4,4), (4,5), (4,6),

(5,1), (5,2), (5,3), (5,4), (5,5), (5,6), (6,1), (6,2), (6,3), (6,4), (6,5), (6,6) }

28




Define three events:

1 = “first roll gives an odd number”

2 = “second roll gives an odd number” = “the sum of the two rolls is odd”

Find the probability of using probability of 1 and 2

29





A 1

A 2

30

S = { (1,1), (1,2), (1,3), (1,4), (1,5), (1,6),

(2,1), (2,2), (2,3), (2,4), (2,5), (2,6),

(3,1), (3,2), (3,3), (3,4), (3,5), (3,6),

(4,1), (4,2), (4,3), (4,4), (4,5), (4,6),

(5,1), (5,2), (5,3), (5,4), (5,5), (5,6),

(6,1), (6,2), (6,3), (6,4), (6,5), (6,6) }





Recap

1. Outcomes, events and sample space:

2. For mutually exclusive events A1, A2 ,…, AN, we have:

3. In general, we have:

37




4. Conditional probability reduces the sample space:

5. Two events A and B are independent only if

6. For independent events:

38

Recap




Four “Rules of Thumb”

1. Whenever you see two events which have an OR relationship (i.e., event A or

event B), their joint event will be their union, { A U B}Example: On a binary channel, find the probability of error?

An error occurs when

A: “a 0 is transmitted and a 1 is received” OR

B: “a 1 is transmitted and a 0 is received”

Thus probability of error is: Pr{ A U B}

T0

T1

R0

R1

Pr{R0|T0}

Pr{R1|T1}

39




2. Whenever you see two events which have an AND relationship (i.e., both

event A and event B), their joint event will be their intersection, { A ∩ B}

Example: On a binary channel, find the probability that a 0 is transmitted and a

1 is received?


A: “a 0 is transmitted” AND

B: “a 1 is received” Thus probability of above event is: Pr{ A ∩ B}

T0

T1

R0

R1

Pr{R0|T0}

Pr{R1|T1}

40





3. Whenever you see two events which have an OR relationship (i.e., A U B),

check if they are mutually exclusive. If so, set Pr{ A U B} = Pr{ A} + Pr{B}

Example: On a binary channel, find the probability of error?




Thus probability of error is: Pr{error} = Pr{ A U B}

Are A and B are mutually exclusive?

T0

T1

R0

R1

Pr{R0|T0}

Pr{R1|T1}

41





3. Whenever you see two events which have an OR relationship (i.e., A U B), check if

they are mutually exclusive. If so, set Pr{ A U B} = Pr{ A} + Pr{B}

Example: On a binary channel, find the probability of error?




Thus probability of error is: Pr{error} = Pr{ A U B}

YES!

A and B are mutually exclusive; transmission of a 0 precludes the possibility of

transmission of a 1, and vice versa. Therefore, we can set

Pr{error} = Pr{ A U B} = Pr{ A} + Pr{B}

T0

T1

R0

R1

Pr{R0|T0}

Pr{R1|T1}

42





4. Whenever you see two events which have an AND relationship (i.e., A ∩ B), check if

they are independent. If so, set Pr{ A ∩ B} = Pr{ A}Pr{B}

Example: On a binary channel, find the probability that a 0 is transmitted and a 1 is

received?


B: “a 1 is received”

Probability of above event is: Pr{ A ∩ B}

Are A and B independent?

T0

T1

R0

R1

Pr{R0|T0}

Pr{R1|T1}

43





4. Whenever you see two events which have an AND relationship (i.e., A ∩ B), check if

they are independent. If so, set Pr{ A ∩ B} = Pr{ A}Pr{B}

Example: On a binary channel, find the probability that a 0 is transmitted and a 1 is

received?


B: “a 1 is received”

Probability of above event is: Pr{ A ∩ B}

Are A and B independent?

No.

Pr ∩ = Pr Pr = 0 ×1

2 =

0

2

Pr Pr =1

2

×1

2

=1

4

T0

T1

R0

R1

Pr{R0|T0}

Pr{R1|T1}

44





Total Probability

B1, B2 ,…, BN form a partition of a sample space we have: S = B1 U B2 U … U BN

Bi ∩ B j = Ø, i ≠ j

B1

B2

B3 B4 s2 s4

s6

s1 s5

s3

45




Total Probability

If B1, B2 ,…, BN form a mutually exclusive partition:

What does this imply?

B1

B2

B3

B4 A

s2s4

s6

s1 s5

s3

A

46




Total Probability


What does this imply? B1 ∩ B2 ∩ ….. ∩ Bn = Ø and B1 U B2 U ….. U Bn = S

B1

B2

B3

B4 A

s2s4

s6

s1 s5

s3

A

47




Total Probability



How to express A in term of Bi?

B1

B2

B3

B4 A

s2s4

s6

s1 s5

s3

A

48




Total Probability



How to express A in term of Bi? A = ( A ∩ B1) U ( A ∩ B2) U … U ( A ∩ BN)

B1

B2

B3

B4 A

s2s4

s6

s1 s5

s3

A

49




Total Probability




What is the probability of A?

B1

B2

B3

B4 A

s2s4

s6

s1 s5

s3

A

50




Total Probability




What is the probability of A? Pr{ A} = Pr{ A ∩ B1} + Pr{ A ∩ B2} + … + Pr{ A ∩ BN}

B1

B2

B3

B4 A

s2s4

s6

s1 s5

s3

A

51




Total Probability

Using the definition of conditional probability:

Pr{ A| Bi } = Pr{ A ∩ Bi } / Pr{Bi }

=> Pr{ A ∩ Bi } = Pr{ A| Bi } Pr{Bi }

B1

B2

B3

B4 As2

s4s6

s1 s5

s3

A

52




The Law of Total Probability

The Law of Total Probability states:

B1

B2

B3

B4

As2 s4

s6

s1 s5

s3

A

If B1, B2,…, BN form a partition then for any event A

Pr{ A} = Pr{ A|B1} Pr{B1} + Pr{ A|B2} Pr{B2} + … + Pr{ A|BN} Pr{BN}

53

’ h




Based on the Law of Total Probability, Thomas Bayes decided to

look at the probability of a partition given a particular event, theso-called inverse probability.

Bayes’ Theorem

B1

B2

B3

B4

As2

s4s6

s1 s5

s3

A

54

’ h




Bayes’ Theorem

Based on the Law of Total Probability, Thomas Bayes decided to

look at the probability of a partition given a particular eventPr{Bi | A} = Pr{ A ∩ Bi } / Pr{ A}

=> Pr{ A ∩ Bi } = Pr{ A|Bi } Pr{Bi }

=> Pr{Bi | A} = Pr{ A|Bi } Pr{Bi } / Pr{ A}

B 1

B 2

B 3

B 4As 2

s 4s 6

s 1 s 5

s 3

A

55

B ’ Th




Bayes’ Theorem

Pr{Bi | A} = Pr{ A|Bi } Pr{Bi } / Pr{ A}

From the Law of Total Probability, we have:Pr{ A} = Pr{ A|B1} Pr{B1} + Pr{ A|B2} Pr{B2} + … + Pr{ A|BN} Pr{BN}

B 1

B 2

B 3

B 4A

s 2s 4

s 6

s 1 s 5

s 3

A

Bayes’ Rule

56

B ’ Th




Bayes’ Theorem

B 1

B 2

B 3

B 4A

s 2s 4

s 6

s 1 s 5

s 3

A

57

A Fi h P bl




Azeem (Iqbal) is a fisherman.

Azeem is an educated man.Azeem builds a fishing robot that will do his work for him.

A Fishy Problem …

58

B 4

B 2

B 1

B 3

A Fi h P bl




Question: If Azeem’s robot catches a fish that is detected red, what species is it?

Answer: It could be any of four species in Azeem’s part of the sea.

A Fishy Problem …

59

B 4

B 2

B 1

B 3

A Fi h P bl




Let’s change the question:

What is the chance that a red fish is a species B1, B2, B3 and B4?

A Fishy Problem …

60

B 4

B 2

B 1

B 3

A Fi h P bl




A Fishy Problem …

61

B 4

B 2

B 1

B 3

lecture01 02 intro probability theory

Documents