ece353: probability and random processes lecture...

ECE353: Probability and Random Processes

Lecture 2 - Set Theory

Xiao Fu

School of Electrical Engineering and Computer ScienceOregon State University

E-mail: [email protected]

January 10, 2018

Set Theory

• Set theory is the mathematical basis of proabability.

• A set contains elements, e.g.,

A = {1, 2, 3}, B = {h, t}.

• In an experiment, S is a set of elementary outcomes.

• A subset of S, i.e., A ⊆ S, is an event, where A ⊆ S reads “A is a subset of orequal to S.”

• We use lower-case letters to denote the elements in sets, e.g., x ∈ A means “xbelongs to A”.

ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 1

Union

• Union of A and B is

A ∪B = {x ∈ S | x ∈ A or x ∈ B},

where A and B are events and S is the entire sample space.

• x ∈ A ∪B if and only if x ∈ A or x ∈ B.


Intersection

• Intersection of A and B is

A ∩B = {x ∈ S | x ∈ A and x ∈ B},

where A and B are events and S is the entire sample space.

• x ∈ A ∩B if and only if x ∈ A and x ∈ B.


Complement

• Complement of A is denoted as

Ac = {x ∈ S | x /∈ A}.

• note that (Ac)c = A.


Empty Set

• Empty set, denoted as ∅, is defined through its properties

1. A ∪ ∅ = A.2. A ∩ ∅ = ∅.3. ∅c = S.


Set Difference

• Set difference of A and B is

A−B = {x ∈ S | x ∈ A and x /∈ B }= {x ∈ S | x ∈ A and x ∈ Bc }

• Note that A−B 6= B −A

• A symmetrical set difference notion of A and B is

A\B = {x ∈ S | (x ∈ A or x ∈ B) and (x /∈ A ∩B)}

• Note that A\B = B\A


De Morgan’s Law

• Theorem: De Morgan’s Law

(A ∪B)c = Ac ∩Bc.

• a Theorem is something that we need to prove from Axioms and Definations.

• Proof: the basic idea is to show (A ∪ B)c ⊆ Ac ∩ Bc and Ac ∩ Bc ⊆ (A ∪ B)c

hold simultaneously.

– Assume x ∈ (A ∪ B)c. This means that x /∈ A ∪ B, which means x /∈ Aand x /∈ B. Together, this means that x ∈ Ac and x ∈ Bc, which leads tox ∈ Ac ∩Bc.

– Assume x ∈ Ac ∩ Bc. Then we know that x ∈ Ac and x ∈ Bc. This isequivalent to x /∈ A and x /∈ B. By definition it means that x /∈ A ∪B, whichmeans that x ∈ (A ∪B)c.


Disjoint Sets

• Sample Space, S, is the set of all elementary outcomes.

• Event, E, is a subset of S.

• Definition: Two sets are disjoint if and only if the intersection is empty.

• Definition: Sets A1, . . . , An are mutually disjoint if and only if Ai∩Aj = ∅, forall i, j.

• Definition: If A1, . . . , AN are collectively exhaustive if their union is S, i.e.,A1 ∪ . . . ∪An = S or ∪ni=1Ai = S.

• The above definition of collectively exhaustive can be generalized to countablyinfinite number of Ai’s, i.e., using

∪∞i=1Ai = S.


Partition of Sample Space

• If {Ai}∞i=1 are both mutually disjoint and collectively exhaustive, then we callthem a partition of S.

• A puzzle pattern, or a mosaic of the sample space S.


Remarks

• The book calls a partition an event space.

• We will not follow this terminology for several reasons.

• We will stick with “partition”.

• Why do we care about partition?

• Basic idea is that using partition we can express any event as a union of disjointevents, which may make the calculation of probability much easier.


Axioms of Probability Theory

• “mathematics cannot tell you what it is, but what would be if ”

• Probability Theory is based on several Axioms.

• Axiom 1: Probability of any event A ⊆ S is greater than or equal to 0, i.e.,

P [A] ≥ 0, ∀A ⊆ S.

• Axiom 2: P [S] = 1. (“something will happen”).

• Axiom 3: For any countable collection of mutually disjoint events A1, A2, . . ., wehave

P [A1 ∪A2 ∪ . . .] = P [A1] + P [A2] + . . .

• Theorems, propositions, lemmas, and corollaries need proof.


Basic Theorems of Probability Theory

• Theorem: P [∅] = 0.

• Proof: Since P [S ∪ ∅] = P [S]. Using Axiom 3, we know that

P [S] + P [∅] = P [S] =⇒ P [∅] = 0.

• Theorem: P [Ac] = 1− P [A].

• Proof: Note that A and Ac are disjoint and that A ∪Ac = S. Hence, we have

1 = P [S] = P [A ∪Ac]

= P [A] + P [Ac]



• Theorem: A ⊆ B =⇒ P [A] ≤ P [B].

• Proof: First note thatB = (B −A) ∪A.

Then, we have, by Axiom 3, that

P [B] = P [B −A] + P [A].

Also note that by Axiom 1 we have P [B −A] ≥ 0. Therefore, it is obvious that

P [B] ≥ P [A].



• Theorem: For every partition {B1, B2, B3, . . .} of S, it holds that

P [A] =∑i

P [A ∩Bi], ∀A ⊆ S.

• This is called Law of Total Probability—extremely useful in engineering.


Conditional Probability

• Example: somebody is rolling a fair die behind a curtain.

• Sample SpaceS = {1, 2, 3, 4, 5, 6}

• P [{1}] = P [{2}] = . . . = P [{6}] = 1/6 is now the probability model.

• Q: what is the probability of P [{2, 3}] (or, equivalently, P [{2} ∪ {3}])?


Conditional Probability

• In practice, we sometimes can get “side information” from somewhere, whichcould change our probability model.

• Suppose that there is a person who tells you that this time you have an evennumber.

– Then, what is the probability of getting 3?

P [{3}|Even] = P [{3}|{2, 4, 6}] = 0.

– What is the probability of getting 2?

P [{2}|Even] = P [{2}|{2, 4, 6}] = 1/3.


More on Conditional Probability

• Consider two events A and B.

• Assume that we have the side information that B has already happened, we wouldlike to know what is the probability of A conditioned on that B has happened,i.e., P [A|B].



• Consider two events A and B.

• Assume that we have the side information that B has already happened, we wouldlike to know what is the probability of A conditioned on that B has happened,i.e., P [A|B].

• Conjecture: P [A|B] = P [A ∩B]?



• P [A|B] = P [A ∩B] is not correct for a couple of reasons:

– P [A|B] should be larger than P [A ∩B] by intuition.– When A is a superset of B, P [A|B] = 1 but P [A ∩B] = P [B].

• Intuition: we should scale (normalize) P [A ∩B] to represent P [A|B].

• Definition: the probability of A conditioned on B is P [A|B] = P [A∩B]P [B] .


More on Conditional Probability• First, 0 < P [B] ≤ 1. Then,

P [A|B] =P [A ∩B]

P [B]≥ P [A ∩B];

i.e., the first bug is fixed.

• Second, when B ⊆ A, then

P [A|B] =P [A ∩B]

P [B]=

P [B]

P [B]= 1.

The second bug is fixed.

• Example: Let’s go back to the rolling dice example.

P [{4}|Even] = P [{4 ∩ Even}]P [Even]

=1/6

1/2= 1/3.


Remarks on Conditional Probability

• Remark 1: The definition of conditional probability automatically assumes that

P [B] > 0.

• Remark 2: P [A|B] itself is a “respectable probability measure”, i.e., P [A|B]satisifes Axioms 1-3 of Probability Theory.

• Remark 3: If A ∩B = ∅, then we have

P [A|B] = 0.


ece353: probability and random processes lecture...

Documents