Elementary Probability Theory

Chapter 5 of the textbook

Pages 145-164

Introduction

Statistical Decision Theory – using the probability of possible outcomes to choose between several available options

Statistical Inference – using samples to infer the probabilities of the population

Definitions

Statistical Experiment– Measuring an elementary outcome that is not known in

advance

Elementary Outcome– Each possible outcome of a statistical experiment– If the experiment was to test gender in this classroom

the elementary outcomes would be male and female

Sample Space– The set of all possible elementary outcomes

Sample Space Examples

Definition: the set of all possible outcomes of an experiment.

Examples of sample spaces:– Outcomes of the roll of a die: {1, 2, 3, 4, 5, 6}– Outcomes of 2 coin flips: {HH, HT, TH, TT}– Outcomes of rolling 2 dice:

Definitions

Events – Subsets of the sample space– Each event contains 1 or more elementary outcomes

Event Space– All the elementary outcomes that constitute an event

Complimentary Event– All elementary outcomes not in the event space

Event

An outcome or a set of outcomesExamples of events:– Roll of one die: {2}– Roll of one die: {2, 5}– Roll of two dice: {2 and 4}, {4 and 3}– Roll of two dice: {1 and 2, 5 and 6}– Flip coin once: {H}– Flip coin twice: {HT}

ExampleAssume I sampled a people on the bus and asked their ages and got the following results

19, 20, 20, 23, 27, 31, 37, 42, 56, 58

How many elementary outcomes do I have?

If I break the sample space into events by decade (e.g., 20s) what are my events?

What is the event space of each event?

What is the complimentary event of the 50s event?

SymbolsP() – The probability of something (usually an outcome or an event)

Ei – An elementary outcome, note the “i” which ranges from 1 to n

(S) – The sample space (you may also see (Ω))

A, B, … – Events are typically assigned to capital letters

– Complimentary events are the event letter with a bar

Ø – Null (i.e., no solution)

A

Relationships Between Events

Remember – each experiment has 1 and only 1 elementary outcome, but an outcome can be in 1 or more events

Intersection: the event space that is shared (i.e., the outcome is in both (or all) event spaces)– Example: overlapping portion of the Venn

Diagram

Union: combination of event spaces, (i.e., the outcome is in at least 1 event)

Venn Diagrams

UNIONA B

INTERSECTIONA B

Probability Postulates

P(Ei) probability of an outcome is between 0 and 1 (0 = impossible, 1 = certain)

P(A) = = sum of probabilities of all elementary outcomes in the event space

P(S) = 1 = certain

P(Ø) = 0 = impossible

)( Ai iEP

Rules Derived From Postulates

The sum of all elementary outcomes is 1 (certain)

The probability of an event is between 1 and zero**

If A and B are mutually exclusive P(A ∩ B) = Ø

n

i

EP1

1 1)(

1)(0 AP

** Note the book incorrectly uses “≤” in this rule

Types of Probability

Subjective: an event probability with accuracy/validity based on the experience of and information available to an observer

Objective: an event probability determined by the frequency of elementary outcomes observed during statistical experimentation

Calculating Probability

When all outcomes are equally likely, the probability of an event A:

m = the number of elementary outcomes in the event space

n = the number of elementary outcomes in the sample space

In other words….

P(A) = Total number of ways to achieve the event Total number all possible outcomes

nmAP /)(

Calculating Probability Example

Experiment: coin toss – P(heads) = 1 / 2 = 0.5

– The number of elements in the event space (m) = 1 (i.e., heads)– The number of elements in the sample space (n) = 2 (i.e., heads or

tails)

Experiment: roll a die.– P(rolling a 6) = 1 / 6 = 0.166667

– The number of elements in the event space (m) = 1 (i.e., a 6)– The number of elements in the sample space (n) = 6 (i.e.,

1,2,3,4,5,or 6)

Complicating Factors

What do we do when all outcomes are not equally likely?

Answer: “the subset of the sample space that comprises the event space must be specified… the [sum] of the elementary outcome probabilities in the event space will yield the event probability”

Conceptually this just means that we back up a step and calculate the probability of the outcomes and add them up for each event (think frequency tables)

A Familiar Example

Assume I sampled a people on the bus and asked their ages and got the following results:

19, 20, 20, 23, 27, 31, 37, 42, 56, 58

What is the probability of getting a result of 31?– P(answer of 31) = 1 / 10 = 0.1

What is the probability of getting a result in the 30s?– P(30s) = P(answer of 31) + P(answer of 37) = 0.1 + 0.1 = 0.2

What is the probability of getting a result in the 20s?– P(20s) = P(answer of 20) + P(answer of 23) + P(answer of 27) = 0.2 + 0.1

+0.1 = 0.4

Counting Rules

These are some useful rules for determining the elementary outcome counts (which are used to determine probabilities)

These are useful for many applications beyond just calculating probability

Symbol Confusion– The book uses “r” in two different ways– For the product rule each “r” is a group and each group has n

elements (i.e., r1 has n1 elements, r2 has n2 elements…)– For the combinations and permutations rules each “r” is a subset of

a larger group and “r” indicates the size (i.e. the number of elements) in the group being formed

Product Rule

Used to calculate all possible combinations available when selecting one member from each available group

Number of possible combinations = n1 * n2 ….

Example: Flipping a coin 3 times– Each flip is a group and each flip has 2 possible outcomes (n1 = n2

= n3 =2)– The number of possible outcomes is 2*2*2 = 8– Outcomes = {HHH}, {HHT}, {HTH}, {HTT}, {THH}, {THT},

{TTH}, {TTT}

Combinations Rule

Used to select all the possible groups of size r from the sample space

Since the sample space has n outcomes, r ≤ n

Example: – 4 cards - A, K, Q, J– How many combinations can you

have if you pick 2 cards (r=2)– AK, AQ, AJ, KQ, KJ, QJ

)!(!

!

rnr

nC n

r

64

24

)!24(!2

!442

C

Permutations RuleUsed to select all the possible groups of size r from the sample space including the order of the elements

Since the sample space has n outcomes, r ≤ n

Example: – 4 cards - A, K, Q, J– How many combinations can you

have if you pick 2 cards (r=2)– AK, AQ, AJ, KQ, KJ, QJ– KA, QA, JA, QK, JK, JQ

)!(

!

rn

nPn

r

122

24

)!24(

!442

P

Hypergeometric RuleThe combination of the product rule and the combination rule

Since the sample space has n outcomes, r ≤ n

Example: – 2 sets of 4 cards (A, K, Q, J) and

(1, 2, 3, 4)– How many combinations can you

have if you pick 2 cards (r=2) from each set?

– Answer = 36

11

11 * n

rnr CC

Probability Theorems

Addition Theorem

Rule of thumb: Union uses addition

)()()()( BAPBPAPBAP

UNIONA B

Examples

Coin Flip: – P(heads) = 0.5– P(tails) = 0.5 – P(heads ∩ tails) = 0

Cards– P(heart) = 13/52– P(king) = 4/52– P(heart ∩ king) = 1/52

)()()()( BAPBPAPBAP

Probability Theorems

Complementation Theorem

Recall that P(S) = 1

)(1)( APAP

Probability Theorems

Conditional Probability

Think of this as the probability of X given Y where both X and Y have their own probability

Intuition should tell you that this will hinge on the intersection of X and Y

)(

)()|(

BP

BAPBAP

Probability Theorems

Statistically Independent Events – the probability of an event remains the same despite the occurrence of another event

Example: The probability of a coin flip being heads is ½ regardless of what the last coin flip was

Based on conditional probability

)()|( APBAP

Probability Theorems

Multiplication Theorem

Rule of thumb: Intersections use multiplication

)(*)|()( BPBAPBAP

INTERSECTIONA B

Statistically Independent Examples

2 Coin Flips – A and B are the probability of getting heads– P(heads) = 1/2– P(heads ∩ heads) = P(A|B) * P(B) = ¼– P(heads | heads on first flip) = P(heads ∩ heads) / P(B) = (¼) / (½) = 1/2

Draw 2 cards– P(heart) = 13/52– P(king) = 4/52– P(heart ∩ king) = 1/52– P(heart | king) = P(heart ∩ king) / P(king) = (1/52) / (4/52) = 13/52 = 1/4– P(king | heart) = P(heart ∩ king) / P(heart) = (1/52) / (13/52) = 4/52 = 1/13

)(

)()|(

BP

BAPBAP

Statistically Dependent ExampleProbability of drawing 2 hearts– Drawing single cards from a complete deck would equate to:

• P(A) = P(heart) = 13/52• P(B) = P(heart) = 13/52• P(A|B) = P(heart|heart on last draw) = 12/51

– Solution 1: imagine drawing 1 card and then the second: – (A ∩ B) = P(A|B) * P(B) = 0.589

– Solution 2: imagine drawing both cards at once• Remember P(event) = m/n• n = number of all combinations (full sample space)• m = number of possible combinations of 2 hearts (event space)• Both m and n are calculated using the combinations rule

• m = n =

• P(drawing 2 hearts) = 78/1326 = 0.589

78)!213(!2

!13132

C 1326

)!252(!2

!52522

C