Discrete probabilitySample spacesProbability distributionsEventsBasic facts about probabilities of events

TheoremConditional probabilityIndependence of two eventsPairwise and mutual independenceBiased coins and Bernoulli trialsThe binomial distribution

TheoremProofDefinition

Pairwise and mutual independenceBayes' theoremRandom variablesBiased coins and the geometric distributionExpectation of a random variable

Better expression for the expectationTheoremProof

Expected number of successes in Bernoulli trialsTheorem

Linearity of expectationTheorem

VarianceDefinition

Discrete probability

Sample spaces For any probabilistic experiment or process, the set of all its possible outcomes is called its samplespace.

E.g. Consider the following probabilistic (random) experiment:

Flip a fair coin 7 times in a row, and see what happens

The possible outcomes of the example above, are all the sequences of Heads ( ) and Tails ( ), oflength 7. In other words, they are the set of strings .

The cardinality of this sample space is , because there are two choices for each of the sevencharacters of the string.

Sample spaces may be infinite.

E.g. Flip a fair coin repeatedly until a Heads is flipped.

However, in discrete probability, we focus on finite and countable sample spaces.

Probability distributions A probability distribution over a finite or countable set , is a function:

such that .

In other words, each outcome is assigned a probability such that and thesum of the probabilities of all outcomes sum to 1.

E.g. Suppose a fair coin is tossed times consecutively. This random experiment defines aprobability distribution on , where for all , .

Since , we have

E.g. suppose a fair coin is tossed repeatedly until it lands heads. This random experiment definesa probability distribution , on , such that, ,

Note that

Events For a countable sample space , and event E, is simply a subset of the set of possibleoutcomes.

Given a probability distribution , we define the probability of the event to be .

E.g. For , the following are events:

The third coin toss came up heads

The fourth and fifth coin tosses did not both come up tails

E.g. For , the following are events:

The first time the coin comes up heads is after an even number of coin tosses

Basic facts about probabilities of events For event , define the complement event to be .

Theorem Suppose are a (finite or countable) sequence of pairwise disjoint events from thesample space . In other words, and . Then

Furthermore, for each event , .

Proof: Follows easily from definitions:

, , thus, since the sets are disjoint,

.

Likewise, since ,

Conditional probability Definition: Let be a probability distribution, and let be two events, such that

.

The conditional probability of given , denoted , is defined by:

E.g. A fair coin is flipped three times. Suppose we know that the event occurs.

What is the probability that the event occurs?

For the example above, there are 8 flip sequences, , all with probability . The event that is . The event . So,

Independence of two events Intuitively, two events are independent if knowing whether one occurred does not alter the probabilityof the other. Formally:

Definition: Events and are called independent if .

Note that if then and are independent if and only if

Thus, the probability of is not altered by knowing occurs.

E.g. A fair coin is flipped three times. Are the events and , independent?

Yes, because , , and , so .

Pairwise and mutual independence Definition: Events are called pairwise independent if for every pair

and are independent. (i.e. )

Events are called mutually independent, if for every subset .

Clearly, mutual independence implies pairwise independence. However, pairwise independencedoes not imply mutual independence.

Biased coins and Bernoulli trials A Bernoulli trial is a probabilistic experiment that has two outcomes: success or failure (e.g. headsor tails).

We suppose that is the probability of success, and is the probability of failure.

We can of course have repeated Bernoulli trials. We typically assume the different trials are mutuallyindependent.

E.g. A biased coin, which comes up heads with probability , is flipped timesconsecutively. What is the probability that it comes up heads exactly times?

The binomial distribution

Theorem The probability of exactly successes in mutually independent Bernoulli trials, with probability ofsuccess and of failure in each trial, is:

Proof We can associate Bernoulli trials with outcomes . Each sequence )with exactly heads and tails occurs with probability . There are such sequenceswith exactly heads.

Definition The binomial distribution with parameters and , denoted , defines a probabilitydistribution on , given by:

Pairwise and mutual independence A finite set of events is pairwise independent if every pair of events is independent - that is,if and only if for all distinct pairs of indices :

A finite set of events is mutually independent if every event is independent of any intersection of theother events - that is, if and only if for every -element subset of :

Bayes' theorem Let and be events such that and .

Random variables Definition: A random variable, is a function , that assigns a real value to each outcome ina sample space .

E.g. Suppose a biased coin is flipped times. The sample space is . The function that assigns to each outcome the number of coin tosses that came

up heads, is one random variable.

For a random variable we write as shorthand for the probability ). The distribution of a random variable is given by the set of pairs

.

Note: These definitions of a random variable and its distribution are only adequate in the context ofdiscrete probability distributions.

Biased coins and the geometric distribution Suppose a biased coin comes up heads with probability each time it is tossed. Supposewe repeatedly flip this coin until it comes up heads.

What is the probability that we flip the coin times, for ?

Answer: The sample space is . Assuming mutual independence of coinflips, the probability of is . Note that this does define a probability distribution on

because:

A random variable , is said to have a geometric distribution with parameter , if for all positive integers .

Expectation of a random variable Recall: A random variable is a function , that assigns a real value to each outcome in asample space .

The expected value or expectation or mean, of a random variable , denoted by , isdefined by:

Here is the underlying probability distribution on .

Question: Let be the random variable outputting the number that comes up when a fair die isrolled. What is the expected value, , of ?

However, sometimes this way can be inefficient for calculating . Take the sample space of 11coin flips, for example: . Our summation will have terms!

Better expression for the expectation Recall denotes the probability .

Recall that for a function ,

Theorem

For a random variable ,

Proof

, but for each , if we sum all terms such that , we get as their sum. So, summing over all we get

.

So, if is small, and if we can compute , then we need to sum a lot fewer termsto calculate .

Expected number of successes in Bernoulli trials

Theorem The expected number of successes in (independent) Bernoulli trials, with probability of success ineach, is .

Linearity of expectation

Theorem For any random variables on ,

Furthermore, for any ,

In other words, the expectation function is a linear function.

Variance The variance and standard deviation of a random variable give us ways to measure (roughly) onaverage, how far off the value of the random variable is from its expectation.

Definition For a random variable on a sample space , the variance of , denoted by , is defined by:

The standard deviation of , denoted by , is defined by

E.g. Consider the random variable , s.t. , and the random variable , s.t. .

Then , but , whereas and .

Theorem

For any random variable ,

Proof