ch1. review of basic probability and statistics terminology

IntroductionSome Basic Statistical Concepts

Some Useful Discrete DistributionsSome Useful Continuous Distributions

Ch1. Review of Basic Probability andStatistics Terminology

Zhang Jin-Ting

Department of Statistics and Applied Probability

July 16, 2012

Zhang J.T. Ch1. Review of Basic Probability and Statistics Terminology

Outline

Introduction

Some Basic Statistical Concepts

Some Useful Discrete Distributions

Some Useful Continuous Distributions

AimsI This chapter is a very brief review of basic properties and

terminology from probability and statistics that we will usein this semester.

I The student is assumed to have seen most of this chapterbefore.

I This material is treated in most introductory texts onprobability and statistics.

Cumulative Distribution FunctionI Given a real-valued random variable X , the Cumulative

Distribution Function (c.d.f.) of X is the functionF : R → [0, 1] defined by

F (x) = Pr(X ≤ x), x ∈ R.

I If there exists a function f : R → [0,∞) such thatF (x) =

∫ x−∞ f (y)dy for every x ∈ R, then x is said to be

continuous with Probability Density Function (p.d.f.) f .

Cumulative Distribution FunctionI Given a real-valued random variable X , the Cumulative

Distribution Function (c.d.f.) of X is the functionF : R → [0, 1] defined by

F (x) = Pr(X ≤ x), x ∈ R.

I If there exists a function f : R → [0,∞) such thatF (x) =

∫ x−∞ f (y)dy for every x ∈ R, then x is said to be

continuous with Probability Density Function (p.d.f.) f .

Probability Mass FunctionI On the other hand, if X only takes values in the set of

integers, or more generally in some countable (or finite) setS, then its c.d.f. is completely determined by its ProbabilityMass Function (p.m.f.), p : S → [0, 1] where

pi = Pr(X = i), i ∈ S.

I Clearly,∫∞−∞ f (x)dx = 1 for any p.d.f. and

∑i∈S pi = 1 for

any p.m.f.

Probability Mass FunctionI On the other hand, if X only takes values in the set of

integers, or more generally in some countable (or finite) setS, then its c.d.f. is completely determined by its ProbabilityMass Function (p.m.f.), p : S → [0, 1] where

pi = Pr(X = i), i ∈ S.

I Clearly,∫∞−∞ f (x)dx = 1 for any p.d.f. and

∑i∈S pi = 1 for

any p.m.f.

Mean and ExpectationI The Mean or Expectation of a real-valued random variable

X is defined by

E(X ) =

{∫∞−∞ xf (x)dx if X has p.d.f. f∑

i∈S ipi if X has p.m.f. p.

I The Variance of X , denoted by var(X ), is defined to beE [(X − E(X ))2] and equals E(X 2)− [E(X )]2 when it isfinite. If we view X as the value of some measurement,then the standard deviation

√var(X ) determines the

magnitude of error in this measurement.

Mean and ExpectationI The Mean or Expectation of a real-valued random variable

X is defined by

E(X ) =

{∫∞−∞ xf (x)dx if X has p.d.f. f∑

i∈S ipi if X has p.m.f. p.

I The Variance of X , denoted by var(X ), is defined to beE [(X − E(X ))2] and equals E(X 2)− [E(X )]2 when it isfinite. If we view X as the value of some measurement,then the standard deviation

√var(X ) determines the

magnitude of error in this measurement.

Sample MeanI Let X1, X2, · · · be a sequence of Independent Identically

Distributed (iid) random variables with mean µ andvariance σ2. We define the n-th Sample Mean by

X̄n =1n

I Then E(X̄n) = µ and var(X̄n) = σ2

Sample MeanI Let X1, X2, · · · be a sequence of Independent Identically

Distributed (iid) random variables with mean µ andvariance σ2. We define the n-th Sample Mean by

X̄n =1n

I Then E(X̄n) = µ and var(X̄n) = σ2

CLT and CII When n is sufficiently large, the Central Limit Theorem

implies X̄n−µσ/√

n has an approximate N(0, 1) distribution.Therefore,

Pr(X̄n − 1.96σ√n≤ µ ≤ X̄n + 1.96

σ√n

) ≈ 0.95

for sufficiently large n.

Sample VarianceI When σ2 is unknown, its value can be estimated by the

Sample Variance

1n − 1

(Xi − X̄n)2.

I The sample variance is an Unbiased Estimator of σ2

whenever the Xi ’s are iid.

Sample VarianceI When σ2 is unknown, its value can be estimated by the

Sample Variance

1n − 1

(Xi − X̄n)2.

I The sample variance is an Unbiased Estimator of σ2

whenever the Xi ’s are iid.

CovarianceI Consider two random variables X and Y (have some Joint

Distribution). The Covariance of X and Y is defined by

Cov(x , y) = E [(X−E(X ))(Y−E(Y ))] = E(XY )−E(X )E(Y ).

I If X and Y are Independent then Cov(X , Y ) = 0 (but theconverse of this statement is false in general).

CovarianceI Consider two random variables X and Y (have some Joint

Distribution). The Covariance of X and Y is defined by

Cov(x , y) = E [(X−E(X ))(Y−E(Y ))] = E(XY )−E(X )E(Y ).

I If X and Y are Independent then Cov(X , Y ) = 0 (but theconverse of this statement is false in general).

CorrelationI It is easy to check that

var(X + Y ) = var(X ) + var(Y ) + 2Cov(X , Y )

I The Correlation of X and Y is defined by the followingnormalization of covariance:

Cor(X , Y ) =Cov(X , Y )√

var(X )√

var(Y ).

CorrelationI It is easy to check that

var(X + Y ) = var(X ) + var(Y ) + 2Cov(X , Y )

I The Correlation of X and Y is defined by the followingnormalization of covariance:

Cor(X , Y ) =Cov(X , Y )√

var(X )√

var(Y ).

1. BernoulliI Let p be a number in the interval [0,1].I The Bernoulli distribution with parameter p is the discrete

distribution on {0, 1} whose pmf is

p0 = 1− p, p1 = p.

I We write X ∼ Bernoulli(p) to say that the random variableX has this distribution.

p0 = 1− p, p1 = p.

2. BinomialI The binomial distribution with parameters n and p ∈ [0, 1]

is the discrete probability distribution whose pmf is given by

)pi(1− p)n−i , i = 0, 1, . . . , n.

I We write X ∼ B(n, p) to say that the random variable Xhas this distribution.

I If X1, . . . , Xn are iid with common distribution Bernoulli(p),then X1 + · · ·+ Xn ∼ B(n, p). The B(n, p) has mean np andvariance np(1− p).

)pi(1− p)n−i , i = 0, 1, . . . , n.

3. GeometricI The geometric distribution with parameter p ∈ [0, 1] is the

discrete probability distribution whose pmf is given by

pi = (1− p)i−1p , i = 1, 2, 3, . . .

I This describe the distribution of the number of tosses of a(possibly unfair) coin needed until the first “heads” appears.

I We write X ∼ Geometric(p) to say that the random variableX has this distribution.

I The distribution has mean 1/p and variance (1− p)/p2.

pi = (1− p)i−1p , i = 1, 2, 3, . . .

4. PoissonI The Poisson distribution with parameter λ > 0 is the

pi =e−λλi

i!, i = 0, 1, 2, . . .

I We write X ∼ Poisson(λ) to say that the random variable Xhas this distribution.

I The mean and variance of this distribution are both equalto λ.

pi =e−λλi

i!, i = 0, 1, 2, . . .

pi =e−λλi

i!, i = 0, 1, 2, . . .

LemmaSuppose that X is a continuous random variable having pdff (x).

(i) If a is a real number, then the pdf of X + a is f (x − a).(ii) If b is a positive number, then the pdf of bX is b−1f (x/b).

1. UniformI Let a and b be real numbers with a < b.I The uniform distribution on the interval [a, b] is the

continuous distribution whose pdf is given by

f (x) =

b−a if a ≤ x ≤ b0 otherwise.

I We write X ∼ U[a, b] to say that the random variable X hasthis distribution.

f (x) =

2. NormalI Let µ be a real number and σ be a positive number.I The normal distribution with mean µ and variance σ2 is the

continuous distribution whose pdf f is given by

f (x) =1√2πσ

exp{−(x − µ)2/2σ2}

I We write X ∼ N(µ, σ2) to say that the random variable Xhas this distribution.

I If Z ∼ N(0, 1), then µ + σZ ∼ N(µ, σ2).

f (x) =1√2πσ

exp{−(x − µ)2/2σ2}

f (x) =1√2πσ

exp{−(x − µ)2/2σ2}

f (x) =1√2πσ

exp{−(x − µ)2/2σ2}

3. ExponentialI The exponential distribution with parameter λ > 0 is the

f (x) =

{λe−λx if x ≥ 00 if x < 0

I We write X ∼ Exp(λ) to say that the random variable Xhas this distribution.

I The distribution has mean 1/λ and variance 1/λ2.

f (x) =

4. GammaI The gamma distribution with parameters a > 0 and b > 0

is the continuous distribution whose pdf f is given by

f (x) =

{baxa−1e−bx/Γ(a) if x ≥ 00 if x < 0

where Γ is the Gamma function

Γ(a) =

∫ ∞

0ta−1e−tdt .

I The Gamma function satisfies the recursionΓ(a) = (a− 1)Γ(a− 1).

I We write X ∼ Gamma(a, b) to say that the random variableX has this distribution.

I This distribution has mean a/b and variance a/b2.Zhang J.T. Ch1. Review of Basic Probability and Statistics Terminology

f (x) =

Γ(a) =

∫ ∞

0ta−1e−tdt .

f (x) =

Γ(a) =

∫ ∞

0ta−1e−tdt .

f (x) =

Γ(a) =

∫ ∞

0ta−1e−tdt .

4. Gamma

Proposition

I Let X1, . . . , Xk be independent random variables, andassume Xi ∼ Gamma(ai , b) for each i. Then

X1 + · · ·+ Xk ∼ Gamma(a1 + · · ·+ ak , b).

I Assume Z ∼ N(0, 1), then Z 2 ∼ Gamma(12 , 1

4. Gamma

Proposition

X1 + · · ·+ Xk ∼ Gamma(a1 + · · ·+ ak , b).

4. Gamma

Proposition

X1 + · · ·+ Xk ∼ Gamma(a1 + · · ·+ ak , b).

4. GammaI As a consequence of the above proposition, assume that

Z1, . . . , Zk are iid N(0, 1) random variables. ThenZ 2

1 + · · ·+ Z 2k has the Gamma( k

2 , 12 ) distribution.

I This arises often in statistics, and is called the chi-squareddistribution with k degrees of freedom, or χ2

k for short.

4. GammaI As a consequence of the above proposition, assume that

Z1, . . . , Zk are iid N(0, 1) random variables. ThenZ 2

1 + · · ·+ Z 2k has the Gamma( k

2 , 12 ) distribution.

I This arises often in statistics, and is called the chi-squareddistribution with k degrees of freedom, or χ2

k for short.

ch1. review of basic probability and statistics terminology

Documents