math5315 applied statistics and probability 2011-2012sta6ajb/math5315/5315lec.pdf · math5315...

MATH5315 Applied Statistics and Probability 2011-2012

Lecturer: Andrew J. Baczkowski – room 9.21i – email: [email protected] updated information about the module is available on the internet at:http://www.maths.leeds.ac.uk/∼sta6ajb/math5315/math5315.htmlModule Objective: The aim of the module is to provide a grounding in the aspects of statistics,in particular statistical modelling, that are of relevance to actuarial and financial work. Themodule introduces and develops the fundamental concepts of probability and statistics used inapplied financial analysis.

Provisional Detailed Syllabus:Part I: Fundamentals of Probability (11 lectures)Summarising data; Introduction to probability; Random variables; Probability distributions; Gen-erating functions; Joint distributions; The central limit theorem; Conditional expectation.Part II: Fundamentals of Statistics (9 lectures)Sampling and statistical inference; Point estimation; Confidence intervals; Hypothesis testing.Part III: Applied Statistics (10 lectures)Correlation and regression (OLS); Analysis of variance (ANOVA); Univariate time series analysisand forecasting (ARMA); Multivariate time series analysis (VAR); Cointegration; Volatility models(ARCH/GARCH).

Booklist: Sections from the two books will form the course notes for this module.1. “Subject CT3 Probability and Mathematical Statistics Core Technical, Core Reading”, publishedby Institute of Actuaries, price ≈ £45. Referred to as CT3.

2. “Introductory Econometrics for Finance (2nd edition)” by C. Brookes, published by CambridgeUniversity Press, 2008, price ≈ £40. Referred to as IEF.

IT is ESSENTIAL that you have access to these books. You MUST prepare the material BEFOREthe lectures, which will consist of examples and further explanation to illustrate the book material.

Timetable:Lectures (weeks 1-5): Tuesdays 10-11 in RSLT14 and Tuesdays 12-1 in RSLT08.Lectures (weeks 1-4, 6-11): Fridays 1-3 in RSLT08.Seminar (weeks 1-4, 6): Fridays 3-4 in E.C.Stoner Building, room 9.90.Practical (weeks 7-11): Fridays 3-4 in Irene Manton North cluster.(RSLT is the Roger Stevens Lecture Theatre Block).

Assessment:70% of marks for two hour examination at end of semester.30% of marks for continuous assessment practical work.

Examination Paper: Format currently planned as follows for the TWO hour paper. Elevensection A questions each worth TWO marks. Eleven section B questions each worth THREEmarks. Eleven section C questions each worth FIVE marks. You attempt TEN section A questions,TEN section B questions, and TEN section C questions.

Exercise Sheets for MATH5315: None. I will introduce examples as we need them. There aresome questions (and answers) available on the module web-page.

1

MATH5315 Applied Statistics and Probability

Lecture 1: Summarising Data

References: CT3 Unit 1.(CT3 denotes “Subject CT3 Probability and Mathematical Statistics Core Technical, Core Read-ing”, published by Institute of Actuaries, price ≈ £45.)

§2 Tabular and graphical methods.

§2.1 Types of data. Discrete and continuous data.

§2.2 Frequency distribution. A line chart is better for discrete data than a bar chart!

§2.3 Histograms.

§2.5 Lineplots. Dotplots. Cumulative frequency is frequency ≤ x.

§3 Measures of Location.

§3.1 The mean. Sample mean x.

§3.2 The median.

§4 Measures of spread.

§4.1 The standard deviation. Sample standard deviation s, sample variance s2.

§4.2 Moments. Sample moments m′k =

1

n

n∑

i=1

xki , mk =

1

n

n∑

i=1

(xi − x)k.

§4.3 The range.

§4.4 The interquartile range. More often people use the semi-interquartile range SIQ =12(Q3 − Q1).

§5 Symmetry and skewness.

§5.1 Boxplots.

2


Lecture 2: Introduction to Probability

References: CT3 Unit 2.

§1 Introduction to sets. Sample space S, event A.

§1.1 Complementary sets. A′. More usually Ac might be used.

§1.2 Set operations. Union ∪ and intersection ∩.

§2 Probability axioms and the addition rule.

§2.1 Basic probability axioms. PS = 1, 0 ≤ PA ≤ 1. If A and B are mutually exclusive,PA ∪ B = PA + PB.

§2.2 The addition rule. In general PA ∪ B = PA + PB − PA ∩ B.

§3 Conditional probability. PA|B =PA ∩ B

PB .

§3.1 Independent events. A and B are independent if and only if PA ∩ B = PAPB.

§3.2 Theorem of total probability. PA =

n∑

j=1

PA ∩ Ej.

§3.3 Bayes’ Theorem. PEi|A =PA|EiPEi

n∑

j=1

PA|EjPEj.

3


Lecture 3: Random Variables


§1 Discrete random variables. Random variable X. Probability function fX(x) = PX = x;notation pX(x) would be better!

Cumulative distribution function (cdf) FX(x) = PX ≤ x =∑

xi≤x

fX(xi).

§2 Continuous random variables. Probability density function (pdf) fX(x).

Pa ≤ X ≤ b =

∫ b

a

fX(x)dx. Cdf FX(x) =

∫ x

−∞fX(t)dt, so fX(x) =

dFX(x)

dx.

§3 Expected values.

§3.1 Mean. E[X] or µ. More generally E[g(X)].

§3.2 Variance and standard deviation. Variance V[X] often denoted σ2. (I prefer Var[X] asnotation!) Standard deviation σ.

§3.3 Linear functions of X. E[aX + b] = aµ + b. V[aX + b] = a2σ2.

§3.4 Moments. µk = E[(X − µ)k] is kth central moment of X about µ. Can measure skewnessusing µ3/σ

3.

§4 Functions of a random variable. Y = u(X).

§4.1 Discrete random variables. If have a 1–1 mapping, PY = y1 = PX = x1 wherey1 = u(x1).

§4.2 Continuous random variables. Y = u(X) so X = w(Y ). fY (y) = fX(x)

∣

∣

∣

∣

dx

dy

∣

∣

∣

∣

.

4


Lecture 4: Probability Distributions I


§2 Discrete distributions.

§2.1 Uniform distribution. PX = x = 1k

for x = 1, 2, . . . , k.

§2.2 Bernoulli distribution. Bernoulli trial.

§2.3 Binomial distribution. If X is number of successes in n Bernoulli trials, X ∼ Bin(n, θ);PX = x =

(

nx

)

θx(1 − θ)n−x for x = 0, 1, 2, . . . , n.

§2.4 Geometric distribution. If X is number of Bernoulli trials until first success, X ∼geometric(θ); PX = x = θ(1 − θ)x−1 for x = 1, 2, 3, . . ..

§2.7 Poisson distribution. If X ∼ Poisson(λ), then PX = x =λxe−λ

x!for x = 0, 1, 2, . . . .

5


Lecture 5: Probability Distributions II


§3 Continuous distributions.

§3.1 Uniform distribution. If X ∼ uniform(α, β), fX(x) =1

β − αfor α < x < β.

§3.2 Gamma distribution. Gamma function Γ(α), Γ(1) = 1, Γ(α) = (α − 1)Γ(α − 1), Γ(n) =(n − 1)! for n ∈ Z.

If X ∼ gamma(α, λ), fX(x) =λαxα−1e−λx

Γ(α)for x > 0. E[X] =

α

λ, V[X] =

α

λ2.

Exponential distribution: X ∼ exponential(λ) ≡ gamma(1, λ).Chi-squared distribution: X ∼ χ2

ν ≡ gamma(12ν, λ = 1

2).

§3.3 Beta distribution. (NOT needed for the exam.) fX(x) =xα−1(1 − x)β−1

B(α, β)for 0 <

x < 1 where B(α, β) =Γ(α)Γ(β)

Γ(α + β).

Mean is µ =α

α + β. Variance is µ =

αβ

(α + β)2(α + β + 1).

§3.4 Normal distribution. If X ∼ N(µ, σ2), fX(x) =1√

2πσ2exp

(−(x − µ)2

2σ2

)

for −∞ < x <

∞.

If Z =X − µ

σ, then Z ∼ N(0, 1). Values PZ < z = Φ(z) are tabulated.

§3.6 t-distribution. If X ∼ χ2ν and Z ∼ N(0, 1) independently, then T =

Z√

X/ν∼ tν .

§3.7 F-distribution. If X ∼ χ2n1

and Y ∼ χ2n2

and are independent, then F =X/n1

Y/n2∼ Fn1,n2

.

6


Lecture 6: Generating Functions


§1 Probability generating functions. If X is discrete taking value k with probability pk

(k = 0, 1, 2, . . .), then GX(t) = E[tX ] =∞∑

k=0

pktk.

GX(1) = 1, GX(0) = p0.

§1.1 Important examples. Uniform. Binomial. Geometric (as special case of negative bino-mial). Poisson.

§1.2 Evaluating moments. G′X(t) =

∞∑

k=1

kpktk−1 so G′

X(1) =

∞∑

k=1

kpk = E[X].

G′′X(t) =

∞∑

k=2

k(k − 1)pktk−2 so G′′X(1) =

∞∑

k=2

k(k − 1)pk = E[X(X − 1)].

§2 Moment generating function. mX(t) = E[etX ].

In continuous case mX(t) =

∫

x

etxfX(x)dx.

m(r)(t) =

∫

x

xretxfX(x)dx so m(r)(0) =

∫

x

xrfX(x)dx = E[Xr].

§2.1 Important examples. gamma(α, λ). N(µ, σ2).

§4 Linear functions. If Y = a + bX, GY (t) = E[tY ] = E[ta+bX ] = taE[(tb)X ] = taGX(tb).Similarly, mY (t) = eatmX(bt).

7


Lecture 7: Joint Distributions I


§1 Joint distributions.

§1.1 Joint probability (density) functions.Discrete case f(x, y) = PX = x, Y = y, though a better notation might be pX,Y (x, y) as in §1.3!Continuous case f(x, y) is joint pdf. Px1 < X < x2, y1 < Y < y2 =

∫ y2

y1

∫ x2

x1

f(x, y)dxdy.

§1.2 Marginal probability (density) functions.

Discrete case fX(x) = PX = x =∑

y

f(x, y) though a better notation might be pX(x) = PX = x

as in §1.3!Continuous case, marginal pdf is fX(x) =

∫

y

f(x, y)dy.

§1.3 Conditional probability (density) functions. Recall PA|B =PA ∩ B

PB .

Discrete case PX = x|Y = y =pX,Y (x, y)

pY (y). Continuous case fX|Y =y(x|y) =

fX,Y (x, y)

fY (y).

§1.4 Independence of random variables.If X and Y are independent, then fX,Y (x, y) = fX(x)fY (y) for all x and y.If X and Y are independent, then g(X) and h(Y ) will be independent.

§2 Expectations of functions of two random variables.

§2.1 Expectations. Discrete case E[g(X,Y )] =∑

x

∑

y

g(x, y)PX = x, Y = y.

Continuous case E[g(X,Y )] =

∫

x

∫

y

g(x, y)fX,Y (x, y)dxdy.

§2.2 Expectations of sums and products. E[ag(X) + bh(Y )] = aE[g(X)] + bE[h(Y )].If X and Y are independent, E[g(X)h(Y )] = E[g(X)]E[h(Y )].

8


Lecture 8: Joint Distributions II


§2.3 Covariance and correlation coefficient.cov(X,Y ) = E[(X − µX)(Y − µY )] = E[XY ] − µXµY . cov(X,X) = V[X].

corr(X,Y ) =cov(X,Y )√

V[X]V[Y ], often denoted ρ, −1 ≤ ρ ≤ 1.

If ρ = 0, X and Y are uncorrelated.

§2.3.1 Useful results on handling covariances.cov(aX + b, cY + d) = ac cov(X,Y ). cov(X,Y + Z) = cov(X,Y ) + cov(X,Z).If X and Y are independent, cov(X,Y ) = 0. Converse not necessarily true.

§2.4 Variance of a sum. V[X + Y ] = V[X] + V[Y ] + 2cov(X,Y ).

§3 Convolutions. Suppose Z = X + Y .

Discrete case PZ = z =∑

x

PX = x, Y = z − x.

Continuous case fZ(Z = z) =

∫

x

fX,Y (x, y = z − x)dx.

Simplifies in independence case.

9


Lecture 9: More on generating functions

References: CT3 Units 5 and 6.

§3 (Unit 5) Cumulant generating function.CX(t) = log mX(t). C ′

X(0) = E[X]. C ′′X(0) = V[X].

CX(t) = κ1t + κ2t2

2!+ κ3

t3

3!+ · · ·.

§3.1 (Unit 6) Moments of linear combinations of random variables.

E

[

n∑

i=1

ciXi

]

=

n∑

i=1

ciE[Xi].

V

[

n∑

i=1

ciXi

]

=

n∑

i=1

c2i V[Xi] + 2

∑∑

1≤i<j≤n

cicj cov(Xi,Xj). Special case is (mutual) independence case.

§3.2 (Unit 6) Distributions of linear combinations of independent random variables.Discrete case via probability generating function (pgf). Let S = c1X + c2Y .GS(t) = E[tc1X+c2Y ] = E[(tc1)X ]E[(tc2)Y ] = GX(tc1)GY (tc2).Binomial case: If X ∼ Bin(m, θ) and Y ∼ Bin(n, θ), X + Y ∼ Bin(m + n, θ).Poisson case: If X ∼ Poisson(λ) and Y ∼ Poisson(γ), then X + Y ∼ Poisson(λ + γ).

Continuous case via moment generating functions (mgf). Let S = c1X + c2Y .mS(t) = E[e(c1X+c2Y )t] = E[e(c1t)X ]E[e(c2t)Y ] = mX(c1t)mY (c2t).

Exponential case: If Xiind∼ exponential(λ), i = 1, 2, . . . , k,

k∑

i=1

Xi ∼ gamma(k, λ).

Gamma case: If X ∼ gamma(α, λ) and Y ∼ gamma(δ, λ), then X + Y ∼ gamma(α + δ, λ).Chi-square case: If X ∼ χ2

m and Y ∼ χ2n, then X + Y ∼ χ2

m+n.Normal case: If X ∼ N(µX , σ2

X) and Y ∼ N(µY , σ2Y ), then X + Y ∼ N(µX + µY , σ2

X + σ2Y ).

10


Lecture 10: Central Limit Theorem


§1 The central limit theorem. For X1,X2, . . . ,Xn iid with common mean µ and common

variance σ2(< ∞),X − µ

σ/√

n≈ N(0, 1) for large n.

§2 Normal approximations.

§2.1 Binomial distribution. If X ∼ Bin(n, θ) and n is large, X ≈ N(nθ, nθ(1 − θ).

§2.2 Poisson distribution. If Xiind∼ Poisson(λ), from lecture 8,

n∑

i=1

Xi ∼ Poisson(nλ). For

large n, this is approximately N(nλ, nλ).

§2.3 Gamma distribution. From lecture 8, if Xiind∼ exponential(λ),

n∑

i=1

Xi ∼ gamma(n, λ).

For large n, this is approximately N(n/λ, n/λ2).Since χ2

k ≡ gamma(12k, 1

2), χ2k ≈ N(k, 2k) for large k.

§3 The continuity correction. If X ∼ Poisson(µ) and µ is large, X ≈ N(µ, σ2 = µ).

PX ≤ x ≈ P

Z ≤ x + 12 − µ

σ

, where Z ∼ N(0, 1).

11


Lecture 12: Poisson process and simulating random variables


§4 The Poisson process. Point-like events ocur randomly and independently in time at anaverage rate λ per unit time. Let N(t) be the number of events in interval [0, t] with N(0) = 0.Let pn(t) = PN(t) = n.pn(t + h) ≈ pn−1(t)[λh] + pn(t)[1 − λh] so pn(t + h) − pn(t) ≈ λh[pn−1(t) − pn(t)].As h → 0, p′n(t) = λ[pn−1(t) − pn(t)].Similarly p′0(t) = −λp0(t).

Define G(s, t) =

∞∑

n=0

snpn(t).

Thus G′(s, t) = λsG(s, t) − λG(s, t) so log G(s, t) = λt(s − 1) as G(s, 0) = 1.Thus G(s, t) = exp λt(s − 1) so N(t) ∼ Poisson(λt).Time T1 to first event satisfies T1 ∼ exponential(λ).Time between events has the same exponential distribution.

§5 Random number simulation.

§5.1 Basic simulation method. Generate U ∼ uniform(0, 1). Then use inverse transformationmethod.

§5.2 Continuous distributions. If F (x) = PX ≤ x, let x = F−1(u). X ∼ exponential(λ)example.

§5.3 Discrete distributions.

13


Lecture 13: Sampling and Statistical Inference I


§1 Basic definitions.

§2 Moments of the sample mean and variance.

§2.1 The sample mean. X =1

n

n∑

i=1

Xi. E[X ] = µ, V[X ] =σ2

n.

§2.2 The sample variance. S2 =1

n − 1

n∑

i=1

(Xi − X)2 =1

n − 1

(

n∑

i=1

X2i − nX2

)

. E[S2] = σ2.

§3 Sampling distributions for the normal.

§3.1 The sample mean. In general Z =X − µ

σ/√

n≈ N(0, 1) for large n.

If Xiind∼ N(µ, σ2), then Z =

X − µ

σ/√

n∼ N(0, 1) for all n.

§3.2 The sample variance. If Xiind∼ N(µ, σ2), then

(n − 1)S2

σ2∼ χ2

n−1.

U = χ2k tabulated values χk(α) satisfy P

U > χ2k(α)

= α.

§3.3 Independence of the sample mean and variance. Xi ∼ N(µ, σ2) case.

14


Lecture 14: Sampling and Statistical Inference II


§4 The t distribution. tk =N(0, 1)√

χ2k/k

for independent N(0, 1) and χ2k distributions.

If Xiind∼ N(µ, σ2), then t =

X − µ

S/√

n∼ tn−1.

Properties of tk distribution. Tabulated values tk(α) satisfy PT > tk(α) = α for T ∼ tk. AlsoPT < −tk(α) = α

§5 The F result for variance ratios.

If U ∼ χ2ν1

and V ∼ χ2ν2

are independent, then F =U/ν1

V/ν2∼ Fν1,ν2

.

If S1 and S2 are based on samples of size n1 and n2 respectively from normal populations with

variances σ21 and σ2

2 respectively, then F =S2

1/σ21

S22/σ2

2

∼ Fn1−1,n2−1.

Tabulated values Fν1,ν2(α) satisfy PF > Fν1,ν2

(α) = α for F ∼ Fν1,ν2. Also P

F <1

Fν2,ν1(α)

= α.

15


Lecture 15: Point Estimation I


§1 The method of moments.

§1.1 The one-parameter case.

§1.2 The two-parameter case.

§3 Unbiasedness. Let g(X) be an estimator of a parameter θ.Bias is Bias(g(X)) = E[g(X)] − θ.Unbiasedness.

§4 Mean square error. Let g(X) be an estimator of a parameter θ. MSE(g(X) = E[

(g(X) − θ)2]

.MSE(g(X) = V[g(X)] + Bias(g(X))2.

16


Lecture 16: Point Estimation II


§2 The method of maximum likelihood.

§2.1 The one-parameter case. Likelihood L(θ) =n∏

i=1

f(xi; θ).

§2.1.1 Example.

§2.2 The two-parameter case.

§5 Asymptotic distribution of MLE.

For large n, θ ≈ N(θ, v) where v =1

nE

[

(

∂

∂θlog f(X; θ)

)2] .

17


Lecture 17: Confidence Intervals I


§1 Confidence intervals in general. Want values θ1 and θ2 such that P

θ1 < θ < θ2

= 0.95.

§2 Distribution of confidence intervals.

§2.1 The pivotal method. Look for a pivotal quantity g(X , θ) such that, for example, ifg(X , θ) increases as θ increases, then g(X , θ) < g2 ⇔ θ < θ2 and g1 < g(X , θ) ⇔ θ1 < θ.

For example, if Xiind∼ N(µ, σ2), then g(X , µ) =

X − µ

σ/√

n.

§2.2 Confidence limits.

The interval

(

X − 1.96σ√n

, X + 1.96σ√n

)

can be written as X ± 1.96σ√n

.

§2.3 Sample size.

§3 Confidence intervals for the normal distribution.

§3.1 The mean.

Recall t =X − µ

S/√

n∼ tn−1 so 95% confidence interval for µ is X ± tn−1(2.5%)

S√n

.

§3.2 The variance.

Recall(n − 1)S2

σ2∼ χ2

n−1 so 95% confidence interval for σ2 is(n − 1)S2

χ2n−1(2.5%)

< σ2 <(n − 1)S2

χ2n−1(97.5%)

.

18


Lecture 18: Confidence Intervals II


§4 Confidence intervals for binomial and Poisson.

§4.1 The binomial. Course text a bit muddled here I think! If Ph1(θ) < X < h2(θ) ≥ 0.95,then PX ≥ h2(θ) ≤ 0.025 and PX ≤ h1(θ) ≤ 0.025. Now X < h2(θ) if θ > θ2(X) and similarlyX > h1(θ) if θ < θ1(X). Thus interval is θ2 < θ < θ1, where, for example, PX ≥ x|θ = θ2 =0.025.

§4.1.1 The normal approximation.

§5 Confidence intervals for two sample problems.

§5.1 Two normal means.

Confidence interval with known variances based on fact that X1 − X2 ∼ N

(

µ1 − µ2,σ2

1

n1+

σ22

n2

)

so

X1 − X2 − (µ1 − µ2)√

(

σ2

1

n1+

σ2

2

n2

)

∼ N(0, 1).

If σ21 = σ2

2 = σ2 is unknown, can be estimated using s2p =

(n1 − 1)s21 + (n2 − 1)s2

2

n1 + n2 − 2and then

X1 − X2 − (µ1 − µ2)√

s2p

(

1n1

+ 1n2

)

∼ tn1+n2−2.

§5.2 Two population variances. RecallS2

1/σ21

S22/σ2

2

∼ Fn1−1,n2−1 soS2

1/S22

σ21/σ

22

∼ Fn1−1,n2−1.

Also, if F ∼ Fν1,ν2with PF > Fν1,ν2

(α) = α, then

P

Fn1−1,n2−1(0.975) <S2

1/S22

σ21/σ

22

< Fn1−1,n2−1(0.025)

= 0.95 re-arranges to give

S21

S22

· 1

Fn1−1,n2−1(0.025)<

σ21

σ22

<S2

1

S22

· 1

Fn1−1,n2−1(0.975)where

1

Fn1−1,n2−1(0.975)= Fn2−1,n1−1(0.025).

§5.3 Two population proportions.X1 ∼ Bin(n1, θ1) ≈ N(n1θ1, n1θ1(1 − θ1)) and X2 ∼ Bin(n2, θ2) ≈ N(n2θ2, n2θ2(1 − θ2)).

Thus θi =Xi

ni

≈ N

(

θi,θi(1 − θi)

ni

)

for i = 1, 2 and so θ1 − θ2 ≈ N

(

θ1 − θ2,θ1(1 − θ1)

n1+

θ2(1 − θ2)

n2

)

.

In practice we assume θ1 − θ2 ≈ N

(

θ1 − θ2,θ1(1 − θ1)

n1+

θ2(1 − θ2)

n2

)

so can obtain confidence in-

terval for θ1 − θ2 by assuming the variance is known.

§6 Paired data. (NOT needed for the exam.) Form pairs Di = X1i − X2i. ThenD − µD

SD/√

n∼ tn−1.

19


Lecture 19: Hypothesis Testing I


§1 Hypotheses, test statistics, decisions and errors.Null hypothesis H0. Alternative hypothesis H1. Critical region.α = PType I error = PReject H0 when H0 true.β = PType II error = PAccept H0 when H0 false.Power = 1 − β = PReject H0 when parameter is θ.

§2 Classical testing, significance and P -values.

§2.1 “Best” tests. Neyman-Pearson Lemma H0 : θ = θ0 vs. H1 : θ = θ1 best test based on

likelihood ratio with critical region C satisfyingL0

L1≤ k.

§2.2 P -values. P = PA value occurs as or more extreme than the one observed|H0 true.

§3 Basic tests – single parameter.

§3.1 Testing the value of a population mean. Testing H0 : µ = µ0.

Test based on Z =X − µ0

σ/√

n∼ N(0, 1) or T =

X − µ0

S/√

n∼ tn−1 if H0 true.

§3.2 Testing the value of population variance. Testing H0 : σ2 = σ20.

Test based on(n − 1)S2

σ20

∼ χ2n−1 if H0 true.

§3.3 Testing the value of a population proportion. Testing H0 : θ = θ0.Test bsed on X ∼ Bin(n, θ0) ≈ N(nθ0, nθ0(1 − θ0)) if H0 true.

§4 Basic tests – two independent samples.

§4.1 Testing the value of the difference between two population means.Testing H0 : µ1 − µ2 = δ.

Test based on Z =X1 − X2 − δ√

σ2

1

n1+

σ2

2

n2

∼ N(0, 1) or T =X1 − X2 − δ

Sp

√

1n1

+ 1n2

∼ tn1+n2−2 if H0 true.

§4.2 Testing the value of the ratio of two population variances. Testing H0 : σ21 = σ2

2.

Test based onS2

1

S22

∼ Fn1−1,n2−1 if H0 true.

§4.3 Testing the value of the difference between two population proportions.Testing H0 : θ1 = θ2(= θ).

Test based onθ1 − θ2

√

θ(1−θ)n1

+ θ(1−θ)n2

if H0 true.

20


Lecture 20: Hypothesis Testing II


§7 χ2 tests.

Test statistic is∑

i

(fi − ei)2

ei

where ei are expected values under H0 and fi are observed values.

I prefer the notation∑

i

(Oi − Ei)2

Ei

!

§7.1 Goodness of fit.

§7.1.1 Degrees of freedom.Number of groups − Number of constraints on ei − Number of fitted parameters.

§7.1.2 The “accuracy” of the χ2 approximation.Ensure all ei > 5 by combining groups (cells).

§7.1.3 Example.

§7.2 Contingency tables.For r × c table, degrees of freedom is (r − 1)(c − 1).

Expected frequencies for any cell areRow total × Column total

Grand Total.

Tests of homogeneity and independence not clearly distinguished.

21


Lecture 21: Correlation and Regression I

References: CT3 Unit 12; IEF chapter 2, pages 27-38, 44-51.

§0 (CT3) Introduction. Scatter plot and summary statistics Sxx, Syy, Sxy.

§1 (CT3) Correlation analysis.

§1.1 (CT3) Data summary. Sample correlation r =Sxy

√

SxxSyy

.

§1.2 (CT3) The normal model and inference.

If ρ = 0,r√

n − 2√1 − r2

∼ tn−2.

If W = 12 log

1 + r

1 − r, then W ≈ N

(

12 log

1 + ρ

1 − ρ,

1

n − 3

)

.

Can re-write this as W = tanh−1 r, so that W ≈ N

(

tanh−1 ρ,1

n − 3

)

.

§2 (CT3) Regression analysis – the simple linear model.

§2.1 (CT3) Introduction. Yi = α + βxi + ei, i = 1, 2, . . . , n. E[ei] = 0, V[ei] = σ2.

§2.2 (CT3) Fitting the model. Fitted line is y = α + βxi where α = y − βx and β =Sxy

Sxx

.

α and β minimise q =n∑

i=1

e2i =

n∑

i=1

(yi − α − βxi)2. Least squares derivation.

E[β] = β, V[β] =σ2

Sxx

. σ2 =1

n − 2

n∑

i=1

(yi − yi)2.

§2.3 (CT3) Partitioning the variability of the responses.∑

i

(yi − y)2 =∑

i

(yi − yi)2 +

∑

i

(yi − y)2. SSTOT = SSRES + SSREG.

SSTOT = Syy. SSREG =S2

xy

Sxx

.

E[SSTOT ] = (n − 1)σ2, E[SSREG] = σ2 + β2Sxx, E[SSRES] = (n − 2)σ2.

Coefficient of determination R2 =SSREG

SSTOT.

Cases where line closely fits the data and where line a poor fit to the data.

22


Lecture 22: Correlation and Regression II

References: CT3 Unit 12; IEF chapter 2, pages 38-39, 51-66.

§2.4 (CT3) The full normal model and inference.

Assumptions. β ∼ N

(

β,σ2

Sxx

)

, independently of(n − 2)σ2

σ2∼ χ2

n−2.

§2.5 (CT3) Inferences on the slope parameter β.

(β − β)/√

σ2/Sxx√

((n − 2)σ2/σ2)/(n − 2)=

β − β√

σ2

Sxx

∼ tn−2.

§2.6 (CT3) Estimating a mean response and predicting an individual response.

µ0 = E[Y |x0] = α + βx0 estimated by µ0 = α + βx0. E[µ0] = µ0. V[µ0] =

1

n+

(x0 − x)2

Sxx

σ2.

µ0 − µ0√

V[µ0]∼ tn−2.

Estimate individual response y0 = α + βx0 + e0, estimated by y0 = α + βx0.

Since E[y0 − y0] = 0, V[y0 − y0] = σ2 + V[µ0], soy0 − y0

√

σ2

(

1 +1

n+

(x0 − x)2

Sxx

)

∼ tn−2.

§2.7 (CT3) Checking the model. Residuals ei = yi − yi. Residual plot ei vs. xi.

§2.8 (CT3) Extending the scope of the linear model. (NOT needed for the exam.)Transformations to give linearity.

§3 (CT3) The multiple linear regression model. (NOT needed for the exam.) Yi =α + β1xi1 + · · · βkxik + ei, i = 1, 2, . . . , n.

23


Lecture 23: Analysis of Variance


§0 (CT3) Introduction.

§1 (CT3) One-way analysis of variance.

§1.1 (CT3) The model. Yij = µ + τi + eij for i = 1, 2, . . . , k, j = 1, 2, . . . , ni, n =

k∑

i=1

ni.

eijind∼ N(0, σ2) so Yij ∼ N(µ + τi, σ

2).

If

k∑

i=1

niτi = 0, then τi is the ith treatment effect, and is µ the overall mean.

§1.2 (CT3) Estimation of the parameters. Minimise q =

k∑

i=1

ni∑

j=1

(Yij − µ − τi)2 to give

µ = Y, τi = Yi − Y

. If S2

i =1

ni − 1

ni∑

j=1

(Yij − Yi)2, then

(ni − 1)S2i

σ2

ind∼ χ2ni−1.

Hence σ2 =1

n − k

k∑

i=1

(ni − 1)S2i satisfies

(n − k)σ2

σ2∼ χ2

n−k.

§1.3 (CT3) Partitioning the variability.k∑

i=1

ni∑

j=1

(Yij − Y)2 =

k∑

i=1

ni∑

j=1

(Yij − Yi)2 +

k∑

i=1

ni(Yi − Y)2,

i.e., SST = SSR +SSB where SST is total sum of squares, SSR is residual sum of squares (within-treatments sum of squares), SSB is between-treatments sum of squares.

If H0 : τ1 = τ2 = · · · = τk = 0 is true, thenMSB

MSR

∼ Fk−1,n−k, where MSB =SSB

k − 1is mean squares

between treatments and MSR =SSR

n − kis residual mean square.

§1.4 (CT3) Example.

§1.5 (CT3) Checking the model. Residuals are rij = eij = Yij − µ − τi = Yij − Yi.

§1.6 (CT3) Estimating the treatment means.

95% confidence interval for µ + τi is Yi ± tn−k(2.5%)σ√ni

.

95% confidence interval for τi − τj is Yi − Yj ± tn−k(2.5%)σ

√

1

ni

+1

nj

.

§1.7 (CT3) Further comments. Linear regression model Yi = a + bxi + ei, i = 1, 2, . . . , n can

be analysed asn∑

i=1

(Yi − Y )2 =n∑

i=1

(Yi − Yi)2 +

n∑

i=1

(Yi − Y )2, i.e., SST = SSR + SSREG.

If H0 : b = 0 is true,SSREG

SSR/(n − 2)∼ F1,n−2.

24


Lecture 24: Univariate Time Series Analysis and Forecasting I

References: IEF chapter 5, pages 206-223.

§5.1 Introduction.

§5.2 Some notation and concepts.

§5.2.2 A weakly stationary process. E[Yt] = µ, V[Yt] = σ2, cov(Yt1 , Yt2) = γt1−t2 .

Autocovariance function is cov(Yt, Yt−s) = γs. Autocorrelation function is τs =γs

γ0.

§5.2.3 A white noise process. E[Yt] = µ, V[Yt] = σ2, γs = 0 for s 6= 0.

If Ytind∼ N(µ, σ2) for t = 1, 2, . . . , T , τs ≈ N(0, 1/T ).

Box-Pierce test that τ1 = · · · = τm = 0; if H0 true, Q = Tm∑

k=1

τ2k ∼ χ2

m.

§5.3 Moving average processes. MA(q) is Yt = µ+Ut+θ1Ut−1+· · ·+θqUt−q with Utind∼ (0, σ2).

Backshift operator L (Most textbooks would use B!) satisfies LYt = Yt−1.Thus Yt = µ + θ(L)Ut with θ(L) = 1 + θ1L + · · · + θqL

q.V[Yt] = γ0 = (1 + θ2

1 + · · · + θ2q)σ

2, γs = (θs + θs+1θ1 + · · · + θqθq−s)σ2 for s = 1, 2, . . . , q.

Example 5.2.

§5.4 Autoregressive processes. AR(p) is Yt = µ + φ1Yt−1 + φ2Yt−2 + · · · + φpYt−p + Ut.Can write as φ(L)Yt = µ + Ut with φ(L) = 1 − φ1L − · · · − φpL

p.

§5.4.1 The stationarity condition. AR(p) process stationary if roots of φ(z) = 0 lie outsideunit circle. Can then write AR(p) process as MA(∞) Yt = φ−1(L)Ut.Example 5.3.

§5.4.2 Wold’s decomposition theorem. (NOT needed for the exam.)All we really need here is that (1 − φ1 − φ2 − · · · − φp)E[Yt] = µ and the autocorrelation functionsatisfies the Yule-Walker equations τr = τ1−rφ1 + τ2−rφ2 + · · · + τp−rφp for r = 1, 2, . . . , p withτ−s = τs.Example 5.4.

§5.5 The partial autocorrelation function. The pacf τkk can be found from fitting the modelYt = µ + τk,1Yt−1 + · · · τk,k−1Yt−k+1 + τkkYt−k + Ut.

§5.5.1 The invertibility condition. MA(q) process is invertible if roots of θ(z) = 0 lie outsidethe unit circle. The process Yt = θ(L)Ut can then be written as an AR(∞) process θ−1(L)Yt = Ut.

25


Lecture 25: Univariate Time Series Analysis and Forecasting II

References: IEF chapter 5, pages 223-238, 247-251.

§5.6 ARMA processes. ARMA(p, q) process is φ(L)Yt = µ + θ(L)Ut.Mean satisfies (1 − φ1 − φ2 − · · · − φp)E[Yt] = µ.AR(p) process: acf (oscillatory) geometric decay, pacf zero after lag p.MA(q) process: acf zero after lag q, pacf (oscillatory) geometric decay.ARMA(p, q) process: acf like AR, pacf like MA.

§5.6.1 Sample acf and pacf plots for standard processes. Correlogram (acf plot) has linesdrawn at ±1.96/

√n to indicate significant τk; pacf plot also has lines drawn at ±1.96/

√n.

§5.7 Building ARMA models: the Box-Jenkins approach. Determine model order; esti-mate parameters; check model validity. Parsimonious models best!

§5.7.1 Information criteria for ARIMA model selection. AIC widely used (with SBIC).

§5.7.3 ARIMA modelling. Data differenced to give stationarity.

§5.8 Constructing ARMA models in EViews. (NOT needed for the exam.) We use R.

§5.11.4 Forecasting with time series versus structural models. (NOT needed for theexam.) Conditional expectation is E[Yt+1|Ωt] = E[Yt+1|Y1, Y2, . . . , Yt].

§5.11.5 Forecasting with ARMA models. (NOT needed for the exam.) ARMA(p, q)

model Yt =

p∑

i=1

aiYt−i +

q∑

j=1

bjUt−j . Forecast at time t + s is Yt+s =

p∑

i=1

aiYt+s−i +

q∑

j=1

bjUt+s−j

where Yk = Yk for k ≤ t, Uk = 0 for k ≤ t, Uk = 0 for k > t. IEF uses notation ft,s ≡ Yt+s.

§5.11.6 Forecasting the future value of an MA(q) process. (NOT needed for theexam.)MA(3) process Yt = µ + Ut + θ1Ut−1 + θ2Ut−2 + θ3Ut−3.eg, Yt+2 = µ + Ut+2 + θ1Ut+1 + θ2Ut + θ3Ut−1. Thus E[Yt+2|Ωt] = µ + θ2Ut + θ3Ut−1.

§5.11.7 Forecasting the future value of an AR(p) process. (NOT needed for theexam.)AR(2) process Yt = µ + φ1Yt−1 + φ2Yt−2 + Ut.eg, Yt+2 = µ + φ1Yt+1 + φ2Yt + Ut+2. Thus E[Yt+2|Ωt] = µ + φ1Yt+1 + φ2Yt.

26


Lecture 26: Multivariate Models I

References: IEF chapter 3, pages 88-93; chapter 6, pages 265-271, 276-277.

§3.1 Generalising the simple model to multiple linear regression.

§3.2 The constant term. Writing the linear model in the form y = Xβ + u.

§3.3 How are the parameters (the elements of the β vector) calculated?β = (X ′X)−1X ′y found by minimising (y − Xβ)′(y − Xβ).

§6.1 Motivations.Structural equations. Reduced form equations.

§6.2 Simultaneous equations bias.

§6.3 So how can simultaneous equations models be validly estimated?

§6.4 Can the original coefficients be retrieved from the πs?

§6.4.1 What determines whether an equation is identified or not?

§6.8 Estimation procedures for simultaneous equations systems.

§6.8.1 Indirect least squares (ILS). (NOT needed for the exam.)

§6.8.2 Estimation of just identified and overidentified systems using 2SLS. (NOTneeded for the exam.)Using R systemfit command.

27


Lecture 27: Multivariate Models II

References: IEF chapter 6, pages 290-293, 294-296, 298, 308-315.

§6.11 Vector autoregressive models.

§6.11.1 Advantages of VAR modelling.

§6.11.2 Problems with VARs.

§6.11.3 Choosing the optimal lag length for a VAR.

§6.11.5 Information criteria for VAR lag length selection.

§6.12 Does the VAR include contemporaneous terms?

§6.14 VARs with exogenous variables.

§6.17 VAR estimation in EViews. (NOT needed for the exam.)We use the R package vars.

28


Lecture 28: Cointegration

References: IEF chapter 7, pages 318-329, 335-341.

§7.1 Stationarity and unit root testing.

§7.1.1 Why are tests for non-stationarity necessary?

§7.1.2 Two types of non-stationarity.

§7.1.3 Some more definitions and terminology.

§7.1.4 Testing for a unit root.

§7.3 Cointegration.

§7.3.1 Definition of cointegration.

§7.4 Equilibrium correction or error correction models.

§7.5 Testing for cointegration in regression: a residuals-based approach.

§7.6 Methods of parameter estimation in cointegrated systems. (NOT needed for theexam.)

§7.6.1 The Engle-Granger 2-step method. (NOT needed for the exam.)

29


Lecture 29: ARCH Models

References: IEF chapter 8, pages 379-381, 383-384, 385-389.

§8.1 Motivations: an excursion into non-linearity land.

§8.1.1 Types of non-linear models.

§8.2 Models for volatility.

§8.3 Historical volatility.

§8.6 Autoregressive volatility models.

§8.7 Autoregressive conditionally heteroscedastic (ARCH) models.

§8.7.1 Another way of expressing ARCH models.

§8.7.2 Non-negativity constraints.

§8.7.3 Testing for ‘ARCH effects’.

§8.7.5 Limitations of ARCH(q) models.

30


Lecture 30: GARCH Models

References: IEF chapter 8, pages 392-399.

§8.8 Generalised ARCH (GARCH) models.

§8.8.1 The unconditional variance under a GARCH specification.

§8.9 Estimation of ARCH/GARCH models.

§8.9.1 Parameter estimation using maximum likelihood. (NOT needed for the exam.)

§8.9.2 Non-normality and maximum likelihood. (NOT needed for the exam.)

31

math5315 applied statistics and probability 2011-2012sta6ajb/math5315/5315lec.pdf · math5315...

Documents