Factor Models > 21. Machine learning > Toward supervised machine learning

Types of regression estimation

ARPM - Advanced Risk and Portfolio Management - arpm.co This update: Mar-26-2017 - Last update

Factor Models > 21. Machine learning > Toward supervised machine learning

Basics

• Xt = outputs or responses

• Zt = inputs or features

• Responses conditioned on the features are independent across time:

Xt|z ∼ fXt|z(x) independent (21.1)

⇒ Functional model relating features and responses:

fXt|z(x) ≡ fθ(x|z) (21.2)

calibrated from the training set {xt,zt}t̄t=1

ARPM - Advanced Risk and Portfolio Management - arpm.co This update: Mar-26-2017 - Last update

Factor Models > 21. Machine learning > Toward supervised machine learning

Basics

• Xt = outputs or responses

• Zt = inputs or features

• Responses conditioned on the features are independent across time:

Xt|z ∼ fXt|z(x) independent (21.1)

⇒ Functional model relating features and responses:

fXt|z(x) ≡ fθ(x|z) (21.2)

calibrated from the training set {xt,zt}t̄t=1

ARPM - Advanced Risk and Portfolio Management - arpm.co This update: Mar-26-2017 - Last update

Factor Models > 21. Machine learning > Toward supervised machine learning

Basics

• Xt = outputs or responses

• Zt = inputs or features

• Responses conditioned on the features are independent across time:

Xt|z ∼ fXt|z(x) independent (21.1)

⇒ Functional model relating features and responses:

fXt|z(x) ≡ fθ(x|z) (21.2)

calibrated from the training set {xt,zt}t̄t=1

ARPM - Advanced Risk and Portfolio Management - arpm.co This update: Mar-26-2017 - Last update

Factor Models > 21. Machine learning > Toward supervised machine learning

Predictive linear models

• Predictive models: factors Zt contain information only up toprevious time period

• Predictive regression LFM’s: Xt+1 = α+ βZt +U t+1

VAR(1) process

The VAR(1) process (2.106) with (εt ∼ N (µ,σ2) i.i.d.) is a predictiveregression LFM

∆Xt+1︸ ︷︷ ︸Xt+1

= µ︸︷︷︸α

+ (b− In̄)︸ ︷︷ ︸β

Xt︸︷︷︸Zt

+ εt+1 − µ︸ ︷︷ ︸Ut+1

(21.3)

Indicator of defaultThe indicator of default 1Dn∈[t,t+1) (1.59) is a predictive model

1Dn∈[t,t+1)︸ ︷︷ ︸Xn,t+1

|zn,t ∼ Bernoulli(pθ(zn,t)) (21.4)

ARPM - Advanced Risk and Portfolio Management - arpm.co This update: Mar-26-2017 - Last update

Factor Models > 21. Machine learning > Toward supervised machine learning

Predictive linear models

• Predictive models: factors Zt contain information only up toprevious time period

• Predictive regression LFM’s: Xt+1 = α+ βZt +U t+1

VAR(1) process

The VAR(1) process (2.106) with (εt ∼ N (µ,σ2) i.i.d.) is a predictiveregression LFM

∆Xt+1︸ ︷︷ ︸Xt+1

= µ︸︷︷︸α

+ (b− In̄)︸ ︷︷ ︸β

Xt︸︷︷︸Zt

+ εt+1 − µ︸ ︷︷ ︸Ut+1

(21.3)

Indicator of defaultThe indicator of default 1Dn∈[t,t+1) (1.59) is a predictive model

1Dn∈[t,t+1)︸ ︷︷ ︸Xn,t+1

|zn,t ∼ Bernoulli(pθ(zn,t)) (21.4)

ARPM - Advanced Risk and Portfolio Management - arpm.co This update: Mar-26-2017 - Last update

Factor Models > 21. Machine learning > Toward supervised machine learning

Generalized linear models

• Generalized linear models:

fθ(x|z) ≡ γg(x;α+ βz) (21.5)

with γ normalizing constant

• If the response is binary (xt ∈ {0, 1}) then

Xt|z ∼ Bernoulli(pβ(z)) (21.6)

• logistic regression (logit) model:

pβ(z) ≡ 11+e−βz

(21.7)

• probit model:pβ(z) ≡ Φ(βz) (21.8)

ARPM - Advanced Risk and Portfolio Management - arpm.co This update: Mar-26-2017 - Last update

Factor Models > 21. Machine learning > Toward supervised machine learning

Generalized linear models

• Generalized linear models:

fθ(x|z) ≡ γg(x;α+ βz) (21.5)

with γ normalizing constant

• If the response is binary (xt ∈ {0, 1}) then

Xt|z ∼ Bernoulli(pβ(z)) (21.6)

• logistic regression (logit) model:

pβ(z) ≡ 11+e−βz

(21.7)

• probit model:pβ(z) ≡ Φ(βz) (21.8)

ARPM - Advanced Risk and Portfolio Management - arpm.co This update: Mar-26-2017 - Last update

Factor Models > 21. Machine learning > Toward supervised machine learning

Generalized linear models

• Generalized linear models:

fθ(x|z) ≡ γg(x;α+ βz) (21.5)

with γ normalizing constant

• If the response is binary (xt ∈ {0, 1}) then

Xt|z ∼ Bernoulli(pβ(z)) (21.6)

• logistic regression (logit) model:

pβ(z) ≡ 11+e−βz

(21.7)

• probit model:pβ(z) ≡ Φ(βz) (21.8)

ARPM - Advanced Risk and Portfolio Management - arpm.co This update: Mar-26-2017 - Last update

Factor Models > 21. Machine learning > Toward supervised machine learning

Generalized linear models

• Generalized linear models:

fθ(x|z) ≡ γg(x;α+ βz) (21.5)

with γ normalizing constant

• If the response is binary (xt ∈ {0, 1}) then

Xt|z ∼ Bernoulli(pβ(z)) (21.6)

• logistic regression (logit) model:

pβ(z) ≡ 11+e−βz

(21.7)

• probit model:pβ(z) ≡ Φ(βz) (21.8)

ARPM - Advanced Risk and Portfolio Management - arpm.co This update: Mar-26-2017 - Last update

Factor Models > 21. Machine learning > Toward supervised machine learning

Further generalizations

Ways to generalize the model:

• non-linear expressions of zt in fθ(x|z)

• reduce dimension of zt via lasso techniques (Section 21.5)

Generalization of logit modelFunctional form:

fθ(x|z) ≡ γe∑s̄

s=1 θsφs(x,z) (21.9)

• fθ(x|z) is an exponential family distribution (Section 22.6)• Large number of feature functions φs(x,z):

• linear φs(x,z) ≡ (x− 12)zk

• quadratic φs(x,z) ≡ (x− 12)zjzl

• . . .

ARPM - Advanced Risk and Portfolio Management - arpm.co This update: Mar-26-2017 - Last update

Factor Models > 21. Machine learning > Toward supervised machine learning

Further generalizations

Ways to generalize the model:

• non-linear expressions of zt in fθ(x|z)

• reduce dimension of zt via lasso techniques (Section 21.5)

Generalization of logit modelFunctional form:

fθ(x|z) ≡ γe∑s̄

s=1 θsφs(x,z) (21.9)

• fθ(x|z) is an exponential family distribution (Section 22.6)• Large number of feature functions φs(x,z):

• linear φs(x,z) ≡ (x− 12)zk

• quadratic φs(x,z) ≡ (x− 12)zjzl

• . . .

ARPM - Advanced Risk and Portfolio Management - arpm.co This update: Mar-26-2017 - Last update