The Business ofPredictive Modeling

December 17, 2013

Christine Hofbeck, FSA, MAAACentroid Analytics, LLC

AGENDA

PART I -- INTRODUCTION

PART II – MODELING 101 (Basic Steps)

PART III – “GOLDEN QUESTION”

PART IV – OPERATIONAL CONSIDERATIONS

2

INTRODUCTION

Predictive modelling [sic] is the process by which a model is created or chosen to try to best predict the probability of an outcome. -- Wikipedia

In practice:

3

VISUALIZECustomers

Most Profitable Lines

OTHERyou are only limited by

your creativity

Identify patterns/ segment

risks

Develop business

rules

Improved decision making

Potential Applications

4

OPTIMIZE Operational Efficiency Distribution ChannelsClaims Management

Pricing / Reserves

DATA

VISUALIZECustomers

Most Profitable Lines or Products

Target Marketing

OTHERyou are only limited by

your creativity

MINIMIZERisk

Fraud

Potential Applications (Life)

5

1. Triage UW decisions; implement STP for (more) applicants

2. Decrease purchase of traditional UW requirements by determining when they may not be necessary

3. Identify & target customers more likely to buy

4. Identify customers more likely to lapse – intervene if profitable, allow unhealthies to lapse

5. Inforce book management

6. Identify most desirable agents

7. Smart customer handling

Predictors

A predictive model is made up of a number of predictors (“independent variables”), which are data elements likely to influence future behavior or results (“dependent variable”).

6 SEEK PARSIMONY

DON’T USE ONE VARIABLE DON’T USE ALL VARIABLES the mean predicts the future but doesn’t tell us why…(“underfit”)

exactly replicates the past… cannot predict the future (“overfit”)

BASIC STEPS (Modeling 101)

7

Define & Scope

Data Prep

Model Build

Model Validation

Implementation

Review & Refine

Define & Scope

8

What is the budget?

Consider IT, staff, data purchase,

training, etc.

Do we have the systems capacity to

implement?

How long do we have to build? To

implement?

Insource or outsource?

How will the results be used?For whom/what

are we trying to predict this? (“unit

of exposure”)

Exactly what are we trying

to predict?

Data Prep* sometimes the most time intensive step of modeling

9

MODELING DATASET

INTERNAL DATAo # yearso accuracyo ability to accesso primary key

EXTERNAL DATAo match rateo cost – to modelo cost – to useo frequency of update

Consider both expected & unexpected relationships – creativity in data exploration can be the key

to your competitive edge!

Data Prep (cont’d)1. COMBINE various data sources2. CONVERT to desired exposure unit or format3. CORRECT inaccurate data4. INSPECT to remove variables:

- Too many blank values that cannot be imputed- All/most values the same- Data cannot be relied upon- Data will not be captured going forward- Legal advice not to use

5. BUCKET (“bin”) values10

Model Build (cont’d)

UNIVARIATE ANALYSIS – test each variable one by one to see which ones may be predictive.

11

MULTIVARIATE ANALYSIS – examine multiple variables in different groups to obtain the best, USABLE results – remember parsimony!

INTERACTIONS – which variables can be combined into a “mega variable” to improve results (i.e., does 1+1 = 1.5? does 1+1 = 3?)

Complicate the model (add variables, interactions) and simplify the model (remove variables, bin)

to find the preferred combination.

Model Build (cont’d)

Various tests can be used to determine variable inclusion:

12

STATISTICAL CONSISTENCY JUDGMENT

P-valuesCramer’s VConfidence

intervalsType III tests

Apply business knowledge to

assess whether suggested

relationships make sense

Of patterns -

Over timeOver random

parts of a dataset

Model Validation

ACTUAL vs. EXPECTED-- how close did we get?

Generally, a subset of the data is withheld during the modeling process for validation:

Model validation graphs are useful for communicating model performance to non-technical audiences.

13

OUT OF TIMEwithhold most

recent data

OUT OF SAMPLEwithhold randomly

generated % of records

Model Validation – Sample Chart

14

1 2 3 4 5 6 7 8 9 100.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

ActualExpectedO

utco

me

Decile

Implementation

BUSINESS RULESWhat decisions will be made based on the prediction?

May vary by location, business, rate group, etc.

SYSTEM BUILDScoring engine (collects data & calculates predictions)

Decision tool (executes business rules)User interface

TRAININGAnyone who will interact with the model must

understand what it does and why15

Review & RefineREPORTING

How close did we get to the goal? How far did we exceed it?

Multiple reporting packages required for varied audiences, for example:

Executives – highlights in aggregate by zone, business unit, product

Actuaries – detailed results by variable, state, rate group Marketing – by broker/agent, location Underwriting – by underwriter as a performance measure

Frequency of update – weekly, monthly, quarterly, yearly?

Method of calculation – automated? ad hoc?

Review & Refine (cont’d)MODEL UPDATES

WHY? As target customer is attained, characteristics of inforce

book will change Business goals/strategies may change New data may become available Tolerance for certain characteristics may change

HOW? Update current variable relativities (“recalibrate”) Start over - search for more predictive variables (“recast”)

HOW OFTEN?17

Advantages of Modeling Over

18

1. Many additional and often unconventional variables may be examined

2. Modeling a particular variable controls for the effects of other included variables – we don’t risk double counting or attributing effects to the wrong variables

3. Traditional approaches segment data into smaller categories which impact credibility

4. Interactions are introduced

The above advantages can lead to improved accuracy, enhanced business and strategic benefits, more reliable

assumptions, improved risk mitigation, etc.

Traditional Approaches

THE “GOLDEN QUESTION”

Through brainstorming, feedback loops, and data review, determine what single characteristic (“golden question”) will define your target

19

OPERATIONAL CONSIDERATIONS

1. Executive & cross-functional support

2. Time/cost versus depth of investigation

3. Strategic modeling process

4. Cross-functional involvement throughout build

5. Thorough training

20

Executive & Cross-Functional Support

21

If target users don’t support the model, they will resist using it.

Gaining complete support can be difficult:1. Resistance to change2. Concern that model results will highlight

current deficiencies3. Lack of understanding of predictive models

Support (cont’d)

22

We’ve always done it this way, and it’s worked.

I don’t see a reason to change

anything.

I will have to take

on additional work associated with

new processes. My workflow will

double (triple).

I don’t know how to explain this to a broker/agent so I don’t want

to use it.

I found one outlier so the

model must be wrong.

I already have an established plan. I

know who our target customer is. The model will suggest

that my current methodis incorrect, which will reflect poorly on my

performance/reputation.

My position will be eliminated if a model is now used to select risks. My expertise

must not be important to the company.

Time/Cost vs. Depth of Investigation

23

The process of building and implementing a model can typically be quite lengthy – longer than most expect

OR

Remember that a simple model does not necessarily indicate a simple study!

• Simpler Study (3-12 months)• Results more conservative• Perhaps internal data only• Generous binning• Limited interactions• May be appropriate if goal is

a general sense of direction

• More thorough investigation• Additional time• Additional development cost• Possible greater payoff

through enhanced segmentation and data exploration

Strategic Modeling ProcessTARGET PREDICTION/USEo Ensure target is appropriate for the intended useo While many ideas are interesting, you may wish to focus on those

which are actionable

STATISTICAL SIGNIFICANCE vs. ULTIMATE IMPACTo The most statistically significant model may not be the most

impactfulo Consider ease of implementation, repeatability, updateso Identify when “less is more”!

FLEXIBILITYo Allow for unexpected insights which could lead to unanticipated

changes in business strategy or processo Sometimes the insights gained from the journey will prove more

important than the planned goal

Cross-Functional InvolvementData, product & IT experts, legal advisors, and model

users must remain engaged throughout the model build

25

Insight from functions

Insight to functions

Eases training and implementation

Keep modelers apprised of changes in strategy

Legal considerations around certain variables

Thorough Training

26

The model isn’t done when it’s done.

Who will provide the training?Who is most appropriate to provide training?

Modeling teamGeneral training team

Functional expertsConsulting team*

Other

No clear answer – but this must be thoughtfully considered and appropriately executed to reap the full benefits of the model

which was built

*Consider what information may be shared (non-proprietary)

Discussion/Q&A

27

Remember…Modeling is a complete business strategy

NOT just a mathematical process

So how will YOU use predictive modeling to improve your business?

Christine Hofbeck, FSA, MAAACentroid Analytics, LLC

christine.hofbeck@centroidanalytics.com 908.884-4103 (c) 908.574-5351 (w)www.centroidanalytics.com