F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 1

ProposedPerformance Metrics

Block Time & Predictability

Marc Rose, MCR/SETA-IIGabriela Rohlck, MCR/SETA-II

15 March, 2006

Presented atNEXTOR - Asilomar

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 2

Objective

An evaluation of two specific metrics: Scheduled Block Time and PredictabilityA general discussion of types of metrics

This work has been sponsored by two ATO Planning offices:1) The Office of Strategy2) The Office of Performance Analysis

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 3

Three Primary Types of Metrics

System Activity or Workload– Number of flights per month, overall average block time– Provides an estimate of workload, generally NOT performance– Impacted by non-performance based changes

• Avg Flight length - more longer flights, not performance related

Performance – Generally a system activity metric in constrained times– Normalized block time, arrival delay (sometimes)

Diagnostic– Needed to understand changes in performance

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 4

Three Primary Types of Metrics

ALL metrics have flaws/limitations:Identification of these limitations is vital to understanding

and proper usage

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 5

Data and DefinitionData Sources

– Airline Service Quality Performance (ASQP)• Data available from 1/1995 to 12/2005

– Aviation System Performance Metrics (ASPM)• Data available from 1/2000 to 12/2005

– Official Airline Guide (OAG)• Data available from 1/1982 to 9/2005

Definition:– Block Time: Airborne+Taxi-Out+Taxi-In (Gate-Out to Gate-in)– Total Flight Time: Block Time + Departure Delay

• Or: Actual arrival (in) - Scheduled Departure– Scheduled: Airline published departure/arrival times

• Based on OAG or ASPM data (pre departure)– Actual: departure/arrival times as flown (post departure)

• Based on ASQP or ASPM data

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 6

Scheduled Block Time

A Performance Metric (a lagging indicator)Metric based on OEP to OEP flights

– A current FAA Executive Council decisionConsidered the “NAS Level” (All* flights) and the OEP-OEP level (only flights to/from an OEP 35 airport)Used ASPM data, limited to those flights that have OAG data

– OAG=“Y” flag– Does not include cancelled flights

Two metrics proposed1) “Normalized” average block time

– Normalization consists of identical set of O-D pairs weighted using the average number of flights/month across all datasets

2) Delta metric - A relative change from the overall average

*Not truly ALL flights – as will be seen in the development

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 7

Normalization of Block TimeMathematical Details

∑∑

=

jiij

ijji

ijk

k n

n

,

,

ˆ

ˆµµ

N

nn

N

kijk

ij

∑== 1ˆ

ijk

lijkl

ijk n

x∑=µ

Where x is the block time for flight lfor O-D pair ij within month k andn is the number of flights for O-D pairWithin all (majority*) of data sets

Combining into a normalized NAS level “Performance” measure:

Two intermediate values are calculated1) the mean time 2) the average number of flights

µijn̂

Where N is the number of months of data

Where ij is an O-D pair,and k is month/year

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 8

Normalized ScheduledBlock Time (OEP to OEP)

Moving average (ttm)

140

141

142

143

144

145

146

147

148

149

150

Jan-0

0Apr-

00Ju

l-00

Oct-00

Jan-0

1Apr-

01Ju

l-01

Oct-01

Jan-0

2Apr-

02Ju

l-02

Oct-02

Jan-0

3Apr-

03Ju

l-03

Oct-03

Jan-0

4Apr-

04Ju

l-04

Oct-04

Jan-0

5Apr-

05Ju

l-05

Oct-05

Source: ASPM

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 9

Normalized ScheduledBlock Time (NAS Level)

Moving average (ttm)

Easy to see missing data

80

85

90

95

100

105

Jan-

82

Jan-

83

Jan-

84

Jan-

85

Jan-

86

Jan-

87

Jan-

88

Jan-

89

Jan-

90

Jan-

91

Jan-

92

Jan-

93

Jan-

94

Jan-

95

Jan-

96

Jan-

97

Jan-

98

Jan-

99

Jan-

00

Jan-

01

Jan-

02

Jan-

03

Jan-

04

Jan-

05

Nor

mal

ized

Ave

rage

Blo

ck T

ime

(min

)

Source: OAG

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 10

Scheduled Block Times A Lagging Metric

130

135

140

145

150

155

160

2000 2001 2002 2003 2004 2005

Flig

ht T

imes

(min

)

Total Flight Time (ttm)

Scheduled Block Time (ttm)

Actual Block Time (ttm)

0.97

0.98

0.99

1

1.01

1.02

1.03

1.04

1.05

Dec-95

Apr-96

Aug-96Dec

-96Apr-9

7Aug-97Dec

-97Apr-9

8Aug-98Dec

-98Apr-9

9Aug-99Dec

-99Apr-0

0Aug-00Dec

-00Apr-0

1Aug-01Dec

-01Apr-0

2Aug-02Dec

-02Apr-0

3Aug-03Dec

-03Apr-0

4Aug-04Dec

-04Apr-0

5

Rel

ativ

e C

hang

ing

Blo

cktim

e

Scheduled

Actual

Source: ASPM, OAG and ASQP

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 11

Delta Metric – Another Approach

ijk

n

lijkl

ijk n

xijk

∑== 1µ

Where x is the block time for flight land O-D pair ij within month k andn is the number of flights

∑∑==

kijk

N

kijkijk

ij n

n1ˆµ

µ A multi-month average block time

∑∑ −

=

ijijk

ijijkijijk

k n

n)ˆ( µµδ

Which is a relative performance metric for the block time that includesALL flights

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 12

Scheduled Block Time Delta Metric (OEP-OEP)

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

Jan-00

Jul-0

0Ja

n-01

Jul-0

1Ja

n-02

Jul-0

2Ja

n-03

Jul-0

3Ja

n-04

Jul-0

4Ja

n-05

Jul-0

5

Min

utes

Data Source: ASPM

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 13

Scheduled Block TimeDelta Metric (NAS Level)

Data Source: ASPM

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

Jan-00

Apr-00

Jul-0

0Oct-

00Ja

n-01Apr-0

1Ju

l-01

Oct-01

Jan-02

Apr-02

Jul-0

2Oct-

02Ja

n-03Apr-0

3Ju

l-03

Oct-03

Jan-04

Apr-04

Jul-0

4Oct-

04Ja

n-05Apr-0

5Ju

l-05

Oct-05

Min

utes

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 14

Historical Scheduled Block Time Delta Metric (NAS Level)

-8

-6

-4

-2

0

2

4

6

Jan-

82

Jan-

83

Jan-

84

Jan-

85

Jan-

86

Jan-

87

Jan-

88

Jan-

89

Jan-

90

Jan-

91

Jan-

92

Jan-

93

Jan-

94

Jan-

95

Jan-

96

Jan-

97

Jan-

98

Jan-

99

Jan-

00

Jan-

01

Jan-

02

Jan-

03

Jan-

04

Jan-

05

Blo

ck T

ime

Cha

nges

(min

)

Moving average (ttm)

Data Source: OAG

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 15

Scheduled Block Time ChangesAccounting for Aircraft Type

-4

-3

-2

-1

0

1

2

3

Jan-

82

Jan-

83

Jan-

84

Jan-

85

Jan-

86

Jan-

87

Jan-

88

Jan-

89

Jan-

90

Jan-

91

Jan-

92

Jan-

93

Jan-

94

Jan-

95

Jan-

96

Jan-

97

Jan-

98

Jan-

99

Jan-

00

Jan-

01

Jan-

02

Jan-

03

Jan-

04

Jan-

05

Data Source: OAG

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 16

Total Flight Time Variabilitya Predictability Metric?

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 17

Percentiles as a Predictability Metric

• Start at origin-destination level:– How does measured block time of airport-pair (O-

D) differ from average timewhere:

is measured time for O-D pair (i, j) is mean time for O-D pair (i, j)

• Flight-level Mk is generated for all flights and the Selected percentiles are output as a NAS level metric

ijijk xM µ−=

ijµijx

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 18

NAS PredictabilityPerformance Metric(Total Flight Time – includes gate delay)

Source: ASQP

-30

-20

-10

0

10

20

30

40

50

60

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Pred

icta

bilit

y M

etric

(min

)

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 19

Predictability Metric OEP Arrival Airports – Sep/2005

-40

-30

-20

-10

0

10

20

30

40

50

ATL

BO

S

BW

I

CLE

CLT

CVG DC

A

DEN

DFW DTW

EWR

FLL

HN

L

IAD

IAH

JFK

LAS

LAX

LGA

MC

O

MD

WM

EM MIA

MSP

OR

D

PDX

PHL

PHX

PIT

SAN

SEA

SFO

SLC

STL

TPA

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 20

Observations• Using 10th, 90th percentiles

–Eliminate significant outliers–Analysis shows distribution right-skewed.

•Average closer to lower bound than upper bound.

–Metric defined as difference between lower, upper percentiles.

•Can be calculated with one month of data.–Does not require historical data.

• Seasonality–This Metric partially accounts for known seasonality

(e.g., winter headwinds) by using current months averages rather than historical

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 21

Backup

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 22

Metric with Normalized Average Flight Time

0

10

20

30

40

50

60

70

80

90

100

Jan-95

Jul-95

Jan-96

Jul-96

Jan-97

Jul-97

Jan-98

Jul-98

Jan-99

Jul-99

Jan-00

Jul-00

Jan-01

Jul-01

Jan-02

Jul-02

Jan-03

Jul-03

Jan-04

Jul-04

Jan-05

Jul-05

Met

ric (m

in)

0

20

40

60

80

100

120

140

160

Ave

rage

Tot

al F

light

Tim

e (m

in)

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 23

Actual Block Time Variability:Percentiles and Weighted Average(10th and 90th percentiles)

100

110

120

130

140

150

16019

95

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

Blo

ck ti

me

(min

)

F E D E R A L A V I A T I O N A D M I N I S T R A T I O N • A I R T R A F F I C O R G A N I Z A T I O N 24

Predictability Metric Calculation - Example

Origin Destination Flight Time Avg Flt tm Delta Sorted DeltaA B 90 98.0 -8.0 -29.9A B 110 98.0 12.0 -14.9A B 92 98.0 -6.0 -14.8A B 100 98.0 2.0 -9.8C D 97 81.8 15.2 -8.0C D 67 81.8 -14.8 -7.9C D 72 81.8 -9.8 -7.8C D 101 81.8 19.2 -6.0C D 80 81.8 -1.8 -4.9C D 74 81.8 -7.8 -1.8E F 200 174.9 25.1 2.0E F 202 174.9 27.1 5.1E F 160 174.9 -14.9 12.0E F 145 174.9 -29.9 15.2E F 170 174.9 -4.9 19.2E F 180 174.9 5.1 25.1E F 167 174.9 -7.9 27.1

Metric = -14.8 to +21.5