bloem defense.v16.slides

Optimization and Analytics

for Air Traffic Management

Enabling Decision-Support Tools

Michael Bloem

Stanford University NASA Ames Research Center

Outline

• Air traffic management background

• Research topics

• Research objective and approach

• Case study:

decision-support tool for area supervisors

• Summary of contributions

2

Air Traffic Management is Important

3


civil aviation

was responsible for

5.4% of US GDP

in 2012

[FAA’s 2014 "The Economic Impact of Civil Aviation on the US Economy"]

3


civil aviation

was responsible for

5.4% of US GDP

in 2012

837 million passengers

carried by airlines

operating in US airspace

in 2012

[FAA’s 2014 "The Economic Impact of Civil Aviation on the US Economy"]

3

Air Traffic Management is Complex

4

Air Traffic Management is Complex

∼50,000 flights/day

5,000+ flights simultaneously

4

Air Traffic Management is

Accomplished Largely by

Human Decision Makers

5

Research Topics: ATM Decisions

6


1 Area supervisors:area configuration selection

– Motivation: prescriptive decision model

6




2 Operations managers:assignment of flights to slots

– Motivation: insight into airline delay costs

6





– Motivation: insight into airline delay costs

3 Flow managers:Ground Delay Program implementation

– Motivation: predictive capability and insights

6

Research Objective

decision-

support

tool

7

Research Objective

solution

algorithm

decision-

support

tool

decision

model

constraints

objective

7

Research Approach

expert input

& feedback

solution

algorithm

decision-

support

tool

decision

model

constraints

objective

8

Research Approach

expert input

& feedback

operational

decision

data

solution

algorithm

decision-

support

tool

decision

model

constraints

objective

8

Research Approach

fast-time

simulations

expert input

& feedback

operational

decision

data

solution

algorithm

decision-

support

tool

decision

model

constraints

objective

8

Research Approach

fast-time

simulations

human-in-

the-loop

simulations

expert input

& feedback

operational

decision

data

solution

algorithm

decision-

support

tool

decision

model

constraints

objective

8

Outline


• Research topics


• Case study:



9

En-Route Air Traffic Control Centers

10

Cleveland Center

11

Cleveland Center Area 4

4645

12

Cleveland Center Area 4

47

48

49

4546

36,000

31,000

24,000Longitude

Latitude

Altitude

13

Sample Area of Specialization Resources

Sectors

47

48

49

4546 4546

4749

48

14


Sectors

Controllers

available to fill

operating

positions

47

48

49

4546 4546

4749

48

09:00–09:30 :

09:30–11:00 :

14


Sectors

Controllers

available to fill

operating

positions

Workstations

47

48

49

4546 4546

4749

48

09:00–09:30 :

09:30–11:00 :

48

14

Modeling Decisions and Developing Algorithms

fast-time

simulations

human-in-

the-loop

simulations

expert input

& feedback

operational

decision

data

solution

algorithm

decision-

support

tool

decision

model

constraints

objective

15


fast-time

simulations

15

Sample Sectors, Configuration, and Traffic

sample sectors:

16


sample sectors: at time step k:

Ck

16


Tk

sample sectors: at time step k:

Ck

16

Decision Model

Configuration Schedule Advisory Problem (CSA)

minimize

K∑

k=1

gk(Ck−1, Tk−1, Ck, Tk)

subject to Ck ∈ Ck, k = 0,1,2, . . . , K

17

Decision Model


minimize

K∑

k=1



gk: single-time step advisory cost

17

Decision Model


minimize

K∑

k=1



gk: single-time step advisory cost

Ck: set of valid configurations at k

17

Sample Scenario

0timestep1 2 3 4 5

18

Configuration Constraints

0timestep1 2 3 4 5

19

Configuration Constraints

0timestep1 2 3 4 5

C2

19

Possible Algorithm Advisory

0timestep1 2 3 4 5

20

Possible Algorithm Advisories

0timestep1 2 3 4 5

21

Possible Algorithm Advisories

0timestep1 2 3 4 5

25 = 32 possibilities

21

Advisory Cost: Static Cost

gSk(Ck, Tk)

20% 40% 60% 80% 100% 120%0

2

4

6

8

10

Open Sector Load[aircraft count/Monitor Alert Parameter]

Two−operating positionstatic costgS

k(Ck, Tk)

22

Advisory Cost: Static Cost

gSk(Ck, Tk)

20% 40% 60% 80% 100% 120%0

2

4

6

8

10


Two−operating positionstatic cost

One−operating positionstatic cost

gSk(Ck, Tk)

22

Advisory Cost: Reconfiguration Cost

gRk(Ck−1, Tk−1,Ck, Tk)

(1)

(1)

(2)

(1)

(2)

(2)

(1)

Ck−1, Tk−1 Ck, Tk

23



sector & flights moving

(1)

(1)

(2)

(1)

(2)

(2)

(1)


between workstations

23



open sector gaining

sector & flights moving

(1)

(1)

(2)

(1)

(2)

(2)

(1)


between workstations

2nd operating position

23



open sector gaining

open sector losingsector & flights moving

(1)

(1)

(2)

(1)

(2)

(2)

(1)


between workstations 2nd operating position


23



open sector gaining


(1)

(1)

(2)

(1)

(2)

(2)

(1)




single-time step advisory cost:

gk(Ck−1, Tk−1, Ck, Tk) = gSk(Ck, Tk)+β

RgRk(Ck−1, Tk−1, Ck, Tk)

23



open sector gaining


(1)

(1)

(2)

(1)

(2)

(2)

(1)




single-time step advisory cost:

gk(Ck−1, Tk−1, Ck, Tk) = gSk(Ck, Tk)+β

RgRk(Ck−1, Tk−1, Ck, Tk)

operational decision data & fast-time simulations

leveraged to select βR

23

Minimum-Cost Advisory

0timestep1 2 3 4 5

0

00 3

0 1

0 0

8

5

0

24


0timestep1 2 3 4 5

0

00 3

0 1

0 0

8

5

0

gSk(C1, T1)

24


0timestep1 2 3 4 5

0

00 3

0 1

0 0

8

5

0

gRk(C3, T3, C4, T4)

gSk(C1, T1)

24

Decision Model and Solution Algorithm


minimize

K∑

k=1

gSk(Ck, Tk) + βRgR

k(Ck−1, Tk−1, Ck, Tk)


25



minimize g(C,T)


25



minimize g(C,T)


Shortest path problem → use A∗ algorithm

25

Decision Model Relative to Previous Research:

Objectives and Constraints

AdvisoryCharacteristic

matchconfigurations to

traffic

few disruptivereconfigurations

number of opensectors

number ofoperatingpositions

26




Cano et al.(2007)

MB et al.(2008–2009)

Tien et al.(2010–2012)


traffic

constraintand

optimizeconstraint constraint


constraintnot

consideredconstraint


minimize minimizenot

considered


notmodeled

not modeled minimize

26




Cano et al.(2007)

MB et al.(2008–2009)

Tien et al.(2010–2012)

DecisionModel


traffic

constraintand

optimizeconstraint constraint optimize


constraintnot

consideredconstraint optimize


minimize minimizenot

consideredconstraint


notmodeled

not modeled minimize constraint

26

Decision Model vs. Operational Decisions:

Problem Instances for Fast-Time Simulations

47

48

49

4546 4546

4749

48

• traffic and configurations from

231 days in 2011 and 2012

• 6 am to midnight local time

• rolling horizon: implement first

hour of two-hour advisories

• five-minute time steps

27


Few Disruptive Reconfigurations

3 6 9 12 150

500

1000

1500

2000

2500

Number ofOpen Sector

Instances

Duration [hours]

operational

model

28


Match Configurations to Traffic

20% 40% 60% 80% 100% 120%0

2

4

6

8

10


Low HighJust Right

gSk(Ck, Tk)

29


Match Configurations to Traffic

Low Just Right High0

20

40

60

80

100

Percent ofOpen Sector-

Minutes

operational

model

30


fast-time

simulations

31

Experts’ Feedback and Suggestion

Limitations of decision model

32



1 difficult to quantify safe and efficient operations

32




2 does not model individual human controllers

32





Suggestions for solution algorithm

32






• return ∼ 3 good advisories

32







• one advisory: optimal for decision model

32








• other advisories: not too sub-optimal

32








• other advisories: not too sub-optimal

• distinct advisories

32

Advisory Difference Metric

Φ(C,C′)

Φ(C,C′) =

K∑

k=1

ϕ(Ck, C′k)

33

Advisory Difference Metric

Φ(C,C′)

Φ(C,C′) =

K∑

k=1

ϕ(Ck, C′k)

Configuration difference metric

ϕ(Ck , C′k) =

¨

1 if Ck and C′kcombine sectors differently

0 else

33

Multiple-Advisories Problem Statement

M ϵ-Optimal d-Distinct Configuration Schedule

Advisories Problem (M-ϵ-d-CSAs)

minimize

M∑

m=1

g(Cm, T)

subject to |CM| =M

Cmk∈ Ck k = 0,1,2, . . . , K, m = 1,2, . . . ,M

C1 ∈ C⋆

CSA( = optimal advisories for CSA)

g(Cm, T)− g(C1, T)

g(C1, T)≤ ϵ m = 2,3, . . . ,M

Φ(Cm, Cm′

) ≥ d ∀m,m′ where m 6=m′

34

Multiple-Advisories Problem Statement

M ϵ-Optimal d-Distinct Configuration Schedule

Advisories Problem (M-ϵ-d-CSAs)

minimize

M∑

m=1

g(Cm, T)

subject to |CM| =M

Cmk∈ Ck k = 0,1,2, . . . , K, m = 1,2, . . . ,M

C1 ∈ C⋆

CSA( = optimal advisories for CSA)

g(Cm, T)− g(C1, T)

g(C1, T)≤ ϵ m = 2,3, . . . ,M

Φ(Cm, Cm′

) ≥ d ∀m,m′ where m 6=m′

M-ϵ-d-CSAs is NP-complete

34

Algorithms for M-ϵ-d-CSAs

35


1 Sequential Distinct A∗ with Shortcuts (SDA∗-SC)

• novel algorithm based on A∗

35

Sequential Distinct A∗ (SDA∗)

Finding advisory C1:

C, T C1shortest path

algorithm

36


Finding advisory Cm:

C, T Cm

{C1, C2, . . . , Cm−1}

constrained

shortest path

algorithm

36



C, T Cm

{C1, C2, . . . , Cm−1}

constrained

shortest path

algorithm

constrained shortest path problem is NP-complete

36

Constrained Shortest Path Algorithm:

Forward Distinct A∗ (FDA∗)

For some λ ≥ 0, find C̃2 that minimizes Lagrangian

L(C2, λ) = g(C2, T)︸︷︷︸

advisorycost

+λ�

d− Φ(C1, C2)�

︸︷︷︸

similaritycost

37

Constrained Shortest Path Algorithm:

Forward Distinct A∗ (FDA∗)

For some λ ≥ 0, find C̃2 that minimizes Lagrangian

L(C2, λ) = g(C2, T)︸︷︷︸

advisorycost

+λ�

d− Φ(C1, C2)�

︸︷︷︸

similaritycost

Another shortest path problem → use A∗

37

FDA∗ and Duality

Proposition

If M = 2, ϵ =∞, and |C⋆CSA| = 1,

then FDA∗ implements the dual objective:

h(λ) =minimizeC2∈C

L(C2, λ)

38

FDA∗ and Duality

Proposition

If M = 2, ϵ =∞, and |C⋆CSA| = 1,

then FDA∗ implements the dual objective:

h(λ) =minimizeC2∈C

L(C2, λ)

Corollary: If M = 2, ϵ =∞, |C⋆CSA| = 1,

λ = λ⋆, and strong duality holds,

then C̃2 satisfies a necessary condition for C2⋆

(C̃2 = second advisory returned by FDA∗)

38



C,T Cm

{C1,C2, . . . ,Cm−1}

constrained

shortest path

algorithm

constrained shortest path problem is NP-complete

39



C,T Cm

{C1,C2, . . . ,Cm−1}

FDA∗

λ

39




40




2 Forward Backward Value Iterationwith Sequential Advisory Search (FBVISAS)

• novel algorithm based on value iteration

40

Forward Backward Value Iteration

with Sequential Advisory Search (FBVISAS)

0timestep1 2 3 4 5

41

Theoretical Guarantees for FBVISAS:

the Good News

42


the Good News

Simple M-ϵ-d-CSAs instance: M = 2, ϵ =∞, |C⋆CSA| = 1,

C1 = C2 = · · · = CK, and C0 ∈ C1

42


the Good News


C1 = C2 = · · · = CK, and C0 ∈ C1

Reconfiguration cost-dominated M-ϵ-d-CSAs instance:

reconfiguring never decreases the advisory cost

42


the Good News


C1 = C2 = · · · = CK, and C0 ∈ C1

Reconfiguration cost-dominated M-ϵ-d-CSAs instance:

reconfiguring never decreases the advisory cost

Proposition

If one exists, FBVISAS finds an optimal C2⋆ for

simple reconfiguration cost-dominated instances

42


the Bad News

Static cost-dominated M-ϵ-d-CSAs instance: βR = 0

43


the Bad News

Static cost-dominated M-ϵ-d-CSAs instance: βR = 0

Proposition

If d > 1, FBVISAS does not return any C2 for

simple static cost-dominated instances

43






44






3 Lowest-Cost Paths (LCP)

• finds optimal solution to relaxation• many efficient algorithms

44






3 Lowest-Cost Paths (LCP)

• finds optimal solution to relaxation• many efficient algorithms

4 Value Iteration Fraction Optimalwith Exhaustive Advisory Search

• finds optimal solution• not computationally efficient

44


fast-time

simulations

45

Evaluation of Algorithms on Small Instances

47

48

49

4546 4546

4749

48


two days in December 2011

• nine two-hour instances per day

• 18 total instances


• C requires same number of open

sectors as were used operationally

• request two advisories (M = 2)

• ϵ = 0.2 constraint on cost of C2

• d requires 30 minutes of different

airspace configurations

46

Small Instances: Properties of C2

SDA∗-SC FBVISAS LCP0

20

40

60

80

100

Percent ofInstances

optimal C2⋆

47

Small Instances: Properties of C2

SDA∗-SC FBVISAS LCP0

20

40

60

80

100

Percent ofInstances

optimal C2⋆

feasible C2

47

Evaluation of Algorithms on Realistic Instances

47

48

49

4546 4546

4749

48


231 days in 2011 and 2012




• 4158 total instances


• C only requires same initial

configuration as was used

operationally

• request three advisories (M = 3)

• ϵ = 0.5 constraint on cost

• d requires 30 minutes of different

airspace configurations

48

Realistic Instances: Advisory Costs

0.7 0.8 0.9 1.1 1.2 1.30

20

40

60

80

100

0.99 1.01

Percent ofInstances

Cost Ratio�g(C2 ,T)+g(C3 ,T) for SDA∗-SC

g(C2 ,T)+g(C3 ,T) for FBVISAS

�

SDA∗-SCworse

SDA∗-SCbetter

mean = 1.02

49

Realistic Instances: Computation Times

(on a Workstation)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Computation

Times

[seconds]

SDA∗-SC FBVISAS

50


fast-time

simulations

51


51

Operational Airspace Sectorization

Integrated System (OASIS)

52

OASIS Screenshot: Multiple Advisories

53

Human-in-the-Loop Simulations Set Up

• eight retired FAA

personnel

• four simulated

scenarios

54

Human-in-the-Loop Simulations Set Up

• eight retired FAA

personnel

• four simulated

scenarios

Three experimental conditions in which users

1 generate configuration schedule

2 select from among algorithm-generated advisories

3 select from among algorithm-generated advisories

and then modify

54

Human-in-the-Loop Simulations Results

[Lee et al., 2013]

55


[Lee et al., 2013]

Advisories enable safe and efficient operations

• average acceptability of selected

algorithm-generated advisories: > 4 out of 5

• user modifications were minor and led to

no significant improvement in acceptability

55


[Lee et al., 2013]






Providing multiple advisories added value

• in more than 60% of instances, users selected

second or third advisory

• when asked how many advisories they wanted the

tool to provide, users requested an average of 2.8

55


[Lee et al., 2013]






Providing multiple advisories added value

• in more than 60% of instances, users selected

second or third advisory

• when asked how many advisories they wanted the

tool to provide, users requested an average of 2.8

Computation times were rated as highly acceptable

55


56

Outline


• Research topics


• Case study:



57

Summary of Contributions

58



– developed prescriptive decision model– designed efficient algorithms to findlow-cost and distinct paths

– implemented algorithm in decision-support tool

58






– designed maximum-likelihood-based algorithm that– generated insights from operational decision data

58






– designed maximum-likelihood-based algorithm that– generated insights from operational decision data

3 Flow managers:Ground Delay Program implementation

– deployed supervised learning and inversereinforcement learning models that

– produced predictive capability and insights

58

Relevant peer-reviewed conference papers1 M. Bloem and P. Gupta, "Configuring Airspace Sectors with Approximate Dynamic

Programming," ICAS, September 2010.

2 M. Bloem and H. Huang, "Evaluating Delay Cost Functions with Airline Actions inAirspace Flow Programs," USA/Europe ATM R&D Seminar, June 2011.

3 M. Bloem and N. Bambos, "Coordinated Tactical Air Traffic and Airspace Management,"IEEE CDC, December 2011.

4 M. Bloem, M. Drew, C. F. Lai, and K. Bilimoria, "Advisory Algorithm for Scheduling OpenSectors, Operating Positions, and Workstations," AIAA ATIO, September 2012.

5 M. Bloem, H. Huang, and N. Bambos, "Approximating the Likelihood of Historical AirlineActions to Evaluate Airline Delay Cost Functions," IEEE CDC, December 2012.

6 M. Bloem and N. Bambos, "An Approach for Finding Multiple Area of SpecializationConfiguration Advisories," AIAA ATIO, August 2013.

7 M. Bloem and N. Bambos, "Ground Delay Program Analytics with Behavioral Cloningand Inverse Reinforcement Learning," AIAA ATIO, June 2014.

8 M. Bloem and N. Bambos, "Infinite Time Horizon Maximum Causal Entropy InverseReinforcement Learning," IEEE CDC, December 2014.

Relevant journal articles1 M. Bloem, P. Gupta, and P. Kopardekar, "Algorithms for Combining Airspace Sectors," Air

Traffic Control Quarterly, Vol. 17, No. 4, 2009.

2 M. Bloem, M. Drew, C. F. Lai, and K. D. Bilimoria, "Advisory Algorithm for Scheduling OpenSectors, Operating Positions, and Workstations," AIAA Journal of Guidance, Control, andDynamics, Vol. 37, No. 4, July–August 2014.

3 M. Bloem and N. Bambos, "Air Traffic Control Configuration Advisories fromNear-Optimal Distinct Paths," AIAA Journal of Aerospace Information Systems, Vol. 11,No. 11, November 2014.

4 M. Bloem and N. Bambos, "Ground Delay Program Analytics with Behavioral Cloningand Inverse Reinforcement Learning," AIAA Journal of Aerospace Information Systems,accepted on 21 December 2014, available online on 03 March 2015.

5 M. Bloem and N. Bambos, "Stochastic Models of Ground Delay ProgramImplementation for Prediction, Simulation, and Insight," Journal of AerospaceOperations, invited submission to special issue, expected publication in late 2015.

59

Acknowledgments

• Stanford

• NASA

• Family & friends

60

Questions?

Backup Slides

62

Why Improve Area Configuration Planning?

1 more efficient flights

2 fewer disruptions to area supervisors and

traffic managers

3 better controller staff management

4 prepare for increased flexibility in future operations

→ more complex planning

63


fast-time

simulations

64

Operational Decision Data Set and

Fast-Time Simulation Setup

47

48

49

4546 4546

4749

48


230 days in 2011 and 2012





• only airspace configurations

• constraint: only use typical

configurations

65

Static Cost versus Reconfiguration Cost

700 800 900 1000 11000

50

100

150

200

Total

Reconfiguration

Cost

Total Static Cost

operational

βR = 0.5

βR = 15

ratio(βR) =total reconfiguration cost(βR)

total static cost(βR)

error(βR) =��ratio(operational)− ratio(βR)

��

66

Selecting βR

0 1.75 5.0 10 155

10

15

20

25

Total Error

βR

set βR = 1.75

67

Uncertainty in Traffic Predictions:

Why Not Incorporated?

• Experts understand prediction errors

• Avoid inconsistency with other deterministic tools

68


Uncertain Future Aircraft Counts

0 5 10 15 200

0.05

0.1

0.15

0.2

0.25

Aircraft Count

Probability

prediction

distribution

69


Impact on Advisory Costs

Advisories generated using predicted traffic with realistic errors

are typically 2.5%–12.5% costlier

than advisories generated with perfectly-predicted traffic

70


Approximate Dynamic Programming Approach

• Finite time horizon Markov decision process (MDP)

– state: sector aircraft counts and predictions thereof– action: current configuration (not a schedule)– state transitions: based on prediction errors inoperational tool

– objective: similar to decision model (CSA)

• Rollouts algorithm performs 15% better than a

heuristic and within 2% of optimal

• Rollouts algorithm computes solutions fast enough

for real-time implementation

M. Bloem and P. Gupta, "Configuring Airspace Sectors withApproximate Dynamic Programming," ICAS, September 2010.

71


Other Approaches

• improve predictions with a statistical model

• discount contribution to cost by predicted aircraft

• stochastic shortest path problem

• risk-averse advisory

• larger MDP formulation

• robust set of advisories

72

SDA∗ with Shortcuts (SDA∗-SC)

Objectives:

1 computation time ≤ that of A∗ for each advisory

→ use data from reverse A∗ for C1

to find "shortcuts"

2 no re-tuning of λ per instance or advisory

→ normalize advisory cost and similarity cost terms

73

Forward Backward Value Iteration

with Sequential Advisory Search (FBVISAS)

• Use value iteration to find minimum-cost advisory

through each Ck ∈ Ck, k = 1, . . . ,K

• Sort these advisories from low to high cost

• Initialize set of advisories to return: CM ← ∅

• Loop:

– select next advisory– if advisory is sufficiently different from otheradvisories in C

M, add it to CM

– if |CM| =M, stop

74

Multiple Advisories Problem is NP-Complete

For an arbitrary instance of the independent set

problem on G(V,E), construct M-ϵ-d-CSAs instance:

• M = required number of independent nodes

• gk() = 0

• K = 1 and C1 = V

• d = 1

• configuration difference metric:

ϕ(Ck,C′k) =

¨

1 if Ck and C′kindependent in G(V,E)

0 else

∃ solution to the independent set instance

⇔ ∃ solution to the M-ϵ-d-CSAs instance

75

Computational Complexity

of Multiple Solutions Problem Algorithms

Algorithm Complexity

VIFOEAS O(n3K)a

FBVISAS O(n2K + nK log(nK + 1))

SDA∗-SC O(Mn2K log(nK +1))

Eppstein lowest-cost paths O(n2K + nK log(nK + 1) +M)

Suurballe lowest-cost node-disjoint paths O(Mn2K log(nK +1))b

a This does not include the complexity of searching through�|Cϵ |

M

�advisory

subsets.b This assumes that Dijkstra’s algorithm is used as a subroutine for findingshortest paths.

76

Airline Delay Cost Model Evaluation

decision-

support

tool

77


Motivation: how costly is a flight delay?

78



Operational decision data:

default assignment

ABC1

10:00

ABC2

10:30

Slot #60

11:00

Slot #90

12:00

78




airline-selected assignment

ABC1

10:00

ABC2

10:30

Slot #60

11:00

Slot #90

12:00

78




airline-selected assignment

ABC1

10:00

ABC2

10:30

Slot #60

11:00

Slot #90

12:00

Contribution: novel algorithm enables determination of

delay cost model and cost noise parameters that

maximize an approximation of likelihood of data

78


• developed novel algorithm for finding cost noise

parameters that maximize an approximation of the

likelihood of minimum-cost matching problem

instance solutions, given a candidate cost model

79






• evaluated more than ten proposed airline delay

cost models using hundreds of slot swap decisions

79








• determined delay cost model that maximizes

approximate likelihood of data for each airline

79








• determined delay cost model that maximizes

approximate likelihood of data for each airline

• estimated cost noise parameters for each delay

cost model for each airline

79

Ground Delay Program (GDP)

Implementation Modeling

decision-

support

tool

80




arrival rate control arrivals

start end per

airport time time hour

SFO 9:00 am noon 30

81





start end per


SFO 9:00 am noon 30

Motivation: predict & understand GDP implementation

81





start end per


SFO 9:00 am noon 30

Motivation: predict & understand GDP implementation

Contribution: produced predictive models and insights

by developing behavioral cloning and inverse

reinforcement learning models of GDP implementation

81



• evaluated behavioral cloning and inverse

reinforcement learning models of GDP

implementation at EWR and SFO using hundreds of

days of operational decision data

82







• demonstrated that a behavioral cloning model can

produce superior predictions of GDP

implementation

82









implementation

• determined that neither class of model provides

much evidence that conditions beyond those in the

next two hours impact GDP implementation

82









implementation

• determined that neither class of model provides

much evidence that conditions beyond those in the

next two hours impact GDP implementation

• estimated a reward function that provides insight

into metrics guiding GDP implementation

82

bloem defense.v16.slides

Documents

prescriptive decision

human decision makers

cleveland center area

sample area of spec

slots motivation

atm decisions

important civil aviation

time simulations human