a new paradigm for environmental prediction · 2019. 5. 28. · a new paradigm for environmental...
TRANSCRIPT
A New Paradigm forEnvironmental Prediction
Charlie EwenExecutive Director Technology
Prof. Alberto ArribasHead Informatics Lab
Met Office (UK)
To work at the forefront of weather and climate
science for protection, prosperity and well-being“
”
UK
Government
International
community
World-leading
science
Commercial
business
It is not the gadget but the thinking that matters
Technology is something that doesn’t quite work yet“”Douglas Adams – Author: Hitchhikers Guide to the Galaxy
Technology is something that wasn’t
around when you were born“
”Alan Kay – Computer Scientist
If only…
ENTERPRISE IT
Derivative Data
Product Generation
Networks
Storage
Corporate IT
SCIENCE IT
Supercomputing
Research
NWP
Post Processing
Observations
Mass Storage
DIGITAL IT
Last mile IT
APP’s & API’s
Desktops (Not research)
If only…
ENTERPRISE IT
Derivative Data
Product Generation
Networks
Storage
Corporate IT
SCIENCE IT
Supercomputing
Research
NWP
Post Processing
Observations
Mass Storage
DIGITAL IT
Last mile IT
APP’s & API’s
Desktops (Not research)
Create Exploit
If only…
ENTERPRISE IT
Derivative Data
Product Generation
Networks
Storage
Corporate IT
SCIENCE IT
Supercomputing
Research
NWP
Post Processing
Observations
Mass Storage
DIGITAL IT
Last mile IT
APP’s & API’s
Desktops (Not research)
PUBLICCLOUD
Physical
App
Binary / Libs
OS
Server
Attachedstorage
Not a ‘cloud’ strategy…well, kind of
All options
have a place
Make active
choices
Very different
characteristics
Virtualisation
NAS
Server
Hypervisor (Type 1)
App
A.0
App
A.1
App
B.0
Binary
/Libs
Binary
/Libs
Binary
/Libs
Guest
OS
Guest
OS
Guest
OS
Containers
Co
nta
ine
r
Binary
/Libs
Binary
/Libs
Guest OS
Server
Ap
p A
.0
Ap
p A
.1
Ap
p B
.0
Ap
p B
.1
Ap
p B
.2
Serverless
Physical
App
Binary / Libs
OS
Server
Attachedstorage
Not a ‘cloud’ strategy…well, kind of
All options
have a place
Make active
choices
Very different
characteristics
Virtualisation
NAS
Server
Hypervisor (Type 1)
App
A.0
App
A.1
App
B.0
Binary
/Libs
Binary
/Libs
Binary
/Libs
Guest
OS
Guest
OS
Guest
OS
Containers
Co
nta
ine
r
Binary
/Libs
Binary
/Libs
Guest OS
Server
Ap
p A
.0
Ap
p A
.1
Ap
p B
.0
Ap
p B
.1
Ap
p B
.2
Serverless
Somebody else’s
data centre
Physical
App
Binary / Libs
OS
Server
Attachedstorage
Not a ‘cloud’ strategy…well, kind of
All options
have a place
Make active
choices
Very different
characteristics
Virtualisation
NAS
Server
Hypervisor (Type 1)
App
A.0
App
A.1
App
B.0
Binary
/Libs
Binary
/Libs
Binary
/Libs
Guest
OS
Guest
OS
Guest
OS
Containers
Co
nta
ine
r
Binary
/Libs
Binary
/Libs
Guest OS
Server
Ap
p A
.0
Ap
p A
.1
Ap
p B
.0
Ap
p B
.1
Ap
p B
.2
Serverless
Somebody else’s
data centreTransformation
Physical
App
Binary / Libs
OS
Server
Attachedstorage
Not a ‘cloud’ strategy…well, kind of
All options
have a place
Make active
choices
Very different
characteristics
Virtualisation
NAS
Server
Hypervisor (Type 1)
App
A.0
App
A.1
App
B.0
Binary
/Libs
Binary
/Libs
Binary
/Libs
Guest
OS
Guest
OS
Guest
OS
Containers
Co
nta
ine
r
Binary
/Libs
Binary
/Libs
Guest OS
Server
Ap
p A
.0
Ap
p A
.1
Ap
p B
.0
Ap
p B
.1
Ap
p B
.2
Serverless
Somebody else’s
data centreTransformation Disruption
Observations
What is a weather forecast?
ObservationsSimulation
Processing &interpretation
Thousandsof forecasts
Processing & Analysis
Millionsof predictions
Probability, confidence, impacts and guidance
ObservationsObservations
Simulation Processing & Analysis
Millionsof predictions
Probability, confidence, impacts and guidance
Nowcasting
IoT sensors
Assimilation
Resolving
Parameter
-isations
Contextual and
Impact
Predictions
Machines as
well as people
Quality
Assessments
Create Exploit
… My views, not my employers’
Heretical Views
Time / Effort
Revenu
e /
Gro
wth
1900s - 1950s. Navier-Stoke PDEs and Computers develop
1960-2010s. Incremental Improvement
Diminishing returns
2018. End of Moore’s Law
WHAT CAN A PHYSICIST LEARN AT BUSINESS SCHOOL?
1960s. NWP dominant design - PDEs on HPC
Time / Effort
Revenu
e /
Gro
wth
1900s - 1950s. Navier-Stoke PDEs and Computers develop
1960-2010s. Incremental Improvement
Diminishing returns
2018. End of Moore’s Law
1960s. NWP dominant design - PDEs on HPC
ML &ScalableCompute
WHAT CAN A PHYSICIST LEARN AT BUSINESS SCHOOL?
Time / Effort
Revenu
e /
Gro
wth
1900s - 1950s. Navier-Stoke PDEs and Computers develop
1960-2010s. Incremental Improvement
Diminishing returns
2018. End of Moore’s Law
1960s. NWP dominant design - PDEs on HPC
Paradigm Shift!
ML &ScalableCompute
WHAT CAN A PHYSICIST LEARN AT BUSINESS SCHOOL?
HERETICAL VIEW #1: HPC-POWERED PROGRESS IS OVER
[ Slide from Bryan Lawrence ]
MO storage
HERETICAL VIEW #1: HPC-POWERED PROGRESS IS OVER
Giving Scientists back their flow. Robinson, Niall et al. 2018. AGU Books.
People
HERETICAL VIEW #2: SCIENCE & SERVICES SLOWDOWN
HERETICAL VIEW #2: SCIENCE & SERVICES SLOWDOWN
Bauer et al. Nature. 2015
Looney Tunes. 1949
DOING THE SAME BUT BETTER IS NO LONGER ENOUGH
There is nothing (…) more perilous or more uncertain in its success (…) than to take the lead
in the introduction of a new order of things.
For the reformer has enemies in all those who profit by the old order, and only lukewarm
defenders in all those who would profit by the new order, this lukewarmness arising partly
from fear of their adversaries (…) and partly from the incredulity of mankind, who do not
truly believe in anything new until they have had actual experience of it.
NOT JUST SCIENCE / TECHNOLOGY BUT “SOCIAL COMPETITION”
Machiavelli, 1532
© 2018 Cray Inc.
Some Reasons Why Machine Learning is Being Applied
When Simulation Is
too Expensive
• Detailed simulation of subatomic particles interactions is essential to High Energy Physics at CERN.
• Monte Carlo approach is not fast enough for the High-Luminosity Large Hadron Collider needs.
• 3D convolutional GAN can generate realistic detector output >2000x faster.
Ref: Dr. Federico Carminati et al, CERN
When data is too big
• Satellites create more data than can be assimilated.
• Only a small % of available data is used today.
• “Deep learning object detection can be used to identify areas of atmospheric instability from satellite observation data, focus extraction of observations on these regions of interest.”
Ref: Jebb Stewart, NOAA, 2018 ECMWF workshop on HPC in Meteorology
[ Slide from Per Nyberg ]
© 2018 Cray Inc.
Convergence?
SIMULATION ARTIFICIAL INTELLIGENCE
Credit: “The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction” Eric Breck et al, Google, Inc.
• Workflows are bespoke: Use case and data-specific
• Code can be disposable
• ML-based system behaviour is not easily specified in
advance - depends on dynamic qualities of the data and on
various model configuration choices
• Maximize data movement-- scan/sort/stream all the data all
the time
• Simulation codes are universal, long lasting and evolve more
predictably
• Focused on making sure the code mirrors the physics
• Architecture specific optimizations are long lasting
• Tightly integrated processor-memory-interconnect & network
storage
[ Slide from Per Nyberg ]
Create Exploit
Explore
Informatics Lab
Strategic Innovation for Met Office Executive (since 2014)
WHAT DO WE DO?
Science Technology Design
www.informaticslab.co.uk
WHAT DO WE DO?
Build partnerships/networks
to develop solutionsBuild prototypes
to understand problems
We learn
WHAT DO WE DO?
PANGEO: Scalable, Interactive, Parallel, and Repeatable Data Science
=
http://pangeo.io/
…
USGS
Berkeley
MIT
UCLA
[ Slide from Joe Hamman ]
Currently Exploring: ML Nowcasting
[ Slide from
Suman Ravuri]
Currently Exploring: ML Nowcasting
[ Chen et al.
Neural ODE ]
[ Slide from Rachel Prudden ]
ill posed problem!
Spatial/Temporal structure is
essential
Use GP to approximate high-res distribution
Currently Exploring: Ultra-High-res Downscaling
AMAZON AI
Rainfall rate Cloud fraction Wet bulb potential temp.
[ Slide from Rachel Prudden ]
[ Slide from
Rachel Prudden ]
“Truth” Orignal Low-res
Reconstructed High-res Samples
Paradigm shift is here
Create
Exploit
Explore
“Social change”not just
Sci/Tech
Ecosystem: Met Office + Tech
+ Academia + Industry
