University of Nebraska - LincolnDigitalCommons@University of Nebraska - LincolnPublications, Agencies and Staff of the U.S.Department of Commerce U.S. Department of Commerce

2017

Weighting strategy for the Fourth National ClimateAssessment (APPENDIX B)Benjamin SandersonNational Center for Atmospheric Research, bsande@ucar.edu

Michael WehnerLawrence Berkeley National Laboratory, MFWehner@lbl.gov

Follow this and additional works at: https://digitalcommons.unl.edu/usdeptcommercepub

This Article is brought to you for free and open access by the U.S. Department of Commerce at DigitalCommons@University of Nebraska - Lincoln. Ithas been accepted for inclusion in Publications, Agencies and Staff of the U.S. Department of Commerce by an authorized administrator ofDigitalCommons@University of Nebraska - Lincoln.

Sanderson, Benjamin and Wehner, Michael, "Weighting strategy for the Fourth National Climate Assessment (APPENDIX B)"(2017). Publications, Agencies and Staff of the U.S. Department of Commerce. 576.https://digitalcommons.unl.edu/usdeptcommercepub/576

Weighting strategy for the Fourth National Climate Assessment

(APPENDIX B)

Benjamin Sanderson, National Center for Atmospheric Research

Michael Wehner, Lawrence Berkeley National Laboratory

Citation: In: Climate Science Special Report: A Sustained Assessment Activity of the U.S.

Global Change Research Program [Wuebbles, D.J., D.W. Fahey, K.A. Hibbard, D.J. Dokken,

B.C. Stewart, and T.K. Maycock (eds.)]. U.S. Global Change Research Program,

Washington, DC, USA (2017), pp. 642-651.

Comments: U.S. Government work

Abstract

This document briefly describes a weighting strategy for use with the Climate Model Intercomparison Project, Phase 5 (CMIP5) multimodel archive in the 4th National Climate Assessment. This approach considers both skill in the climatological

performance of models over North America and the interdependency of models arising from common parameterizations or tuning practices. The method exploits information

relating to the climatological mean state of a number of projection-relevant variables as well as long-term metrics representing long-term statistics of weather extremes. The weights, once computed, can be used to simply compute weighted mean and significance

information from an ensemble containing multiple initial condition members from co dependent models of varying skill.

Our methodology is based on the concepts outlined in Sanderson et al. (2015), and the specific application to the Fourth National Climate Assessment (NCA4) is also described

in that paper. The approach produces a single set of model weights that can be used to combine projections into a weighted mean result, with significance estimates which also treat the weighting appropriately.

The method, ideally, would seek to have two fundamental characteristics: • If a duplicate of one ensemble member is added to the archive, the resulting mean

and significance estimate for future change computed from the ensemble should not change. • If a demonstrably unphysical model is added to the archive, the resulting mean

and significance estimates should also not change

CSSR 5OD: FINAL CLEARANCE Appendix B

Subject to Final Copyedit 28 June 2017642

Appendix B. Weighting Strategy 1

for the Fourth National Climate Assessment 2

Introduction 3

This document briefly describes a weighting strategy for use with the Climate Model 4Intercomparison Project, Phase 5 (CMIP5) multimodel archive in the 4th National 5Climate Assessment. This approach considers both skill in the climatological 6performance of models over North America and the interdependency of models arising 7from common parameterizations or tuning practices. The method exploits information 8relating to the climatological mean state of a number of projection-relevant variables as 9well as long-term metrics representing long-term statistics of weather extremes. The 10weights, once computed, can be used to simply compute weighted mean and significance 11information from an ensemble containing multiple initial condition members from co-12dependent models of varying skill. 13

Our methodology is based on the concepts outlined in Sanderson et al. (2015), and the 14specific application to the Fourth National Climate Assessment (NCA4) is also described 15in that paper. The approach produces a single set of model weights that can be used to 16combine projections into a weighted mean result, with significance estimates which also 17treat the weighting appropriately. 18

The method, ideally, would seek to have two fundamental characteristics: 19

• If a duplicate of one ensemble member is added to the archive, the resulting mean 20and significance estimate for future change computed from the ensemble should 21not change. 22

• If a demonstrably unphysical model is added to the archive, the resulting mean 23and significance estimates should also not change. 24

Method 25

The analysis requires an assessment of both model skill and an estimate of intermodel 26relationships— for which intermodel root mean square difference is taken as a proxy. The 27model and observational data used here is for the contiguous United States (CONUS), 28and most of Canada, using high-resolution data where available. Intermodel distances are 29computed as simple root mean square differences. Data is derived from a number of 30mean state fields and a number of fields that represent extreme behavior—these are listed 31in Table B.1. All fields are masked to only include information from CONUS/Canada. 32

CSSR 5OD: FINAL CLEARANCE Appendix B

Subject to Final Copyedit 28 June 2017643

The root mean square error (RMSE) between observations and each model can be used to 1produce an overall ranking for model simulations of the North American climate. Figure 2B.1 shows how this metric is influenced by different component variables. 3

[INSERT FIGURES B.1 AND B.2 HERE] 4

Models are downweighted for poor skill if their multivariate combined error is 5significantly greater than a “skill radius” term, which is a free parameter of the approach. 6The calibration of this parameter is determined through a perfect model study (Sanderson 7et al. 2016b). A pairwise distance matrix is computed to assess intermodel RMSE values 8for each model pair in the archive, and a model is downweighted for dependency if there 9exists another model with a pairwise distance to the original model significantly smaller 10than a “similarity radius.” This is the second parameter of the approach, which is 11calibrated by considering known relationships within the archive. The resulting skill and 12independence weights are multiplied to give an overall “combined” weight—illustrated in 13Figure B.2 for the CMIP5 ensemble and listed in Table B.2. 14

The weights are used in the Climate Science Special Report (CSSR) to produce weighted 15mean and significance maps of future change, where the following protocol is used: 16

• Stippling—large changes, where the weighted multimodel average change is 17greater than double the standard deviation of the 20-year mean from control 18simulations runs, and 90% of the weight corresponds to changes of the same sign. 19

• Hatching—No significant change, where the weighted multimodel average 20change is less than the standard deviation of the 20-year means from control 21simulations runs. 22

• Whited out—Inconclusive, where the weighted multimodel average change is 23greater than double the standard deviation of the 20-year mean from control runs 24and less than 90% of the weight corresponds to changes of the same sign. 25

We illustrate the application of this method to future projections of precipitation change 26under RCP8.5 in Figure B.3. The weights used in the report are chosen to be 27conservative, minimizing the risk of overconfidence and maximizing out-of-sample 28predictive skill for future projections. This results (as in Figure B.3) in only modest 29differences in the weighted and unweighted maps. It is shown in Sanderson et al. (2016b) 30that a more aggressive weighting strategy, or one focused on a particular variable, tends 31to exhibit a stronger constraint on future change relative to the unweighted case. It is also 32notable that tradeoffs exist between skill and replication in the archive (evident in Figure 33B.2), such that the weighting for both skill and uniqueness has a compensating effect. As 34such, mean projections using the CMIP5 ensemble are not strongly influenced by the 35weighting. However, the establishment of the weighting strategy used in the CSSR 36

CSSR 5OD: FINAL CLEARANCE Appendix B

Subject to Final Copyedit 28 June 2017644

provides some insurance against a potential case in future assessments where there is a 1highly replicated, but poorly performing model. 2

[INSERT FIGURE B.3 HERE] 3

4

CSSR 5OD: FINAL CLEARANCE Appendix B

Subject to Final Copyedit 28 June 2017645

TABLES 1

Table B.1: Observational Datasets Used as Observations.2Field Description Source Reference Years

TS Surface Temperature (seasonal)

Livneh,Hutchinson (Hopkinson et al. 2012; Hutchinson et al. 2009; Livneh et al. 2013)

1950-2011

PR Mean Precipitation (seasonal) Livneh,Hutchinson (Hopkinson et al. 2012; Hutchinson et al. 2009; Livneh et al. 2013)

1950-2011

RSUT TOA Shortwave Flux (seasonal)

CERES-EBAF (NASA 2011) 2000-2005

RLUT TOA Longwave Flux (seasonal)

CERES-EBAF (NASA 2011) 2000-2005

T Vertical Temperature Profile (seasonal)

AIRS* (Aumann et al. 2003)

2002-2010

RH Vertical Humidity Profile (seasonal)

AIRS (Aumann et al. 2003)

2002-2010

PSL Surface Pressure (seasonal) ERA-40 (Uppala et al. 2005)

1970-2000

Tnn Coldest Night Livneh,Hutchinson (Hopkinson et al. 2012; Hutchinson et al. 2009; Livneh et al. 2013)

1950-2011

Txn Coldest Day Livneh,Hutchinson (Hopkinson et al. 2012; Hutchinson et al. 2009; Livneh et al. 2013)

1950-2011

Tnx Warmest Night Livneh,Hutchinson (Hopkinson et al. 2012; Hutchinson et al. 2009; Livneh et al. 2013)

1950-2011

Txx Warmest day Livneh,Hutchinson (Hopkinson et al. 2012; Hutchinson et al. 2009; Livneh et al. 2013)

1950-2011

rx5day seasonal max. 5-day total precip.

Livneh,Hutchinson (Hopkinson et al. 2012; Hutchinson et al. 2009; Livneh et al. 2013)

1950-2011

CSSR 5OD: FINAL CLEARANCE Appendix B

Subject to Final Copyedit 28 June 2017646

Table B.2: Uniqueness, Skill and Combined weights for CMIP5 1

Uniqueness weight Skill Weight Combined

ACCESS1-0 0.60 1.69 1.02

ACCESS1-3 0.78 1.40 1.09

BNU-ESM 0.88 0.77 0.68

CCSM4 0.43 1.57 0.68

CESM1-BGC 0.44 1.46 0.64

CESM1-CAM5 0.72 1.80 1.30

CESM1-FASTCHEM 0.76 0.50 0.38

CMCC-CESM 0.98 0.36 0.35

CMCC-CM 0.89 1.21 1.07

CMCC-CMS 0.59 1.23 0.73

CNRM-CM5 0.94 1.08 1.01

CSIRO-Mk3-6-0 0.95 0.77 0.74

CanESM2 0.97 0.65 0.63

FGOALS-g2 0.97 0.39 0.38

GFDL-CM3 0.81 1.18 0.95

GFDL-ESM2G 0.74 0.59 0.44

GFDL-ESM2M 0.72 0.60 0.43

GISS-E2-H-p1 0.38 0.74 0.28

GISS-E2-H-p2 0.38 0.69 0.26

GISS-E2-R-p1 0.38 0.97 0.37

GISS-E2-R-p2 0.37 0.89 0.33

HadCM3 0.98 0.89 0.87

HadGEM2-AO 0.52 1.19 0.62

HadGEM2-CC 0.50 1.21 0.60

HadGEM2-ES 0.43 1.40 0.61

IPSL-CM5A-LR 0.79 0.92 0.72

IPSL-CM5A-MR 0.83 0.99 0.82

IPSL-CM5B-LR 0.92 0.63 0.58

MIROC-ESM 0.54 0.28 0.15

MIROC-ESM-CHEM 0.54 0.32 0.17

MIROC4h 0.97 0.73 0.71

MIROC5 0.89 1.24 1.11

MPI-ESM-LR 0.35 1.38 0.49

MPI-ESM-MR 0.38 1.37 0.52

MPI-ESM-P 0.36 1.54 0.56

MRI-CGCM3 0.51 1.35 0.68

MRI-ESM1 0.51 1.31 0.67

NorESM1-M 0.83 1.06 0.88

bcc-csm1-1 0.88 0.62 0.55

bcc-csm1-1-m 0.90 0.89 0.80

inmcm4 0.95 1.13 1.08

2

CSSR 50D: FINAL CLEARANCE Appendix B

1 FIGURES

2

3 Figure B.1: A graphical representation of the intermode1 distance matrix for CMIPS and

4 a set of observed values. Each row and column represents a single climate model (or

5 observation). All scores are aggregated over seasons (individual seasons are not shown) .

6 Each box represents a pairwise distance , where wann (red) colors indicate a greater

7 distance . Distances are measured as a fraction of dIe mean intemlodel distance in the 8 C:rvtIP5 ensemble. (Figure source: Sanderson et al. 20 16b).

Subject to Final Copyedit 647 28 June 2017

CSSR SOD: FI NAL CLEARANCE Appendix B

• HadGEM2 AD

~ HadGEM2-CC

0 , , HadGEM2- ES

.~ 0 ., , IPSL CM5e LR .. ~

MRI "CGCM3" 1.8 MRI-ESMl

inmcm4 0 0 CMCC CESM

~ CMCC- CM

1.6 CMCC- CMS

0 <1 , '. + g~8~~~~~2 • x 1.4 • o~ q.,~ ~ IPSL "CMSA LR

:E t;. IPSL -CMSA- MR ~ V MPI ESM LR ~ I> MPCESM:=MR

0 <l MPI ESM P

'" 1.2 •• ~~ o CESlA1 CAMS (f) it; o NorESfl1 M c o bee csm 1-1 m ~ o o ~ § ~:~gH~ CHEM ·c

Os q, ~~ w 1.0 <> t;. MIROC4h -E « CNRM_CM5 ~ 0* . 0 0 8 CSIRO Mk3 6 0

" MIROC5 --0 08 o ACCESS1_D Z

D ACCESS1_3 CCSM4

~o 8 CESM1 8GC 06 CESt .. WFASTCHEM x+ CanESM2

0, ~ 0. 5 ________ 0 BNU_ESM 0 bee csml 1

0.4 0 GIS"S_E2J-Cpl

~ GISS_E2_H_p2

0 0 3~ GISS_E2_Ryl GISS E2 R p2 0, 0 0 FGOAlS~2"""

0.4 0.5 0.6 0.7 0.8 0.9 1.0 o HadCM3 ~ Combined Weight 1

2 Figure B .2: Model skill and independence weights for the CMIPS archive evaluated over

3 the Nordl American domain. Contours show dIe overall weighting, which is the product

4 of the two individual weights . (Figure source: Sanderson et al. 2016b).

Subject to Final Copyedit 648 28 June 2017

CSSR 50D: FINAL CLEARANCE Appendix B

20

200 220 240 260 280 300 320

80 (b) Independence Weighted

60

40

20

200 220 240 260 280 300 320

BO

60

-1_5 -0.5 o Precipition change In mmlday (2080-2100) -(1980-2000)

1

2 Figure B3: Projections of precipitation change over North America in 2080- 2100,

3 relative to 1980--2000 under RCP8.5. (a) shows the simple "nweighted CMIPS multi-4 model average , using dIe significance methodology from (IPee 2013) , (b) Shows dIe

5 weighted results as oudined in Section 3 for models weighted by uniqueness only . and (c) 6 shows weighted results for models weighted by both uniqueness and skill. (Figure source: 7 Sanderson et al. 20 16b).

Subject to Final Copyedit 649 28 June 2017

CSSR 5OD: FINAL CLEARANCE Appendix B

Subject to Final Copyedit 28 June 2017650

REFERENCES 1

Aumann, H.H., M.T. Chahine, C. Gautier, M.D. Goldberg, E. Kalnay, L.M. McMillin, H. 2Revercomb, P.W. Rosenkranz, W.L. Smith, D.H. Staelin, L.L. Strow, and J. 3Susskind, 2003: AIRS/AMSU/HSB on the Aqua mission: Design, science objectives, 4data products, and processing systems. IEEE Transactions on Geoscience and Remote 5Sensing, 41, 253-264. http://dx.doi.org/10.1109/TGRS.2002.808356 6

Hopkinson, R.F., M.F. Hutchinson, D.W. McKenney, E.J. Milewska, and P. Papadopol, 72012: Optimizing Input Data for Gridding Climate Normals for Canada. Journal of 8Applied Meteorology and Climatology, 51, 1508-1518. 9http://dx.doi.org/10.1175/JAMC-D-12-018.1 10

Hutchinson, M.F., D.W. McKenney, K. Lawrence, J.H. Pedlar, R.F. Hopkinson, E. 11Milewska, and P. Papadopol, 2009: Development and Testing of Canada-Wide 12Interpolated Spatial Models of Daily Minimum–Maximum Temperature and 13Precipitation for 1961–2003. Journal of Applied Meteorology and Climatology, 48, 14725-741. http://dx.doi.org/10.1175/2008JAMC1979.1 15

IPCC, 2013: Climate Change 2013: The Physical Science Basis. Contribution of Working 16Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate 17Change. Cambridge University Press, Cambridge, UK and New York, NY, 1535 pp. 18http://dx.doi.org/10.1017/CBO9781107415324 www.climatechange2013.org 19

Livneh, B., E.A. Rosenberg, C. Lin, B. Nijssen, V. Mishra, K.M. Andreadis, E.P. Maurer, 20and D.P. Lettenmaier, 2013: A Long-Term Hydrologically Based Dataset of Land 21Surface Fluxes and States for the Conterminous United States: Update and 22Extensions. Journal of Climate, 26, 9384-9392. http://dx.doi.org/10.1175/JCLI-D-2312-00508.1 24

NASA, 2011: CERES EBAF Data Sets. 25https://ceres.larc.nasa.gov/products.php?product=EBAF-TOA 26

Sanderson, B.M., R. Knutti, and P. Caldwell, 2015: A Representative Democracy to 27Reduce Interdependency in a Multimodel Ensemble. Journal of Climate, 28, 5171-285194. http://dx.doi.org/10.1175/JCLI-D-14-00362.1 29

Sanderson, B.M., M. Wehner, and R. Knutti, 2016b: Skill and Independence weighting 30for multi-model Assessment. Geoscientific Model Development. 31https://doi.org/10.5194/gmd-10-2379-2017 32

Uppala, S.M., P.W. KÅllberg, A.J. Simmons, U. Andrae, V.D.C. Bechtold, M. Fiorino, 33J.K. Gibson, J. Haseler, A. Hernandez, G.A. Kelly, X. Li, K. Onogi, S. Saarinen, N. 34Sokka, R.P. Allan, E. Andersson, K. Arpe, M.A. Balmaseda, A.C.M. Beljaars, L.V.D. 35

CSSR 5OD: FINAL CLEARANCE Appendix B

Subject to Final Copyedit 28 June 2017651

Berg, J. Bidlot, N. Bormann, S. Caires, F. Chevallier, A. Dethof, M. Dragosavac, M. 1Fisher, M. Fuentes, S. Hagemann, E. Hólm, B.J. Hoskins, L. Isaksen, P.A.E.M. 2Janssen, R. Jenne, A.P. McNally, J.F. Mahfouf, J.J. Morcrette, N.A. Rayner, R.W. 3Saunders, P. Simon, A. Sterl, K.E. Trenberth, A. Untch, D. Vasiljevic, P. Viterbo, and 4J. Woollen, 2005: The ERA-40 re-analysis. Quarterly Journal of the Royal 5Meteorological Society, 131, 2961-3012. http://dx.doi.org/10.1256/qj.04.176 6