CONCURRENCY: PRACTICE AND EXPERIENCE, VOL. 8( lo), 73 1-740 (DECEMBER 1996)

Experiences using high performance computing for operational storm scale weather prediction A. SATHYE, G. BASSETT, K. DROEGEMEIER, M. XUE AND K. BREWSTER

Centerji)r Analysis and Prediction of Storm, University of Oklahoma, Norman, OK 7301 9-0628, USA

SUMMARY The Center for Analysis and Prediction of Storms (CAPS) has developed a storm-scale predic- tion model named the Advanced Regional Prediction System (ARPS). CAPS has been testing the ARPS in an operational mode each Spring since 1993 to evaluate the model and to in- volve the operational community in CAPS’ development efforts. In this paper we describe our experiences with using high performance computing in a operational setting.

1. INTRODUCTION

The Center for Analysis and Prediction of Storms (CAPS) at the University of Oklahoma (OU) is a National Science Foundation (NSF) Science & Technology Center, which was one of the first 11 centers to be established in 1988. The center’s mission is to develop techniques for practical prediction of weather phenomena ranging from individual storms to storm complexes, and demonstrate the practicability of storm-scale weather prediction for operational, commercial and research applications. The center’s ultimate goal is to develop and test a fully functioning, multi-season storm-scale Numerical Weather Prediction (NWP) system around the turn of the century.

CAPS started a yearly evaluation and testing of its prediction system, known as the Advanced Regional Prediction System (ARPS)[ 11, in 1993. These tests, termed Cooper- ative Regional Assimilation and Forecast Test (CRAFT), were a means of involving the operational community in CAPS’ development efforts[2]. The tests in 1993 and 1994 were patterned after project STORMTIPE[3], while in 1995 ARPS was tested in a full NWP mode. The experiments were very useful in presenting the constraints and challenges of an operational environment.

Numerical weather prediction (NWP) problems place a heavy demand on computational and storage resources for modeling the complex behavior of the atmosphere. Severe storm models, which typically work on single digit kilometer scales of 1-3 km, increase the computation requirement multi-fold. Operational weather prediction, producing before- time weather forecasts, increases the complexity, given that storm lifetimes are usually less than a few hours. A complete storm prediction in real time is not limited just to the numerical prediction phase, but involves a series of pre-processing such as accessing the current observations, to post-processing, such as converting the model output into a form that end users can interpret. Other issues like data assimilation also add to the logistical complexity of the exercise, and to the turnaround time of an actual prediction.

CCC 1040-3 108/96/1oO73 1-10 01996 by John Wiley & Sons, Ltd.

Received 18 December 1995 Revised 22 March 1996

732 A. SATHYE ETAL.

High performance computing using Parallel Vector Processors (PVP) is quite common in the operational medium range weather prediction environment, e.g. the use of multiproces- sor Cray PVPs at the National Meteorological Center (NMC) and the European Center for Medium-Range Weather Forecasting (ECMWF). A significant improvement in the micro- processor technology, coupled with a similar increase in hardware and software reliability in emerging parallel architectures, a stagnation in PVP technology and the relative ease with which weather models adapt to parallel execution, first envisioned by L. F. Richardson in 1922[4], made a strong case for a move towards Massively Parallel Processing (MPP) technology[5-81.

2. THE ARPS MODEL

The ARPS is a non-hydrostatic atmospheric prediction model appropriate for use on scales ranging from a few meters to hundreds of kilometers[9,10]. It is designed as a scalable code that takes advantage of the parallelism inherent in fluid flow equations. The code has been designed in a highly modular form which allows for ease of modification, learning, and porting to various shared memory (SGI Power Challenge, Cray T3D using Cray Adaptive Fortran) and distributed memory machines (Cray T3D using Parallel Virtual Machine (PVM)/Message Passing Interface (MPI), IBM SP-2, Intel Delta, Thinking Machines CM5, network of workstations)[ 11-13].

The governing equations of the atmospheric model component of the ARPS include momentum, heat (potential temperature), mass (pressure), water substances, and the equa- tion of state. The ARPS solves prognostic equations for u, w, w, B’,p’ and q+, which are, respectively, the z, y and z components of the Cartesian velocity, the perturbation potential temperature and perturbation pressure, and the six categories of water substance (water vapor, cloud water, rainwater, cloud ice, snow and hail).

The governing equations are transformed into a terrain-following curvilinear co-ordinate and solved numerically using finite difference methods on a rectangular computational grid. The ARPS is second order accurate in time, and has a choice of being second or fourth order accurate in space. The model variables are staggered on an Arakawa C-grid with scalars defined at the center of the grid boxes and the normal velocity components defined on the corresponding box faces.

A mode splitting technique[ 141 is used for the time integration. For each step of time integration, u, w , w, p’ are integrated forward from t - At to t + At through n, number of small time steps, where At is the big time step, 2At = n&-, and AT is the small time step size. One boundary update is required for each of these variables during each small time step. The rest of the prognostic variables require one boundary update each large time step and an additional boundary update is necessary for fourth order advection during the large time step.

2.1. Parallel implementation

vpically atmospheric models, including ARPS, are composed of two main parts, the dynamics or hydrodynamics (i.e. the solution of equations for momentum, thermodynamic energy and mass) and the physics (which is concerned with solar radiation, surface physics, moisture precipitation, etc.). The dynamics part has data dependencies in both the horizontal and the vertical, while most physical processes have dependencies along a column of air, i.e.

STORM SCALE WEATHER PREDICTION 733

along the vertical. Additionally, the ARPS can use a vertically implicit solution technique. Given the penalties of distributing data along the vertical, we selected a horizontal two dimensional domain decomposition strategy.

As a means of porting our code to emerging standards (High Performance Fortran (HPF) and MPI), and to aid us in rapid prototyping and independence from local constraints, we focussed on translation tools which would help us preserve our original source form, while providing us the necessary performance gain. Previously, for a data parallel port, we developed a translator which converted Fortran 77 to array syntax, while inserting data distribution directives. This data parallel translator supports code conversions to the Cray Adaptive Fortran environment and HPF. For our initial message passing port a simple translator converted comment ‘markers’ into Fortran subroutine calls to the local message passing library. The initial version of the ‘markers’ were comments in English which would describe the action that had to be performed at that line. We wrote these comments in a standard format, and an auk translator was written to convert these comments, usually of the form ‘Send buffer X to Processor to the left’, into Fortran 77 code. Additional code modifications, such as initialization, were done by hand.

The translator included with the current ARPS source code uses message passing mark- ers contained in the ARPS Fortran source code (in the form of ‘cMP’ Fortran comments) to create a message passing version of the ARPS model. To keep the translator simple, it is not designed to search out on its own the portions of code needing changes to implement a message passing version. That task is carried out by the programmer, who then commu- nicates the specific tasks needed to the translator through the ‘cMP‘ markers in the source code. The translator is then used as a tool to perform the code modifications that have been identified for it.

There are markers for inserting, removing and modifying the original source code. Code immediately following a ‘cMP insert:’ is inserted into the source code. This marker is commonly used to include calls to message passing initialization routines and adding additional code necessary to carry out global reduction operations. The ‘cMP remove’ marker instructs the translator to remove the next Fortran command in the source code. The ‘cMP if’ marker is used to identify if statements which need to be altered so that they are executed only for physical boundaries and not internal boundaries between processors. Lastly, the ‘cMP bc’ marker is used to identify areas which implement boundary conditions so that the translator can add code to send data to, and receive data from, the neighboring processors. The translator currently generates calls to the Parallel Virtual Machine (PVM) or the Message Passing Interface (MPI). As a result, the model can be run on all platforms which support these environments, including the Cray T3D, the IBM SP-2, and networks of workstations. Since most parallel platforms lack VO support, each processor manages its own file read/write operations. We have created tools that split/merge the input/output data files, which carry out their operations based on the parameters specified in the model input file.

A Unix script, makearps, which was written to handle the entire translation, compilation and linking process, presents a common interface across all platforms. It can be controlled by user supplied options which specify the compilation options, external libraries to link, etc. Generally, a user can modify the Fortran code freely without much knowledge about message passing paradigms and practices, In the case that the modifications do affect boundary communication, the user can very easily learn to add the directives.

734 A. SATHYE ETAL.

3. 1993 AND 1994 WEATHER PREDICTION EXPERIMENTS AND LOGISTICS

The CRAFT-93 and CRAFT-94 predictions were conducted on a 2 h 8-processor dedicated queue on the Pittsburgh Supercomputing Center (PSC) Cray C90. The horizontal grid spacing was set at a constant 1 kin with 500 m vertical resolution over a 128 x 128 km2 domain. The model was initialized with a 10 km radius warm ‘bubble’ of air with the temperature increased to a maximum of 5 K. The bubble was placed in the center of the domain and its buoyancy triggered the initial moist convection. Aside from the bubble, the atmosphere was initially horizontally homogeneous with the vertical variability specified by a forecast sounding. No terrain was included in the model. Open boundary conditions were used for the lateral boundary, and explicit warm rain microphysics was included. The model initialization, execution and output retrieval was automated resulting in a complete 4 h prediction within 90 min.

The primary goal of these experiments was to gain experience in evaluating storm- scale models and the logistics of integrating them into an operational environment as a prototype for future forecast offices. Each day the Experimental Forecast Facility (EFF) forecasters would evaluate the potential for convection in the forecast domain. In the case that thunderstorms were possible they would alert CAPS personnel and prepare a forecast sounding. Based on the forecast sounding a prediction was generated by the CAPS scientist.

These experiments, though simplistic, were crucial in testing the procedures for real time prediction and uncovering a few problems with the model running in an operational setting. We were also able to focus on the need to tune the model for better efficiency and to develop output products tailored to the needs of operational forecasters.

4. VORTEX-95 OPERATIONS

4.1. Model configuration

The operational runs during the Spring of 1995 were closer to the storm-scale prediction concept envisioned by CAPS. The model runs were more comprehensive than previous years and an MPP was used, for the first time ever, in operational severe weather predic- tion[l,l5]. A 3 km resolution storm scale forecast was one way nested inside a 15 km resolution large domain run. These model runs used surface parameterization (soil type, vegetation type) and terrain data, and were initialized from observations and a National Meteorological Center (NMC), now known as National Center for Environmental Predic- tion (NCEP), forecast. Computing resources for these experiments were provided by the Pittsburgh Supercomputing Center (PSC). They provided us dedicated time on two proces- sors on their Cray C90, and a dedicated 256-node partition on their Cray T3D MPP system, which was used to demonstrate the feasibility of MPP machines operational weather pre- diction. The experiment was conducted in collaboration with the Verification of the Origins of Rotation in Tornadoes Experiment (VORTEX)[ 161.

4.1.1. Coarse grid

Two 6h forecasts were made with the ARPS on each operational day, between 23 April through 15 June. The first run utilized a 15 km horizontal grid resolution (coarse grid) over an area of 1200 x 1200 km2 centered over Western Oklahoma. In Figure I , most of Oklahoma is shown covered by the 3 km grid, while the 15 km grid includes parts of

STORM SCALE WEATHER PREDICTION 735

Figure 1. VORTEX-95 physical domain location: {he IS km grid was centered over western Ok- lahoma and covers parts of neighboring states. The 3 km grid, shown in one of the few optional

locations, was located within the OLAPS domain

neighboring states, primarily, Kansas to the north, Texas to the south, New Mexico to the west, and Colorado to the north-west. The vertical grid resolution for the 35 vertical levels varied from 100 m near the ground to 900 m at the top of the domain. The NMC 60 km Rapid Update Cycle (RUC) forecast produced at 122 (or Greenwich Mean Time) and valid at 18,21 and 0 Z (Ipm, 4pm and 7pm CDT) provided the initial and boundary conditions for this run. The initial fields were interpolated in space directly onto the ARPS grid, while the boundary conditions were interpolated in time throughout the forecast period. The coarse grid forecasts were run on a dedicated two processor queue on a Cray C90. The code achieved about 350 MFLOPS per processor and took about 17 min to produce a 6 h forecast. Post-processing and data transfer took about 28 min, and the forecast was ready at around 10:45am CDT. 45 min after the start time.

4.1.2. Fine grid

The second forecast utilized a 3 km horizontal grid resolution (fine grid) over a 336 x 336 km2 area, the location of which was based on a daily severe weather target area as decided by the VORTEX forecasters. The 3 km grid, shown in Figure 1, was placed within the Oklahoma Local Area Prediction System (OLAPS) domain and one way nested within the 15 km resolution domain described above. Similar to the coarse grid, the vertical grid resolution varied from 100 m near the ground to 900 m at the top of the domain over 35

736 A. SATHYE ETAL.

levels. Hourly data dumps from the 15 km grid forecast were used as boundary conditions for the fine grid run. The initial data on the fine grid was supplemented by analyses from the 10 km resolution OLAPS, which included data from the Oklahoma Mesonet and optionally from the KTLX (Twin Lakes, Oklahoma City) WSR-88D (previously called NEXRAD) radar (as shown in Figure 2)[ 17,181. The fine grid was run on a dedicated 256 node partition of a Cray T3D in about 74 min. Due to the lack of flexible YO support on the Cray T3D, post-processing of the data such as joining the individual processor files, plotting, etc. was done on the two dedicated processors of the Cray C90. The Cray C90 processor provides VO and file system support to the Cray T3D, and was, in the interest of time, the ideal choice. Post-processing and data transfer took about 46 min, and the forecast would be ready around 3:30pm CDT, 2 h after the start time.

4.2. Logistics

The model execution and product generation was automated through the use of UNIX shell scripts and cron tabs. A CAPS ‘duty scientist’ monitored the flow of execution and co-ordinated CAPS’ activities for that day.

Operational runs would begin with the creation of initial data for the coarse grid run from the 122 RUC output from NMC on a workstation at CAPS. Next, the initial data files would be remote copied to the PSC to be used with the coarse grid run. The same script would then launch the ARPS model and on completion would run utilities to convert the numerical output to graphical form. The output would then be transferred back to the CAPS workstation where it would be displayed on the World Wide Web (http : //wwwcaps . uoknor . edu/Forecasts). The gridded output data would also be converted to a format that could be easily manipulated by the forecasters, and sent to the VORTEX-95 operations center.

The fine grid runs began at 1:30pm CDT when the 18Z OLAPS data were available. These data were then converted on the CAPS cluster into a form readable by the ARPS, and then sent to the PSC. The various input data files, soil data, surface data, initial fields, and boundary data, which were either obtained from the coarse grid run or from independent sources, were then split so they could be read in by all the Cray T3D processors on an individual basis. The same script then submitted the ARPS fine grid job to the T3D and then waited for the output from the model. Each individual processor wrote its own history file, and the script joined them to create a single history file. The post-processing utilities would then perform the data conversion, image generation for the WWW page and data transfer to the VORTEX operations center, as was done for the coarse grid data.

5.

An operational model not only demands good computational and I/O performance, but depends heavily on software functionality and software and hardware reliability of the ma- chine. Traditionally, most operational weather prediction centers have used PVP machines largely due to their tried and trusted nature. Recent purchases by the National Oceano- graphic and Atmospheric Administration (NOAA), which bought a Cray Triton, and the ECMWF, which bought a Fujitsu VPP 500, indicate a strong reliance on PVP machines. However, increasing performance on MPP machines, and the lack of budgetary support, has forced groups to look at MPP machines. The High Performance Computing Section at

USING HPC FOR OPERATIONAL WEATHER PREDICTION

STORM SCALE WEATHER PREDICTION 737

VORTEX-95 ARPS Real-Time Flow Chart

Start ARPS

12 z 18Z OOZ 06CST '2 12CST 18CST

I I I I i I J

I I I I

12 z RUC init.

I I

1 1 1 1 1 1 15-km ARPS

. . . . . . . . . . . - - . . . . .t. -t. $y . . 3-km ARPS A

init. condit.

(I"'""") 0-3 h fcst \

w Observations

Figure 2. Operationalflowchart for VORTEX-95: the coarse grid run was initialized by the 1BZ RUC output. The$ne grid run was initialized using the output from OLAPS, and boundary conditions

were provided by hourly data dumps from the coarse grid forecast

the Forecast Systems Laboratory of the NOAA has been working on porting and testing the parallel versions of the Rapid Update Cycle (RUC) and Eta models in collaboration with the NMC. They have developed a library called the Nearest Neighbor Tool (recently renamed the Scalable Modeling System)[ 191, to speed up the porting process. A group consisting of researchers from Scandinavian countries has been working on porting the operational HIRLAM2[6] model onto various architectures and programming paradigms. Researchers at the ECMWF have ported the Integrated Forecast System model to parallel machines[20]. However, with the exception of a few models, and those too backing up the PVP run, none of the centers runs the parallel versions in an operational mode.

For CAPS, these experiments are a means to build a platform for a portable severe storm prediction environment, and the operational tests serve as a reality check. But due to the evolving nature of the model, the performance of the model suffers greatly and we are currently working with translation tools which will help bridge the conflicts arising from the same model being a research and a teaching tool, and also a production code.

738 A. SATHYE ETAL.

ARPS model timings on various parallel machines

1 10 Number of Processors

Figure 3. Comparison of ARPS erformance on various parallel machines: the timings shown above are for a 64 x 64 x 35 km domain (1 km horizontal resolution), and the points below the P

horizontal dashed line indicate model output faster than real time

The model performed at about 13 MFLOPS /processor, on the Cray T3D, during the real time experiment; about 10% of the machines peak performance. We have obtained far better performance on the IBM SP-2, as can be seen in Figure 3, but further performance will still need to increase for a full scale operational run. CAPS believes that a 1000 x 1000 x 32 km3 domain will be required to produce a regional forecast. The ARPS does about 5000; this number will increase once a full physics package is added, floating point computations per grid point per large time step. To produce a 6 h forecast on the ideal domain, within an hour, one would need a machine capable of performing at speeds exceeding 250 GFLOPS, sustained.

The Cray T3D lacks flexible parallel I/O support and we had to write our own tools to exchange data with the MPP. Parallel visualization tools do exist, such as Cray Animation Theater and Data Explorer, but most weather models rely on tested graphics libraries, which are available only on sequential machines. We used National Center for Atmospheric Re- search (NCAR) graphics, which forced us to rely on sequential high performance machines for our post-processing needs, which added to our time to prediction. We barely achieved 5% of our peak network bandwidth during the Spring of 1995, and the data transfer time threatens to dominate our turnaround time, based on our intention to make large domain runs.

6. CONCLUSIONS

CAPS conducts operational severe weather prediction experiments as a means of demon- strating the practicality of such an approach and as a means of reaching their goal of producing a fully fledged operational environment by the turn of the century. We have been successful in demonstrating the capabilities of high performance machines, more sig- nificantly MPP machines, in producing forecasts a few hours before the actual event. We

STORM SCALE WEATHER PREDICTION 739

have also demonstrated the high availability and reliability of the new generation of MPP machines. We have also succeeded in producing a single source code for all platforms due to our work on translation tools.

With the standardization of programming environments and paradigms (HPF and MPI), moving to the more suitable architecture or machine will be fairly simple. We are looking into translation tools to generate hardware specific optimized code to improve cache usage on MPP systems. We are also looking into simplifying our boundary condition strategy, which will also reduce the instances and the amount of communication.

A standardized high performance parallel VO library is required before we can use MPP’s to their full potential. Performance analyzers which report cache behavior too are necessary for improving single node performance and so are tools to translate Fortran 77 code to HPF and MPI.

ACKNOWLEDGEMENTS

CAPS is supported by grant ATM91-20009 from the NSF, and a supplemental award through the NSF from the Federal Aviation Administration. Computer resources were pro- vided by the Pittsburgh Supercomputing Center, which is also sponsored by the NSF. Fund- ing and support for VORTEX field operations was provided by the National Severe Storms Laboratory (National Oceanic and Atmospheric AdministrationEnvironmental Research Laboratory-NOAAERL), CAPS at OU (NSF Grant ATM9 1-20009), Cooperative Institute for Mesoscale Meteorological Studies, Office of Research Administration and Graduate College at OU, Southern and Central Region Headquarters of the National Weather Service (NWS), the National Severe Storms Forecast Center, the Experimental Forecast Facility at NWS/Norman, local N W S offices throughout the VORTEX region, NOAAERL Fore- cast System Laboratory, and the Mesoscale and Microscale Modeling and Atmospheric Technology Division of the National Center for Atmospheric Research.

We would also like to express our gratitude to the staff at the Pittsburgh Supercomputing Center, especially Ken McLain, Chad Vizino, David O’Neal and Sergiu Sanielevici. For providing us access to their machines, and going out of their way to solve our problems, we would like to thank Mike Tuttle of Cray Research Inc., Steve Anderson of the Laboratory for Computational Science and Engineering, University of Minnesota, the support staff of the Maui High Performance Computing Center and the Arctic Region Supercomputing Center. Locally, we would like to thank Sue Weygandt for drafting the figures and Norman Lin for the first version of the scripts to convert markers into parallel code.

REFERENCES 1. M. Xue, K. Brewster, F. Cam, K. Droegemeier, V. Wong, Y. Liu. A. Sathye, G. Bassett, P. Janish,

J. Levit and P. Bothwell, ‘Realtime numerical prediction of storm scale weather during Vortex- 95, Part 11: Operations summary and example predictions’, Preprints, 18th Con5 on Severe Local Storms, 19-23 February 1996, Amer. Meteor. SOC., San Francisco, CA.

2. P. R. Janish, K. K. Droegemeier, M. Xue, K. Brewster and J. Levit, ‘Evaluation of the advanced regional prediction system (ARPS) for storm scale operational forecasting’, Preprints, Conf: on Wearher Analysis and Forecasting, 15-20 January 1996, Amer. Meteor. SOC., Dallas, TX, pp. 224-229.

3. H. E. Brooks, C. A. Doswell I11 and L. J. Wicker, ‘STORMTIPE: A forecasting experiment using a three dimensional cloud model’, Weather Forecast., 8,352-362 (1993).

4. L. F. Richardson, Weather Prediction by Numerical Process, Cambridge University Press, 1922.

740 A. SATHYE ETAL.

5 .

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

L. Hart, T. Henderson and B. Rodriguez, ‘Evaluation of parallel processing technology for operational capability at the forecast systems laboratory’. Proceedings of the Sixth ECMWF Workshop on the Use of Parallel Processors in Meteorology, G. R. Hoffman and N. Kreitz (Eds.), World Scientific, London, UK, 1995. T. Kauranne, J. Oinonen, S. Saarinen, 0. Serimaa and J. Hietaniemi, ‘The operational HIRLAM 2 model on parallel computers’, Proceedings of the Sixth ECMWF Workshop on the Use of Parallel Processors in Meteorology, G. R. Hoffman and N. Kreitz (Eds.), World Scientific, London, UK, 1995. A. Dickinson, P. Burton, J. Parker and R. Baxter, ‘Implementation and initial results from a parallel version of the meteorological office atmosphere prediction model’, Proceedings of the Sixth ECMWF Workshop on the Use of Parallel Processors in Meteorology, G. R. Hoffman and N. Kreitz (Eds.), World Scientific, London, UK, 1995. A. Sathye, G. Bassett, K. Droegemeier and M. Xue, ‘Towards operational severe weather prediction using massively parallel processors’, Preprints, Proceedings of the International Conference on High Pet$ormance Computing, Tata McGraw Hill, India, 1995. M. Xue, K. Droegemeier and V. Wong, ‘The advanced regional prediction system and real-time storm-scale weather prediction’, Preprint, International Workshop on Limited-area and Variable Resolution Models, Beijing, China, 1995. M. Xue, K. Droegemeier, V. Wong, A. Shapiro and K. Brewster, ARPS Version 4.0 User Guide, Center for Analysis and Prediction of Storms, University of Oklahoma, 1995. K. Droegemeier, M. Xue, K. Johnson, M. O’Keefe, A. Sawdey, G. Sabot, S. Wholey and K. Mills’, ‘Experiences with the scalable-parallel ARPS cloud/mesoscale prediction model on massively parallel and workstation cluster architectures’, Proceedings of the Fith ECMWF Workshop on the Use of Parallel Processors in Meteorology, European Centre for Medium Range Weather Forecasting, Reading, England, 1992. K. W. Johnson, J. Bauer, G. A. Riccardi, K. K. Droegemeier and M. Xue, ‘Distributed processing of a regional weather model’, Mon. Weather Rev., 122,2558-2572 (1994). G. Sabot, S. Wholey, J. Berlin and P. Oppenheimer, ‘Parallel execution of a Fortran 77 weather prediction model’, Proceedings of the Supercomputing ’93 Conference, IEEE, Piscataway N. J., 1993. J. B. Klemp and R. B. Wilhelmson, ‘The simulation of three-dimensional convective storm dynamics’, J . Atmos. Sci., 38, 1558-1580 (1978). K. Droegemeier, M. Xue, A. Sathye, K. Brewster, G. Bassett, J. Zhang, Y. Liu, M. Zou, A. Crook, V. Wong, R. Carpenter and C. Mattocks, ‘Realtime numerical prediction of storm scale weather during Vortex-95: Goals and methodology’, Preprints, 18th Con$ on Severe Local Storms, 19-23 February 1966, Amer. Meteor. SOC., San Francisco, CA. E. N. Rasmussen, J. M. Straka, R. Davies-Jones, C. A. Doswell 111, F. H. Carr, M. D. Eilts and D. R. MacGorman, ‘Verification of the origins of rotation in tornadoes experiment’, Bull. Am. Meteorol. SOC., 75,995-1005 (1994). K. Brewster, E Carr, N. Lin, J. Straka and J. Krause, ‘A local analysis and prediction system for initializing realtime convective-scale models’, Preprints, loth Conference on Numerical Weather Prediction. 18-22 July 1995, Amer. Meteor. SOC., Portland, OR, pp. 596-598. K. Brewster, S. Albers, F. Carr and M. Xue, ‘Initializing a non-hydrostatic forecast model using WSR-88D data and OLAPS’, Preprints, 27th Con$ on RadarMeteor., 9-1 3 October 1995, Amer. Meteor. SOC., Vail, CO. 8. Rodriguez, L. Hart and T. Henderson, ‘A library for the portable parallelization of operational weather forecast models’, Proceedings of the Sixth ECMWF Workshop on the Use of Parallel Processors in Meteorology, G. R. Hoffman and N. Kreitz (Eds.), World Scientific, London, UK, 1995. S. R. M. Barros, D. Dent, L. Isaksen and G. Robinson, ‘The IFS model - overview and parallel strategies’, Proceedings of the Sixth ECMWF Workshop on the Use of Parallel Processors in Meteorology, G. R. Hoffman and N. Kreitz (Eds.), World Scientific, London, UK, 1995.