FUTURE DATA ANALYSIS TECHNIQUES FOR ATMOSPHERIC REMOTE SENSINGMEASUREMENTS: FIRST STEPS TOWARDS GRID COMPUTING

L. Hoffmann 1, M. Kaufmann1, C. Lehmann1, R. Spang1, M. Riese1, D. P. Moore2, and J. J. Remedios2

1Forschungszentrum Julich, ICG-I, 52425 Julich, Germany2University of Leicester, SRC, University Road, Leicester, LE1 7RH, United Kingdom

ABSTRACT

Within two projects it is studied how the rapid radia-tive transfer model JURASSIC (Juelich Rapid SpectralSimulation Code) performs in cluster computing environ-ments. We do so, having data analysis of future atmo-spheric satellite experiments in mind, which are likely toprovide 10 – 100 times more measurement data than to-days instruments. First studies presented here are basedon radiance measurement provided by Envisat MIPASduring its continuous measurement period in July 2002 toMarch 2004. We concentrate on retrievals of long-livedtrace gases and aerosol extinction.

Key words: Envisat MIPAS, fast forward modelling, op-timal estimation retrieval, chlorofluorocarbons (CFCs),aerosol extinction, grid processing (G-POD).

1. INTRODUCTION

Satellite remote sensing experiments provide a wealthof information on the Earth’s atmosphere. In its nomi-nal measurement mode Envisat MIPAS [1, 2] performedabout 1100 vertical soundings of mid-infrared limb ra-diance spectra per day. Several important atmosphericparameters were retrieved operationally from the MIPASmeasurements (pressure, temperature, volume mixing ra-tio of six key species) [3, 4, 5]. However, there is still alarge number of trace gases left, which are not covered bythe ESA operational retrievals due to limited resources.Future experiments, e.g. the I-MIPAS [6], currently un-der investigation in an ESA pre-phase A study, will pro-vide even more measurement data (applying a 2D detec-tor array, I-MIPAS is designed to perform around 500 000vertical soundings of mid-infrared limb radiance spectraper day). Retrieval of atmospheric parameters from thesecomprehensive measurements will be a challenge as ex-tensive data-processing resources are required.

The Juelich Rapid Spectral Simulation Code (JURAS-SIC) [7, 8] is a new mid-infrared radiative transfer model,recently developed at the research center in Julich. Eventhough the model relies on sophisticated techniques to ac-

celerate the computations involved, processing of satel-lite data still remains a costly task. About 100 days ofCPU-time on a conventional workstation are required toretrieve the abundance of a single trace gas from En-visat MIPAS measurements for the time period coveredby continuous measurements (July 2002 to March 2004,covering about 500 000 vertical scans).

Aiming on more comprehensive analysis of Envisat MI-PAS measurements and having future experiments inmind, we search for better methods to run the retrievals.First attempts were made to apply JURASSIC on a mas-sive parallel computer operated in Julich. Within theproject JURASSIC4GPOD our radiative transfer modeland an accompanying retrieval processor are imple-mented as applications in the ESA Earth ObservationGrid Processing-on-Demand environment. Within thesetwo projects retrievals of long-lived trace gases andaerosol extinction from Envisat MIPAS measurementsare envisaged. These data are required in many scien-tific applications (e.g. to study atmospheric transport pro-cesses or to validate atmospheric models). This text sum-marizes first outcomes of the projects.

2. JURASSIC FORWARD MODEL

The Juelich Rapid Spectral Simulation Code (JURAS-SIC) is a fast radiative transfer model for the mid-infraredspectral region (4 – 15 micron). Computations are basedon local thermodynamic equilibrium which applies in thetroposphere and stratosphere in most cases. Scatteringis neglected, i. e. forward model computations are re-stricted to cloud-free conditions. The main characteris-tics of JURASSIC are summarized in Tab. 1.

Fig. 1 illustrates the concept used by JURASSIC to fa-cilitate rapid radiative transfer calculations: Instead ofcomputing monochromatic transmittance and radiance ina line-by-line approach (Fig. 1a), spectral mean emissiv-ity is obtained from precomputed look-up tables and usedto compute mean radiance for selected spectral ranges(Fig. 1b). This approach allows to avoid the most time-consuming part of the radiative transfer calculations, i. e.the calculation of line strength and line profile for thou-

_____________________________________________________

Proc. ‘Envisat Symposium 2007’, Montreux, Switzerland 23–27 April 2007 (ESA SP-636, July 2007)

Table 1. List of basic characteristics of the JURASSICfast forward model.

Flexible handling of observation geometries:

• Interpolation routines for 1D, 2D or 3D atmosphericdata (e. g. individual vertical profiles, profiles alonga satellite track, or atmospheric model output)

• Observer within or outside Earth’s atmosphere

• Nadir, limb, or zenith viewing mode

Methods used for fast radiative transfer calculations:

• Look-up tables for spectral mean emissivity [9]

• Band Transmittance Approximation [9, 10]

• Emissivity Growth Approximation [9, 10, 11, 12]

• Continuum Approximation [9, 10]

Modelling of instrument effects:

• Arbitrary spectral response functions

• Arbitrary field of view (restricted to limb-mode)

• Radiometric offset- and gain-calibration

Retrieval interface:

• Definition of state-, parameter-, and measurement-vector to interface retrieval code

• Jacobians computed by means of numerical per-turbation (tangent altitude, pressure, temperature,trace gases, aerosol extinction, offset- and gain-calibration factors)

Tools based on the JURASSIC software library:

• Generic retrieval processor utilizing the standard op-timal estimation approach [10, 13, 14]

• Retrieval preprocessors for different instruments

• Post-processing tools (plotting, data distribution)

• Drivers for many library functions (interpolation ofatmospheric data, ray-tracing, data conversion, etc.)

a)

0

0.005

0.01

0.015

0.02

0.025

840 860 880 900 920 940

radi

ance

[W/(m

2 sr c

m-1

)]

wavenumber [cm-1]

mid latitudes / 15 km

totalCFC-11CFC-12Aerosol

b)

10

20

30

40

50

60

1e−06 1e−05 1e−04 0.001 0.01 0.1

tang

ent h

eigh

t [km

]

radiance (844−851 cm−1) [W/(m2 sr cm−1)]

mid latitudes

totalAerosol

CO2

CFC−11CFC−114

H2OHNO3

O3

OCS

Figure 1. a) Forward modelling for Envisat MIPAS basedon line-by-line computations utilizing the RFM. b) Com-putation of spectral mean radiance based on the emissiv-ity growth approximation utilized by JURASSIC.

sands of molecular emission lines. In addition, quite of-ten knowledge of detailed spectral structures is found notto contribute significant additional information to solve aretrieval problem.

JURASSIC was thoroughly optimized. The source codeitself was analyzed by means of profiling tools to findand remove computational bottlenecks. The benchmarktools also helped to select the best algorithms for individ-ual sub-problems during model development (e. g it wasfound that linear interpolation works far more effective inhandling emissivity look-up table data than the more real-istic logarithmic relationship between transmittance andemitter column amount).

Several studies were carried out to optimize differentmodel parameters. The most important parameter is theray-tracing step length, which in our model also definesthe atmospheric sampling along the ray paths. The re-lationship between step length and reciprocal computa-tional time is nearly linear over a wide range (Fig. 2a).Doubling the step length will reduce the CPU-time re-quired for forward model computations by a factor 2. Onthe other hand increased step lengths lead to increasedmodel errors as atmospheric inhomogeneity along the raypaths is not sampled properly (Fig. 2b). Sampling errorsare found to be sufficiently small for step lengths of 1 –

a)

0.01

0.1

1

10

100

0.1 1 10 100

mea

n C

PU

−tim

e (t)

[mse

c]

segment length (ds) [km]

mid latitudest[msec] = 4.89 / ds[km]

b)

0

0.2

0.4

0.6

0.8

1

0.1 1 10 100

radi

ance

err

or (o

ne s

igm

a) [%

]

segment length (ds) [km]

tropicsmid latitudes

polar summerpolar winter

Figure 2. Results of parameter optimization study for ray-tracing step length. Plots show dependence of a) CPU-time and b) model accuracy on step length.

10 km in limb mode and 0.1 – 1 km in nadir mode. Basedon these estimates the computational times are typically2 – 5 orders of magnitude below line-by-line referencecomputations (the actual speed-up factor depending onthe spectral width of the radiance channels involved).

Errors are introduced in the JURASSIC forward modelcomputations as approximation are applied to the radia-tive transfer equation. Due to the approximations spec-tral correlations between different emitters as well as cor-relations between transmittance and the Planck functionalong the ray paths are neglected. These correlationsare often small, but they depend on the distribution ofmolecular emissions lines in the specific spectral rangesas well as the atmospheric conditions (pressure, temper-ature, emitter amounts). Hence, the errors cannot be es-timated in general and model accuracy has to be deter-mined by reference model comparisons. JURASSIC hasbeen compared against the MIPAS Reference ForwardModel (RFM) [15, 16] and the Stand-alone AIRS Radia-tive Transfer Algorithm (SARTA) [17]. The JURASSICcomputations typically agree within 0.5 – 2 % with refer-ence model data, see Fig. 3 for an example.

JURASSIC was successfully applied in forward modeland retrieval studies for several remote sensing experi-ments, e. g. the satellite limb sounders Envisat MIPAS,GLORIA (proposed) [18, 19], I-MIPAS (proposed) and

10

20

30

40

50

60

−1 −0.5 0 0.5 1

tang

ent h

eigh

t [km

]

radiance difference (JURASSIC − RFM) [%]

tropicsmid latitudes

polar summerpolar winter

Figure 3. Comparison of forward model computations fordifferent climatological conditions made with JURASSICand the MIPAS Reference Forward Model.

HIRDLS [20, 21], the air-borne limb sounders CRISTA-NF [22, 23] and GLORIA-AB (in development), and thesatellite nadir sounder AIRS [24]. JURASSIC will beprovided to interested scientific users. For download andfurther documentation please register on this web site:https://jurassic.icg.kfa-juelich.de/JURASSIC

3. G-POD PROJECT

Within a call for proposals in fall 2006 JURASSIC hasbeen selected as one of ten scientific applications to beimplemented and operated in the ESA Earth Observation(EO) Grid-Processing on Demand (G-POD) environment[25]. Being a new G-POD application, JURASSIC willbe used to carry out comprehensive mass retrievals oflong-lived trace gases and aerosol extinction from EnvisatMIPAS measurements.

The ESA EO G-POD environment integrates high-speedconnectivity, distributed processing resources and pro-vides storage for large volumes of data (e. g. ESA datacatalogue or third-party data). It provides a generic in-frastructure where specific data handling and applicationservices are seamlessly plugged in. The environmentmainly aims to:

• Support science users for focused collaborations asneeded for calibration and validation, developmentof new algorithms, generation of high level andglobal products.

• Provide the reference environment for the genera-tion of systematic application products coupled witharchives and near real time data access.

The G-POD environment currently provides the follow-ing computing and storage resources:

• 64 Dual Pentium XEON 2.4 GHz CPUs

Figure 4. Web interface for the JURASSIC application onthe ESA EO G-POD environment. Find the web site at:http://eogrid.esrin.esa.int

• 27 Pentium III (Coppermine) 1000 MHz CPUs

• about 28 TByte of hard disk storage space

By mid of April 2007 the implementation phase of theJURASSIC4GPOD project is nearly completed [26, 27].The JURASSIC software was adapted to the specific re-quirements of the ESA EO G-POD environment. A webinterface was designed and implemented by ESA (Fig.4), allowing the user to select specific time periods of MI-PAS measurements for retrievals.

The retrieval setups (i. e. control files, correspondinglook-up tables, climatological data, etc.) need to be pro-vided by the G-POD users. We currently provide setupsfor the retrieval of the chlorofluorocarbons CFC-11 andCFC-12. Setups for aerosol extinction at selected wave-lengths are in preparation. Retrieval results and diagnos-tic output (error estimates, characterization of a priori in-fluence) are stored as new data products in the widelyused netCDF-format on the G-POD environment. A nicefeature of the new G-POD application is the automaticgeneration of quick-look plots for retrieval results of in-dividual orbits (see Fig. 5).

Within the production phase of the project (starting May2007) systematic processing of CFC-11 and CFC-12 re-trievals for all available MIPAS measurements made inthe time period July 2002 to March 2004 is envisaged.The project plan includes several validation activities toensure high quality of the new data products.

4. JUMP PROJECT

Since January 2004 the computing center at the researchcenter in Julich operates the massive parallel computerJUMP (Juelich Multiprocessor). The system is based onthe IBM p690 frame architecture. It contains 1312 CPUsand provides 8.9 T-FLOPS peak performance.

A parallelization strategy for the JURASSIC genericoptimal estimation retrieval processor was developed

a)

b)

c)

Figure 5. Example quick-look plots for a) CFC-11, b)CFC-12, and c)12.0µm aerosol extinction retrieval re-sults. Quick-look plots are automatically generated bythe JURASSIC application on the G-POD environment.

based on the Message Passing Interface (MPI). A sim-ple scheduling scheme is used to distribute individualretrievals to the available nodes. As no interactions be-tween the individual retrievals are involved, the problemscales nearly linear over a wide range.

The MPI version of the retrieval software was success-fully tested on a Linux cluster. However, implementationon the JUMP IBM architecture is still in preparation asproblems with software portability occurred.

5. SUMMARY

First studies were made to investigate how the rapid ra-diative transfer model JURASSIC performs in clustercomputing environments and on massive parallel com-puters. Adapting JURASSIC to the ESA EO G-POD en-vironment was successful. Within the production phaseof the JURASSIC4GPOD project new Envisat MIPASatmospheric data products will be generated (CFC-11,CFC-12, aerosol extinction). We also expect to get bet-ter performance estimates for our software. Implemen-tation of JURASSIC on the Julich massive parallel com-puter JUMP is still in preparation as porting of the exist-ing software to this architecture was found to be rathercomplicated compared to standard Linux clusters.

ACKNOWLEDGMENTS

We gratefully acknowledge the work and assistance pro-vided by O. Colin and E. Mathot, ESA to implementJURASSIC in the ESA EO G-POD environment. We ac-knowledge the support given by D. Koschmieder and I.Gutheil, Forschungszentrum Julich to adapt JURASSICon the JUMP. We thank A. Dudhia, University of Oxfordfor providing the MIPAS Reference Forward Model.

REFERENCES

[1] H. Fischer and H. Oelhaf. Remote sensing of ver-tical profiles of atmospheric trace constituents withMIPAS limb-emission spectrometers.Appl. Optics,35:2787–2796, 1996.

[2] M. Endemann. MIPAS instrument concept and per-formance. InProceedings of ESAMS ’99 - Eu-ropean Symposium on Atmospheric Measurementsfrom Space, pages 29–43, 1999.

[3] Herbert Nett. MIPAS data products. InProceed-ings of ESAMS ’99 - European Symposium on At-mospheric Measurements from Space, pages 45–49,1999.

[4] Marco Ridolfi, Bruno Carli, Massimo Carlotti,Thomas von Clarmann, Bianca M. Dinelli, AnuDudhia, Jean-Marie Flaud, Michael Hopfner,Paul E. Morris, Piera Raspollini, Gabriele Stiller,

and Robert J. Wells. Optimized forward model andretrieval scheme for MIPAS near-real-time data pro-cessing.Appl. Optics, 39:1323–1340, 2000.

[5] B. Carli, D. Alpaslan, M. Carlotti, E. Castelli,S. Ceccherini, B. M. Dinelli, A. Dudhia, J. M.Flaud, M. Hopfner, V. Jay, L. Magnani, H. Oel-haf, V. Payne, C. Piccolo, M. Prosperi, P. Raspollini,J. Remedios, M. Ridolfi, and R. Spang. First resultsof MIPAS/ENVISAT with operational level 2 code.Adv. Space Res., 33:1012–1019, 2004.

[6] B. Kerridge. Premier - process exploration throughmeasurements of infrared and millimetre-waveemitted radiation. Proposal to ESA, 2005.

[7] Lars Hoffmann. Schnelle Spurengasretrieval fur dasSatellitenexperiment Envisat MIPAS. Technical Re-port JUEL-4207, Forschungszentrum Julich, Ger-many, 2006. ISSN 0944-2952.

[8] L. Hoffmann, R. Spang, M. Kaufmann, andM. Riese. Envisat MIPAS measurements of CFC-11and CFC-12: an add-on to ESA operational data. InProceedings of the First ESA Atmospheric ScienceConference, number ESA SP-628, Frascati, Italy,2006.

[9] Benjamin T. Marshall, Larry L. Gordley, andD. Allen Chu. BANDPAK: Algorithms for model-ing broadband transmission and radiance.J. Quant.Spectrosc. Radiat. Transfer, 52:581–599, 1994.

[10] Clive D. Rodgers.Inverse Methods for AtmosphericSounding: Theory and Practice, volume 2 ofSerieson Atmospheric, Oceanic and Planetary Physics.World Scientific, 2000.

[11] Michael P. Weinreb and Arthur C. Neuendorffer.Method to apply homogeneous-path transmittancemodels to inhomogenous atmospheres.J. Atmos.Sci., 30:662–666, 1973.

[12] Larry L. Gordley and James M. Russel. Rapid in-version of limb radiance data using an emissivitygrowth approximation.Appl. Optics, 20:807–813,1981.

[13] Clive D. Rodgers. Retrieval of atmospheric temper-ature and composition from remote measurementsof thermal radiation. Rev. Geophys. Space Phys.,14:609–624, 1976.

[14] Clive D. Rodgers. Characterization and error analy-sis of profiles retrieved from remote sounding mea-surements.J. Geophys. Res., 95:5587–5595, 1990.

[15] Thomas von Clarmann, M. Hopfner, B. Funke,M. Lopez-Puertas, A. Dudhia, V. Jay, F. Schreier,M. Ridolfi, S. Ceccherini, B.J. Kerridge, J. Reburn,and R. Siddans. Modelling of atmospheric mid-infrared radiative transfer: the AMIL2DA algorithmintercomparison experiment.J. Quant. Spectrosc.Radiat. Transfer, 78:381–407, 2003.

[16] A. Dudhia. RFM Software User’s Manual. De-partment of Atmospheric, Oceanic and PlanetaryPhysics, University of Oxford, United Kingdom,2004. http://www.atm.ox.ac.uk/rfm.

[17] L. L. Strow, S. E. Hannon, S. De-Souza Machado,H. E. Motteler, and D. C. Tobin. Valida-tion of the Atmospheric Infrared Sounder ra-diative transfer algorithm. J. Geophys. Res.,111:doi:10.1029/2005JD006146, 2006.

[18] M. Riese, F. Friedl-Vallon, R. Spang, P. Preusse,C. Schiller, L. Hoffmann, P. Konopka, H. Oelhaf,T. von Clarmann, and M. Hopfner. GLObal limbRadiance Imager for the Atmosphere (GLORIA):Scientific objectives.Adv. Space Res., 36:989–995,2005.

[19] F. Friedl-Vallon, M. Riese, G. Maucher, A. Lengel,F. Hase, P. Preusse, and R. Spang. Instrument con-cept and preliminary performance analysis of GLO-RIA. Adv. Space Res., 37:2287–2291, 2006.

[20] J. Gille and J. Barnett. Conceptual design ofthe High Resolution Dynamics Limb Sounder(HIRDLS) for the EOS Aura Mission. InProceed-ings of SPIE, volume 2830, pages 190–201, 1996.

[21] M. A. Dials, J. C. Gille, J. J. Barnett, and J. G. Whit-ney. Description of the High Resolution DynamicsLimb Sounder (HIRDLS) instrument. InProceed-ings of SPIE, volume 3437, pages 84–91, 1998.

[22] A. Kullmann, M. Riese, F. Olschewski, F. Stroh, andK. U. Grossmann. Cryogenic Infrared Spectrome-ters and Telescopes for the Atmosphere - New Fron-tiers. InProceedings of SPIE, volume 5570, pages423–432, 2004.

[23] Fred Stroh, Armin Afchine, Jochen Barthel, Ver-ena Cals, Vincenzo Fracassi, Klaus Großmann,Hans-Peter Heuser, Achim Heynen, Martin Kauf-mann, Andreas Kullmann, Peter Knieling, Fried-helm Olschewski, Peter Preuße, Martin Riese,Axel Schonfeld, Reinhold Spang, Nicole Spel-ten, Vicheith Tan, Katja Weigel, and Jorg Wolters.CRISTA-NF: development and perfomance of anew airborne mid-IR limb emission instrument.Adv. Space Res., 2007. in preparation.

[24] Hartmut H. Aumann, Moustafa T. Chahine, Cather-ine Gautier, Mitchell D. Goldberg, EugeniaKalnay, Larry M. McMillin, Hank Revercomb,Philip W. Rosenkranz, William L. Smith, David H.Staelin, L. Larrabee Strow, and Joel Susskind.AIRS/AMSU/HSB on the Aqua mission: Design,science objective, data products, and processingsystems. InIEEE Transactions on Geoscience andRemote Sensing, volume 41, pages 253–264, 2003.

[25] ESA G-POD Team. Earth Observation GridProcessing-on-Demand Portal User Manual. Tech-nical Report GRID-GODS-EOPG-MA-06-0004,European Space Agency, 2006.

[26] ESA G-POD Team. Earth Observation GridProcessing-on-Demand, application design for G-POD project 3973 (JURASSIC for the ESAEarth Observation G-POD). Technical Re-port GRID-GDOS-EOPG-DD-07-0003, EuropeanSpace Agency, 2007.

[27] ESA G-POD Team. Earth Observation GridProcessing-on-Demand, implementation plan for

G-POD project 3973 (JURASSIC for the ESAEarth Observation G-POD). Technical ReportGRID-GDOS-EOPG-PL-07-0004, European SpaceAgency, 2007.