the language, libraries and culture of python in meteorologyscipy in py-art the language, libraries...

The Language, Libraries and Culture of Python in Meteorology

JonathanHelmus

ArgonneNa*onalLaboratory,USA

This presentation has been created by UChicago Argonne, LLC, Operator of Argonne National Laboratory (“Argonne”). Argonne, a U.S. Department of Energy Office of Science laboratory, is operated under Contract No. DE-AC02-06CH11357. This research was supported by the Office of Biological and Environmental Research of the U.S. Department of Energy as part of the Atmospheric Radiation Measurement Climate Research Facility.

Technology Powers Scientific Discovery

§  Scien*ficadvancementsaredrivenbystate-of-the-arttechnologies.

§  Newtechnologiesleadthewaytoscien*ficbreakthroughsandamorecompleteunderstandingoftheworldaroundus.

TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA

2

Computers Prime Technology for Science

§  Computersarethemostsignificanttechnologyavailabletoscien*ststoday.

§  Theabilitytoperformfastcomputa*onshasvastlychangedmeteorologyandwillcon*nuetochangethefieldinthefuture.


3

Source:R.G.Strauchelal,JTech,1984,1,37ARMCSAPRRadarhRp://www.arm.gov/instruments/csapr

Programming: Controlling Computers for Science

§  Thedailyopera*onofscienceinvolvescomputers.

§  Computersrequirespecificdirec*onsintheformofsoUwaretorealizetheirpoten*altopowerscien*ficresearchandopera*ons.

§  TheabilitytocreatesoUwarecustomizedtoaspecificscien*fictaskis

oUenneeded.

§  Wri*ngsoUwarecanbea*meconsumingandburdensometask.–  Scien*stses*matethat35%oftheirresearch*meisspentdevelopingsoUware[1].–  Mostscien*stslearntodevelopsoUwareinformallyfromtheirpeers[2].–  Thechoiceofprogramminglanguageaffectsthe*merequiredfordevelopment.


4

[1]P.Prabhu,etal,ASurveyofthePrac*ceofComputa*onalScience,2011,10.1145/2063348.2063374[2]J.E.Hannay,etal,HowDoScien*stsDevelopandUseScien*ficSoUware,2009,10.1109/SECSE.2009.5069155.

Python is Ideal for Writing Scientific Software

§  Pythonistheidealprogramminglanguageforwri*ngsoUwaretomeetthecomputa*onalchallengesinmeteorologyandatmosphericscience.

§  DevelopingsoUwareinPythonissimpleandfastdueto:

–  Thedesignandphilosophyofthelanguage.

–  Theavailabilityofanumberofhigh-quality,efficientscien*ficPythonlibraries.

–  Thewelcoming,vibrant,andgrowingPythoncommunity.


5

Python: the Language

Python,theprogramminglanguageis:

§  Ahigh-levellanguage.–  Allowsonetofocusonthethetaskconceptsratherthanthedetailsofthecomputer.–  Makesdevelopmentsimpleandquick.

§  Interpretedandinterac*ve.–  Nocompila*onstep,speedsupdevelopmentcycle.–  Interac*veshell(REPL)fortes*nganddebugging.

§  Generalpurpose.–  CanbeusedtowritesoUwareacrossawiderangeofapplica*ons,notdomainspecific.–  Largeruseranddeveloperbase.

TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA6

Python: the Language

Pythonisanideallanguageforscien*ficprogrammingbecauseitis:§  Conciseandreadable

§  OpenSourceandFree

–  NocompanycontrolsthesoUware,nolicensingcosts.

§  Hasalargenumberofthird-partylibraries.–  PyPIcontainsmorethan72,000packages.–  Manyscience-focusedlibrariesexistcoveringavarietyoffields.

§  Easytointerfacewithexis*ngC,C++andFortrancode.–  ToolslikeCython,cffi,ctypes,andF2PYmakeinterfacingwithexis*ngcodesimple.


7

colors=['red','blue','green']forcolorincolors:print(color)

@colors=("red","blue","green");foreach(@colors){print"$_\n”;}

Python Perl

Python: the Challenges

Pythondoesfacesomechallenges:

§  Execu*onspeedisslowcomparedtosomeprogramminglanguages.–  Manytasksarenotlimitedbyexecu*onspeed.–  Op*onsexisttospeedupPythoncode.

§  Thelanguageisnotthebestatsometypesofparallelcomputa*ons.–  TheGlobalInterpreterLock(GIL)poseslimitsofmul*-threadedperformance.–  Handlingconcurrencyisnotasrobustassomelanguages(Go,Erlang,Clojure).–  Supportforcomputa*ononGPUsandotherhighlyparallelarchitecturesislimitedand

requiresaddi*onallibraries(Numba,PyCUDA)§  Thelanguageiscurrentlyinflux.

–  Python3isbackwardsincompa*blewithPython2.–  SomelibrariesdonotsupportPython3.


Scientific Python Libraries: Introduction

TheScien*ficPythonEcosystemorSciPystackisacollec*onofopensourcesoUwareforscien*ficcompu*nginPython.Someofthecorepackagesare:§  NumPy:Providespowerful,efficientmul*-dimensionalarraysinPython§  SciPy:Fundamentalnumericalalgorithmsforcommontasksinscience.§  matplotlib:comprehensivepublica*on-quality2Dplolng

§  Jupyter/IPython:Rich,interac*veinterfacesforprocessingdataandtes*ngideas.§  pandas:Highperformance,easytousedatastructures.§  SymPy:Symbolicmathema*csandcomputeralgebra


9

The Python ARM Radar Toolkit: Py-ART

§  Py-ARTisamoduleforvisualizing,correc*ngandanalyzingweatherradardatausingpackagesfromthescien*ficPythonstack.

§  DevelopmentbegantoaddresstheneedsoftheARMprogramwiththeacquisi*onofmulPplescanningcloudandprecipitaPonradars.

§  Theprojecthassincebeenexpandedtoworkwithavarietyofweatherradars,includingNEXRAD,andhasawideuserbaseincludingradarresearchers,weatherenthusiastsandclimatemodelers.

§  AvailableonGitHubasopensourcesoQware,arm-doe.github.io/pyart/.§  Condapackagesareavailableatanaconda.org/jjhelmus


10

Scientific Python Libraries: NumPy

§  NumPyisthefundamentalpackageforscien*ficcomputa*oninPython.

§  NumPyprovides:–  Powerful,efficient(fast)mul*-dimensionalarrayobject,thendarrayclass.–  RobustmethodsformanipulaPngthesearrays.–  Maskedandrecordarrayobjects.–  Rou*nesforlinearalgebra,Fouriertransform,andrandomnumbers.–  Comprehensive,well-wriRendocumenta*on,hRp://docs.scipy.org/doc/.

§  Otherscien*ficPythonpackagesbuildonNumPytoaddaddi*onalfeaturesandabili*es.


11

NumPy’s ndarray details

§  NumPy’scorefunc*onalityisprovidedbyitsndarrayobject.§  Andarrayisahomogeneous,stridedviewonaconPguousblockof

memory.

§  Althoughsimple,thendarrayisapowerfulconstructastheloca*onofunderlyingmemorycanbepassedtootherlanguages(C,C++,CythonandFortran)withouttheneedtocopydata.


4bytes 4bytes 4bytes

16bytes

16bytes

(3,4)ndarrayoffloat32strides(16,4)

ndarray views

§  SlicingaNumPyndarrayalmostalwayscreatesa“view”ofthedata.

§  Nocopyingofdataisneededwhenaccessingormodifyingviews.


13


16bytes

16bytes


8bytes

16bytes

a=np.zeros((3,4),np.float32)(3,4)ndarray

a[0,:](4,)ndarray

a[:2,::2](2,2)ndarray

NumPy in Py-ART

NumPyisusedextensivelythroughoutPy-ARTasndarraysaretheprimaryobjectsusedtostoreandmanipulatenumericaldata.


14

elifdata_type_name=='PHIDP2':out[:]=360.*(data.view('uint16')-1.)/65534.mask[data.view('uint16')==0]=Trueelifdata_type_name=='HCLASS2':out[:]=data.view('uint16')elifdata_type_name=='XHDR':#Herewereturnanarraywiththetimesinmilliseconds.returndata[...,:2].copy().view('i4')#onebytedatatypeselifdata_type_name[-1]!='2':#makeaviewoflefthalfofthedataasuint8,#thisistheactualraydatacollected,therighthalfisblank.nrays,nbin=data.shapendata=data.view('(2,)uint8').reshape(nrays,-1)[:,:nbin]

NumPy in Py-ART


15

#decoderunlengthencodingrle_size=radial_header['nbytes']*2rle=np.fromstring(buf2[pos:pos+rle_size],dtype='>u1')colors=np.bitwise_and(rle,0b00001111)runs=np.bitwise_and(rle,0b11110000)//16radial[:]=np.repeat(colors,runs)

Scientific Python Libraries: SciPy

SciPyisacollec*onofmathema*calalgorithmsandfunc*onswhichbuilduponNumPytoprovideefficientsolu*onstocommonnumericaltasks.SciPyisdividedintosubpackageswhichcoveranumberofscien*ficdomains:§  Imageprocessing§  Signalprocessing§  Interpola*on§  Spa*aldatastructuresandalgorithms§  Clustering§  Numericalintegra*on§  Differen*alequa*ons§  Sta*s*cs§  Sparsematrices


16

SciPy in Py-ART


17

SciPyfunc*onsforimageprocessing,numericalintegra*on,interpola*on,spa*alanalysisandsparsematrixstorageareallusedinPy-ART.

importscipy.ndimage…def_find_regions(vel,gfilter,limits):"""Findregionsofsimilarvelocity."""mask=~gfilterlabel=np.zeros(vel.shape,dtype=np.int32)nfeatures=0forlmin,lmaxinzip(limits[:-1],limits[1:]):#findconnectedregionswithinthelimitsinp=(lmin

Scientific Python Libraries: matplotlib

matplotlibisaplolnglibrarywhichworkswithNumPy.

§  Comprehensive2D,publica*onqualityplots.–  Mul*pleplottypes:line,scaRer,image,contours,pseudocolor,…–  Manyoutputformats:png,jpg,svg,ps,pdf,…

§  Alimitedsetof3Dplots.–  Line,scaRer,wireframe,tri-surface,contour,polygon,…

§  Plotscanbeexaminedinterac*velyorembeddedinapplica*ons.

–  ExploredatainaGUI–  ARTView:GUIviewerbuiltontopofthePy-ARTwhichembedsmatplotlibplots


18

matplotlib in Py-ART


19

importmatplotlib.pyplotaspltimportpyartradar=pyart.io.read('sgpxsaprrhicmacI5.c0.20110524.015604_NC4.nc')fig=plt.figure(figsize=(12,3))display=pyart.graph.RadarDisplay(radar)display.plot('reflectivity_horizontal',vmin=-20,vmax=40,cmap='pyart_NWSRef',title='XSAPR',colorbar_label='Refl.(dBZ)')display.set_limits(ylim=(0,15),xlim=(-42,42))plt.tight_layout()plt.savefig('figure.png')

Scientific Python Libraries: Jupyter/IPython

ProjectJupyter(previouslyIPython)isasetofrich,interac*veinterfacesandtoolsforprocessingdataandtes*ngideasinPython.§  UserInterfaces

–  JuypterConsole:Terminalbasedinterac*vePythonenvironment–  JuypterNotebook:Webbasedplauormforauthoringrichdocuments–  Bothhaveexcellentintegra*onwithmatplotlib

§  Kernels–  IPython:interac*vecompu*nginPython–  ipyparallel:Lightweightparallelcompu*ngwithnotebookintegra*on–  IJulia,IRKernel,IRuby,IPerl,…

§  Manyotherinteres*ngtools:nbviewer,nbconvert,nbgrader,jupyterhub…


20

Jupyter/IPython in Py-ART


21

Scientific Python Libraries: Cython

Cython§  PythontoCcodetranslator.§  GeneratesaPythonextensionmodule.§  CanbeusedtospeedupPythoncodeby

addingsta*ctypeinforma*on.§  AlsocanbeusedtointeractwithC/C++

func*onandclassesinexternallibraries.


22

Cfile(.c)

CCompiler(gcc)

Externallibraries(*.so,.a)

Cythonfiles(.pyx,.pxd)

Cython

CompiledPythonmodule(.so)

Cython in Py-ART: wrapping libraries


23

cdefexternfrom"rsl.h":ctypedefstructRadar:Radar_headerhVolume**vctypedefstructRadar_header:intmonth,day,yearinthour,minutefloatsec...ctypedefstructVolume:Volume_headerhSweep**sweep...

...

Radar*RSL_anyformat_to_radar(char*infile)...

voidRSL_free_volume(Volume*v)voidRSL_free_radar(Radar*r)

cimport_rsl_hcdefclassRslFile:

cdef_rsl_h.Radar*_Radarcdef_rsl_h.Volume*_Volumedef__cinit__(self,filename):self._Radar=_rsl_h.RSL_anyformat_to_radar(filename)ifself._RadarisNULL:raiseIOError('filecannotberead.')def__dealloc__(self):_rsl_h.RSL_free_radar(self._Radar)defget_volume(self,intvolume_number):rslvolume=_RslVolume()rslvolume.load(self._Radar.v[volume_number])returnrslvolume...propertymonth:def__get__(self):returnself._Radar.h.monthdef__set__(self,intmonth):self._Radar.h.month=month

_rsl_h.pxd _rsl_interface.pyx

Cython in Py-ART: speeding up Python code


24

107secondsvs.0.234seconds,x450performanceimprovement.

@cython.boundscheck(False)@cython.wraparound(False)def_fast_edge_finder(int[:,::1]labels,float[:,::1]data,intrays_wrap_around,intmax_gap_x,intmax_gap_y,inttotal_nodes):"""Returnthegateindicesandvelocitiesofalledgesbetweenregions."""cdefintx_index,y_index,right,bottom,y_check,x_checkcdefintlabel,neighborcdeffloatvel,nvelcollector=_EdgeCollector(total_nodes)right=labels.shape[0]-1bottom=labels.shape[1]-1forx_indexinrange(labels.shape[0]):fory_indexinrange(labels.shape[1]):label=labels[x_index,y_index]iflabel==0:continuevel=data[x_index,y_index]#leftx_check=x_index-1ifx_check==-1andrays_wrap_around:x_check=right#wraparoundifx_check!=-1:neighbor=labels[x_check,y_index]nvel=data[x_check,y_index]...#addtheedgetothecollection(ifvalid)collector.add_edge(label,neighbor,vel,nvel)

More Scientific Python Libraries…

§  pandas:datastructuresandanalysis§  xray:labeledarrayanddatasets§  Iris:meteorology/climatedatamodel

§  basemap:plot2Ddataonmaps§  pyproj:cartographictransforma*ons§  Cartopy:cartographictoolsforPython

§  netCDF4-python:ReadandwriteNetCDFfiles.§  h5py:ReadandwriteHDF5files

§  scikit-learn:machinelearning§  scikit-image:imageprocessing§  Andmany,manymore…


25

Python Community: Online

ThePythoncommunityprovidesawelcoming,vibrantandhelpfulculture.Muchofthecommunityinterac*onsoccuronline:§  Websites

–  Python.org:Documenta*on,tutorial,PyPIandawiki.–  Otherwebsites,scipy.org,pyaos.johnny-lin.com

§  Socialmedia–  Facebook,TwiRer,Google+,IRC,YouTube–  Blogs:planetpython.organdplanet.scipy.org.–  Podcasts:TalkPythontomeandpodcast.__init__

§  Mailinglists–  NearlyalltheSciPystackpackageshavetheirownmailinglists–  PyAOSmailinglist(pyaos.johnny-lin.com)


26

Python Community: in Person

ThePythoncommunityalsomeetsinperson.§  Conferences

–  SciPy–  PyCon–  Localandspecializedconferences(PyData,AMS)–  ConferencetalksoUenavailableatpyvideo.org

§  Localusergroups–  wiki.python.org/moin/LocalUserGroups

§  Meetupsandhackathons.–  python.meetup.com


Python Community: for Developers

§  Mailinglists–  Manyprojectshavea–devmailinglist

§  Socialcodingsites–  GitHub–  Bitbucket

§  NumFocus

§  Scien*ficPythonfocusedcompanies–  Con*nuumAnaly*cs–  Enthought


28

My Own Path Through the Python Community

§  Undergraduate-ChemistryatMichiganTechnologicalUniversity

§  Ph.D.atTheOhioStateUniversity-ChemicalPhysics.–  LearnedtoprograminPython–  Wrotenmrglue,alibraryforworkingwithNMRdatainPython

§  Post-docatUConnHeathCenter–  Con*nuedtoprograminPython–signalprocessingforNMR–  ARendedfirstSciPyconference

§  AdvancedAlgorithmsEngineeratArgonneNa*onalLaboratory–  DevelopmentleadofthewidelyusedopensourcePy-ARTproject.–  Contribu*ngtootherlibrariesintheSciPystack.–  ARendandat*mespresentatSciPy,PyCon,ChiPy,…

TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA 29

Careers in Meteorology with Python Programming

Doyouenjoyprogramming?Solvingproblems?Python?§  HighPerformanceCompu*ng(HPC)§  Informa*onProcessingTechnologies§  Instrumenta*onandprocessing§  DataAssimila*on§  NumericalWeatherPredic*on§  Climateandatmosphericmodeling

Regardlessofthecareerpathyouchoose,learningPythonwillenhanceyourpoten*al.


30

Ques*ons?

TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA 31

Resources for Learning Python

§  Websites–  Python.orgtutorial–  ScipyLectureNotes,www.scipy-lectures.org

§  Books–  “ThinkPython:HowtoThinkLikeaComputerScien*st”byAllenB.Downey–  “Effec*veComputa*oninPhysics:FieldGuidetoResearchwithPython”byAnthony

Scopatz,KathrynD.Huff

§  Inperson–  Tutorialsatconference(PyCon,SciPy,AMS)–  SoUwareCarpentry,soUware-carpentry.org

§  Videos–  ManyPythonconferencesmakestheirtutorialsavailableonYouTubeorpyvideo.org


32

the language, libraries and culture of python in meteorologyscipy in py-art the language, libraries...

Documents