Scanned by CamScanner

B.ProjectSummary:

WeproposetoobtainanewLinuxmachine,toshadowandthenreplace(uponfailure)

ourcurrentagingmachine(weather.rsmas.miami.edu),asignificantgeographicalnodein

Unidata’sIDDnetwork.Weatherisalsoalargepublicdatarepository,ashowcasefor

UnidatatechnologiesbothtotheMiamiresearchcommunityandtotheonlinecommunity

atlarge.Thecontinuityofweather.rsmasisimportant:Forinstance,itisadvertizedinmy

YouTubevideosenthusingaboutUnidata’sIDVandTheMapesIDVCollection.That

CollectionnowincludesvisualizationsforournewLabManualsupplementtoapopular

synopticmeteorologytextbook(Lackmann).ThemanualgoesonsalesoonfromAMSpress,

andmanyusersoftheLackmanntextbook.Aweather.rsmasfailureasfallclassesstart

hittingtheserverwouldbeembarrassingandinconvenient,sowewanttoobtainahot

sparethissummer.Wehavehadonescare,andthereisaconcerningupswinginopaque

RAIDcontrollerwarningmessages.

Wewillputtheagingmachine(whichwillberefurbishedatMiamiexpenseuponfirst

failure)togooduse.Forintellectualmerit,itwillserveasabackupofoursubstantial(~50

TB)communitydatarepository,includinglargedatasetofferingsthatshowcaseUnidata

technologiestoclimate,oceanography,field-campaign,andotherpotentialuserbasesat

themarginofUnidata’straditionalrealtimeweathercommunity.Communityservice

functionssuchasInternetDataDistributionwillbesustainedandenhanced,asnew

capacitymayallowlongerrollingarchivesandlargernewLDMfeeds(GFS0.25deg,HRRR,

GOES-R).RedundantcapacitywillalsobeusedasatestplatformforthePI’s(collaborative

withUnidata)EarthcuberesearchprojecttointegrateUnidata’spowerfulIDVwithother

technologies(notebooksandrepositories),tomakeUnidatamorerelevanttostatistical

andclimaticresearch.High-capacityinternaltesting(withgoodwill)ofanysoftwarestack

iscrucialtoavoidspoilingthebroadercommunity’spatienceviaadisappointingrollout.

Thesparecapacitywillfacilitatesuchtesting,usingclassroomsassynchronousloads.

Thesestudents,atthebleedingedgeofJupyterNotebook-IDVintegration,willalsobe

entrainedintotheproductionofpromotionalmaterialssuchasthePI’sYouTubevideos.

C.ProjectDescription

WeproposetoobtainahotspareandreadyreplacementforouragingLinuxserver

(weather.rsmas.miami.edu).Thenewmachinewillbesetupasaquasi-cloneoftheexisting

machine,includingadailysynchronizedcopyofmanyTBofhighvaluedata,whichwas

itselfsetupasamotherlode.ucar.educlone,forrapidfailoverandminimalmaintenance

burden.Unidatastaffwhohelpmanagetheresourceforcommunityaswellaslocalgood

willhavecapableMiamisysadminstaffassistance,whereagoodworkingrelationship

existsalready.After4yearsofservice,theweather.rsmasmachineisgivingoffmoreand

moreRAIDwarnings.Itcouldstillhaveyearsofservicelife,butredundancyisneededto

preventamajorserviceinterruptionuponfailure.Infactwehavehadonescarealready

(twodrivesfailedintheRAID,requiringanervousweekortwotorebuild).Thevirtuesof

suchredundancyforreliability,includingcrossmounteddiskswithadailysynchronized

copyofmanyTBofthehighestvaluedata,willnotbebelaboredhere,buttheyare

significant.

Inthemeantime,beforefailureoftheoldsystem,theoverlappingcomputerandstorage

capacitywillbehighlyvaluedandputtogoodusetowardintellectualmerit,community

dataservices,andpromotionofUnidatatechnologiestobothresearchersinadjacentbut

under-engagedcommunities(climate,oceanography)andtheworldatlarge(through

YouTubevideos,andthehoped-forreliabilityofthePI’snewtextbook-relatedIDV

exercises,intheupcomingLackmann,Mapes,Tylelabmanual,see

https://tinyurl.com/LMTmanual.

SpecialConsiderationsfor2017

ThisproposaladdressesSpecialConsiderationitemsdetailedintheRFP

http://www.unidata.ucar.edu/community/equipaward/RFP2017.html,especiallythese:

• Production of online training materials (code notebooks, video tutorials, online documentation, or similar resources) that can be shared with the Unidata community

• Installation of equipment that provides student access to modern visualization tools such as AWIPS, IDV, or Jupyter notebooks

Inparticular,PIMapeshasasubstantialpresenceinonlineIDVtrainingmaterials.

BesidestheLMTLabManualdescribedabove,hehasproducedmanyIDV-basedresources,

underthenameMapesIDVCollection

(http://www.rsmas.miami.edu/users/bmapes/MapesIDVcollection.html).Theseresources

includewell-receivedintroductoryYouTubevideos,butalsoasubstantialdeeperbodyof

tools.Alibraryofcuratedbundlesisthemostvisible.Myself-updatingIDVpluginincludes

notonlyaFavoritesfolderpointingtothosebundles,butalsoadvancedformulasfor

statistics,filtering,interactivescatterplot,publication-qualityverticalsectionsona

pressurecoordinate,andmanyotherusefuladditionstotheIDV’sbasiccapabilities,aimed

atkeepingitrelevantformodernresearchandnotjustweatherdepiction.ThePIiseager

tokeepuptheeffort,butworriedaboutthefrailtyofitshardwareunderpinnings.Cloud

solutionsdonotseemreadyyettoreplace“baremetal”ofLinuxrackshostedlocally,but

thefrailtyofmycurrentmachine(froma2013Unidataequipmentgrant)iswearingonme.

AnothertrulynovelcontributionbythePI’sgroupistheIDVnotebook(seeexamples

andcodeathttps://github.com/suvarchal/JyIDV).ThisallowsanIDVsessiontobe

operatedandannotatedfromaJupyterNotebookenvironment(http://jupyter.org).This

couplingtransformsanIDVsessionfromapersonalexperienceintoadocumentedand

repeatableexploration.ThePIhascollaboratedwithcurrentandformerUnidatastaff,

includingsomeconsultingpaidatmyownexpenseaswellasanEarthcubegrantwithco-PI

YuanHoatUnidata(themainIDVdeveloper),tohelpfirmupthiscoupling,andbringIDV

stronglyintotheworldofJupyter.Thesetechnologieswillneedexplaining,orreally

showing,andcontinuityoftheplatformfortheirdevelopmentandtestingisessential.

Figure1:IDVNotebook,aninnovationfromthePI’sgroup(https://github.com/suvarchal/JyIDV).It

couplestheIDVtoJupyternotebookstocreateasaved,replicable,literate(human-reader-oriented)

wrapperaroundtheotherwiseephemeralGUIexperience.

Basedpartlyonthisprototype,thePIreceived(collaborativewithUnidata)anNSF-

EarthcuberesearchgranttofurtherintegrateUnidata’sIDVwithnotebooksand

repositories,tobuildasoftwarestackthatismuchmoreusefulthanitspartstoanalysis

challengesintheBigDataage.Specifically,ourgoalistofulfillthevisionofVisualizationfor

AlgorithmDevelopment,theoriginalbasisfortheIDV’sunderpinnings(VisADisinfactthe

correspondingJavalibraryname).Algorithmsareessentialtoextractmeaningfrombig

datasets,butedgecasesoftheirperformanceneedtobeexamined(Visualized)carefully,in

ordertofurthertheirDevelopment.Inthissense,wearefulfillingveryelementalgoalsat

thebaseofUnidata’shistory.Astablecomputingplatformwithsomeavailablecapacity

willallowbothresearchandclassroomtestingofthese“Unidata-plus”softwarestacks.

Classroomtestingisespeciallypowerful:Besidesbeingeducationalforthestudents,it

quicklyexposesjargonandworkaroundsthatexpertshaveceasedtonoticetheyareusing.

Researchusesoftheweather.rsmas.miami.eduRAMADDArespositorywillbesustained

bythesafebackupandenhancedwithanysurpluscapacity.Inadditiontotherealtime

weatherdatastreams,wehostanumberofdatasetsthatutilizeRAMADDA’sservicestoaid

communitygoals.Readersareinvitedtobrowseathttp://tinyurl.com/Miami-RAMADDA-

datasets,buthereisasample:

• Largenestedhurricane“naturerun”simulationdatasets,withassociatedIDVbundles–valuableforshowcasingmultiscalehurricanedynamics,andforcomputingandillustratingvortexdynamicsquantitiesonarealistic3Ddataset.

• Huge“Giga-LES”(2048x2048x128)convectionsimulationoutputs,andcoarse-

grainings(like256x256),allwithassociatedIDVbundlesdisplayingbothsets--valuableforturbulencestudies,allowingstatisticstobesuperposedonthegridpoint-leveldetailstheysummarize.

• DatasetsfromtheDYNAMO-AMIEfieldcampaign,allgatheredtogether,withIDVbundlesthatoverlayandjuxtaposethem.Whilethesedatacanbedownloadedonebyonefrom“catalogs”elsewhereontheweb,dataintegrationissomucheasierwhentheyarehostedonarepositorywithservices.

• EnergyandWaterCycledatasetsarchive,synthesizingmultipleestimatesofdifficulttoobservequantitiesinoneplaceforeasycomparison(incollaborationwiththeNASANEWSprogramanditsresearchcommunity).

• Oceanographicdatasets(comingsoon)

AcommonthemeisthatresearchcommunitiesatthemarginsofUnidata’s

traditionalmeteorologyrootsarebeingeducatedaboutthepowerofthesetools,andsome

ofthelimitationsoftheold“download,folderize,andwritecode”modelofsciencefordata

integration,anincreasinglyimportantpartofresearch.

Personnel,expertiseandinstitutionalsetting

ProfessorMapesisaneducatorandresearcherwithexperienceandskillsincomputing

andscientificprogramming.HeservesontheIDVsteeringcommittee,anduntilrecentlyon

Unidata’sStrategicAdvisoryCommittee.HeisalsoPIonanEarthCubeproject,withUM

oceanographerMohammedIskandaraniandUnidataco-PIYuanHo,designedtokeepthe

strengthsoftheIDVrelevantintheageofPythonandJupyternotebooks,assketched

brieflyabove.

TheDepartmentofAtmosphericScienceconsistsof12facultyandmorethan50

students,undergraduateandgraduate.Thereisnocentralcomputingforthisgroup.The

RosenstielSchoolofMarineandAtmosphericSciences(RSMAS)hasafacultyofabout80,

andtotalresearchcomputerusersinthehundreds.TheRSMASComputingFacility(RCF)

hasUniversity-leadingskillsduetothedata-intensivenatureofourfield.Inaddition,the

UniversitymaintainsaCenterforComputationalScience(CCS).Togetherthesefacilities

andexpertiseandservices(frompowertobandwidthtomaintenance)aremorethan

adequatetosuccessfullydeployandmaintaintheproposedpurchase.

D.BudgetThetotalrequestedfundingforthisproposalis$18,649,acapitalequipmentlineexempt

fromindirectcosts.Thisquotewasbasedontheexistingweather.rsmas.miami.edumachine,

andusesthesamevendor.Again,ourgoalislow-maintenancecontinuationofexisting

services,enablingasafeplatformforresearchandoutreachnovelties,whilecontinuingto

fosterthecoreUnidata-relatedcommunitygoods.

Server-roomrackspace,electricityandcoolingwillbeprovidedbytheSchoolasan

expenseofitsresearchandinstructionalmission.Optionally,theUniversityhasofferedto

hostthemachineatitsUniversity’sCenterforComputationalScience,inacrosstown

facilitywithmajorregional-infrastructuregradeinternetlinesandelectricalandphysical

security.Thisdecisionwillbemadeuponrecommendationforfunding,inconsultation

withexpertsatUnidata,theSchool,andtheUniversity.ThePIcontributestimeandtravel

costs,includingfrequentvisitstoUnidata,andisactive(tothepointofapersonalmission)

intransferringknowledgeaboutUnidatatechnologiesandservicestofaculty,studentsand

otherinterestedmembersoftheUM-RSMAScommunityandbeyond.Thistimeis

consideredaninvestmentinmyteachingandresearch.

E.ProjectMilestonesWeexpectatimelinelikethefollowing:

•May2017–Notificationofawardandpurchaseofquotedmachine.

•June2017–Installequipmentandsoftware,perhapsliterallycloninganimageofexisting

weather.rsmasasastartingpoint.ConsiderbroadeninganddeepeningIDD

subscriptions(LDMfeeds)toutilizeincreasedcapacities.

•July2017:Prepareandtestfailoverschemesforweather.rsmas.miami.eduIP.

•Summer2017:LMTtextbooklabmanualreleased...Inpreparationforautumnsemester

synopticmeteorologycourses,retestallmaterialsforreadinessandpreparebackup

plans,withAlbany(viacollaboratorK.Tyle)asafurtherfailsafe.Prepareandpostmore

YouTubevideosandothertrainingmaterials.

•Autumn2017:Problem-solveanyLMTtextbookmanualissuesastheyarise.

•Ongoing:Utilizeserverforhostingofadditionalopenlyshareddatasourcessuchashigh-

resolutionoceangrids,testingandsharingimprovementstotheMapesIDVCollection,new

capabilities(suchasIDVNotebook),etc.

•April2018–Submitafinalreport,intheformofashortarticle,totheUnidataProgram

Centerthatdescribes:1)Lessonslearnedaboutcomputersetupforfailureproofing,2)

ActivitiesandservicestotheMiamiresearchandbroadercommunity,madepossibleby

theavailabilityofoverlappingcapacity,3)Assessmentsonapproachesthat

worked/failedinordertoimprovetheprogram,and4)Recommendationsandbenefits

ofparticipationintheprogram.