1

Transcriptionof“TheGreatDebate”:JohnGustafsonvs.WilliamKahanonUnumArithmeticHeldJuly12,2016ModeratedbyJimDemmelDEMMEL:Asyouknow,JohnGustafsonrecentlypublishedabook,calledTheEndofError,Unum,inwhichheproposesanewfloating-pointformat,called“unums,”

whichisshortfor“universalnumber,”and,asareplacementforconventionalfloatingpoint.Andasthebook’stitlesuggests,hemakessomeratherstrongclaimsaboutwhatunumcando.Soforexample,thattheyareeasiertousethanfloatingpoint,moreaccurate,lessdemandingonenergy,onmemoryandbandwidth,orpower,andguaranteedtogivebitwiseidenticalresults,andinmostsituations,faster.

So,inmanyplacesthroughoutthebook,JohnacknowledgesthecontributionsofWilliamKahan,offloatingpointarithmeticofcourse.AndsoheusesmanyofKahan’sexamples,throughoutthebookabouthowfloatingpointcangowrong,toillustratethepotentialadvantageofunum.So,forexample,onesectionofGustafson’sbookisentitled“TheWrathofKahan”[laughter]andanotherisentitled“AnotherKahanBoobyTrap”.So,TheGreatDebateisforeshadowedinSection18.1,whereGustafsonrespondsatlengthtoachallengeproposedbyProfessorKahantoanearlydraftofthebook.Now,thisdevelopmenthasnotstoppedwiththis,andJohnhascontinuedtodeveloptheseideas,andmostrecentlyhe’sproposedsomethingcalledSORNs,forSetsOfRealNumbers.AndIbelieve,hewillexplainthatnewproposalhere,andthatwillalsobeasubjectofthedebate.Sothewaythisisgoingtobeorganizedisasfollows:Soeachspeaker,startingwithJohn,willgetthirtyminutes,justtoexplaintheirposition.Andyoucanonlyaskexplanatoryquestions,clarifyingquestions.Debatequestionswillcomelater.AndthenProfessorKahanwillgethisthirtyminutes.Andthenwe’llhavethirtyminutesofafree-for-all.ButI’mtheModerator[laughter],andIwillmakesureeverybodystickstotheirthirtyminutesandthenwe’llaskquestionsandthetwogentlemencanspeakonit.Sowithoutfurtherado…John?

2

GUSTAFSON:(takesthepodium)Thankyouforthisopportunity.KAHAN:Ifpeoplemovecloser,they’llfinditeasiertoreadtheslides.[commotion]

GUSTAFSON:SoIalsohaveatakeonwhythisdebateisoccurring.It’salittlebitdifferentfromwhatyoujustheard.TheEndofError:UnumComputinghaddozensofreviewers,whichincludedDavidBailey,HorstSimon(colleaguesofDr.KahanatBerkeley),alsoGordonBell,JohnGunnels,whohaswonthreeGordonBellawards,andabouthalfadozenotherPhDnumericalanalysts.Andthebookwascritiquednotjustbasedonitsmathematicalcontentbutalsoonitsstyle,andits

approach,sinceitwasaimedatthe,let’ssay,pre-collegereader,whodidnotnecessarilyhavetoknowcalculusinordertounderstandalotofwhatthebookisabout.[audiencerequestsadjustmentstomicrophone]SoIwaspurposelytargetingpre-calculuspeople,peoplewhohadhadagoodhighschooleducationinmathematics.So,ProfessorKahanhashadthemanuscriptsinceNovember2013,atleastapreliminaryversion,Part1,andwehademailconversationsaboutit,butallemailcutoffsuddenlyinJuly2014[laughter],goingbackandforth,butthenIposedacounter-questiontohim,Isaid,“Canyousolvethisone?”Andthenthelinewentdead.Ididn’tknowwhatIsaid,butIthinkit’spossiblehealthproblemswereinvolved,butIwasreallymissinghiscontributionbecauseIwantedtomakesureIwantedtomakesurehewasontopofeverythingIwasabouttopublishandwouldbeabletogivemehisfeedback.Thenthishappened.ThebookcameoutinFebruary,and,fairlyquicklybecamethenumberonebestsellerinNumericalSystems,admittedlyasmallcategorybutthere’sfourthousandbooksonNumericalSystemsonAmazon,anditgot…eightreviewsoffiveoutoffivestars,andtheykeptrunningoutofthething.Theycouldn’tkeepitinstock.AndIthinkthisisprobablywhatrousedtheattentionofDr.Kahan,isbecausepeoplewereactuallyreadingthisthing,soheshoulddosomething…tocautionpeopleaboutit[chucklingsounds].

3

So.“TheWrathofKahan:ABitterBlog”.Asyoumayknow,Kahandoesnotsubmitpaperstojournals,atleastnotveryoften.Whathedoesinsteadis,hewritesblogarticles,andsomeofthemare…verycolorful,let’ssay.Youknow,like,“HowJava’s

FloatingPointHurtsEveryoneEverywhere.”Andthingslikethat.Andhepreparesthesediatribes,andhelovesthem,andusuallylabelsthemas“Workinprogress.”Idon’tknowwhetherhelaterchangesthemlaterornot.Buttheissuewe’rediscussingtoday,thefutureofnumberrepresentationoncomputers,Ithinkistooimportanttobelefttothebickeringoftwooldmen[chucklingsounds].

So.Hewaskindenoughtosharewithmethe38-pageattackhewantstopostaboutTheEndofError:UnumArithmetic.AndsoIwasabletoseesomeofthethingsheplanstosay.Iexpecthim,probably,tobecoveringsomeofthesethingstoday.Sincethisistheonlythingthat’sreallyfinished,Iwon’tbetalkingaboutwhatarenowcalled“unumstype1”or“unums1.0.”Idon’thaveanymaterialpreparedabout2.0,becauseinfactI’mnotfinishedwithityet,readytoshowtheworld.It’sstillinprogress.SoIwillrespondinpartheretothe“BitterBlog”thatIhaveseen.Youseethe“TableofContents”there;I’msureyoucan’treadthosewords.Firstofall,Ithinkeverybodyhasthisreaction:Variablebitsizeisjusttoodarnedexpensive.Tobeabletochangethewidthsandsizesofrepresentationsisgoingtobeaheadacheforhardwaredesigners,andforsoftware,andstorage,andsoon.AndIalwayssighandsay,well,itdoesn’thavetobevariablesize;youcandothesethingsfixedsize.Butifyoudohavetousevariablesize,youdon’tneedanadditionalsetofpointersinordertopackandunpackthesethings.There’sthesethreebits—threefieldsintheutag.Theyserveasapointertothenextunum,soyouuntangleitasalinkedlist;theyalreadyhavetheoverheadnecessaryforpackingandunpacking.Noadditionalbitsrequired.So,Chapter7:thetitleoftheChapteris“FixedSizeUnumStorage.”AndIdon’tknowhowpeoplecanmissit,pages93to102,talksabouthowyoucanmakesurethatyoucanfillmatrixarrayswithunumsandstillbeabletohaveconstant-timeaccesstoanythinganddoexactlywhatyouwoulddowithcurrentfloating-pointarithmetic,noproblems.Thereisstillenergyandpowersavingswhenyouunpackthings,leavingafewbitsunused.Hardwaredesigners

4

knowhowtoturnoffbitsiftheyknowthatcertainfieldswithina64-bitregisterarenotgoingtobetouched.Soevenwithinthechipyoucanstillgetenergy-powersavingsintheunpackedform.Now,thisbringsupanexamplethatKahancalls“ABogusAnalogy.”Here’swhatIput,aversionofit.InCourier,16-pointtype,‘Unumsofferthesametradeoffversusfloatsasvariablewidthversusfixedwidthtypefaces.Harderforthedesignengineerandmorelogicforthecomputer,butsuperiorforeveryoneelseintermsofusability,compactness,andoverallcost.’DothatinTimes,andnoticehowmuchspaceittakesuponthescreen.Orhowmuchspaceitwouldtakeupifprinted.EverybodyIknowwhohasreadthis,understoodtheanalogy.Except…ProfessorKahan.Hesaid,‘BUNKUM!Gustafsonhasconfusedthewaytextisprinted,or

displayed.ItwouldoccupymoreDRAMmemory,notless!’…Uh,yeah,ittakesmoreDRAMmemorytotakelessspaceonapieceofpaperoronascreen,whichmightbethemostvaluableresourcethesedays.So,Iwassayingthatifyouhavemorememory,youcando,certainly,tighterfittingofthingsiftheyonlyhavetotakeuptheamountofspacethattheyeachrequire,insteadofeverythinghastoasbigasthelargestletter.Sothesameway,ifyouhave

variablesizearithmetic,youcanshrinkandadjustandingeneralyouwillsavespaceusingonlyasmanybitsasareneededinmemoryforthenumbers.Ihavenotmetanyotherreaderwhomisunderstoodthis,andthenattackedit.Buthere’sthefallacy,andit’sa‘willfulmisunderstanding’technique.YoutakesomethingI’dwritten,oranybodyhadwritten,andmisreadittobesomethingthat’sclearlyafalsestatement,andthenyouattackthat,eventhoughit’snotwhatIsaid.SoasIreadthroughthis38-pagedocument,imagine…38pagesofattacksjustlikethisone.Ididn’tsaythat!Sohe’sattackingthingsthatwerenotthepositionIwastakingatall.

5

IwanttogooveraclassicKahanexample,theareaofatrianglethatisvery,veryskinny.Solet’stakeatriangle,sidesa,b,andcwherecisthelongside,andaandbarejustbarely,barelymorethanhalfthelengthofc.Soitpopsupalittlebit,likebuckledconcreteinthesun.Andit’sonly3ULPslarger,andthenyou’regoingtotrytousetheformula,theclassicgenericformula,ofthesquarerootofs,sminusa,sminusb,sminusc,afterfindingthesumaplusbpluscovertwo.Thisishazardous,becauseofthat:[pointstothe(s–c)term]sandcareveryclose;infactthey’reonlysixULPsapart.Soyou’regoingtobetakingtwoverysimilarnumbersandsubtracting,notnecessarilyaproblemifit’sexact.Youdon’tloseanyinformationifthosetwoareproperlyrepresented.Butifthere’sanyproblemthatyouroundedswhenyoucalculatedit,that’sgoingtoamplifytherelativeerrorinamassiveway.SoinIEEEquadprecision,let’spulloutallthestopsandusethebeststuffyoucangetrightnow:128bits,34decimals.We’llletaequalb,sevenhalves,andaddthreeULPswhereanULPistwotothenegative111thpower.That’sareallysmallnumber.Andcequalsexactlyseven.Andjusttomakeitalittlebitmoreinteresting,let’ssaythattheunitsoflengtharelight-years.So,sevenlight-yearsaway,that’slike,AlphaCentauriineachdirection,ourneareststar.ThreeULPsis1/200ththediameterofaproton.Amazingly,thatmanagestopopthetriangleupbyroughlythewidthofastandarddoor,85centimeters.Sofinallywehaveanumberwecankindofcomprehend.Andthatarea,ofthatextremelylongstrip,isabout55timestheareaoftheearth.It’sgettingalmostuptothesurfaceareaofoneofthegasgiants,likeNeptune.Sohereistheanswercorrectto34decimals,insquarelight-years.Justrememberthefirstfewdecimals,please.It’seasytoremember3.14,timestentotheminus16th,squarelight-years.That’swhatweshouldget.

6

Thequadprecisionfloatresultis,itgetsonedigitright.Alotofworkforonedigit!Threepointsix,timestentotheminus16th.Aboutfifteenpercenttoohigh.That’s—

intermsofULPs,that’s250-somepeta-ULPs.Theresultdoesnotadmitanyerror,norboundit.Itgivesnowarningtotheuser.You’reexpectedtodoerroranalysistofigureoutthatthishappenedtoyoubecauseit’scompletelysilent.Nothingprotectsyoufromthiserror.ProfessorKahansuggeststhatyousortthesidessothatthey’reagreaterthanbgreaterthanc,andthenapplythisformulawhichisalgebraicallyequivalent.Iwouldlovetoknowhow

manyhoursittookhimtofindthatformula.Orhowmanyhoursitwouldtakeyoutofigureoutthatformula.KAHAN:Halfanhour.GUSTAFSON:Halfanhour.[laughter]HalfanhourofaBerkeleyprofessor’stime,conservativelylet’ssupposeyourtimeisworththreehundreddollarsanhour,it’sprobablymore.Soitcostaboutonehundredandfiftydollarstogetthatformulaout,inordertosavethecomputerfromhavingtodoalittlebitmorework.That’snowprovablywithinelevenULPsofthecorrectarea.Butitusuallytakeshourstofigureoutthatkindofapproach.Italsousestwiceasmanyoperations.Ihavenoobjectiontousingtwiceasmanyoperations;we’vegotoperationstoburn,butthat’snottheissue.It’sthepeoplecostoftheapproach.It’sthewrongdirection.Weneedtomakethingseasierforusers.Let’stryunumarithmetic.Andlet’smakesurethatwe’renotcheating,soI’mnotgoingtouseanymorethan128bits,period.Wewerelimitedonthequads,128bitspernumber,butitcanadjust,itwillsometimesuselessthan128butitwillnevergoover.Theexponentismoreflexible.Itcangofromonetosixteenbits,soI’veactuallygotabiggerdynamicrange.Thefractioncanbeoneto128bits,plusthehiddenbit.That’shigherprecisionthanquad.

7

Theresultisarigorousbound,accurateto31decimals.Theareaisstrictlyboundedbythosetwonumbers.Theydon’tstartdiffering(youseetheorangeoverthere)untilthe32nddecimal.Thesizeofthatboundisasquareabouteightnanometersonaside,sowe’regettingdowntoaboutthelimitsofpresentlithography,inordertofindtheerrorinfindingthisthingthathas55timestheareaoftheplanetEarth.OK,soinsummary,let’stakealookatquad-precisionIEEEfloatsversusunums,also128bits.Thedynamicrangeisalmostthesquareofwhatyougetwithquad-precisionfloats.Insteadof34decimals,Icouldhaveupto38.8.Andthen,tryingit

onthetriangle,Iusedanaverageof90bits;theyshrink.Andtheresultis,thefloatresultsays,areaexactlyequalsthiswrongnumber,versusunumsthatsaytheanswerisdefinitelybetweenthesetwonumbers,provably.Thetypeofinformationlossinthecaseoffloatsisinvisibleerror,veryhardtodebug.WhenIactuallydidthisexample,Iusedunumstodothequad.Ijustsettheexponenttothe

rightsizeandthefractiontotherightsize,andsimulateditwithmyenvironment.Andatthepointwhereyougettheroundingerror,wheresuddenlysixULPsbecomeseightULPs,it’sinstantlyvisible.Itwassoeasytospot.Therewasnoproblem.WhereasIthinkifitjustkeepscrankingaway,alwaysgivingwhatlookslike34decimalsofprecision,you’regoingtohavetowalkthroughit,onestepatatime,verylaboriously,sobyhavingmoreinformation,itmakesiteasiertodebugandfigureoutwhat’sgoingoninyourcomputation.Here’sanotherexampleof,“Well,justre-writethecodethisway.”NowIdidanexamplejusttoshowhowtheroundingerrormightnotberandom,andalsohowthesystemcouldautomaticallyadjusttogiveyouhigheraccuracyandself-manageaccuracy.ThesimplestexampleIcouldthinkofwasjust,counttoabillion.Justkeeponaddingone,addingone,addingone,abilliontimes,andseehowyoudowith

8

floating-pointarithmetic.Floatingpoint,floats,ofcourse,canrepresentthenumberonebillion,buttheywillgetstuck,asweheardinanearliertalk.Intryingtocounttoabillion,IEEEfloatsgetstuckattwotothe24th,sixteenmillionsome.Theycan’tgetanyhigherbecauseasyouaddone,itkeepsonroundingdown.Ah,well,thisisawell-knownproblem,andDr.Kahansaysinhisbigwrite-up:“CompensatedsummationwillbeillustratedbyapplicationtoasillysumGustafsonusesonpage120tojustifywhatunumsdoasintervalsdo,namely,conveynumericaluncertaintyviatheirwidths.”Idon’tthinkthat’ssillyatall.Iliketoknowwhensomethinghassuddenlylostitsabilitytogivemeapreciseanswer.Andthat’sactuallynotwhatthissectionwasabout.Butlet’stryit.So.ThisisascreenshotfromKahan’spaper.Withnequaltotentotheninth,compensatedsummation,allinfloats.Setthesum.Nowyou’veintroducedtwonewvariables:oldsum,andcompensation,andyoukeeptrackofthis.Yoursummationinaprogramoftenwillbescatteredallovertheprogram,soforeverysumyou’reaccumulatingyou’regoingtohavetokeeptwoauxiliaryvariableslikethisandkeeptrackofthem.Whatdosupposethisdoesforthemaintainabilityofthecode?Andtheportability?Andthechancethatyoumighthavemadeabugsomewhere,maybeanerror?I’dsayitgoesskyhigh.

SoIcodeditupinMathematica,exactlyasyouseehere;it’saslighttranslationfromC,fromwhateverpidginlanguagethatistoMathematica,butIthinkitisexactlywhatyousee.AndIsummedfromonetoten.Igotthenumber2036.Itriedaddingoneplusone.Itgavemefour[painedexpression].Itriedgoingallthewaytoabillion.NaN.Ithinksumminusoldsumisinfinityminusinfinity,anditsaid,notanumber.Whatisgoingon?ThisisaFAIL.Hegotanerrorwhenhecodeditup!Andheputitrightthere,inthecode,inthetext.SoevenKahandidn’tgetitright.Inrewritingcode

9

totrytocompensateforthesethings,andsay,“Let’sjustrearrangeallthisstuff”isveryerror-prone.Italsomakesyourcodecompletelycryptictoanyonetryingtounderstand“whatareyoudoing?”Theywon’tbeabletoseeit.Plus,itgetshuge,potentiallyhugemistakes.Soanapproachthatusesmorecodingeffortandthreetimesasmanybitstoproduceawildlywronganswerdoesnotimpressme.Exampleslikethisneedtobetested,notmerelyasserted.Hesays,“Sumequalstentotheninth,exactly!”No,it’snot.Iwonderhowmanyotherthingshesays,“Thisistrue,forthisformula.”WhenIwrotethebook,Ididnotuseawordprocessor.IusedMathematicatogeneratethebookasacomputerprogram.Alltheexamples,therefore,arecomputedoutput.SoanymistakeIwouldmakewouldbe,like,transcribingaresultsomewhereintothecommentfield.Imightmakeamistakedoingthat,butotherthanthat,it’sallprettymuchbombproof.NowIhada—oneofmyfavoriteexamplesfromKahan,is,Icallithis“monster,”andthat’smeantasaflatteringtermbecauseIthinkit’sareallycleverexample.Butit’saverytoughoneforfloats.There’sthreefunctions,T,Q,andV,Igotthelettersrightthistime,Ithink,andQis,shouldbeidenticallyzero.Butfloatscan’tdothesquarerootsandreciprocalsexactly,sotheytendtogetalittleslop.Yousquarethatnumberinthebottomfunction,G,andthatmeansyou’regoingtogetsomethingalittlebitbiggerthanzero.Andthenyouusetheremovablesingularityfunctionetothezminusonealloverz,andfloatswilltendtogetzeromostofthetime.Theymightgetluckyandactuallygetone,whichisthecorrectanswer.OK.SowhenIcodedthisup,Iusedtheapproximately-equalssign,whichactuallymeansintersects,soifaunumintervalintersectszero,thenthat’sthewobbly“notnowhereequal”sign.Ididn’tuseexactequality,there,showninbrightblue,justthewayheshoweditinbrightbluebecausehewantedtoemphasizethatIwascheating.Sohegotbentoutofshapebythat,soIcodeditupwithanequalssign,andyoucanseeitrightthere:Ifzequalszero,one;elseetothezminus1overz.Idon’tknowhowmoredirectIcouldcodeexactlywhathewantedashisexampleproblem.Theotherthreelinesarejustputtingitinunumform,asunumoperations.

10

Here’stheresult.Insteadofgettingexactly1,Igot1plusanULP.I’mshowingyouthefirstnineandthelastnine,butit’sallveryboring.Thelistofalltenthousandnumbersisallexactlyone,andanULP.Andthey’realloftheformoneplusepsilon,andthisisahalf-openinterval.Itincludesthenumberone,andisalittlebigbigger,goesalittlebittotheright,butit’sneverzero.Itnevermakesthatmistake.

Nowhewouldhavefoundthatoutifhe’dtriedit.Hehasaccesstoallmycodeathisfingertips,asdoallofyou;it’safreedownload.JustgototheCRCpublicationsiteanduseit.OrifyougetitinJulia,or—there’saboutfiveversionsofunumarithmeticnowinJuliaontheweb,andoneinPythonthat’saline-by-linetranscriptionoftheprototype,Cversionsarepoppingupeverywhere,C++…itwon’tbelongbefore,nameyourlanguage,it’sgoingtohavesomeunumsupportfromsomebody.SoIdon’tknowwhyhedidn’tjusttrythesethingsandhejustsaid,“That’snotgonnawork.”Hespeculatedthatitwouldfail.Sohemadeoneup,andIfeelsohonoredthatI’veearnedmyfirst,targeted,“ThiswillBREAKunums!”fromDr.Kahan.Whathedidwas,headdedonanumber,tentothenegative300thpower,allraisedtothepoweroftenthousand,timesxplus1,whichcouldbeaslarge—orIshouldsayassmallas—tentothenegativethirtybillion.SoI’mveryflatteredthathethoughtitwouldtakeanumberliketentothenegativethirtybilliontobreakunumarithmetic.Oratleastslowitdown,soitwouldtakeanintractableamountoftime.“What,ifanything,doesunumcomputinggetforthis?Howlongdoesittake?Itcannotbesoonnorsimply1.0.”Wannabet?

11

Ithadabsolutelynotroublehandlingthis,anditdidn’tslowitdownonebit.Foraverysimplereason:Whathecalled—thatverysmallnumber—isimmediately

turnedinto…theunumzerotoepsilon.Openinterval.It’snotzero,it’stoosmalltorepresent.Thisisthewayunumshandlewhatnormallywouldbecomeunderflow.Theydon’tunderflow.Theydon’toverflow,andtheydon’tround.Theyjustsay,“It’sinthatinterval,zerotoepsilon,andIwillkeeptrackofitthatway.”Youwanttoexponentiateit?Fine.Onceagain,itevaluatestoone(gesturesthestartofaninterval),tooneplusepsilon(gesturestheendofaninterval).Itdidn’tgoanyslower.Itdidn’trequirehighprecision.It—Iguessyoucouldsayitjustgaveup.Itdidn’tevenattempttorepresentthisnumber.AndDr.Kahanreferredtoitasan“infinitesimal,”whichIthinkisastrangeuseoftheword“infinitesimal”sinceit’sclearlyanumber

andnotaninfinitesimal.Anyway.Here’swhatIcall“AnInconvenientInfinity.”AndIjustwasfindingaverysimpleexample—asIsay,thisisaverybroadaudiencebook,soIsaidwhat’stheareaofacircle?Let’sgiveitanarcofacircle,multiplybyfourandseehowclosetopiwecangetwithone-decimalprecision,withvery,verycrudelimits.So,ifyoudivide—ifyoupickapointontheleft,itwillkindoflikeaburningfusemoveovertotherightandenclosethegapofwhatevaluatestoxsquaredplusysquaredequalsone.Allyouhavetoknowishowtomultiplyandaddandtestforequalitywithone.Theerrortendstobeoneovern,andit’sarigorousbound.Thisworksonanycontinuousfunction.Itdoesn’tcare;infact,withslightmodification,itcanevenworkondiscontinuousfunctions.SoIwaslookingforgenerality,notfor“HowtightcanImakethebound.”SohereisoneofthethingsthatKahanwroteupinhisattackonmybook.Hementionedthatthemidpointrulehasanerrortermassociatedwithit,thetrapezoidalrulehasanerrorassociatedwithit,andifyouusethosetwotogether,youcanputaquote,bound,aboutthenumericalevaluationofintegrals.Prettygood!Itdoesrequirethatthesecondderivativeexistandbeboundedthroughout.AndasIreadthroughhiswrite-up,hementionedseveraltimesthatf-prime-primeisboundedthroughout.Andthenhesortofnoticedthat…it’snot.It’snotbounded.Forthequartercircle,itgoestoinfinity.

12

Tomyastonishment,heusedtheformulaanyway.Heusedthebounds,ignoringthefactthathis“thisbound,plusinfinity,islessthan…”Youcan’tdothat!Ifmystudentsdosomethinglikethis,Itakeoffatleasthalfpointsonthetest.Therewereveryclearconditionsforthistheoremtowork.Andheignoredthembecausehewantedanice,tightbound.Healsosaidthatmymethodisordernsquared,whichIthink,ifyoujustlookatthemethod,you

canseethatit’snotordernsquared.Idogiveanearliermethodwhichpopulatesallthesquaresunderneath,andquicklysay,“That’snottherightwaytodoit;youdoitthisway”buthestillstuckwith“Oh,that’sareallyslowway!”Itwouldbevery,veryslowjusttofillinallthesquares.There’stoomanymistakestoreallycoverhere,butperhapsthebiggestoneishesays,“Thebookclaimsitendsallerror.”Itdoesnothingofthekind.Itsaysitendsaspecifickindoferror.It’stheerrorofroundingandsampling.Alwayssubstitutingaspecificexactnumberwhenyoureallywantedsomethingdifferent.Andhesays,“Unumsaretartedintervals.”Well,unumssubsumefloatsandintervals.Youcanusethemtobefloatingpointnumbers;youcanusethemtobeintervalnumbers…Thisisawholeenvironment.It’snotjustanumberformat.It’sasetofrulesforhowtousethemaswell.Whatyoucandointhescratchpad,forexample;that’swhatgivesyoubitwisecompatibleanswers.Butyouknow,“tarted”means,like,toomuchlipstick,andmakeup,andthingslikethat?AndwhenIlookatthese34-digitnumbersthatclaimtohaveallthisprecision,andonlyoneofthemiscorrect,Ithinkthat’stheonethatlookslikeit’swearingtoomuchmakeup.Yousee,unumsadmitwhattheydon’tknow,whereasfloatsalwaysclaimallthisprecision,whichmightnotbetrue!“Gustafsonregardscalculusasevil.Heisnotjoking!”Iwasamazed,becauseIusedaraccoonmemethatsaysDIYLOLonit.Idon’tknowhowanyonecouldknowthatI’mnotjoking.“I’veinventedsomethingevil.Iwillcallitcalculus.”Ithinkmostpeoplewouldlaughwhentheyseethis,andthatwasmyintent.Iwastryingtomakethepointthatpeopletendtoconfusewhentheyarecalculating,and

13

whenthey’reusingcalculus,andtheygetinalotoftroubledoingthat.There’salotofproblemsthatcanbesolved,withoutusingcalculus.Iwantedtokindofpushthelimits:Howfarcanwegowithoutrequiringcalculus?…Anyway.

Iwouldsayinafewplacesthatgradeschoolmathwouldsuffice.Andyousay,“That’snotgradeschool!That’strigonometry!”And,well,12thgradeisagrade.[chucklingsounds]11thgradeisagrade.That’swhythey’recalled…grades.IfImeantelementaryschool,Iwouldhavesaidelementaryschool.Thisisnotabookwithelementaryschoolmath.“Unumswillcostthousandsofextratransistors!”…Really?Whichwillcostthousandthsofapenny?Thisis2016;thisisnot1985.Athousandtransistorswillcostyouroughlyoneone-hundredthofacent.Rightnow.Andit’sgettingcheaper,still.“Hisapproachisveryinefficient.Here’safasteronethatusuallyworks.”HowmanytimeshaveIheardthat.Iamso

tiredofmethodsthat“usually”work,andrequireanexperttotellyouwhentheydoworkandwhentheydon’twork.I’minterestedinveryrobustmethodsthatyoucanhandtopeoplewhoarenotexperts,thatworkeverytime,evenifit’sslowly.“GustafsonsuffersfromamisconceptionaboutfloatingpointsharedbyvonNeumann.”ItpleasesmenoendtosharemisconceptionswithJohnvonNeumann.Oneofthemisconceptionshewastalkingaboutisactually,vonNeumannwasconcernedthatifyouhandfloating-pointhardwaretopeople,theywouldmisuseit;theywouldjustuseitblindly,withoutdoingpropererroranalysis,andtheywouldgetintrouble,becausethey’dgetwronganswersandtheywouldn’tknowit.That’smy“misconception.”Ithinkit’sperfectlyaccurate.

14

Letmejustgiveyousomememes.Thatwascomputersthen.1980s.Rememberthosedays?Thisiscomputersnow.Athousandtimescheaper.Athousandtimesfaster.Here’swhatarithmeticlookedlikebackthen.Single,double,extended.Here’swhatarithmeticlookslikenow.Doyouseemypoint?Isn’tittimetodosomethingaboutfloating-pointarithmetic?MaybenotwhatIdid,butsomethingtoimprovethequality?Withamilliontimesasmuchpoweratourfingertips,whycan’twedosomethingtomakethingsalittlelesspainfulforpeople?

15

Thebiggestblindspotofallintheentire38pagesofcriticismhe’slevied—andwatchforthiswhenhespeaks,becauseIhopehedoesn’tmakeitnow.Isayseveraltimes:“Remember:Thereisnothingfloatscandothatunumscannot.”Theyareperfectlycapableofmimickingeveryfloatthatthereis,alltheIEEEfloats,andallthesizesinbetween.That’sjustspecialcasesofIEEEfloats.It’sthelastlinebecauseIwantedtomakesurepeoplesawthat.There’sachapterdedicatedtothis,called“PermissiontoGuess.”Becausewhatyoudowithunumsis,youexplicitlyaskforaroundoperation.Itdoesn’thappeneverytime.So,youdon’thavetothrowawayallthesegreatfloatalgorithmsthatwe’vebuiltupoverthedecades.I’mnotsayingweshould.Therearemanythingsthat,well,quiteafewthingsyoucanfind,whereafloating-pointalgorithmisperfectlystable.Itworks,it’sgotasolidbaseoftheorybehindit,andthere’sabsolutelynoreasontouseunums,infactunumsmaycomplicatethingstothepointwhereyoureallywishyoudidn’tusethem.Thenjustusetheunumasafloat.It’sasuperset.Whenyourequestrounding,it’sverymuchmorehonestthanwhenithappenseverytimewithoutyoubeingundercontrol.Unumsendtheerrorofmandatory,invisiblesubstitutionofincorrectexactanswersforcorrectanswers.Itgivesyoubackthecontroloverwhat’sgoingoninthecalculation,andmakesitsovisible,thatyouhaveamuchbetterchanceofknowingwhatwentwrong,ifsomethinggoeswrongwithyourcalculation.It’seasiertodebug.Fareasier.Floatmethodsareanespeciallygoodwaytodealwith“thecurseofhighdimensions.”Ifyouaren’tcareful,youdogetkindofatwo-to-the-ntheffectwhenyou’retryingtosolvesomethingthathasndegreesoffreedom,whereasmaybewithafloatmethod,likeGaussianelimination,thatsolvestheequationswithorderncubed.Youcanusethattogetafirstguess,andifyouwanttodomorewithitusingunums,youhavethatoption.Ifnot,you’vegotwaymoreflexibilityintheprecisionandthedynamicrangethanyou’veeverhadwithIEEEfloats.AUDIENCEMEMBER:Sorry,doyoumeanfloatorunummethods?GUSTAFSON:DidIsaysomethingwrong?Floatmethods,inotherwordstakeamethodthatexistsasafloatmethod;usetheunumstodothefloatmethod.Usethatalgorithm.Usethatnumericalmethod.Like,Gaussianeliminationdoesnotwork

16

wellwithintervals.Badidea,OK?Itjustexplodes;yougetahugeboundandit’snotuseful.ButIdidmeanfloatmethods,here.So,useunumstoperformfloatnumericalmethods.

AsIlookthroughthesepages,Iseethingsthatyounormallydonotfindinamathreview.AndIknowinvectiveworkedforDonaldTrump[laughter],butisthisreallytherightwaytodiscuss…mathematics?Everytimeanyoneposes,seriously,“Let’sdosomethingaboutfloatingpoint,”there’sthisthing,that—inmyheadIenvisionitthisway.“TheLordoftheReals…DoesNOTSharePower.”[laughter]Prettysoontheblogsare,youknow,dartingoutsaying“That’swrong!”“Thatwillneverwork!”I’llleaveyouwiththatand…Dr.Kahan?It’sallyours.[applause]KAHAN:Well,whileyou’reupgettingmyslidesup,whatJohnhasdonehereistotakeexamples,outoftheircontext,startingwiththetriangle.Theflattriangleisnottheissue.Theideawastoshowthatfloatingpointwillgetresultsthatareexcessivelywrongcomparedwiththeconditionoftheproblem.Andsoifhehadchosentosetupadifferenttriangle,thenwecouldhaveaskedthequestion,well,howdoesyouranswer’serrorvarywiththeuncertaintyinthedata?Supposethedata’suncertainbyaunitinthelastplace?Well,itturnsoutthatforthatflattriangle,theuncertaintiesareoutright.Foradifferenttriangle,yougetthesamehorribleuncertaintyfromHeron’sRule,butit’snotdeserved.Inotherwords,youcan’ttellwithordinaryfloating-pointarithmeticunlessyoudoanerroranalysis,whetherwhatyou’regettingisdeserved,ornot.AndwhatJohnissaying,ineffect,is“Youdon’thavetodoanerroranalysis!Youjust

17

havetouseunums!”Well,I’llanswertheotherissuesthatJohnraisedonourtime,butmytimeI’mgoingtouseforsomethingelse.Let’sobserve—(audienceclamorsforhimtousemicrophone)DEMMEL:Areyouwearingamicrophone?KAHAN:Sayagain?DEMMEL:Youneedtousethemicrophone.KAHAN:DoIneedamicrophone?[audienceshoutsyes]It’sOK,I’llusethisone.It’snotaproblem.IfIcanjustgetitturnedon.[audiencechuckling]Now.Isitworkingnow?DEMMEL:Holditup.KAHAN:DoIhavetoholdithere?DEMMEL:Yeah,that’sgood.KAHAN:That’sfine.Let’s,let’sobserve,andyoucanreadthisatyourleisureafterit’sposted,onmywebpage.Andifyou’llgotothenextpageplease.

Thedetailedcritiquesarehere,soyoucanseewhetherJohnhasrepresented,ormisrepresented,thecriticismsthatI’velevied.Youcanreaditforyourself,butthesearethedocumentsthatweexchanged.ThoseareJohn’sdocuments,andthenIexplainedweexchangedthefirstdraftsofthesedocuments.Let’sgoontothenextpagenow.

18

Here’stheissue:Ittakesalongtimetocorrectamistake.Andinanyevent,thearithmeticisasmallpartoftheissuesthatwehavetoface.Let’sgoontothenextpage,please.

Firstofall,whatareunums?Well,Isaidthey’retartedintervals.Andsotheyare.Floatingpointnumbers,eachhasitsownwidth,range,andprecision,youcanvarythemasmuchasyoulike.Eachistaggedtoshowifit’suncertainornot.Usuallyyouusetheminpairs,becausethey’reendpointsofintervals.Sotheymakeupandareincludedintheinterval.Andthey’reallegedtosavestoragespace.Ah!That’sgoingtobethecriticalissue.Dotheyreally

savestoragespace?Andideally,thewidthswouldvaryautomatically.Ofcourse,thatwouldhappenwithintervalarithmeticifwecouldgetit,withvaryingprecision.Butthepeoplewhoaretryingtostandardizeintervalarithmeticsaytheyhaveenoughtroublegrandfatheringinoldschemessotheyaren’tinterestedinthat.

ButaboutSORNs.SORNsdon’thavevariableprecision;you’vegottocommityourself.Theyarefinitecollectionsofextendedrealnumbers;thatmeanstheyincludeinfinity.Theyallowopenintervals.They’realsoserveinpairs.Andthey

19

typicallysavestoragespacebecausewhatyou’remanipulatingarenotthenumbers,butpointerstothevalues.Andso,it’spossibletorunthisarithmeticvery,veryfast.Now,bothschemesaretypicallyproducedwithmassiveparallelism.Andwhenwelookatthiswe’llseehowmassivelyparallelismtheyhavetobe.Next,please.

Theissuehere,is…notsomuchwhethertheywouldbeworthwhile,ifwhatJohnsaysweretrue.Theywouldbe!Itwouldbelovelytobeabletogetawayfromerroranalysis.Iwouldratherdosomethinginteresting.[laughter]Buttheycan’t.Andit’spossibletoproveitfromfundamentals.HoweverI’drathernotboreyouwithfundamentals;I’lltrytogiveyoucounterexamples.Let’sgotothenextpage.

Whataretheextravagantclaims?Well,hesaysyoudon’thavetohaveanunderstandingofdifficultmathematicalanalysis;inparticular,thediscretizationofthecontinuum,whenyousolveadifferentialequation.Youdon’treallyhavetounderstandwhat’sinvolvedhere,youdon’thavetoworryaboutunderestimatingtheuncertaintyinyourdata,andyoudon’thavetoworryaboutroundoff.Well,nowthat’sinteresting,butisittrue.Noneofthatistrue.Now,SORNcomputations,theyhaveadifferentsystem.And,unums,andSORNs,arenotindependentofthedependencyproblemofintervalarithmetic,norofthewrappingeffect,althoughJohndoesclaimtheyare.Heclaimsalltheerrorgoesawayandthat’snotquitetrue.Asyou’llsee.Heclaimsthatcalculusisevil,OK,sohe’sgotafunnycartoon.Buthereallydoesdisparageit!Hesaysyoudon’thavetoknowittosolvedifferentialequations.He

20

saysyoudon’thavetoknowmodernnumericalanalysisinordertodoerroranalysis.That’swhathesaysinhisbook.Youcanreaditforyourself!Andherewe’vegotaproblemherewithdiagnostics.“AplethoraofNaNs”hesays,forinstance.We’vegottoomanyNaNs.Hedoesn’tknowwhatthey’refor,Iguess.And,“Flags:Inaccessiblefrommostprogramminglanguagesandtheyjustannoypeople.”Althoughifthey’reinaccessible,andyoucan’tusethem,whywouldtheybeannoying?GUSTAFSON:Becausethey’reslow.KAHAN:Well,there’snothingslowaboutthem.Butdirectedroundings,well,you’regoingtohavetoworryaboutthosebecauseunumsandSORNsdon’thavedirectedroundings.There’sthetrouble.Andthat’swhatwehavetocombat.Let’slookatthenextpage.

“CanwebeLiberatedfromError-Analysis?”Well,inordertounderstandtheissues,wehavetolookattheprocess.Firstyouchooseanalgorithm.Youexpressitinamathematicallanguage,andyouprovethatitwouldworkifyoudiditinidealarithmeticorarithmeticwithunlimitedprecision.Thenyoutranslateitfaithfullyintoaprogramming,someprogramminglanguage.Thenyouexecutetheprogramandyougetaresult.Now,thesethreethingsaredifferent:The

result,R,mightnotbeaccurateenough.Youhavetocopewiththat.Andusuallyitis.But,oftenitisn’t.Sothere’sthisquestion:IfP’sarithmeticisfloatingpointbutofaprecision,dowehaveanerroranalysis,todetermineforsure,iftheprogramPimplementsthealgorithm,accuratelyenough,andtheansweris,maybeyoudo,andmaybeyoudon’t.And,ifyoudo,itmaycostyouafortune!First,tofindit…youhavetopaysomePhDguyforquiteawhiletolookforit,andthenifhefindsit,itmightcostyouafortunetocomputethaterrorbound.Sothat’swhypeopleareunhappyaboutfloating-pointarithmetic.Andifyourarithmeticdidawaywiththat,wouldn’tthatbelovely?Butitdoesn’t.Nextpageplease.

21

We’regoingtobedisappointed.We’regoingtobedisappointed,because,whenweaffectthediscriminationuponthefunctionwewanttocompute,straightF,andthealgorithmwehopetocomputeit,itcouldbeaformula,curlyF,andthenwehavebigFwhichistheprogram,they’reall…different.Whenyouexecutethisprograminordinaryfloatingpoint,itcouldbearbitrarilywrongsometimesforallweknow.ButI’minerroranalysis,andofcourse

whatwe’dliketodoissomethingaboutthat.AndwhatJohnhasinmind,istouseunums.Butthereareotherwaystodealwiththisproblem,whichI’llgettolater.Infact,theywereevenmentionedatthisconferenceyesterday.NowwhataboutSORNandunumintervalarithmetic?Well,theproblemthereisthatFcanbeaninterval,thatalthoughenclosestheidealmathematicalvalue,itcanbeawfullybig.Andifitisterriblybig,toooften,thenyouwon’tdoit.Ifthebigwidthisduetoroundoff,there’ssomethingyoucandoaboutit;youre-dothecomputationwithappropriatelyhigherprecision.Ofcoursethatmeansyouhavetohaveintervalarithmeticwithvariableprecision.Well,goodluckfindingthat!Unumcomputationletstheprecisionbeincreasedbytheprogrammer.Andoften,itworks.Ifthebigwidthisduetouncertaindata,nowyouhaveadifferentproblem.Andwhatyouhavetodoisbreakupthedataintosmallsubregions,andreplaceFevaluatedonceoverablockofdatabytheunionofalltheintervals,Fwithlittlexinhere.Andoftentheunionissmaller.Thatoftenworks.Thetroubleis,theydon’talwayswork.That’stheissue.Iftheyalwaysworked,wecouldcelebrate.Buttheydon’t.Nextplease.Sohereareacoupleoffailuremodes.Howcanincreasingtheprecisionofunumsfailtoovercomeroundoff?Well,itcan.Thereareexamples;thetroubleis,they’resosubtle,andI’llexplainthemonlyinthequestionperiod.Ittakestoomuchtime,now.Whataboutthis“union”issue?HowcantheunionofFassmallintervalsfailtobesmallerthantheoversideF?Well,here’sasimpleexample.Thisisarationalfunction,whichhastwoexpressions.Thisexpressionandthatexpressionarethe

22

samerationalfunctionsinthefieldofrationalfunctions.Well,whywouldyouusethisone?It’smoreaccuratewhenxandyhaveapproximatelyequalmagnitudes.Whywouldyouusethatone?It’sabetterchoicewhenxandyaresensitivetooverorunderflowthatmightthreatenyou,soyoumightyouonehereandanotheronethereinyourprogram.Nextlookattheopensquare.Itturnsout,withboththeSORNandunumarithmetic,youcan

representopenintervals.So,thissquareisrepresentable.Andnow,ifyousubdivideitintolittlebits,nomatterhowtinytheyare,you’llalwaysfindthattheunionoftheregionswithlittlebitsisalwaysthisinterval.Whereasthecorrectwaytosizeitexceptforamisprint;thoseshouldberoundparentheses,itshouldbeopeninterval,notclosed.That’swhatthisthingremindsmeof.So,thetroubleis,thatifyoudon’tusetherightexpression,youmayendupwithterriblywideintervals.Andthatcanhappentounums,andtointervals,andtoSORNs.Next,please.Well,howoftendoessomethingthisbadhappen?Well,it’saninstanceofthedependencyproblem.Theintervalarithmeticpeopleknowaboutthis.Thebook’schaptersclaimthattheyovercomethisproblem.And,andthedocumentsaboutSORNsclaimtheyhavenodependencyproblem.Thosearetheirpages.Well,itturnsoutthatthisisn’tquitetrue.Nowremember,youwantstraightF,that’swhatyouwanttocompute,butyou’regoingtouseanalgorithmcurlyf,andyou’regoingtowriteaprogram,calledbigF,andthetroubleisthatbigFmaybeperfectlyhappywithfloating-pointnumbers.Itjustmisbehaveswhenyouuseintervals.That’swhathappened,toour—andonthepreviouspage.Idon’tknowhowyoueverfigurethisoutifyoudidn’thaveanalternativewayofdoingthecomputation.

23

Butthere’sanothertheoremormode.Yousee,it’spossiblethatthealgorithmyouchose,althoughitwould,ininfiniteprecision,computetherightfunction,straightF,thealgorithmyouchosegaveyouareallybadwaytocomputewhatyouwant,ifyouhaveroundingerrors.Andthatreallycouldnomatterhowthisisprogrammed,intervalarithmeticnotwithstanding.Andthere’sanexample.Theincenterofatetrahedronisthecenterofthebiggestspherethatfitsinthetetrahedron.Andnomatterhowyouprogramthefunction,unums,intervals,whateveryoulike,asthevolumeofatetrahedronshrinks,thevalueyoucomputegoesofftoinfinity.Andthisexampleisinsomeofmynotes,andit’salsoinoneofthepagesputupofthedocumentsthatItoldyouabout.Anotherplacethingsgoingwrongisthatcurlyfmaybethesolutionofanequation.Now,thisequationmaybesatisfiedbytheidealfunction,that’swhatyouwant,andwhatyou’vegotisanequationmanufacturedwithcurlyf,that’syouralgorithm,solvethisequation.Thetroubleis,thatthecoefficientscoulddependonyourdata.Andifyourdataisalittleuncertain,thenthecoefficientswillbeuncertain,butthey’recorrelated,andthey’recorrelatedinawaythatallintervalarithmeticschemesignore.Sothatmeanstheequationsolutionmayreactfarworsetoperturbationsthanitdoestothetruefunction.Andthishappensoften:loadonelasticstructures,crashtests,leastsquares;itusedtohappentoeigenvalueproblemsuntilwegotbetteralgorithms.So,there’saremedy.Usehigherprecision.Don’tuseSORN,orordinaryintervalarithmetic.You’vegottodosomethingelse.Nextpage,please.

Athirdfailuremodeiscalledthe“wrappingeffect.”Now,we’recalling—thewordwrappingeffectdoesn’tappearinthebook.Andit’ssuggesteditdoesn’thappenbecausethere’snoexponentialgrowthintheerror.AndSORNisthesamething,onthesamepage,“Uncertaintygrowslinearlyingeneral.”Itain’ttrue!Ifyouhavealongcomputation,andespeciallyifyouhaveloops,yourunariskofhavingexponentialgrowth.Andhereisasimplifiedexample.AllIwantis,multiply

24

avectorbyamatrix.Thismatrixisatwentybytwentymatrix.Youcangetitfrom,well,Matlab,it’saHadamardmatrix.Everyelementisoneoverthesquarerootoftwenty.Butthesquareofthematrixistheidentitymatrix.So,whenyoucomputeX1,X2,inorder,inturn,whatyoufindis,you’vegotabigdimensionthatcan—thepropagationisafairlength.ButX2NisX0;there’dbenogrowth.Butinanyintervalarithmeticschemethatwehavenowadays,yourcomputedX2Nwouldbetoobigbythisfactor:Twentytothepowernminusahalf.Andnissupposedtobeprettybig.Well,let’ssee.Whatshouldwedotocutthatdown?Well,whatwedois,wesay,let’ssupposeI’lltolerateanerrorboundthat’s400timesbiggerthanitoughttobe.SoIhavetobreakuptheinitialdata,intolittlebits!Thisishowbig—howmanylittlebitsyou’dhavetohave,andthat’showbigthey’dhavetobe,butwhenyoucomputethisnumber,thisexceedsthenumberofparallelprocessorsthatanybodyhas,includingtheChinese.[chuckles]Soitisn’tgonnawork.“Mindless,large-scaleparallelcomputing,”that’sthethemeofthebook.Itdoesn’t—always—work!Andthat’stheproblem.Itdoesn’talwayswork.Nextpage,please.Ingeneral,theintervalarithmeticcommunity,andJohn,figuredoutthatifyouranswer’sneverwrong,thenitmustbealwaysright.Butthatisn’tquitetrue.Becauseyoucangetintervalsthatareextremelybig,muchtoobig…Now,how,howbadisthat,isthatreallyabadthing?No,notifweknowtheintervalstoobigbecauseifweknowthey’retoobigyou’lldisregardthemorperhapsnotcomputethematall.Butifyoudon’tknowthey’retoobig,thenyou’vegotaproblem.Shouldyoudiscardaworthwhileprojectbecauseitappears,thatitjustdoesn’twork?Intervalsarejustenormous?So,youaren’tgonnadoit?Or,maybewhatyouwanttodoissomeextraworkyoubetterundertakeextrawork,ormaybebecausetheintervalsaretoobig;youbelievethem.Thisisthehorrorthat’sdonebyoversizeintervals.Withoutthaterroranalysis,or,anotherwayofdoingthecomputation,thereisn’tawaytodiagnosethesekindsoffailures.Andthenjusttomakemattersworse,theproposalforSORNarithmeticlackssomethingcalledalgebraicintegrity,whichunderminesaprogrammer’sfaith,thatarithmeticisatleastanapproximationtoalgebra.Let’sseehowthatis.

25

AlgebraicintegrityisatleastsomethingthattheIEEEStandardhas.Ifthere’snoroundingerrors;ifithasseveraldifferentrationalexpressionsforthesamefunction…thentheyproducedifferentvalues,withoutbreakingit!Butiftheydo,youcouldgetatmosttwodifferentvaluesnomatterhowmanyexpressionsyougot,youhaveplusorminusinfinity,that’sapossibility,orelseyoumayhaveatleastoneNaN,andyoucandetectthat.YoucandetectitbecauseNaNsareobvious,oryoucandetectit—detectitbecausetheInvalidOperationflagisnotraised.ButSORNdoesn’thavethat.Yousee?Itsays“Noroundingerrors…Noexceptions…”andthosearethepagesinwhichyoufindthoseboasts.

SodifferentSORNexpressionsforrationalfunctionswillproducedifferentSORNs,andthereitis,now,uandvhappentobethesamerationalfunction…x,y,z,they’rethesamerationalfunction.Asfunctionsinthefieldofrationalfunctions.ButSORNarithmetic,that’sthreedifferentvalues.Andthere’snoroundingerror,andthere’snoindicationthatthere’sanythingwrongwiththeseguys.Let’sseewhythathappens.

Yousee,ithappensbecause:SORNarithmeticdoesn’tproduceaNaNfortheseexceptionalcases.Itproducesthewholerealnumberset,extendedrealnumbersetincludinginfinity.Andofcourseifyousquarethat,whatyougetisallthenon-negativenumbersincludinginfinity,andnowunlessyou’vegotanintervalthatisn’teverywhere,youcansquishitinanynumberofways,andthat’swhathappensfortheseexamples.But,ofcourseyoucouldask,whywouldgetallthesedifferentexpressionsforthesamefunction?Well,it’sbecause,asnumericalanalysts,weknowthatthereareoccasionswhereoneformula…doesn’tworkverywell.It’llworkinthisdomainbutnotinthatdomain.

26

Andsowedevelopdifferentformulasfordifferentsubdomains,andifneedbe,iftheformulamalfunctions,aslongasyoucandetectit,youcandecideOK,IguessIshouldusetheotherformulawhichmaycostmore,that’swhyIusedthefirstonefirst.

Ifanarithmeticsystemhidesmalfunctions,you’regoingtogetintotrouble.Andit’sgoingtohappentoSORNarithmetic.Let’sseehowthathappens.Well,ithassomethingtodowithwhatarecalledC-solutions.That’sJohn’snameforanimportantapplicationofparallelintervalarithmetic.Yousee,whatwe’dliketofindareallthesolutionsofanequationwithinagivenregion.Icalltheseboxes“coffins”[laughter],itcanbecalledaubox;orit’sjustavectorofintervals.Butitlookslikeacoffin.

Andso,givenanintervalprogramthathasverymoderaterequirements,youhavetobeabletowriteaprogramthatmajorizesthetrueequation,andideally,theintervalversionshouldtakethezeroasthewidthoftheintervaltendstozero,exceptforroundingerrors,soultimatelythere’salimit.It’showsmallyoucanmakeit.Well,here’stheprocedure,andthisisaprocedureforintervalarithmetic,usablebySORNs,andusablebyunums.If,whenyoucomputethisexpression,atacoffin,ifyouknowthatitexcludeszerothenyouknowthecoffindoesn’tcontainthesolutionsoyouthrowthatparticularintervalaway.Youpartitiontheoneyoustartwithintosmallcoffins;youdiscardalltheonesthatcan’tcontainthesolution,andthenyoupartitionalltheremainingonesintosmallercoffins,anddoitagain,justthrowawaytheonesthatcan’tcontainthesolutionandyourepeatit,untilitisn’tworthfurthersubdivision.Andthisis—thebookcallstheseC-solutions.Here’sanexample.I’dliketofindthesolutionsofthisequation.I’llstartmysearchontheintervalthatgoesfromzerotofour,well,that’sabadchoice,becausewhenyoucomputethis,yougetaNaN!OrinSORNarithmeticyougeteverything.Sodespitethat,wemustn’tdiscardthisinterval.Instead,yousubdivideit.Yousee,youcan’tsaythatitexcludeszero.Soyou’vegottosubdivideit.Andwhenyoudothat,finallytheintervalsgetsmallenough,you’llfindsomethatexcludezero,you’llthrowthemaway,andyou’llconvergetothesethreevaluesofz.Ah,buttwoandthreearetheonlysolutions;zequalsoneisnotasolutionofthisequation.Inunumsintervals,youwouldgetaNaN.Thatwouldtipyouoffthatthere’ssomethingwrong.InSORNs,you’dgeteverything;there’snothingwrongwitheverything.Youwouldn’tknowthatthere’ssomethingwrong.SothatsaysthatC-solutionscan

27

includesingularitiesofanequationunlessyoufilterthemout.Letmegotothenextpage,please.

Well,thetroubleisthatifSORNsareneverexceptional,it’sgoingtobeverydifficulttoknowwhetheryoushouldfiltersomethingout.Hereabetterexample.ItusesthatfunctionR.Andit’stwoequationsintwounknowns.Anditturnsoutthatwheneverzerobelongstoabox,it’s—it’llbearectangle,sothere’szero…inSORNarithmetic.Andzerobelongstoaboxwheneveronecornerisalotclosertozerothantheothers.SoitturnsouttheC-solutionprocesswillconvergetotinyrectanglesclusteredaroundzero,plusin

SORNarithmeticyougetonethatencloseszero.That’snotasolution,andyoucanseeit’snotasolutionbylookingatthefactthat,“Oh!Rcan’tbeanybiggerthanone,andIwantittobenine-eighths.”Can’tdothat.Therearenosolutionstotheequation.C-solutionscansolveequationsthatdon’thaveasolution.Andthereareotherwaysforthistohappen.So,what,howmuchshouldwehavedonehereinstead?Wellatthispoint,iftheunumintervalarithmetichadreplacedSORNs,doesn’tmatterifit’sunumsorintervals,andifwe’dusedSinsteadofR,wewouldhavegottenridofbadintervalsandthenwewouldhavegottentheC-solutionprocesstoconvergeto,nointervalatall.But,youchoosethewrongformula…toobad.OK,let’sgoontothenextproblem.Nowlook:Hesayshedoesn’tdisparagecalculus,well,look,here’swhathe’ssaying:Theyshouldworkwhenusedthenaïveway,thewayfloatsareusuallyused.Well,that’strue.Floatsareusuallyusedbyignorantpeople.Afterall,youcangetadegreeincomputersciencewithoutevertakingacourseinnumericalanalysis,atleastyoucaninthiscountry.And,here’s

28

theessenceoftheuboxapproach:Mindless,brute-forceapplication.Youthinkthatmindlessbrute-forceapplicationwillcompensateforignorance.Idon’tthinkso.Calculusconsideredevil,discrete—hey,look!That’sthetitleofthechapter!Andthenhesays,calculusdealswithinfinitesimalquantities.Computersdon’t.Theydon’tcalculatewithinfinitesimals.It’snottrue!Calculusdoesnotdealwithinfinitesimalquantities;itdealswithlimits!Andtheinfinitesimalquantitiesarejustashorthand…fordescribingwhathappenswithlimits.Theyshould—BishopBerkeley,inthe16th—inthe17thcentury,hadthesameproblem,heusedtoworryaboutthe“ghostsofvanishedquantities”…butcalculusisaboutlimits,notinfinitesimals.Ifyouwritewithinfinitesimals,you’rewritingashorthandforcertainkindsoflimits.Wecangointoitlaterifyouwant.Andthenhere:Whenphysicistsanalyzependulums,theyprefertotalkaboutsmalloscillations—whydophysicistsprefertotalkaboutsmalloscillations?Theydothatinordertogetalimitingcase,toshowyouwhathappenswithoscillationsofsmallamplitude.Wecanevenestimatetheerrorintheperiod,ifyouwanttousebiggeramplitude,that’swhygrandfatherclockshavesuchlongpendulums.Becausethatway,theperiodofthependulumdoesn’tvarymuch.Andthat’sthewholepointofagrandfatherclock.Whataboutthis,everyphysicaleffect,nowthisishisboldface,thatheusesthewordsampling,whereIusediscretizationerror.Thisisstraightoutofhisbookonthatpage!Well,hedoesn’tdothat!Hedoesn’tsolvephysicalproblemswithadiscretemodel.Hesayshedoes,buthedoesn’t.Ifyouwant,wecandiscusswhyhedoesn’t.Sohere’swhathereallydoes.Hetreatstimeasafunctionoflocation.Butthatworksonlyifyouknowwhatisconserved,andthereareenoughthingsconserved.Buthowdoyouknowthatenergyandmomentumareconserved?Well,thetotalenergy,kineticandpotential,didn’texistascompu–asconcepts,beforethecalculus!And…Well!It’sjustanunreasonablewayofthinkingofthetimedependencyofphysicalsimulations.He’ssaying,insteadofsayingthephysicalpropertiesdependontime,whathe’sgoingtodoissay,we’regoingtotreatthetimeasafunctionoflocation.Thatmeansyouhavetoknowwherethisthingisgonnago.Butwedon’talwaysknowwhereasystemisgonnago.Eventhethree-bodyproblem,wedon’tknowwhereit’sgonnago.Soyoucan’tdothat!Andinanyevent,youcan’tcopewithdragorfriction.Becausewithdragorfriction,youcan’tpredictinadvancewherethemechanicalsystemisgonnago.Nextpageplease.OK,…yes?DEMMEL:Fiveminutes.

29

KAHAN:Well,Iwillprobablyfinishinfiveminutes.Sohearme.[chuckling]

Thebooksays,here,Iwillofferanarbitrarilyprecisesolutionmethod,nocalculus!Justelementaryalgebra,wellthat’swrong.Here’sacounterexample.Yousee,here’sadifferentialequation.Itlooksreallyeasy,doesn’tit,Imean,there’sno…highermathematicshere,that,gravitythat’sconserved,istranscendental.Youcan’tgetfromhere…tothere,withoutthecalculus.Andifyoudon’tgetthere,thenyouhaveamorecomplicatedcalculationthanyou’dlike.

Well,youcanlookatthenotes,andyou’llseehowthatworksout.Now,youhavetoevaluateanintegral,ifyou’vegot,onceyou’vegottheseconservedquantities,you,whatyouneedistoreducethesolutiontotheevaluationofanintegral,sometimesinclosedform,asfor,theKeplerianorbits,they’reinclosedform,butfortheswing,youhavetoactuallyevaluateanintegral,so,let’sseewhathesaysaboutthat.Hesays,“Hereisanerrorestimate.”It’sanerrorbound,actually.Andit’sforthemidpointrulebuthedoesn’ttellusthat.Andthis,thisisaliteralquotefromhisbook!“Whatthehellisthat?”Itturnsout,thatyoudon’thavetoknowcalculustodifferentiate,becausethereareprogramsthatwilldoitforyou!Ifyouwriteaprogramtocomputeafunction,youcansubmitittoanotherprogramthatwilldifferentiateyoursandproduceanewprogramthatinterleavesthecomputationofyourfunction,anditsfirst,andifyouwantit,second,derivative.Soyoudon’thavetoknowhowtodifferentiate,youjusthavetoknowthatyouneedone.Andyoudon’thavetofindthemaximumvalueofthesecondderivativeinordertogetanerrorbound.Nextpage,please.Yousee,therearealgorithms—now,thebook’salgorithm,theworkisoforderoneovererrorsquared…forreasonablefunctions.NowhesaysIappliedacasewhen,oh,thesecondderivativewasinfinite.AndImentionedthat,becausethesecond

30

derivativeisinfinite,insteadofgettingworktogodownlikeoneoverthesquarerootoferror,itgoesdownalittlebitslowerbecausethesecondderivativeisunbounded.Butthatdoesn’tmatter;it’sstillfasterthanthisbyanorderofmagnitude.Andinanyevent,thefastestalgorithmstheygothisway,andthisisfasterbymanyordersofmagnitudethantheotherstuff.But:theseschemesgiveyouintervalbounds,andthatonedoesn’t.Youhavetoknow

somethingaboutasymptotics.Youhavetofollowestimates,inordertosee,asymptotically,howtheerrorisdiminishinginordertogetanyerrorestimate.But,yougetitsomuchsooner!Andso,younowhaveatradeoff:Wouldyoulikeananswerrealsoon,orwouldyoulikeitalotlater,butabsolutelyguaranteed?Well,sometimesyoupreferonetotheother,andinanyevent:It’snottimetooverthrowacenturyofnumericalanalysis,notyet,andit’sallirrelevant.Allthisstuffabout,youdon’tlikecalculus,andyoushouldusethisnumericalmethodorthat,thathasnothingtodowithcomparingunumcomputingwithwhatyoushouldcompareitwith,whichisintervalarithmetic,butintervalarithmeticwithprecisionvariableatruntimeandsupportedbyanappropriateprogramminglanguage.Thatisthefaircomparison.Let’slookatthat.Whatdoesunumcomputingcost?It’stoomuch!Becausethewidthscanvaryalmostarbitrarily.They’reintendedtobepackedtogethertightly.Solet’sseewhathappens.Well,thearithmeticisgoingtohavelargelatencybecausetheunpackingwillrequiremorepipelinestages.Oh,it’snotjustthatwehavemoretransistors;it’sthewires!Whenyouhavealotmoretransistorsthenyouhavecomplicatedstuff.You’regoingtohavetolayoutwires,that’swhat’skillingyou.

31

Memorymanagement.Thebookoverlooksthecost—itsays“doestheprogrammerhavetomanageit,no,no,no,thecomputercandoit”,yeah,foraprice.Thepricecomeswhenyouwanttowriteunums;fetchingisnottoobad.Idon’tthinkyou’regoingtowanttoskipthroughtheunumsasalinkedlist.Youwouldprobablyprepareatableofaddressesinadvance,togowithanysetofunumsyou’regoingtoread,ifallyou’reevergonnaeverdoisreadthem.Thatwouldspeeduptheprocess,andwouldbetolerable.Butifyou’regonnawriteunums,thatcanchangetheirwidth,oh,that’samuchmoredifficultproblem,becauseyoucan’tputthewiderunumbackwhereyougotit.Sothatmeansyou’vegottouseaheap.Andifyouuseaheap,thenyou’regoingtohavememorydefragmentationandgarbagecollectionproblems,ohcomeon.You’vejustignoredthecostofthosethings,andinanyeventthecostdependscruciallyonhowyourprogramminglanguagemanagesdiversewidths.Nextpage,please.DEMMEL:So,so,you’verunoutofyourthirtyminutes,so—KAHAN:Well,justonemoreminute,IthinkI’malmostonthelastpage.[laughter]

WhatdoesSORNarithmeticcost,itturnsout,SORNarithmeticcouldwork,ifyouhavesufficientlylowprecision.Anditwouldn’tbeanunreasonablething.Youwouldn’tdoitinthefastestpossibleway,butyoudoitwithaslowerway.Nextpage.

32

Howmuchprecisionisenough?Youseethatlittleruleofthumb?Well,thatruleofthumbhasworkedforaverylongtime.Andofcourse,itdoesn’talwayswork.Butitworkssooften,thatwhatyouhavetoaskyourselfisthis:Wouldyoulikearesultsoon,orwouldyouliketowaitforitandbeabsolutelysure?Formanyresultsitisn’tworthbeingabsolutelysure,becauseit’llcosttoomuch.Imeanafterall,forcomputergames,whocares?Supposeyouhaveatransientroundingerrorprobleminyourcomputergame.It’llbegoneinamoment.

Theonlysurewaytododealwithit,is,erroranalysis.Andwhatcanbedoneinstead?Well,we’llhavetodiscussthatlater.Thereissomethingtodo.

Buthere’stheimportantthing.TherearesomethingsthatIdon’twanttoknow.AndwhatIdon’twanttoknow,Idon'twanttopaymuchfor.Here’swhatIdowanttoknow,butitturnsoutthatthiskindofthing,uncertaintyquantification,thisiswhatisnothandledwellbyinterval,SORN,orunumarithmetic.It’sableedingsore.

33

Andthenthere’sdebugging.Ah,well.Wespendatleastthreetofourtimesaslongdebuggingaswespendwritingourengineeringandscientificsoftware.Andsothatshouldweighonourminds.ButdebugginginunumarithmeticandSORNarithmetic,thatisgoingtobearealheadache.

OK,Icanquitnow.GUSTAFSON:MayIrespond,rightaway,tothatlastone?

DEMMEL:Letusthankboththespeakers.[applause]DEMMEL:Letthedebatebegin.GUSTAFSON:So,regardingdebugging;ifyouaddinformationtoanumber,doesthatmakeithardertodebugoreasier?BecauseallI’vedoneistakenfloatingpointsandI’veaugmentedthemwiththreefieldsthatareself-descriptive,thatassistinfindingoutwhat

34

wentwrongwiththecalculation.I’msurprisedbythisattack.Ithinkyoushouldjoinmeinactuallysupportingthiskindofadditionofinformationtoanumbersoitwillactuallymakeiteasierforpeopletofindoutwhat’swrongwiththecalculation.KAHAN:Theymayfoundout…thatit’swrong.Butthatdoesn’treallytellthemwhattheywanttoknow:What’swrong.[struggleswithmicrophoneswitch]GUSTAFSON:MayIhelpyou?AUDIENCEMEMBER:Here…isn’titon?KAHAN:It’sgotatinylittleswitch…GUSTAFSON:Pointitatyou.KAHAN:[morestruggling]OK,Ithinkit’sworkingnow.OK,it’sonethingtoknowsomethingiswrong.It’sanotherthingtofigureoutwhyit’swrong.GUSTAFSON:Sodoesithurtorhelptohaveextrafieldsthattellallyouaboutwhat’sgoingon,withtheuncertaintybit,andwhereithappened,andhowbigitisandatwhichbititwasnotgood…doesthatmakeithardertoactuallyusenumericalmethodsandnumericalanalysis?KAHAN:Well,whatIwanttoknowiswherethishappened,inmyprogram.GUSTAFSON:SoifyouhaveasignalingNaN,itwillstopthesystem,andwillgototheoperatingsystemwherethereisacallstack,andyoucanfindoutexactlywhatroutinecalledit,andwhatinstruction,whatlineofthecode,infactyoucanbringupthesourcecode.Thisiswherethishappened.Soinsteadoftryingtoencodenumbers,hashpointerstowherethingswentwronginthefivequintilliondifferentkindsofNaNthatissupportedin64-bitfloats,allyouhavetodoisstop,andlettheoperatingsystemtell,“Thisiswhereyourproblemis.”SoIbelievewhatweneedisaquietNaNthatsays,“Keepongoing,”oranoisyNaN,asignalingNaN,thatwillstopitandgiveyouallthedebuginformationthatyouneed.Thewishthatsomehowpeoplearegoingtostartusingallthesemany,manydifferentkindsofNaNtoencodedebuginformation…neverhappened.Peopledon’tuseit.Itdoesn’texist.

35

We’vewaitedthirtyyearsforthistohappenandithasn’thappened.Idon’tthinkit’sgoingtohappen.

KAHAN:Well,ithashelpedme,onsomeofmyoldcomputers.[chuckles]Itdoesn’thappenonmoderncomputersbecauseMicrosoftwon’tgiveyouthesourcecodefortheiroperatingsystemsoIcan’tfiddlewithit.[laughter]Butwait;whatyou’resayingabouttheotherwillworkifyou’redebuggingyourowncodeandit’ssimpleenough.Butwhenyou’redebuggingyourcodeandit’sacomposite—it’sgotyourstuffinit,andit’sgototherpeople’sstuffinit—youcan’tjuststop.Because,ifyoujuststop,it’llbetoolate.Youwon’tknowwhat’sgoingon.You’llstopinthemiddleofsomebodyelse’sprogram!Andyoudon’tknowwhereitis.Youdon’thavesourcecode.Andifyoudid,youwouldn’twannareadit.GUSTAFSON:Sowhatisyourconstructivesolutiontothis?KAHAN:Well,thesolutionis,toknowfirstofallwhosecodecausedtheproblem,andsecondly,whatkindofproblemwasit.Andifthedebuggercooperates,youcaneventellaguywhereithappened,whenyouaskhimtofixit.GUSTAFSON:Soundsgreat.Idowanttocommentonsomeoftheotherthings.Fromtheveryfirstpagethatyoupresentedwhereyousayaunumiseitheranexactnumberorwithinone-halfULP,IknewIwasgoingtohearanotherhalfhourofcompletemisunderstandingsofeverydefinitioninthebook.Thatisnotwhataninexactunumis.Aninexactunumistheopensetbetweentwoexactunums,wherethoselookexactlylikefloating-pointnumbers.Icannowmarkandfillinalltheopenspacesoftherealnumberline.There’snothinglikeaplus-or-minus-halfroundingthatweget.Butthehabitofthinkinginthosetermsisbendingto,“Let’sbendthistosomethingIcanattackandcritique”eventhoughthat’snotwhatIsaid.

36

“Auboxisthesameasacoffin.”“Coffin”isaclassicKahandysphemism.“Let’sfindthemostnegativewordwecanusetodescribesomething.”Ifyouhaveintervalarithmetic,youdogetthesegreatbigboxes.Theygetbiggerandbigger;like,fivetoten,andthensuddenlytheyjustblowup.Prettysoonyourboundisminusinfinitytoinfinity.UboxesareoneULPonaside,orareexact,inwhatevernumberofdimensionstheyare.Theyarenot,like,fromthreetofour,orwhatever,andtheysplitupintosmallerboxes.Soit’slikeconfusingaboxofLegoswiththeLegosthemselves.YoucanbuildalotofthingsoutofLegos.Youcanbuildallkindsofshapes.It’snotjust“abox.”And…thedefinitionsthatyousawhere,ofwhatis—it’salwaysjustaslightmisstatementofwhatIsaid.“Let’swarpitoverhere,andthenmakeitsaysomethingthatwasn’tsaid.”Itwasn’tsaidthat,“Icansolveallordinarydifferentialequations.”Isaid,“Here’soneyoucansolve,andit’saninterestingapproach.Itworksforthisone.”Ofcoursethere’stonsofthingsIcan’tsolvethatway.There’slotsofproblemsIcan’tsolve.And,Ievenmentionsomeoftheminthebook.Ididnotexaggeratetheuniversalityoftheseapproachesthewayyou’redoinghere.Isaid,“Itworksforthis,itworksforthis…thinkabouttheotherwaysyougetthemethodtowork.”KAHAN:Well,theonlythingIcandoisquotefromthebookasaccuratelyasIcan,andIthinkIhave.IthinkI’verepresentedyourideasfairly.You’reprotestingbecausenowyouseetheconsequencesofyourideas.You’retryingtopersuadepeoplethattheycandocomputations,someofthemverycomplicated,withouthavingtounderstandwhat’sgoingoninsideofthemachine,what’sgoingonmathematically.AndIclaimthatthatisafalsepromise.GUSTAFSON:Forallthings,yes.KAHAN:For…?GUSTAFSON:Itwon’tworkoneverything.Ofcourseitwon’t.KAHAN:That’sright.Well,andthen,ifyoudon’tknowwhenitwillwork,andwhenitwon’t—GUSTAFSON:Here’stheproblem.Asyou’vesaidmanytimes,wehaveevermoreambitiouscalculationsthatweareattemptingwitheverlessnumericalexpertise.WehaveamilliontimesmorecomputingpowernowthanwehadwhentheIEEEStandardwasformed.Whatcanwedotoimprovethissituationandmakecomputersalittlebitmorerobustandlesserror-proneandmoreself-managingwhenitcomestonumericalcalculations?Ithinkwehavenowtheabilitytodothingswithallthosetransistorsthatcanactuallysavebandwidth,whichisanotherthingIwanttocorrectyouon.It’sasubtlething,butyoutalkedaboutmemorymanagementbeingveryexpensive,it’s

37

thewires?Themanagementisnotexpensive.ThemanagementisinsidetheCPU.That’scheap.It’sactuallymovingthedatathatcostsyou.Itistryingtoreducethenumberofbits.ThereasonIinventedunumswastoreducethenumberofbitsgoingbackandforthbetweentheDRAMandtheCPU,totheabsoluteminimum,becauseotherwisewe’renotgoingtogettoexascale.Itwillactuallyblowourpowerbudget.WejustheardfromtheStanfordprofessor,actuallythequestion-and-answer-session,Isaid“whataboutDRAM?”andhesaid“Oh,that’ssixtytoseventypercentofalltheenergy.”Soifyoucancutthatinhalf,nowyou’regettingsomewhere.

KAHAN:Ohno,no,you—you’reignoringthefactthatiftheunumsvaryinwidthinthecourseofthecomputation,you’regoingtohavetoputthemontoaheap.Andifyouhavetoputthemontoaheap,thatmeanswithaunum,you’vegottafetchanaddress,also.GUSTAFSON:Right.Soyouwillgobetweenfixed-sizestorageunums,andoccasionallywewillwrapthemallupandsend

themofftodiskorsendthemofftoDRAM,butwewilldoitintelligentlyasneeded,butmostofthetimetheywillprobablyexistinunpackedform.KAHAN:WelltheexampleinyourbookwiththeFastFourierTransformdidn’tdothat.GUSTAFSON:TheFastFourierTransformactuallycanbearrangedsoyouarealmostalwaysaccessinglinear-ordered,contiguousgroups.Itispossibletodoanin-placeFourierTransformthatkeepsonrearrangingthedata,alwayspacked.Butthatsaid,Iwouldcertainlyunpack,probably,anFFTtomaketheindexingsimple.Itwoulduselessenergyanddoabetterjobofboundingtheanswerwithfewerbits.KAHAN:Sothatmeanssomebodycan’tusethisthingmindlessly,canthey?GUSTAFSON:Notcompletelymindlessly,no.Ididmaketools,aswhich—certainthingsactuallycanbemanagedwithlessefforttofindwhat’sgoingwrong.Because,asIsaid,I’maddinginformationtoanumber;I’mnottakingitaway.So,ifyourobjectionisthat,Iusedtheword“mindlessly,”andofcourseyoucan’tcompletelyuseacomputermindlessly—wewish!—youhavetousesomeamountofthought.Butdoeseverybodyhavetoknownumericalanalysis?Doeseveryonehavetoknowcalculus?Youknow,onlyabouteightpercentofthepeopleintheworldknowcalculus?Therehavebeenstudiestotrytoestimatewhatistheoverallnumericalliteracy

38

ofthehumanrace,andifonlyeightpercentofthepeoplecandonumericalmethodsoncomputers,uh,we’reintrouble.KAHAN:Areninety-eightpercentofthepopulationgoingtowritenumericalprograms?[laughter]Look…GUSTAFSON:Therearelotsofpeopleusingspreadsheetswhodon’tknowwhatthey’redoing.KAHAN:Iunderstandthedesiretomakeiteasierforpeopletousecomputers…tomakeitpossibleforthemtodomore,andknowless.Andthere’sthislittleboxhere,theHP-15cwhichisexactlythat.

Yes,youcanevaluateanintegral.Youcansolveanequation.Youcandocomplexvariables…onthistinycalculator.SoI’minfavor,ofmakingthingseasier.ButI’mnotinfavoroftellingpeoplethatyoudon’thavetoknowanything.Youhavetoknowsomething,andit’sdifficulttosaywhatyouhavetoknow,butunlesspeoplearewillingtolearnalittlebitaboutnumericalanalysis,theyareverylikelytoharmthemselves,or,you.Weallaregoingtodependuponsoftwarethatwegetthroughtheweb.Andifanyidiotcanwritethesoftware,andthensenditoutandsay,“Hey,I’musingunums,soitshouldbeOK,”well…GUSTAFSON:EitherOKoritwillsay,“Look,youjusthadabigerror!You’dbetterfigurethisout,”whereaswithfloats,itwon’ttellyouathing.Atleastyougetthewarningwithunums.Youknowsomethingwentwrong.KAHAN:Butthereisaway,withfloats,wesawadiscussionaboutyesterday,whensomeonedescribedMonteCarlorounding.Now,he’sgottherightidea,exceptawrongimplementation.Let’sseeifyoucansimplifytheimplementationalot,andreducetheprice,alot,andstillgetthebenefit,whichsays,withveryhighprobability,you’lldiscoverwhetheryourresultsarecontaminatedexcessivelywithhighroundoff.And,whenitcomestouncertaindata,that’sthesamesortofthing,thatifyoure-curvethedataandrunalotofexperiments,youcanfindthatout.Butwith

39

unumcomputation,youcanconclude,becauseyousay,“Well,Iputinmydataasunums,eachoneisuncertainbyahalfaunitinthelastplace,becausethat’sasmanybitsasyouhaveperunum[laughter]work,howuncertaintheresultis,becauseoftheuncertaintyofdata,wellthat’sveryoptimistic.

DEMMEL:SoIwouldliketojustbrieflyask,arethereanyquestionsfromtheaudience?Wecanletthedebategoon,butyouare,youknow,closetosix-thirty,andIjustwantedtomakesurethat—AUDIENCEMEMBER:Well,Ihaveaquestionforyou,Dr.Demmel.Alotofpeopleuselibraries,suchasLAPACK,thatIknowyou’reexpertin,orstatisticalpackageslikeRthatmaydependonlargelibrariesunderneath.What’sgonna—whatdoyouthinkmighthappeninaworldwithadifferentnumberrepresentationwitherrorencodedinit?Runningthroughtheselargelibraries,whichyou’vedone,alotoferroranalysisyourself.

DEMMEL:SothisisonequestionthatyoualreadyhaveaddressedandIwasgoingtoaskthat,butletmeaskyou:SoimagineIwanttowriteamatrixalgorithm.AndIrefertosubmatricesallthetime,andaskfortheminotherroutines,andsoifalloftheobjectsarevariablesize,referringtoasubmatrixsuddenlybecomesacomplicatedkindofthing.Wouldyousaythat’ssomethingtobeunpacked?

40

GUSTAFSON:Absolutelyyouwouldunpackthat.Andaslongaswe’retalkingaboutlinearalgebra,thebiggestbenefitoftheunumdefinitionanditsenvironment,isthesupportofthingslikefuseddotproduct.Soyoudon’tdoanykindofroundingorapproximationuntilyou’vedoneacompletedotproduct.ThisistheideafromUlrichKulischthatheunfortunatelypatented,whicheffectivelypreventedtheworldfromgettingtoitforhoweverlongthepatentlasted,andnowthepatenthasexpiredsoIthinkitdeservessomeveryseriousconsiderationagain.Becausethisisavery,verygoodwayto,atapossiblyslightcostoftime,getamuchmorereliableanswer,thatwillobeyassociativityforexample,aslongasyou’redoingthecalculationexactly,inthescratchpad.Sothequalityoflinearalgebracouldimprovevastly,simplybyusingexactdotproductsthroughoutthekernels,andnotmakingthatroundingerror.DEMMEL:So,soIjustwantedtopointoutthatourphilosophyinbuildingLAPACKhasalwaysbeenthatifwecandosomethingthatwillinfactkeeptheerrordown,and,butthenumberofpeoplewhoactuallyusethoseerrorboundsisincrediblytiny,comparedtothenumberofpeoplewhotypeAback-slashb.GUSTAFSON:Yes.DEMMEL:Andsothat’swhyI’mjust,andso—we’renotgoingtogiveupthatphilosophy,butyouknow,it’s—it’shadnotasmuchimpactaswewouldhaveliked.And,sothequestioncomesupin,youknow,alltheapproacheswe’retalkingaboutheretoo,isthat,isitworththecost.KAHAN:Ithinkyou’veusedtheword“unpacking”inaglibway.Because,youcanunpackifallyou’reevergoingtodoisread.Butifyou’regoingtowriteunums,youhavetoallowthatunumsmayvaryinlength.Andifthat’sthecase,you’vegottounpackthem,andthenyou’vegotsomethingwider,whereareyougoingtoputit?GUSTAFSON:You’retalkingaboutthemanagementofaheap.You’vetakensomethingoutoftheheap,andnowit’sgottenbigger,soyoucan’tputitback,that’swhatyou’resaying.KAHAN:Youcan’tputitbackwhereitwas,that’sright.GUSTAFSON:You’renotunderstandingwhat“unpacking”is.Anunpackedisatthemaximumsize.Soyouallowthemaximumsizethataunumcanbe.Whichstillmightbeaveryusablenumberofbitsandaverylargeexponentrange.TheextrabitsthatyougetcanbeusedtomarkwhethersomethingisaNaNorinfinitysuchthatasingle-bittestactuallyisfasterthanafloat.

41

KAHAN:Soyou’vegottamakeprovisionforwhatmaybethelargestwidththatwillturnupinacomputation.GUSTAFSON:Whenit’sontheCPU,itprobablywillpaytodothat.DEMMEL:So,soletmejustcontinuethatline…ifyouwanttoimplementGaussianelimination?Andyouwanttouseblocking,whichiswhatthestandardoptimization—KAHAN:Isyourmicrophoneon?Wecan’thearyou.DEMMEL:OK,letmetryonemoretime.So,let’saskhowwewouldimplementGaussianeliminationwherewewanttousethestandardoptimizationtogetittorunatdecentspeed,whichisblocking,soitseemslikewe’dhaveto—sowe’regoingtobewritingtothatmatrixallthetime,becausethat’showGaussianeliminationworks,soitseemslikethealgorithmwouldhavetoruninsortofstandardarithmetic,basically,becauseyou’regoingtoreadingandwritingthematrixoverandoveragain,andso,you’reeffectivelyrunninginstandardfloatingpoint.Maybewithyourrepresentation,butthesemanticswouldbethesame.GUSTAFSON:Itwouldbesimilar.IthinkI’dfindawaytodoituptoacertainpointwhereassoonasthematrixgetssolarge,thatyoudiscoveryouhavetostoreit,say,ondisk,thenit’sworthpayingthepricetopackandunpackthesethings.It’salwaysanoption.Youdon’thavetodoit.Butrightnow,memoryistwohundredtimesslower.IttakestwohundredtimesaslongtofetchanoperandfrommainDRAMasitdoestodoamultiply-addonit.Youcangetanawfullotofmanagement-typethingsdoneinthattime,andfigureoutwhat’sgoingtogowhere.Inyourgutyousaid,“Oh,that’sgoingtobereallyexpensive!”Specitout!Actuallydothecalculationofhowlongthesethingstake.It’snotsomethingthatyoucanjust,fromanarmchair,say,“Oh,that’sgoingtobereallyexpensive.”You’dbeverysurprisedatwhatthedesignrulesof2016sayyoushoulddo,asexpensiveversusnotexpensive.MovingdatabetweenDRAMandaCPUisachoke.Youwanttodoaslittleofthataspossible,andyouwanteverybittocount.That’swhatI’mtryingtoaccomplish.Soanytimeyoucan—DEMMEL:So,Iwouldhopethat,maybethatafuturechallengeistowriteoptimizedblockGaussianeliminationusingyourarithmeticandseewhattheoverheadmightbe.GUSTAFSON:Sure.

42

KAHAN:Well,there’sasecondproblem.Whenyouminimizecommunicationscosts,youendupusingalgorithmswhich,insomesense,havetoanticipatewhatisgoingon.Inthatcase,youmayrequirerathermoreprecisionthanyouthought,inordertosurvive.AndIgiveacitationtothatononeofthedocumentsonmywebpage,whereagraduatestudentworkeditoutandfound,“Mygoodness!Youmayhavetousedoubleprecisionormore,whenactuallyyourdataisinfloats!Or,smaller!”Thismeansit’shardforyoutopredicthowmuchprecisionwillbeneeded,ifyou’reusinganalgorithmthateconomizesverygreatlyonmemorymovement.Thealgorithmsthateconomizeonmemorymovementaresufficientlystrange,thatsomeofthemrequireextraordinaryprecisioninordertosurvive.DEMMEL:SoIguesswe’retalkingaboutsolvingleast-squarednormalequations,justtobe—KAHAN:Exactly.DEMMEL:—properly,becauseyouwanttocomputeafewxtransposexandhelp,yousortofhavetodothataheadoftime.Andthere’saquestion?AUDIENCEMEMBER:Yes,Ihaveaquestionaboutthecostofperformingwithyourbasicsetunum.Supposeyouhadaniterativebigsolver,whereyoumultiplyamatrix.Youhavethisnicesize,let’ssay,128by128.Chancesare,everytime,ifyouimposealimitonthenumberofbitsyourepresent,everytimeyoudoanFMA,practically,youwellenduprounding.Butwithunumsyou’regoingtohaveimprecisebitset.Right?OK.NowformatrixmultiplicationIhavetodo,inthiscase128FMAs,oneaftertheother,andchancesareveryhigh,everystep,I’mgoingtohaveanimprecisevalue.SoifIunderstandcorrectly,aftercomputingoneelementofmymatrixI’mgoingtoendupwith128boxes.GUSTAFSON:You’retalkingaboutthedotproduct?AUDIENCEMEMBER:—representationbutthepointI’mmakinghere—GUSTAFSON:Thisisthedotproductyou’retalkingabout?You’redoingmatrix-vectormultiply?AUDIENCEMEMBER:Youknow,matrixmultiplication,whereyoutaketwovectors,youknow,togetoneresult,right?

43

GUSTAFSON:Thereisoneroundingerrorattheveryend.Or,itwouldn’tbecalledaroundingerror.It’sdoneasifitwasanintegermultiplication.AUDIENCEMEMBER:Mymainquestionis,ifyouhaveanimpreciseresulteverytimeyoudoaplus,howmanyunitsofstoragedoyouneed,tostoretheresult?GUSTAFSON:Myanswerisyoudonotdoitinaninexactaftereveryplus.Youdotheentirevector;youdoadotproductallinexactarithmetic.Then—AUDIENCEMEMBER:Soyouareproposingimplementinganarithmeticunitthattakestogether128values—GUSTAFSON:It’scalledtheExactAccumulator.AUDIENCEMEMBER:Andthat’s,exact—KAHAN:Look!Youdon’tuse—AUDIENCEMEMBER:Becauseifyouuse1024matrices,Ishouldbeefupmy—KAHAN:OK,look.Mostofthetime,withextremelyrareexceptions,youcandoascalarproduct,accumulatingtheproductswithevermanysifyouwant,oryoucanaccumulatetheminvariablesthathavetwicethewidthoftheoperands.SoyouhaveAtimesB,sayAandBarefloats,AtimesB,youcomputethatindoubleandadditintoadoublesum.Andalmostalways,whatyou’llgetwillbeperfectlysatisfactory.

GUSTAFSON:“Almost.”KAHAN:Andthat’soneofthereasonswhyKulisch’sscheme,whichwouldevaluateitexactly,isn’tverypopular,isbecauseithardlyaddsanythingtothevalue.Onlyinextremelyspecialcircumstances.Soforyourpurposes,ingraphics,Ithinkthat’swhatyouhaveinmind,right?AUDIENCEMEMBER:Not,notreally.wehaveGEMMs,allovertheplace—

44

KAHAN:Ican’thearyou.AUDIENCEMEMBER:GEMM?Right?MatrixMultiply?Ineedmatrixmultiplication,andmybigquestionis,“howmuchmemorydoIneedtostoremyresult,underrandomcircumstances?”KAHAN:Well,well,whateveritis—AUDIENCEMEMBER:IfIuseunums.Iunderstandwhatworksforfloatingpointandhowtodealwitherror.I’mjustcuriousforunum,howmuchmemorydoIneedtostoremyresults,orhowmuchpriceIneedtopayfortheexactimplementation?GUSTAFSON:Areyouafraidthatsomehowit’sgoingtobecomeagiganticnumber?Isthatyourfear?Becauseyousettheenvironment;yousay,“Iwantamaximumofthismanybitsofexponent—AUDIENCEMEMBER:Soit’sareallybigproblemI’mfacing.BasicallyIunderstandthebenefits.Idonotunderstandthecost.GUSTAFSON:AndI’mtellingyouthat—AUDIENCEMEMBER:WhatIheardtoday,it’seithergoingtobeanextremely,extremelycostlyarithmeticunit,orit’sgoingtobeanextremecostbecauseofstorage.ButifIuseunums.Imeanifit’smy—GUSTAFSON:Idon’tthinkyouunderstand.It’snotunlikesaying:“I’mgoingtodothisindoubleprecision.”Youknowtheresult’sgoingtobe64bits.I’mgoingtosay,“ThisisthebiggestexponentIcanhave;thisisthebiggestmantissaIcanhave;that’sthesizeyoucangetto.”It’snotgonnagobeyondthat,withoutyourcontrol.Youcanset—youcantrythat,andseeifitworks.Andifitdoesn’twork—AUDIENCEMEMBER:Letmemakethissimpler.SupposeyouhavevalueA,andvalueB.Andtheyarebothinexact,right?Whichmeans,theonlythingto,youhavetoworryabout,whatevertheUnitintheLastPlaceyougetwiththeseunums,Iaddthem.AplusB.Allofasuddeninsteadof,let’ssay,oneinexactbitattheend,oneunumbitattheend,I’mgoingtohavetwounumbits.GUSTAFSON:What?DEMMEL:Whatwidthhavetheintervals.Whenyouhavetwointervals.Width—AUDIENCEMEMBER:SoI’maddingtwonumbersthatarenotexact,Icannotstoretheresultusingonlyoneunum.GUSTAFSON:Sometimes,youcan.

45

AUDIENCEMEMBER:Ihave—Sometimes,butifyoudon’tknowtheupdate,no,youcannot.SoIwillhave,forsimpleaddition,twoinexactnumbersIhavetostore;Ihavetodoublemystorage!GUSTAFSON:Yeah.AUDIENCEMEMBER:Nowthisisonestep—GUSTAFSON:Youreallyneedtoreadthebook.It’s—AUDIENCEMEMBER:Andwe’retalkingabouttheverysamematrixmultiplication,whichwillenduphavinganNcubedcostintermsofstorage.GUSTAFSON:No.AUDIENCEMEMBER:Ontheaveragecase.Thatseemsalittlebitextreme.GUSTAFSON:That’snotthewaytheywork.AsIsaid—AUDIENCEMEMBER:Andmyquestionisthen,how?GUSTAFSON:You’regettingintosomethingwhereyou’djustabouthavetoreadtwoorthreechaptersofthebooktounderstand—AUDIENCEMEMBER:Can,canyouexplaininafewwordwhyamIwrong.GUSTAFSON:Well,firstofall,ifyouhave—let’ssayyournumbersare—whereyouwouldstorefloats,youhavereplacedthemwithunums.Soyouhaveinexactquantities,here.You—AUDIENCEMEMBER:It’snotaboutreplacingwithunums.It’sabouthowunumswork.Idon’tthinkyouexplainedthat.Youmadeaclaimabouthowwonderfulitwouldbe—GUSTAFSON:Ingeneral,aunumisnotaclosed—AUDIENCEMEMBER:—tohaveaworldinwhichthingsworkmagically,butIdo

notunderstandthemagic.That’smyproblemrightnow.KAHAN:Ithinkintervalarithmeticwasdesignedtohelpyouwithyourproblem.Namely,eachofyourargumentsisuncertain,andifyou

46

use,nottheconventionaltworealnumber,brackets,forintervals,butyouusethecenterandradiusschemeforintervals,thenyou’llfindthatnomatterhowwideyouroperandsare,iftheuncertaintiesaresmall,youwillnothavetoworryaboutagreatdealofstorage,ortheaccumulationofuncertainty.But:youwillhavetoworryaboutthefactthatifyouruncertaintiesarecorrelated,thecorrelationwillnotbetakenintoaccount,andyoumayendupwithaverypessimisticestimate.Butotherwisethanthat,intervalarithmeticwasdesignedtocopewithyourproblem.IthinkIunderstandit.Andthereisn’tasimplerwaytodealwithit.Ifyouknewinadvancethatnoneofyourargumentswasmoreuncertainthan,say,onepartinamillion,thenitwouldbepossibletosaysomethinginadvanceabouttheinherentuncertaintyofthescalarproduct.Butitwouldbeverypessimistic.DEMMEL:Soletmejustsay,we’vebeenremindedthatdinnerisatseven,[chuckles]andsoIwouldliketoinvitebothofourspeakerstomakesomeclosingremarks.Somaybe,oneminuteofclosingremarkseach.Andthenofcourse,thedebatecancontinueatdinner,ifyoudesire.[laughter]GUSTAFSON:AsIsaid,wehavemuchmorepowerthanweusedtohave.MaybeIdidn’tgetitright,notonallcounts,butItriedtoconvincepeople—getpeopletothinkaboutthefactthatweneedtousethevastadvancesintechnologytoimprovethequalityofcomputerarithmetic,andwehavetheopportunitytodoso.IwouldverymuchliketodosowithDr.Kahan’shelp,notasanadversary.HehasbeenamanI’veadmiredallmylife.I’vebeendoingnumericalworksinceIwas15.I’vebeenmakinguseofhiscalculatorssinceIwasin—probablyafreshmanincollege.I’velearnedsomuchfromthisman.Idon’twantto—I’mnothappybeingontheothersideofthefencefromhim,becauseIthinkwedowanttoaccomplishmanyofthesamethings.

Hemayarguewithmystylealittlebit,oralot,butIthinkwereallydowanttoseethesamethingcomeoutofnumericalanalysis.

47

KAHAN:OK.Mypositionisthatweshouldnottrytooverselltheschemes.Weshouldtrytobeconservativeinwhatweoffer,becausepeoplewilldependonit,andtheydependonthingsthatcanfailfromtimetotime,thenit’snecessaryforustoestimate:Whatistheinsurancepremiumthatshouldbepaid,incasethesystemfails?So,ifasystemisn’tfoolproof,whatistheprobabilityoffailure,howmuchwillthefailurecost,andwhenyouputthatalltogether,youaddthatupoverallthepossiblefailuremodes,andthatgivesusanideaofhowmuchaninsurancepremiumshouldcost.There’sthechoice.Eitherthesystemhastobefoolproof,or,ifyousaythatit’sfoolproof,butitisn’t,thenIwouldliketoknowwhatinsurancepremiumwillLloyd’sofLondonchargeyou?Or,youcantellpeople,“Ifyouwanttheanswerinahurry,you’regoingtohavetotakeacertainrisk.Andwe’reaccustomedtothatinlife.Therearesomesituationsinlifewhere,youhavetoacceptcertainrisksinordertogetthingsdoneinareasonabletime.Innumericalcomputation,there’salotofthat.Anditdoesn’tmeanthatwe’rejustguessing.Itmeansthatwetrytostrikeabalancebetweengettingsomethingdoneinareasonablelengthoftimeandhavingreasonableconfidence.

Idon’tthinkthattheworldofcomputationisquiteasuncertainasJohnsaysitis,butthen,sinceI’manumericalanalyst,Iwouldseethingsdifferently.Idoagree,however,thattherearepeoplewhowillwritecomputerprogramswhoarecompletelyinnocentofanyexposuretothehazardsoffloatingpoint.Iagreethatthatisgoingtohappen.Andunfortunately,itisgoingtohappennomatterwhatJohnandIsayordo.[Applause]DEMMEL:So,hopefullythisvideowillbemadeavailabletoallofyousoyoucanreviewalltheclaimsandcounterclaims,andreadallthedocuments.■

48

/ / / / /

Afterword Perhaps the strongest evidence that Kahan did not actually read the book The End of Error: Unum Computing that he is so critical of is this part of his position statement (page 23 of this document):

“Athirdfailuremodeiscalledthe‘wrappingeffect.’Now,we’recalling—thewordwrappingeffectdoesn’tappearinthebook.”

This is untrue. Here it is in the Table of Contents:

To make sure the reader does not miss it, it is clearly defined in a bright blue definition box:

It is discussed thereafter in six different places in the book, with examples and an exercise for the reader showing the limitations of the ubox method. It also appears In the Glossary:

The diagrams of the wrapping effect fill entire pages and are similar to ones Kahan himself uses in his “coffin” discussions:

49

Had he even glanced through the book, spending one second per page, this diagram would have caught his eye instantly as an example of the wrapping effect. Yet he stated with confidence that the book shows no awareness of the problem. We could speculate: Did Kahan read only the first part of the 400-page book, stopping before he got to that part of the discussion? No, that cannot be either. Another mistake, here visible only in his statement, visible on his next-to-last slide (page 33 of this document) and repeatedly asserted in his posted documents, that

“unumshaveonlyoneNaN.” That means he did not even get as far as page 23, or he would find the fact that there are two kinds of NaN, italicized for emphasis. The three times the word “two” appears just on that page for the number of NaN types are highlighted in the following excerpt:

50

The point that there are two unums is emphasized throughout the book and is visible in the code listings in the Appendices as well. When reading through this transcription and thinking about what Kahan asserts, bear in mind the above examples. There are many other examples of Kahan statements about unum arithmetic or the book that are blatantly incorrect. Any statements Kahan makes about the contents or meaning of The End of Error: Unum Arithmetic are probably incorrect, since he provably did not read it. Similarly, his statements about what his own software would do show that he did not even test it by running it, as shown by the bug in his Compensated Summation code. His statements about what unum arithmetic would do show that he did not try running the prototype unum environment, or he would have discovered that his speculations were false.

—John Gustafson 7 January 2017