deepqb: deep learning with player tracking to quantify

1

2019ResearchPapersCompetitionPresentedby:

DeepQB:DeepLearningwithPlayerTrackingtoQuantifyQuarterbackDecision-Making&Performance

BrianBurke,ESPNAnalytics,[email protected]

DeepQBisaproposedapplicationofdeepneuralnetworkstoplayertrackingdatafrom

overtwofullseasonsofAmericanprofessionalfootball.Thisnovelapproachdemonstratestheabilitytosuccessfullyunderstandcomplexaspectsofthepassinggame,mostnotablyquarterbackdecision-making.Itcanassessandcompareindividualquarterbackpasstargetselectionbasedonasnapshotpresentedtothepasserbythereceiversanddefenders.Assessmentsofquarterbackdecision-makingaremadebycomparingactualtargetselectiontothatpredictedbyourmodel.Themodelperformswell,correctlyidentifyingthetargetedreceiverin60%ofcross-validatedcases.Whenpasserstargetthepredictedreceiver,passesarecompleted74%ofthetime,comparedto55%whentheQBtargetsanyotherreceiver.Thisperformanceissurprisinglystrong,giventhattheoffenseoftenconcealsitsintentbydesign,whiledefensestrynottoallowanysinglereceivertobeopen.Further,quarterbackpassingskillsseparateandapartfromhisreceiversanddefenseareisolatedandassessedbycomparingmetricsofactualplaysuccesstothemetricsofsuccesspredictedbythesituationpresentedtothepasser.Thisapproachrepresentsanewwayforteams,media,andfanstounderstandandquantitativelyassessquarterbackdecision-making,anaspectofthesportwhichhaspreviouslybeenopaqueandinaccessible.

1. IntroductionPerhapsthemostenigmaticandyetmostimportantplayerattributeinallofAmericanfootballisthedecision-makingabilitiesofquarterbacks.Althoughmentalabilitiesareimportantforeveryposition,quarterbackisuniqueinthatpsycho-cognitiveabilitiesrivalphysicalabilitiesinregardstosuccessfulperformance.Measurableandphysicalattributesareeasilyobservedthroughthescoutingprocess,butaprofessionalquarterback’sabilitytoprocessandexploithighlydynamicinformationduringthecourseofaplayisnotwellunderstood.Previouslyonlyrelativelycrudeaggregatestatisticalmethods-thatcannotfullyseparatethequarterback’sindividualimpactfromthoseofhisteammates,playdesign,andopponents-haveexistedtoquantifythisskill.Earlyeffortstoexploitfootballtrackingdataonlyscratchedthesurfaceofwhatispossiblewithsuchrichinformation.Typicalapplicationsoftrackingdatamerelyinvolvedmeasuringthemaximumspeedsofthefastestplayers,ortotalingthedistancetraveledbyaplayeronaplay.DeepQBisadeeplearningapproachtoevaluatequarterbackdecision-makingandperformance.Thisapproachprovidesasuiteofmodelsthatpredictandevaluatepassdecisionsandoutcomesattheplaylevel.Thesemodelsarerelativelystraightforwardneuralnetworksthatarecapableofproducingusefulanalysisandinsightsinnearreal-timeduringlivegames.Thispaperdemonstratesthatsuchanapproachisaviableandpromisingwaytoanalyzethepassinggame,themostcriticalaspectofprofessionalfootball.Althoughthereremainlimitationswiththisapproach,DeepQBprovesthatsomeofthemostcomplexaspectsoffootballcanbeunderstoodthoughadeeplearningapproachtomulti-agentfeatures.Itoffersafullyquantitative

2


abilitytomeasureindividualperformanceforthemostimportantposition,andoffersfans,media,andteamsaninnovativenewwaytoevaluatethesport.

2. RelatedResearchDuetothecomparativelylateadventoftrackingdatainfootball,thisthefirstmajorprojecttodevelopmethodsofunderstandingfootballplayertrackingusingadvancedmachinelearningtechniques.However,several publicized research efforts that exploit player tracking have applied neural networks and other advanced techniques to data from basketball, soccer, and hockey. Cervoneetal[1]builtaframeworkbasedonaMarkovmodelusingplayer-trackingdatatoassignanexpectedpointvaluetoeachtimesegmentofabasketballpossession.Theycomparedaplayer’sexpectedcontributiontoactualoutcomestoassessplayerperformance.WangandZemel[2]usedbothafeed-forward(FFN)andarecurrentneuralnetwork(RNN)basedonopticaltrackingdataforplaytyperecognitionandclassificationinbasketball.Leetal[3]used“deepimitationlearning”basedonLongShortTermMemory(LSTM)networkstomimicsoccerdefenders’movementsbasedonthedynamicsoftheotherplayers.Theauthorsstressedtheimportanceofmeaningfulrolerepresentationsforeachplayerinthenetwork.Themodelallowedindividualteam-specificplayingstylestoberepresentedaswell.Harmonetal[4]transformedplayertrackingdatafrombasketballintomulti-channelpictorialrepresentationsthataconvolutionalneuralnetwork(CNN)couldclassify.TheauthorscombinedtheCNNwithafeed-forwardnetworktopredictshotmaking.Luceyetal[5]usedaconditionalrandomfieldmodelwithsoccerplayertrackingdatatoestimatetheprobabilityofashot’ssuccess.Thiswasusedtoassessteamperformanceatboththeseason-andgame-level.Mehrasaetal[6]appliedone-dimensionalconvolutionalnetworkstomodelplayertrajectoriesonbothbasketballandhockey,forplayrecognitionandteamclassification.Thesinglerelevantpaperonfootballattemptedtoassessquarterbackdecision-makingusingVoronoitessellation.Hochsteler’s[7]techniquepartitionedtheareaoftheplayingfieldaccordingtowhichplayerisclosesttoeachpointofthefieldandassignexclusiveownershipofeacharea.Thesizeofeachreceiver’sownedareaandproximitytothenearestdefenderwasusedtoassesstheopennessofareceiver.Althoughtheresearchhadonly224playstoanalyzeandusedrelativelyrudimentarytechniques,ithintedatthepotentialofmorepowerfulmethodstounderstandandassessquarterbackplay.

3. ApproachTheDeepQBsuiteofmodelstakesadvantageofacomprehensiverepresentationoftrackingdataaspresentedtoaquarterbacktopredictreceivertargetselection,completion-incompletion-interceptionoutcomeprobabilities,andyardageexpectation.Thesepredictionsarethencomparedtoactualoutcomestoassessquarterbacktargetselectionandoverallperformance.Modelinputisasnapshotofplayerconfigurationatthetimeofpassrelease,includingreceiverposition,velocity,acceleration,andorientation,aswellasthoseofthesecondary.Hand-builtfeatures,specificallyrelativepositionsandanglesofreceiverswithrespecttootherplayers,wereengineeredandusedasinputs.Additionally,playmetadata,(down,distance,andyardstotheendzone)werefedtothenetwork.Lastly,datafromESPN’sVideoAnalysisTracking(VAT)projectwasusedasinputs,specificallywhetherthequarterbackwasunderduressfromthepassrushandwhethertherewasaplay-actionfakefollowingthesnap.

3


TheheartofDeepQBisamodularfeed-forwardartificialneuralnetwork(FFN)architecture.Itismodularinthattheinputsandthearchitectureofthehiddenlayerscanremainidenticalregardlessofwhattypeoftargetvariablethemodelisbeingaskedtopredict.Thefinaloutputlayerismodifiedtoaccommodatewhetherweareaskingthemodeltoestimateprobabilitiesfordiscreteoutcomes(suchaswhichreceivershouldbetargetedorwhatthepassoutcomewouldbe),orweareaskingthemodeltoestimateacontinuousvalue(suchastheexpectedyardagegainedbyapassplay).Therearefourvariantsofthegeneralmodeldiscussedinthispaper:

• Variant1estimatestheprobabilitythequarterbackwilltargeteachofthefiveeligiblereceiversgiveneachplay’sconfigurationofthereceiversanddefense.

• Variant2estimatestheexpectedyardsgainedforeachofthefiveeligiblereceiversoneachplay.

• Variant3estimatestheprobabilitiesofthethreetypesofoutcomesontheplay:complete,incomplete,orinterception.

• Variant4isanexperimentalmodelthatproducestargetpredictionslikeVariant1,butusestransferlearningtocapturethedecision-makingofindividualquarterbacks.

3.1. DataTrainingandvalidationdataconsistsofeverytargetedpassattemptwithfromthe2016and2017NFLregularandpost-seasons.PlayertrackinginformationcomesfromtheNFL’sNextGenStatssystem,whichusestwoelectronicRFIDchipsineachoftheplayers’shoulderpads.Comparedtotheopticalmethodoftrackingcurrentlyusedforbasketball,hockey,andsoccer,theRFID-basedmethodhastheadvantageofrecordingplayershoulderorientation,whichishelpfulforunderstandingthedynamicsofapassplay.Playerposition,velocity,acceleration,andorientationisprovidedat10Hz,andisavailableinnear-realtimeviaanAPI.Overall,45,501passattemptswereusedtotrain,andvalidatethemodels.Thedatawassplitbetweentraining,validation,andtestsetstoaccuratelymeasuremodelperformanceandminimizeoverfitting.Training(n=24,414)andvalidation(n=10,464)datawerebuiltasa70%/30%splitofpassesfromthe2016and2017NFLregularandpost-seasons.Thetestdatawasbuiltfromthe2018regularseasonthroughweek12(n=10,623).Allresultsstatedherearefromthe2018testsetunlessotherwisespecified.

Figure1ExampleofhowDeepQB"sees"apassplayandtherelationshipsamongreceivers(blue)anddefenders(red).Arrowsindicatevelocity.Theblackbarsindicateshoulderorientation.

4


3.2. ArchitectureThegeneralmodelconsistsofafour-layernetworkasdepictedinfigure2:-Aninputlayerof230nodes-Adensehiddenlayerof256neurons.-Aseconddensehiddenlayerof128neurons.-Adenseoutputlayer,withactivationdependentonthetypeoftargetvariablethemodelisbeingaskedtoestimate.

Eachdenselayer,asidefromtheoutputlayer,usesrectilinearunits(relu)fornonlinearactivation.Batchnormalizationanddropoutareemployedbetweeneachdenselayertopreventoverfittingandimproveperformancewhengeneralizingwithout-of-sampledata.Basedontheinsightsof[5],theorderofinputstothemodelweregivenmeaningandkeptconsistent.Foreachplay,thethesetoffeaturesassociatedwitheachofthefivepotentialtargetreceivers(position,velocity,acceleration),wereorderedfromshallowesttodeepestdownfield.Withineachsetoffeaturesforeachreceiver,thefeaturesassociatedwiththetwonearestdefenderswereincluded.1Withinthesetoffeaturesforeachreceiver,thefeaturesofthenearestdefenderto

1NotethatDeepQBisintendedtoanalyzeplaysprimarilyfromtheperspectiveofthequarterback.Passesclassifiedas“drops”byESPN’sVATanalysisarecountedassuccessfulcompletionsfromthepasser’spointofview.Additionally,allinputsareasofthetimeofpassrelease,sofactorssuchas

Figure2DeepQB'smodular,feed-forward,artificialneuralnetworkarchitecture.

5


thatreceiverappearsfirstintheinputvector,andthefeaturesofthenext-nearestdefenderappearssecond.Anadditionalsetoffeaturesspecifictothequarterbackwasconcatenatedtotheinputs.Thesefeaturesincludepositionandvelocityinformation.Theorientationofthequarterbackwasdeliberatelyexcludedfromtheinputs.Theorientationofthepasser’sshouldersatthetimeofpassreleasewouldbe“cheating”forthismodel’spurposes,asitwouldveryoftencorrelatestronglywiththedirectionoftheintendedtarget.Lastly,playinformationwasconcatenatedtotheinputvector.Thisinformationincludeddown,distancetogain,andyardlineoftheplay.Thetrainingsetwasaugmentedwithitsmirrorimageplays,rotatedaboutthelongitudinalaxisofthefieldofplay,effectivelyincreasingsamplesize.Theassumptionisthattargetselectionandotheroutcomesofapassplayareinvariantwithrespecttolateralsymmetry.Areceiveropenontheleftsideofthefieldwouldbejustasopenifhewereontherightside.Thedrawbackisquarterbackhandedness,especiallyifhethrowswhileontherun.Despitethislimitation,augmentationimprovedout-of-sampleperformance.3.3. LearningEachvariantofthemodelwastrainedusingKeras[8]withaTensorFlowbackend.Trainingwasstoppedattheoptimumperformanceonthevalidationset.Modelhyper-parameterswereselectedusingagridsearchprocedure.

4. ResultsResultsfromeachvariantofthemodelarepresentedhere,alongwithgeneralinsights.4.1. Variant1:TargetProbabilityTheprimarypurposeofthisvariantistoverifythatthistypeofmodelcantrulymakesenseofthetrackingdataofapassplay.Inessence,themodelanswersthequestion,“Giventhepicturepresentedtoatypicalquarterback,whowouldhechoosetotarget?”Themodelpredictstheactualtheproximityofdefendersatthetimeofpassarrivalareintentionallyexcluded,eveniftheywouldimprovepredictionaccuracy.

Figure3Thefeaturevectorusedasinputforthegeneralmodel.

6


targetwithanaccuracyof59.8%–threetimesthenaïveestimateforaone-in-fiveproposition.Thisperformanceshouldbeconsideredinlightofactualfootballtactics.Offensesgotogreatlengthstoconcealtheintentoftheirplaydesigns,whiledefensesendeavortonotallowanysinglereceiverfrombeingadisproportionatelyrewardingpasstarget.

Figure4Actualplaysfrom2018withassociatedreceivertargetprobability.Thewhitevectorsshoworientation.Actualtargetsareindicatedbygreendiamonds.Play(1)illustratesanobvioustargetwithahighprobability.Play(2)showsacheck-down,althoughDeepQBrecommendsthedeeperemergingtargetwithp=.50.Play(3)demonstratesthemodel’sabilitytoseethepropertargetongoal-lineplays.Play(4)suggestsanothertargetingerrorbytheQB.Play(5)showsalogicalcheck-down,asallotherreceiversaresmothered.Play(6)illustratesthatdefenderproximityalonedoesnotdetermineifareceiveris“open.”Relativepositionandvelocityarekeyfactors.

Further,whenquarterbackstargetthereceiveraspredictedbyDeepQB,thecompletionrateis74%.Thiscomparestoacompletionrateof55%whenquarterbackstargetanyotherreceiver.Theseresultsindicatethatthemodelcansuccessfullymakeinferencesusingthetrackingdata,andiscapturingsomedegreeofunderstandingofeachpassplay.However,theYardsPerAttempt(YPA)whenquarterbackstargetthepredictedreceiverissignificantlylowerthanwhenhetargetsanotherreceiver(7.8vs8.0YPA).Thisisasurprisingandcounterintuitiveresult.Ifcompletionpercentagesarehigher,howcouldYPAbelower?Thisleads

7


tothefirstmajorinsight:thetypical(leagueaverage)quarterbackismostlikelytoocautiousinhisreads.Thetheorythatexplainsbothresultsisthatthetypicalquarterbackfavorsasaferchoiceattheexpenseofoptimumefficiency.4.2. Variant2:ExpectedYardsThesecondvariantoftheDeepQBmodeldirectlyestimatestheexpectedyardsgiventheoverallpicturethatispresentedtothequarterback.Inadditionitestimatestheexpectedyardsgiventhatthequarterbacktargetseachofthefiveeligiblereceivers.Thisallowsustoevaluateindividualquarterbacks,aswellastheirteam’sreceivingcorpsandpassingschemes.OverallquarterbackperformancecanbeassessedbymeasuringthedifferencebetweentheiractualYPAandtheexpectedYPAproducedbytheoverallpictureofreceiversanddefenderspresentedtothequarterbackoneachpassplay.Thisrevealsanestimateofthevalueaddedbythequarterback,overandabovethecapabilitiesthathisreceivingcorpsandschemeprovide.Forexample,aquarterbackmayhavealowYPA,buthisbindingconstraintmaybeschemeorreceivers.Foreachplay,quarterbackdecision-makingcanbeassessedbydeterminingwhetherthequarterbacktargetedthereceiverwiththemaximumexpectedyardage.Perhapsabetterwaytomeasurethesameideainaggregateistocalculatetheproportionofthemaximumavailableexpectedyardagerepresentedbytheexpectedyardageofthereceiverthequarterbackactuallytargeted.(Forexample,ifthemaximumavailableexpectedyardageonaplaywas9.0yards,andthequarterbacktargetedareceiverwith7.5expectedyards,theproportionwouldbe7.5/9.0=83%.)Table1liststheresultsforallquarterbacksin2018(throughweek12)withatleast80passattempts,alongwiththeexpectedYPApredictedbythemodel.TheYPAabove/belowexpectedcolumnisthedifferencebetweenactualandexpectedYPA.TheproportionofmaximumexpectedYPAtargetedforeachquarterbackisalsolisted.Table1ComparisonofExpectedYPAtoactualYPAfor2018quarterbackswithatleast80targetedpassattemptsthroughweek12.Thetopandbottom5passersarelisted.

Rank QB ExpectedYPA

YPAabove/belowexpected

Propor-tionofmaxYPAavailable

Rank QB ExpectedYPA

YPAabove/belowexpected

Propor-tionofmaxYPAavailable

1 Goff 7.8 +2.1 70.3% 33 Smith,A 7.6 -0.4 66.7

2 Mahomes 7.9 +2.0 65.6 34 Allen 7.0 -0.5 65.53 Brees 7.1 +1.9 66.7 35 Darnold 7.1 -0.5 63.0

4 Fitzpatrick 7.6 +1.8 70.1 36 Flacco 7.7 -0.6 69.55 Watson 7.2 +1.7 69.8 37 Taylor 7.6 -1.5 70.4

8


Figure5plotstheYPAAboveExpectedmetricagainstthepredictabilityofeachquarterback,whichismeasuredbyhowoftenvariant1(targetprediction)iscorrect.Thisisessentiallyameasureofhow“typical”aquarterbackisinhistargetingdecisions.Noticethatmanyoftherookies(Allen,Darnold,Rosen)aretowardtherightofthechart(highlytypical),withMayfieldastheexception.Someofthemoreaggressivepassers,suchasAaronRodgers,aretowardtheleft.AsdiscussedwithVariant1,thereisaweaknegativecorrelationbetweenthe“typicality”ofaquarterback’stargetingdecisionsandthevalueheaddsabovethepotentialprovidedbyhisreceivers(r=-0.30).Unfortunately,wecannotescapethepossibilityofselectionbias.Forexample,AaronRodgersmayattemptmoreunorthodox,aggressivetargetingbecausehecan,whileotherquarterbacksmaynothavetheskillstodosoreliably.4.3. Variant3:PassOutcomesThethirdvariantofourmodelestimatestheprobabilitiesofcompletion,incompletion,andinterceptionforeachplay.Consideringjustcompletionsandincompletions,themodelappearswell-calibrated(figure6),althoughitrarelyconsiderspasscompletionsashighlyimprobable.Thisissubstantiallyduetotheexclusionofdeliberate“throw-away”passesfromthedata,whicharetechnicallyattemptsbutdidnothaveatargetedreceiver.

Figure5AscatterplotofqualifiedQBsfrom2018throughweek12.TheverticalaxisindicatesthevalueaddedbytheQBaboveorbelowexpectedgivenhisreceiversandopposingdefenses.(Notethat2018isan“up”yearforoffense,sothemeanhereisabovezero.)Thehorizontalaxisindicateshowwellthemodelwasabletopredictthetargetingdecisionsofeachpasser.More“typical”QBsaretotheright.

Table2Modeledoutcomeprobabilitiesbyactualoutcometypes.

9


Passingaccuracyisawidelymisunderstood

attribute,primarilybecausecompletionpercentageisusedubiquitouslyasaproxyforaccuracy.Completionpercentagedoesnotaccountforthedifficultyofthethrow,nordoesitexclude“throw-away”passattemptsandreceiverdrops.ThisvariantofDeepQBcanbetterisolateandassesstheaccuracycomponentofpassingbycalculatingaquarterback’sexpectedcompletionratebasedoneachplay’sdynamicsandwhichreceiverwastargeted,andthencomparingthattohisactualcompletionrate.Notethatthesenumbersareontargetedpassesonly,sothey

willbehigherthancommoncompletionpercentagestatistics.Ofparticularinterestareinterceptions,whichhavelargeimpactsonthegame,anddoubtlesslyconveyagooddealofinformationaboutquarterbackdecision-making.Interceptionsareeventswithaverylowbaserateandofteninvolvetippedballsorotherrandomfactors.Theabsolutenumberofinterceptions,oreveninterceptionsperpass,willthereforebeverynoisymeasuresofaquarterback’struepropensitytothrowthem.Anaccurateestimateofinterceptionprobabilitywouldbeatruermeasure.Theplaysnapshotsinfigure7portrayexamplesofhowthemodelassessesinterceptionprobability.Noticethatthefirsttwoexamplesshowariskyconfigurationofdefenders,andtheinterceptionprobabilitiesareaccordinglyveryhigh.Thethirdexampleshowsasafeandopenpass,withacorrespondinglylowerprobabilityofinterception.Table3Passingaccuracyasassessedbyeachquarterback’sactualcompletionrateabovetheexpectedrateestimated.Thetop5andbottom5qualifiedpassersof2018throughweek12arelistedhere.

Rank QBExpectedComplRate

ComplRateAbvExp

Rank QBExpectedComplRate

ComplRateAbvExp

1 Brees 67.0% 15.0% 33 Bortles 67.6% -0.2%2 Watson 63.0 8.9 34 Rosen 68.1 -0.83 Cousins 67.8 8.6 35 Beathard 71.2 -2.14 Winston 66.8 8.2 36 Darnold 68.1 -5.75 Mahomes 68.4 7.6 37 Taylor 69.3 -10.5

Meaninterceptionprobabilityforthetopandbottomfiveindividualquarterbacksarelistedintable4.Keepinmindthatgamesituation,primarilytimeandscore,hasalargeinfluenceontheoptimum

Figure6Calibrationplotofobservedvs.predictedbinsofcompletionprobability.

10


risklevel,andthatisnotaccountedforintheaggregaterates.However,themodel’sestimatescanprovideevenmorevaluableinsightonapass-by-passbasis.

Table4Meaninterceptionprobabilities,alongwithactualrates,forthetop5andbottom5qualifiedpassersof2018throughweek12.

Rank QB MeanIntProbActualIntrate

Rank QB MeanIntProbActualIntRate

1 Foles 1.23% 1.2% 33 Mayfield 1.97% 2.3%

2 Trubisky 1.31 2.8 34 Stafford 1.99 2.53 Beathard 1.34 4.1 35 Fitzpatrick 2.00 4.9

4 Keenum 1.37 2.6 36 Watson 2.03 2.75 Wentz 1.38 1.8 37 Winston 2.28 5.4

Assumingthepassoutcomeestimatesfromthemodelarereasonable,animportantinsightisrevealed.Becauseexpectedinterceptionrates,basedonthegeometriesofthereceiversanddefense,arelowerthantheactualinterceptionratesofhigh-interceptionquarterbacks,itfollowsthatinterceptionsintheNFLareprimarilyaresultofinaccuratethrowsandrandomsampleerrorratherthanbadtargetingdecisions.4.4. Variant4:IndividualQuarterbackModelsThefinalvariantoftheDeepQBsuiteisahighlyexperimentalversionintendedtocapturethedecision-makingtendenciesofindividualquarterbacks.Thesamplesizeofpassattemptsforindividualquarterbacksisnotsufficienttoadequatelytrainamodeltoaccuratelygeneralizeonout-of-samplecases.However,itispossibletocaptureindividualdecision-makingusingtransferlearning.Transferlearningisatechniquethattrainslowerlayersofaneuralnetworkonabroadtrainingset,buttrainsthehigherlevelsofthenetworkonamorespecificsetofcases.Inthiscontext,the

Figure7Exampleplaysillustratingthemodel'sinterceptionestimates.Onlythetargetedreceiverisshown.Fromlefttoright:ThefirstplayshowsaJameisWinstonpasson2ndand12intotightcoverage.Thesecondshowsa2ndand2attemptfromDerekCarrtowardareceiversurroundedbyfourdefenders.Thethirdshowsanexampleofapasswithaverylowprobabilityofinterception,apasstotheflatbyBenRoethlisberger.

11


weightsofthelowerlevelsoftheneuralnetworkaretrainedonplaysfromthefullsetofquarterbacks.Thebaselevelofthenetworkrepresentthefundamentalrelationshipsamongpositionsandvelocities—understandingandprojectingthegeometriesoftheplayersonthefield,whilethehigherlevelsofthenetworkrepresenttheexecutivefunctions—choosingthepassingtarget.Thisisanalogoustoabiologicalbraininthatmostathletessharesimilarunderstandingsofinstinctive,intuitivephysics—basicslikegravity,distance,andmotion.Butindividualsmaydifferintheirdispositionstowardvariousdecisionsdespitehavingcommonintuitivephysics.Thisvariantofthemodelexploitssuchinstinctivecommonalitiesbytrainingitslowesthiddenlayerusingaverylargesampleofpassesbyallquarterbacks,andthenfreezingtheresultingweightsofthatlayer.Then,thenetwork’shigherdecision-levellayersweretrainedonthesmallersampleofpassattemptsofasinglequarterback.Theresultingnetworkisamodelthatcombinescommonintuitivephysicssharedbyallquarterbackswiththeexecutive-leveldecision-makingofanindividualpasser.Justoneexampleofsuchamodelispresentedhere,thatofSaints’quarterbackDrewBrees.ThetwohigherlayersofVariant1oftheDeepQBmodel(targetselection&prediction)wereretrainedusingjustBrees’passes.OverallaccuracyspecifictoBrees’2018passattemptsroseto64%.Itisinterestingtoseehowotherquarterbacksmightcomparetoacyber-Brees.Table5liststhepassersof2018whosetargetselectionsalignthebestandworstwiththeBreesmodel.

Table5Howpassers’targetingtendenciesalignwith"cyber-Brees"basedonmodelvariant4.

Rank QuarterbackTarget

PredictionAccuracy

Rank QuarterbackTarget

PredictionAccuracy

1 Brees 64.3% 33 Goff 49.7%

2 Smith,A 57.5 34 Mahomes 49.23 Allen 57.3 35 Wilson 48.7

4 Carr 57.1 36 Ryan 47.75 Darnold 56.3 37 Brady 46.1

SomeoftheworstperformingquarterbacksoftheseasonaremostalignedwiththeBreesmodel,whilemostofthebestquarterbacksoftheseasonaretheleastaligned.Thisshouldmakeoneskepticaloftheseresults.Thisisanadmittedlyexperimentalapplication,butoneplausibleinterpretationoftheresultsisthatBreeshasfrequentlytargetedhisrunningbacks,similartohowpooroffensesrelyoftenonshallowcheck-downroutestocompletepasses.InBrees’case,thisislikelybydesignduetotheexceptionalreceivingskillsofhisbacks,butformanyothersthistactictendstobealastresort.

5. LimitationsandFutureDevelopmentAllvariantsofthemodelcurrentlyrelysolelyonabinary“QBduress”variabletoaccountfortheconstraintsontheQBduetothepassrush.Thismodelingdecisionwasmadeattheoutsetoftheprojecttoensuretractabilityofthemodel,butsincethenitbecameclearthemodelcanacceptalargersetofinputswithoutcausingaprohibitivelylongtrainingtime.Futureiterationsofthe

12


modelwilluseamorecompletepictureofpassprotectionbasedonadvancesinmodelingpassprotection[9].Thisshouldenhanceaccuracy,asquarterbacksaresometimesnotabletotargetopenreceiversduetobeingobscuredbypassrushers.Limitingthepicturetothetwonearestdefenderstoeachreceiverwasanothermodelingdecisionmadeforthesamereason.Reviewofhundredsofoutputplaysindicatedthemodelcouldstillpiecetogetherarelativelycompletepicturefromthisstructure—understandingthepresenceofalldefenderstoallreceiversinthefeaturevector.However,futureversionsofthemodelcanincludeuptoallelevendefendersandtheirrelationshipstoeachreceiver.Receivertalentsandskillsarenotfullyaccountedfor.Althoughaspectsofreceivingsuchasrouterunningandspeedcanbecapturedbythemodel,otheraspectssuchasthesizeandstrengthofreceiversisafactorintheprobabilityofreceptionandtheyardageestimateofeachplay.Interceptionsareacriticalaspecttodecision-making.AlthoughVariant3addressesinterceptionprobabilitydirectly,Variant2addresseseachpassplay’sexpectedyardsseparatelyfrominterceptionchances.Thismaymakesomepassingtargetsappearmorelucrativeonnetthanisthecaseduetoapossiblyhighchanceofinterception.FutureversionsofthisapproachwillincorporateconceptssuchasExpectedPointsAdded(EPA)[10],whichaccuratelymeasuresthecomplexinteractionsofdown,distance,fieldposition,andpossession,andistheconsensusutilityfunctionoffootballevents.Thegeneralmodelisbasedonasnapshotoftheplayatthetimeofpassrelease.Openreceiversearlierinthedevelopmentofaplaymaylaterbecomecoveredbythetimeofreleaseifthequarterbackfailstorecognizethem.Indeed,thisisanaturalconsequenceofhowthepositionistaught.Quarterbacksaretypicallyinstructedtomaketheirreadsinaprogression,fromoneprimaryreceivertoasecondary,andthentoanalternate.Thismodelmaybepenalizingaquarterbackwhomakesthecorrectreadintheplannedprogression,butbecauseareadbecomesopenafterhisstepintheprogression,itappearsthatthequarterbackhasmissedanopenreceiver.FuturevariantsoftheDeepQBmodelmayusetechniquessuchasanLSTMnetworktoassesspassingoutcomesonacontinuousbasisthroughouttheplay.

6. SummaryThispaperproposesanewapproachtoexaminingpassinginprofessionalfootball.Despitethechaosanddeceptionintrinsictothesport,aneuralnetworkapproachtoplayertrackingdataperformssurprisinglywell,andhasbeendemonstratedasaneffectivetoolforanalysis.Themodeldemonstrateshowquarterbackeffectivenessanddecision-makingcanbeassessedbothforindividualplaysandinaggregate.VariantsoftheDeepQBsuiteofmodelsfocusontargetselection,expectedyardage,andpassoutcomes.Afourthexperimentalvariantexploresthepossibilityoftailoringthemodeltoindividualquarterbacksusingtransferlearning.Thisoverallapproachoffersthepromiseofmakingapreviouslyopaqueaspectofthesportaccessibletoteams,media,andfans.

13


References[1]Cervone,D.,D’Amour,A.,Bornn,L.,Goldsberry,K.POINTWISE:PredictingPointsandValuingDecisionsinRealTimewithNBAOpticalTrackingData.MITSloanSportsAnalyticsConference2014.2014.[2]Kuan-ChiehWang,RichardZemel.ClassifyingNBAOffensivePlaysUsingNeuralNetworks.MITSloanSportsAnalyticsConference2016.2016.[3]Le,H.,Carr,P.,Yue,Y.,Lucey,P.Data-DrivenGhostingusingDeepImitationLearning.MITSloanSportsAnalyticsConference2017.2017.[4]Harmon,M.,Lucey,P.,Klabjan,D.PredictingShotMakinginBasketballusingConvolutionalNeuralNetworksLearntfromAdversarialMultiagentTrajectories.AssociationfortheAdvancementofArtificialIntelligence,2016.[5]Lucey,P.,Bialkowski,A.,Monfort,M.,Carr,P.,Matthews,I.“QualityvsQuantity”:ImprovedShotPredictioninSoccerusingStrategicFeaturesfromSpatiotemporalData.MITSloanSportsAnalyticsConference2015.2015.[6]Mehrasa,M.,Zhong,Y.,Tung,F.,Bornn,L.,Mori,G.“DeepLearningofPlayerTrajectoryRepresentationsforTeamActivityAnalysis.MITSloanSportsAnalyticsConference2017.2017.[7]Hochstedler,Jeremy.FindingtheOpenReceiver:QuantitativeGeospatialAnalysisofQuarterbackDecision-Making.MITSloanSportsAnalyticsConference2016.2016.[8]Chollet,F.(2015)Keras,GitHub.https://github.com/fchollet/keras[9]Burke,Brian.“Wecreatedbetterpass-rusherandpass-blockerstats:Howtheywork.”espn.com/nfl/story/_/id/24892208/creating-better-nfl-pass-blocking-pass-rushing-stats-analytics-explainer-faq-how-work.2018.[10]Burke,Brian.“ExpectedPointsandExpectedPointsAdded.”AdvancedFootballAnalytics.archive.advancedfootballanalytics.com/2010/01/expected-points-ep-and-expected-points.html.2010.

deepqb: deep learning with player tracking to quantify

Documents