ResearchExperienceandPlansAllieReinsvold Hall

PostdocPizzaMeetingFebruary24,2021

Overview• Third postdoc talk (previous: Feb 2020, Sep 2018)• Received PhD in May 2018 from the University of Notre Dame• Joined Fermilab in June 2018 as part of the Scientific Computing Division (SCD)

• Supervised by Nhan Tran• Mentored by Anadi Canepa• Analysis guided by Bo Jayatilaka• Technical project guided by Giuseppe Cerati

• Technical project: vectorized Kalman Filter (mkFit) project• Analysis: DM searches in LPC-DM group

February2021 2

SearchesforDarkMatter

February2021 3

DarkMatterSearches– LPC-DM• One of the leaders of the LPC-DM analysis group

(https://lpc-dm.github.io/), along with Bo, Matteo, and Doug• Collaborating with Cornell, Nebraska, Kyungpook, SPRACE,

UC Riverside and others

• Since last year: focus on leading LLP searches• iDM approaching preapproval

February2021 4

Ongoing projects within EXO LLP• Self-interacting dark matter (SIDM)• Inelastic dark matter (iDM)

Ongoing projects within EXO MET+X• Dark Higgs• Mono-top (not directly involved)

DarkHiggs• Search for dark Higgs model: spin-1 Z’ and

a spin-0 dark Higgs (hs)• Final states: monojet, mono-V, mono-hs(bb)• Mono-hs expected dominate sensitivity

• Collaborators: F. Dolek (Cukurova), S. Dogra, J. Lee, J. Hong, C.S. Moon (Kyungpook), T. Tomei (SPRACE)• Select hs→ bb with DeepAK15 tagger• Model main backgrounds using jet mass

sidebands + control regions• Fit is now working; plan to move

quickly to preapproval after finishingdocumentation

February2021 5

InelasticDarkMatter• Search for dark matter excited state χ2 →

ground state χ1 and a displaced lepton jet• Final state: lepton jet, aligned with

MET, back-to-back with an ISR jet• Based on arXiv:1508.03050 from

E. Izaguirre, G. Krnjaic, B. Shuve• First search at collider• Timeline: preapproval in the next month• Collaborators: A. Frankenthal (Princeton),

T. Reid, P. Wittich, J. Alexander (Cornell)• CADI: EXO-20-0120

February2021

Theoretical Models

08/10/2018 Yangyang Cheng | LPC-DM Meeting 2

• Astrophysical observations & DM-baryon density suggest a more complex dark

sector with dark matter interactions

• Mediators between the dark and visible worlds: Higgs-portal, vector-portal(Z’, A’…)

• Feeble couplings & no definite mass scaleÆ long-lived particles

Self-interacting Dark Matter

Cosmic connection

Collider production displaced

lepton jet

Darkonium

Inelastic Dark Matter

DM coannihilation

Excited state decay (through off-shell dark photon)

Collider Signature

Self-Interacting Dark Matter Inelastic Dark Matter

08/10/2018 Yangyang Cheng | LPC-DM Meeting 3

• Two-body decay of DM bound state• m_pseudoscalar = 2m_DM• m_darkphoton << m_DMÆ two displaced lepton jets back-to-back

mediator (on-shell): prompt decay

mediator (off-shell): displacement

ISR jet for triggering

• Heavier (“excited”) DM* has longer lifetime, decays to lighter (“ground”) DM, emitting a soft lepton pair

• O(MeV) – O(10GeV) DM; O(10%) mass splitting• m_darkphoton ~ m_DM (set to 3x m_DM) Æ one displaced lepton jet collimated with Etmiss

6

Analysisstrategy• Trigger on MET, since muons are soft (< 30 GeV)• Use displaced standalone (dSA) muons to extend

sensitivity for longer lifetimes• Match dSA muons with global muons if possible,

which have improved pt and vertex resolution• Derived scale factors for dSA ID and reco, which

are now officially supported as part of the muon POG

• Data-driven ABCD background estimation using Δφ and displacement of di-muon vertex• Sensitivity to exclude parameter space with 10%

mass splitting

February2021 7

iDM 2.0• Jose Monroy, Sam Bright-Thonney (Cornell) interested in iDM 2.0 • Adding electron final state, probe smaller mass splittings• Explore low pt electron electron reconstruction developed for b parking algorithm

• Collaborated with Christian last summer with our two virtual summer students

February2021 8

Self-interactingdarkmatter• Search for dark matter resonance Bps decaying

to long-lived dark photons Zd• Final state: two displaced lepton-jets• Require 1 lepton-jet to decay to 2μ for

trigger. Other can decay to 2μ or 2e• Based on arXiv:1811.05999 from Y. Tsai,

T. Xu, H. Yu• Timeline: approval in summer 2021• Collaborators: W. Si, G. Hanson (UC

Riverside), D. Claes, J. Castaneda (Nebraska)

February2021 9

SIDManalysisstrategy• 2018 only due to trigger constraints

• Large sensitivity increase possible in Run 3!

• Cluster PF muons, dSA muons, electrons, and photons using anti-kt algorithm• 2μ2e and 4μ final states

• ABCD method with Δφ between lepton-jets and lepton-jet isolation• Validation method: control region with

single muon as a proxy for a lepton-jet

February2021 10

mkFit tracking

February2021 11

MotivationTracking• Essential for all CMS physics (jets, MET,

PU mitigation, particle flow …) • Most computationally complex and time-

consuming part of reconstruction • Exponential increase in time/event with

respect to increased PUComputing• CPU frequency and single-thread performance

no longer exponentially increasing• Number of transistors and number of logical

cores still increasingConclusion: Need parallel algorithms

February2021

Figure 1. CPU time per eventversus instantaneous luminosity,for both full reconstruction andthe dominant tracking portion.Simulated data with pile-up of25 primary interactions perevent (PU25) corresponds to thedata taken during 2012, whilepile-up of 140 (PU140)corresponds to the low end ofestimates for the HL-LHC era.

as Intels Xeon Phi and NVIDIA general-purpose graphics processing units (GPGPUs). In thisinvestigation we have followed a staged approach, starting with Intel Xeon and Xeon Phi KnightsCorner (KNC) architectures, an idealized detector geometry, and a series of simpler “warm-up”exercises such as track fitting. This simplified problem domain was used to develop our tools,techniques, and understanding of the issues scaling track finding to many cores. The warm-upexercises let us develop useful components while also allowing the physicists to become familiarwith the computational tools and techniques, while the computational experts learned about theproblem domain. Armed with the results of those initial investigations, we are now addressingmore realistic detector geometries and event content, as well as adding new platforms. Thispaper gives an overview of our progress to date and assesses the e↵ectiveness of our stagedapproach.

2. Kalman Filter Tracking

Our targets for parallel processing are track reconstruction and fitting algorithms based on theKalman Filter [3] (KF). KF-based tracking algorithms are widely used to incorporate estimatesof multiple scattering directly into the trajectory of the particle. Other algorithms, such asHough Transforms and Cellular Automata [4][5], are more naturally parallelized. However,these are not the main algorithms in use at the LHC today. The LHC experiments have anextensive understanding of the physics performance of KF algorithms; they have proven to berobust and perform well in the di�cult experimental environment of the LHC.

KF tracking proceeds in three main stages: seeding, building, and fitting. Seeding providesthe initial estimate of the track parameters based on a few hits in the innermost regions of thedetector; seeding is currently out of scope for our project. Track building projects the trackcandidate outwards to collect additional hits, using the KF to estimate which hits represent themost likely continuation of the track candidate. Track building is the most time consuming step,as it requires branching to explore multiple candidate tracks per seed after finding compatiblehits on a given layer. When a complete track has been reconstructed, a final fit using the KF isperformed to provide the best estimate of the track’s parameters.

To take full advantage of parallel architectures, we need to exploit two types of parallelism:vectorization and parallelization. Vector operations perform a single instruction on multiple data(SIMD) at the same time, in lockstep. In tracking, branching to explore multiple candidates per

12

VectorizedKalmanFilterProject(mkFit)• Mission: Implement parallelized and

vectorized Kalman filter tracking algorithm • Collaboration with USCD, Princeton, and

Cornell; SciDAC-4 project U. Oregon• Focus on track building • Wrote new “Matriplex” library to perform

SIMD (single-instruction multiple data) operations on small matrices

• Test algorithm on highly parallel CPU architectures. Primarily using Intel Xeon or Xeon Phi• Current goal: prepare mkFit for use in

offline CMS reconstruction in Run 3

February2021

Achieved same track building efficiency as CMSSW, tested on

TTBar PU 50 sample

13

mkFit withinCMSSW• mkFit is integrated within CMSSW as an

external package• Can be run in place of the nominal track

building iteration 0• Achieved > 6x speedup with respect to

CMSSW• Caveat: CMSSW is compiled with gcc,

mkFit is compiled with icc• Includes all overheads, duplicate removal,

seed cleaning, data conversion, etc• Track fitting now takes more time than

track building• Work ongoing to extend mkFit to other

iterations

February2021 14

Recentaccomplishments• mkFit paper published in JINST

• “Speeding up particle track reconstruction using a parallel Kalman filter algorithm”, S. Lantz et. al., JINST 15 (2020) 09, doi 10.1088/1748-0221/15/09/p09030, arXiv:2006.00071

• Published under CMS “limited authorship” rules

• Main task in last year: various physics improvements, collectively referred to as Pandora’s Box• Necessary to reproduce full

performance of current CMS tracking

• Material effects, parametric magnetic field, invalid modules, multiple hits per layer from overlapping modules

• Requires re-tuning of track building parameters to optimize performance

February2021 15

11

12

13

14

15

16

17

18

0.225 0.250 0.275 0.300

Bin 5: Low pt, 1.7 < eta < 2.5

Fake rate

Num

ber

of la

yers

Previous performanceCurrent head of develNew proposed tuning

Codeportabilitystudies• Most current HPCs and all three future exascale HPCs rely on different GPUs

• Need code that can efficiently on CPUs and a variety of GPUs• Continued collaboration with Future Technologies Group at Oak Ridge

National Laboratory • Pre-covid, spent 25% of my time at ORNL

• Testing performance of one function from the mkFit algorithm• Propagation of track parameters to a

specific z location (“p2z”)• Implement using a variety of parallelization

strategies for CPUs and GPUs, including OpenMP, OpenACC, Eigen, and Kokkos

• Collaborating with HEP CCE-PPS (Matti, Martin) on related test kernel (“p2r”)

• Plan to submit paper to CS journal February2021 16

BigPictureandFuturePlans

February2021 17

PresentationsandOutreach• LHC Physics Conference, virtual, May 2020

• Long-lived particle searches in CMS• Seminar: “Preparing CMS for the HL-LHC and the future of computing”

• Presented seminars at MIT, Harvard/Brandeis, SLAC, Brookhaven, UMass Amherst (scheduled)

• Colloquium at TRIUMF• QuarkNet masterclasses, 2021 Expanding Your Horizons workshop• Lecturer for Saturday Morning Physics• Developed and taught QuarkNet Summer Session for Teachers, a six-week

course for high school teachers• Official mentor for a summer intern in 2020

• Virtual internship with UChicago student based in Hungary• Unofficially mentored several other students

18February2021

Otherroles

• LPC Distinguished Researcher in 2020 and 2021• On the organizing committee for the LPC Physics Forum

• Convener, HEP Software Foundation (HSF) Data Analysis Working Group • Started in November 2021• Organized a series of three meetings on analysis metadata and how it is handled

across different HEP experiments• Future meeting topics: columnar and declarative analysis, improving analysis-

related training across experiments, initiating more discussions with the nuclear physics and neutrino physics communities

• ARC member for “Search for HH->bbγγ” (HIG-19-018) and “Search for dark matter in the dark Higgs to WW MET+diplepton final state” (EXO-20-013)

February2021 19

Jobapplications• Goal: professor at a college or university with a focus on teaching

• Establish CMS research program at a Primarily Undergraduate Institution (PUI) as a CMS “affiliate” member

• Last year: applied to several PUIs and a few research institutions• One interview at a research institution, 0 at PUIs

• Second round of applications has been much more successful so far• Adjusted application materials based on feedback from others at PUIs (Julie

Hogan, Matt Bellis, faculty at my undergrad)• Added teaching experience through QuarkNet course• Four virtual “on-campus” interviews

• Two job offers!

February2021 20

ChallengesforPUIapplications• Emphasizing that I’m not a “flight risk”

• Highlight that this is my actual goal, not a backup plan• Heard from several schools that my letters did a good job reinforcing this

• Mentoring undergrads in research through Fermilab internship programs• Gives confidence that my research program will actually work• Asked my references to point out my mentoring experience in letters

• Get creative with teaching experience • Considered teaching a course as an adjunct at a local community college, instead

made my own course through QuarkNet

• Roles outside CMS might be viewed more favorably• eg HSF convenership• Difficult to explain how CMS works and what my individual contributions are to the

collaboration

February2021 21

PUIinterviews• Teaching demos

• Often the deciding factor in who gets the offer• Either choosing my own topic or on an assigned topic

• Research talk at the level of sophomore undergraduates • Also used to evaluate teaching – outreach experience helped here!

• Common questions:• Why do you want to be here specifically/at a PUI in general?• How will you involve undergrads in research? • What courses do you want to teach? Any ideas for new courses?• What active learning techniques do you use in the classroom?• How will you promote an inclusive/equitable environment?

February2021 22

Thanksforeverything!

Futureplans• Keep working on CMS, especially dark matter searches• Continue to collaborate with Fermilab on analysis and computing• Will visit Fermilab regularly, hopefully with students

February2021 23

EspeciallytoAnadi,Bo,Giuseppe,andtheLPC-DM,mkFit,Coffeaandteams