Monday, 14 September – Pre-Symposium Tutorials 8:00 AM – 12:00 PM

Design of Experiments (DOE) for Promoters and Practitioners Mark Kiemele, Ph.D., Air Academy Associates

T&E in Support of Agile: Test and Evaluation for Information Technology Acquisition Robin Poston, Ph.D. University of Memphis and Wayne Dumais, Department of Homeland Security

Introduction to Cybersecurity Test and Evaluation Pete Christensen, CTEP, American Systems

Incorporating T&E into Acquisition Contracts Terry Murphy and Patrick Kastner, Department of Homeland Security, T&E

The Evolving Role of T&E Matt Reynolds, Consultant

T&E Across the DoD Acquisition Lifecycle Michael Flynn, PhD. CTEP, Defense Acquisition University

Monday, 14 September – Pre-Symposium Tutorials 1:00 PM – 5:00 PM

Application of Scientific Test and Analysis Techniques (STAT) Beyond DOE in Test and Evaluation Gina Sigler and Sarah Burke, Ph.D., Scientific Test and Analysis Techniques Center of Excellence

Ethically-aligned Experimentation and T&E for Decision Making with Complex Systems (This Tutorial is presented by the ITEA Southern Cross Chapter in Australia) Keith Joiner and Malcolm G. Tutty, University of New South Wales, Australian Defence Force Academy

Overcoming Challenges in Distributed T&E Gene Hudgins, Test Resource Management Center, Joint Mission Environment Test Capability (TRMC/ JMETC)

Laser System T&E Challenges Douglas H. Nelson, Teknicare, Inc. and Mark Stevens and Robert Harney, Naval Postgraduate School

Hacking Methods and Their Effects Patrick Lardieri, Lockheed Martin and Paola Pringle, NAVAIR

Tutorial Descriptions

Tuesday, 15 September – Day 1 Morning Session 8:00 AM – Noon

On the Main Stage 8:25 AM Opening Ceremony – Pete Crump GTRI, ITEA President Welcome and Introduction, Steve “Flash” Gordon, PhD. GTRI, Symposium Chair Opening Remarks, C. David Brown, PhD. CTEP, Chesapeake Systems Engineering, Deputy Chair, Symposium 9:00 AM Opening Speaker – Major General Maria Gervais, USA, Director of the Synthetic Training Environment Cross-Functional Team, Army Futures Command (invited) 9:30 AM Keynote Speaker – Raymond O'Toole, PhD, Principal Deputy Director, Office of the Secretary of Defense – Director, Operational Test and Evaluation 10:00 AM Virtual Exhibits – EXPO 10:30 AM Panel Discussion: Test & Evaluation Leadership Panel Chair: Dr. C. David Brown, CTEP, Chesapeake Systems Engineering

• Mike Knoll, Vice President, IT, FedEx • James Wells, Acting Director, Test & Evaluation Division, Department of Homeland Security • Robin Poston, PhD, Director of the System Testing Excellence Program, FedEx Institute of

Technology, The University of Memphis • Devin L. Cate, Director of Test & Evaluation, United States Air Force • Cathy O’Carroll, Global Campaign Director, Integrated Test & Evaluation QinetiQ • John Frederick, Federal Aviation Administration (FAA), Manager V&V Strategies & Practices Branch

Afternoon Session: 1:00 PM – 5:00 PM

Reconvene – Main Stage 1:00 PM Keynote Speaker – The Honorable Kevin M. Fahey, Assistant Secretary of Defense for Acquisition 1:45 PM Featured Speaker – Mitch Crosswait, PhD, Deputy Director, Office of the Director of Operational Test and Evaluation, Office of the Secretary of Defense 2:15 PM Wrap up the day – Symposium Chair, Flash Gordon 4:00 PM Virtual Happy Hour (Networking) with our Exhibitors and Guests

Wednesday, 16 September – Day 2 Morning Session 8:00 AM – Noon

7:30 AM Coffee & Connect (Networking)

On the Main Stage

8:00 AM Welcome – Steve Woffinden, General Dynamics, Symposium Technical Chair

8:05 AM Guest Speaker – John Fitzgibbon, President, TEQ Games

8:50 AM Featured Speaker – Greg Zacharias, PhD, Chief Scientist, DOT&E OSD

9:40 AM Featured Speaker – Daniel Serfaty, PhD, Founder & CEO, Aptima

10:10 AM Virtual Exhibits – EXPO

10:30 AM Panel Discussion: Testing for Safety and Security Panel

Chair: Wayne Dumais, Deputy Director, Immigration and Border Security Programs, Office of Test & Evaluation, Science & Technology Directorate, Department of Homeland Security

• Steve Rowson, PhD, Associate Professor, Department of Biomedical Engineering and Mechanics, Virginia Tech

• Christopher Mocella - Senior Science Officer, DHS, U.S. Customs & Border Protection, Laboratory and Scientific Division

• John Fitzgibbon - President, TEQ Games• To Be Announced

• To Be Announced

Afternoon Technical Track Sessions: 1:00 PM – 3:00 PM – Each Session 30 MinutesOn Tech Sessions

Testing for Subterranean Tunnels • Lessons Learned to Address Subterranean Testing Challenges: Unknown Equipment

Performance, Unknown Targets, Unknown Environments• Evaluating on the Southwest Border• Geophysical and Operational System Performance Tool for the Detection of Clandestine Tunnels

Advanced Instrumentation • Leveraging Commercial Cellular for Operational Testing• JASA – Connecting the (Test) Battlespace• Signal Noise and SNR in Test Planning• Digital Twins and Digital Threads in Defence Through the Lens of T&E

Autonomous Systems DT / Future Ranges • Enabling Effective T&E of Autonomous Systems using M&S• Overwater Impact Location (OWIL)• Spectrum and Future Range

• SENSR Future Range Infrastructure

Model-Based T&E and Other Advancements • A Case Study on the Evaluation of Algorithms for Declaring the Presence of Anomalous Aerosol

Signals Indicative of a Potential Terrorist Attack • The Developmental Evaluation Framework and its Effect on T&E • Iterative Development, Testing, and Analytic Process for a 3-Dimensional Common Graphical

User Interface Across Multiple Baggage Screening Systems • Model-Based Systems Engineering for T&E

International T&E

• Joint Strike Missile Qualification Testing • UK T&E Technology Challenges Industry Day • Operational Suitability: It’s Not Rocket Science, Why So Difficult? • HMS Queen Elizabeth Rotary Wing Trials

STEP Reports on T&E Research #1

• Cyber-Security Testing for Hybrid Energy Storage System Located in Smart Grid • A Testing Framework for The Computational Imaging Process In State-Of-The-Art Microscopy

Systems • Automated Assessment of The Quality of Jupyter Notebooks Using Artificial Intelligence and Big Code • Developing a Vehicular Network Emulator to Bridge the Gap in Network Architecture and

Application Testing

Thursday, 17 September – Day 3 Morning Session 8:00 AM – Noon

7:30 AM Coffee & Connect (Networking) On the Main Stage 8:20 AM Welcome – Steve Woffinden, General Dynamics, Symposium Technical Chair 8:30 AM Panel Discussion: T&E of Artificial Intelligence and Autonomy Panel Chair: Darryl Ahner, P.E., PhD - Director, Scientific Test and Analysis Techniques Center of Excellence • Bill Biggs, QinetiQ • Jane Pinelis, PhD, Chief of the Test, Evaluation and Assessment Branch, Department of Defense Joint Artificial Intelligence Center (JAIC) • David Tate, PhD, Institute for Defense Analysis • Mark Gillenson, University of Memphis • Jared Freeman, Chief Scientist, Corporate Fellow, APTIMA, Inc. • Greg Zacharias, PhD, Chief Scientist, DOT&E OSD 10:00 AM Virtual Exhibits – EXPO 10:30 AM Panel Discussion: Future of the T&E Workforce Panel

Chair: Terry Murphy - Deputy Director, Policy and Workforce Development, Department of Homeland Security (DHS), Office of Test and Evaluation

• Robin Poston, PhD - Director of the System Testing Excellence Program, FedEx Institute of Technology, The University of Memphis

• Darryl Ahner, P.E., PhD - Director, Scientific Test and Analysis Techniques Center of Excellence • Laura Freeman, PhD - Director, Intelligent System Lab, Virginia Tech Hume Center • Kenneth Stefanek, Learning Director for Test and Evaluation Defense Acquisition University (DAU) • Anne Sparrenberger, Former Executive, FedEx • Timothy Sienrukos, Test Director, Network Communications Branch, Federal Aviation Administration

Afternoon Technical Track Sessions: 2:30 PM – 4:00 PM – Each Session 30 Minutes

On Tech Sessions Agile Methods

• Advancing a Defense DX Framework for the Modernization Age • Enhancing the T&E Process Through the Incorporation of Agile Dev, MBSE, and Test Automation • Microservice Architecture Benefits to Test Systems

Directed Energy / Hypersonics T&E

• Live Testing of LASER Systems – Safe Execution • Design of Experiments in Characterizing Hypersonic Flow on a Wind Tunnel Model – A Case Study • Advanced Weapons Effects Test Capability (AWETC)

LVC T&E

• Virtual Reality in a Distributed Open-Air Test Environment • The Need for Simulation Software ‘Technology Readiness Levels’ (TRLs) in the Development of

Simulation Software for Training Test and Evaluation Systems • Overcoming the Challenges of Distributed Test & Evaluation

Cyber Security T&E

• Cyber T&E Tools: I’ll Show You Mine if You’ll Show Me Yours; A Realm of Possibility • Hacking Methods and Their Effects

STEP Reports on T&E Research #2

• How to Motivate System Developers to Enact Privacy By Design: A Regulatory Focus Perspective • DevSecOps Automated Security Testing Framework: Opportunities & Barriers • Testing Machine Learning Algorithms with Consistent Explanations

Friday, 18 September – Day 4

8:00 AM – 1:00 PM On the Main Stage 7:35 AM The State of ITEA – Pete Crump, GTRI, President ITEA 7:45 AM Keynote Speaker – J. George Rumford, Principal Deputy Director, Department of Defense, Test Resource Management Center 8:30 AM Panel Discussion: Integrating T&E with Agile Development Panel

Chair: Robin Poston, PhD, Director of the System Testing Excellence Program, FedEx Institute of Technology, The University of Memphis

• Wayne Dumais, Deputy Director, Immigration and Border Security Programs, Office of Test & Evaluation, Science & Technology Directorate, Department of Homeland Security

• Mike Knoll, Vice President, IT, FedEx • Antonia Pulley, IT Specialist • Dean Ptaszynski, Managing Partner, Agile Leadership Consulting • Naveen Kumar, PhD, Faculty Member, University of Oklahoma, Management Information Systems Division 10:00 AM Virtual Exhibits 10:30 AM Panel Discussion: Future of T&E Panel Chair: Eileen Bjorkman, Executive Director, Air Force Test Center, Edwards AFB • Cathy O’Carroll, Global Campaign Director, Integrated Test & Evaluation, QinetiQ • Libby Vallance-Bull, Head of Air Labs, BAE Systems • Mark Gillenson, PhD, University of Memphis • Devin Cate, Director of Test & Evaluation, United States Air Force • Terry Murphy, Deputy Director, Policy and Workforce Development, Department of Homeland

Security (DHS), Office of Test and Evaluation 12:10 PM Closing Speaker – James Wells, Acting Director, Test & Evaluation Division, Department of Homeland Security 12:45 PM Final Remarks – Steve “Flash” Gordon, PhD. GTRI, Symposium Chair