April 3, 2017 Sam Siewert

SE420 Software Quality Assurance

Lecture 12 – Industry Systems and Analysis by Type

http://dilbert.com/strips/comic/2009-06-22/

http://dilbert.com/strips/comic/2009-06-22/http://dilbert.com/strips/comic/2009-06-22/http://dilbert.com/strips/comic/2009-06-22/http://dilbert.com/strips/comic/2009-06-22/http://dilbert.com/strips/comic/2009-06-22/http://dilbert.com/strips/comic/2009-06-22/

Reminders Assignment #5 – Grading with emphasis on feedback on what you can improve in A#6 and your Final Inspection

Remaining Assignments [Top Down / Bottom-Up] – #6 – Complete Code, Refine and Run all V&V Tests and Deliver

Track Bugs with Bugzilla - http://prclab.pr.erau.edu/ Update your Project Code on GitHub - https://github.com/ Assignment #6 Posted – Questions, 2nd Walk-through?

Sam Siewert 2

http://prclab.pr.erau.edu/http://prclab.pr.erau.edu/https://github.com/https://github.com/

Regression Testing and Test Automation

Re-test Units After Defects are Fixed Re-run I&T and System Tests Automation from Checkout, Build, Test Sets, Verification & Validation of Expected Results

Sam Siewert 3

Reg

ress

ion

Test

ing

(Tes

t Aut

omat

ion)

Discussion and Q&A Industry Specific Guidelines, Standards, and Audit Mission Critical Systems (Life, Financial and Property Risk)

– Enterprise Data Systems – Financial, Medical Records, Government, HR, etc. Storage, Networking, Web Access, DBMS Security and Data Integrity is Critical, Business Continuity

– Aerospace – Civil Aviation, Defense Systems, NASA, Commercial Space – Medical Systems – Diagnostics, Therapeutics, Monitoring, Laboratory

Automation – Energy – Refineries, Nuclear Power Generation, Distribution, Exploration and

Mining – Security – National, Corporate, Personal – Transportation – Public, Private

Cyberphysical Systems - http://cyberphysicalsystems.org/

NSF Conference – 2006, Austin Texas, http://varma.ece.cmu.edu/cps/, focus on position papers (similar to 1968 NATO Software Crisis Conference) Original Paper(s) Introducing Concept - http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-8.pdf Sam Siewert 4

http://cyberphysicalsystems.org/http://varma.ece.cmu.edu/cps/http://varma.ece.cmu.edu/cps/http://varma.ece.cmu.edu/cps/

Cyberphysical Systems

Sam Siewert 5

Emergent Cyberphysical Application – ITS [Intelligent Transportation System]

Intelligent Transportation - http://sites.ieee.org/itss/ Intel in Intelligent Transportation - http://www.intel.com/content/www/us/en/automotive/experiencing-future-intelligent-transportation-video.html Is this a Good Idea? - It’s a Complex System of Systems with Major Security Challenges Opens new Markets and Can Optimize Traffic, both Personal and Commercial [E.g. Truck Fleeting with Self-Driving Trucks, I-5 California]

Sam Siewert 6

http://sites.ieee.org/itss/http://www.intel.com/content/www/us/en/automotive/experiencing-future-intelligent-transportation-video.htmlhttp://www.intel.com/content/www/us/en/automotive/experiencing-future-intelligent-transportation-video.html

Emergent Cyberphysical Application – Commercial Space Transportation

NASA Has Outsourced Space Station Servicing and Access in General – http://www.spacex.com/ – http://www.sncorp.com/ – OSC - http://www.orbital.com/ [Antares 2014 Accident]

Space Tourism and Commercialization – http://www.virgingalactic.com/statement-from-virgin-galactic/

[SpaceShip Two Accident in 2014] – XCOR Aerospace - http://www.xcor.com/ – Boeing CST-100 - http://en.wikipedia.org/wiki/CST-100

A Complex System of Systems with Major Safety Challenges

Sam Siewert 7

http://en.wikipedia.org/wiki/Commercial_Resupply_Serviceshttp://en.wikipedia.org/wiki/Commercial_Resupply_Serviceshttp://www.spacex.com/http://www.sncorp.com/http://www.orbital.com/http://www.nasa.gov/press/2014/october/nasa-s-wallops-flight-facility-completes-initial-assessment-after-orbital-launchhttp://en.wikipedia.org/wiki/Space_tourismhttp://www.virgingalactic.com/statement-from-virgin-galactic/http://www.xcor.com/http://en.wikipedia.org/wiki/CST-100

Software and Public Safety New Challenges for Software Defined Systems and Public Safety (SQA) 1. Self-driving cars (Uber, Google,

Tesla, …) – Integration with drivers and traditional cars, potential SQA issues?

2. Space tourism – SpaceX Mission around the moon

3. Material Internet – delivering packages via small UAS

4. Cyber security threats for CPS

Sam Siewert 8

SpaceX Moonshot

Uber self-driving cars in AZ

Amazon Prime Air

Commercial Mission Critical Software Complexity on the Rise Hackers and Banking

http://www.space.com/35876-how-spacex-moon-flight-will-work.htmlhttp://www.theverge.com/2017/2/21/14687346/uber-self-driving-car-arizona-pilot-ducey-californiahttps://www.amazon.com/Amazon-Prime-Air/b?node=8037720011https://www.wired.com/2017/04/hackers-emptying-atms-drill-15-worth-gear/?mbid=nl_4317_p1&CNDID=46315705

A Decade of Space Tourism

Sam Siewert 9 http://en.wikipedia.org/wiki/Space_tourism

Most Recent

Early Space Tourists

http://en.wikipedia.org/wiki/Space_tourismhttp://en.wikipedia.org/wiki/Space_tourism

Traditional Mission Critical Systems Standards and Guidelines for Systems and Software Testing Enterprise Data Systems – Commercial Standards and IV&V Labs (E.g. Medusa Labs Test Tools), Standards for Protocol (SCSI T10 and SAS, Fiber Channel T11, FCoE, iSCSI) and Performance Test Standards (TPC, SPC), SNIA RAID Aerospace – NASA Standards (https://standards.nasa.gov/documents/nasa ), Military Standards (MIL-STD-498), FAA Launch and Reentry, FAA Medical Systems – IEC 62304, E.g. Commercial Support Energy – NIST Testing, DoE STD-1172 Security – TCG, NIST ITL, NIST FISMA, NIST FIPS, NIST TIRs – Overview Transportation – FAA DO-178B, FAA DO-178C, Software Assurance, V&V, NHTSA Policies on Automated Vehicles, FAA Publications

Sam Siewert 10

http://www.jdsu.com/Downloads/TestToolsGuide.pdfhttp://www.t10.org/http://www.t11.org/index.htmlhttps://tools.ietf.org/html/rfc3720http://www.tpc.org/http://www.storageperformance.org/home/http://www.snia.org/tech_activities/http://www.snia.org/tech_activities/standards/curr_standards/ddfhttps://standards.nasa.gov/documents/nasahttp://en.wikipedia.org/wiki/MIL-STD-498http://en.wikipedia.org/wiki/MIL-STD-498https://www.faa.gov/about/office_org/headquarters_offices/ast/media/AST_Guide_to_Software_Safety_final_070706.pdfhttps://www.faa.gov/about/office_org/headquarters_offices/ast/reports_studies/media/DMurray_SW%20REQTS_IAASS07_FINAL.pdfhttp://en.wikipedia.org/wiki/IEC_62304http://www.vectorcast.com/testing-solutions/software-testing-embedded-medical-devices-fda-iec-62304http://www.nist.gov/software-testing-metrics-portal.cfmhttp://energy.gov/sites/prod/files/2013/06/f1/DOE-STD-1172-2011.pdfhttp://www.trustedcomputinggroup.org/http://www.nist.gov/itl/http://csrc.nist.gov/groups/SMA/fisma/overview.htmlhttp://csrc.nist.gov/publications/PubsFIPS.htmlhttp://csrc.nist.gov/publications/PubsNISTIRs.htmlhttps://www.cs.purdue.edu/homes/xyzhang/fall07/Papers/sw-test.pdfhttp://en.wikipedia.org/wiki/DO-178Bhttp://en.wikipedia.org/wiki/DO-178Chttp://www.faa.gov/documentLibrary/media/Order/1370.109.pdfhttps://www.faa.gov/aircraft/air_cert/design_approvals/air_software/cast/cast_papers/media/cast-11A.pdfhttp://www.nhtsa.gov/About+NHTSA/Press+Releases/U.S.+Department+of+Transportation+Releases+Policy+on+Automated+Vehicle+Developmenthttp://www.faa.gov/aircraft/air_cert/design_approvals/air_software/publications/

Infamous System Software Defects Infamous & Famous Field Defects

1. Toyota ABS Software Recall, 2010 2. Windows Genuine Advantage Outage, 2006 3. Mars Climate Orbiter Units Interoperability,

Loss, 1999 4. Mars Pathfinder Priority Inversion,

Recovered, 1997 5. Ariane-5 501 Cluster Launch Software

Failure, 1996 6. Pentium FPU Bug – Software Acceleration

Co-Processor Recall, 1994 7. ATT 4ESS Upgrade, Outage, 1990 8. Therac-25 Radiation Therapy Patient

Overdosing, 1985-86 9. NORAD False Alarms, User Error for Test

Mode, Device Failures, 1979/1980 10. Apollo 11 Guidance Computer Overload,

1969 - Recovered

Sam Siewert 11

http://en.wikipedia.org/wiki/Software_bug

RCA, Patch, Test, Fix!

http://history.nasa.gov/SP-350/ch-11-4.html

Rear Admiral Grace Hopper

http://www.computerworld.com/article/2515483/enterprise-applications/epic-failures--11-infamous-software-bugs.htmlhttp://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle_recalls#Anti-lock_brake_software_recallhttp://blogs.msdn.com/b/wga/archive/2006/10/05/wga-service-outage.aspxhttp://blogs.msdn.com/b/wga/archive/2006/10/05/wga-service-outage.aspxhttp://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Cause_of_failurehttp://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Cause_of_failurehttp://research.microsoft.com/en-us/um/people/mbj/mars_pathfinder/Authoritative_Account.htmlhttp://en.wikipedia.org/wiki/Cluster_(spacecraft)#Launch_failurehttp://en.wikipedia.org/wiki/Pentium_FDIV_bughttp://www.phworld.org/history/attcrash.htmhttp://en.wikipedia.org/wiki/Therac-25http://en.wikipedia.org/wiki/North_American_Aerospace_Defense_Command#False_alarmshttp://history.nasa.gov/SP-350/ch-11-4.htmlhttp://en.wikipedia.org/wiki/Software_bughttp://en.wikipedia.org/wiki/Software_bughttp://history.nasa.gov/SP-350/ch-11-4.htmlhttp://history.nasa.gov/SP-350/ch-11-4.htmlhttp://history.nasa.gov/SP-350/ch-11-4.htmlhttp://history.nasa.gov/SP-350/ch-11-4.htmlhttp://history.nasa.gov/SP-350/ch-11-4.htmlhttp://history.nasa.gov/SP-350/ch-11-4.htmlhttp://history.nasa.gov/SP-350/ch-11-4.htmlhttp://history.nasa.gov/SP-350/ch-11-4.htmlhttp://en.wikipedia.org/wiki/Grace_Hopper

Importance of SQA and SE Practices Technology ahead of Government Regulations? Policy (Guidelines for Industry to Define/Refine) – E.g. FAA ASSURE – to study sUAS and Civil Aviation Shared

Airspace – DoT – ITS Strategic Plan – NASA Advisory Council on Commercial Space and HEO – NIST – Department of Commerce, Cybersecurity Framework

Independent QA and Certifications – SEI – Think Tank for Software Engineering – ISTQB – Certification for Testing Methods, Process,

Practitioners

Sam Siewert 12

http://www.assureuas.org/https://www.its.dot.gov/research_areas/strategicplan2015.htmhttps://www.nasa.gov/directorates/heo/nac-heochttps://www.nist.gov/cyberframeworkhttp://www.sei.cmu.edu/http://www.istqb.org/

Discussion Does an SEI CMM Rating Guarantee Results? Improve Probability of Success (Defense Acquisition – Past Performance) Are Standards and Guidelines Sufficient? What Can be Done to Improve? Could there be One Standard? NTSB Aviation Accidents (RC - SW?) Coding Practices – Linux Programming Top Errors – Design Principles – E.g. Cohesion and Coupling, Static Metrics

(Cyclomatic Complexity, McCabe, Fan-in, Fan-out) – Papers on SE Metrics – What’s up with software metrics?

Sam Siewert 13

http://www.dau.mil/Research/default.aspxhttp://www.dau.mil/research/symposiumdocs/BRADSHAW%20-%20PAST%20PERFORMANCE%20slides.pdfhttp://www.dau.mil/research/symposiumdocs/BRADSHAW%20-%20PAST%20PERFORMANCE%20slides.pdfhttps://www.ntsb.gov/investigations/reports_aviation.htmlhttp://mercury.pr.erau.edu/%7Esiewerts/se420/documents/Linux/Linux-Programming-Top-Errors.pdfhttp://mercury.pr.erau.edu/%7Esiewerts/se420/documents/Linux/Linux-Programming-Top-Errors.pdfhttp://mercury.pr.erau.edu/%7Esiewerts/se420/documents/Papers/SE-Metrics-Status-and-Mapping.pdf

SE420�Software Quality AssuranceRemindersRegression Testing and Test AutomationDiscussion and Q&ACyberphysical SystemsEmergent Cyberphysical Application – ITS [Intelligent Transportation System]Emergent Cyberphysical Application – Commercial Space TransportationSoftware and Public SafetyA Decade of Space TourismTraditional Mission Critical SystemsInfamous System Software DefectsImportance of SQA and SE PracticesDiscussion