software quality assurance program and rollout at pppl .Ł sqa plan should define standard format,

SoftwareQualityAssuranceProgramandRolloutatPPPL

KeithEricksonkerickso@pppl.gov

NLIT2018Nashville,TNMay2124,2018

Agenda

PPPLBackground,SQAMotivation InitialPlan,Assessment,Surveyofotherlabs Industrystandards,firstattempt Adjustedmethods,inventory,andculturechanges Applicablecontrols SummaryandLessonsLearned

2

PrincetonUniversityPlasmaPhysicsLaboratory

ADOENationalLaboperatedbyPrincetonUniversity

FocusedonPlasmaFusionscienceandresearch

Oneprimaryfusionexperiment(NSTXU)

Severalsmallerprojects,including7USITERdiagnostics

3

MultipleProjectsFueledNeedforSQA

MotivatingProjects

CatalystCorrectiveActions

QualityAssuranceProgram

RootCauseAnalysis

NSTXURecoveryProject

ITERAcceptanceRequirements

ITERDiagnostics

4

NSTXURecoveryhighlightedopportunitytoimproveexistingSQA

ITERacceptanceofdelivereddiagnosticsrequiresQAofanalysissoftwaresuchasATILLA

DOEOrder414.1DreviewidentifiedneedforstrongerSQAcontrols

NationalSphericalTorusExperiment UpgradeOneofthetwolargestfacilitiesintheglobalSTresearchprogram

TS3/4, JapanUTST, JapanTST-2,

Japan

LTX- / CDX-U, USA

LATE,

Japan

QUEST/CPD, Japan

MAST-U, UK

HIST, Japan

PEGASUS, USA

NSTX-U, USA

ST40, UKSUNIST,

ChinaETE, Brazil KTM,

Kazakhstan

Proto Sphera,

Italy

VEST, Korea

17internationalfacilitiesforSTresearchandbroaderfusionscience

GLOBUS-M2, Russia

*SlidecourtesyofJ.Menard5

NSTXUResearchisHighlyCollaborative

362 data users

40 international

29 graduate students

25 post-doctoral researchers

54 collaborating institutions

32 domestic, 22 international

*SlidecourtesyofJ.Menard6

MajordamagetointernalcomponentsinFY16

7

Coppercoilinsulationdeterioratedovertime

Section of failed PF1AU

NSTX-U Machine

NSTX-U Centerstack

*SlidecourtesyofJ.Menard

AchievingNSTXUPerformanceGoalsrequiresRecoveryProject

FY2017:ExtentofConditionreview,resultingCorrectiveActionPlancoversentirefacility,addressesproceduralissuesalongwithaQualityAssuranceProgram

*SlidecourtesyofJ.Menard8

ITER TheWaytoNewEnergy

9*SlidecourtesyofJ.Klabacha

ITER TheWaytoNewEnergy

10

Worlds largest tokamak being built in southern France

Bringing the power of the sun down here to earth

7 ITER Members: China, EU, India, Japan, Korea, Russia, USA

Nothing on this scale has ever been done before

Achieve a burning plasma

Produce 500 [MW] fusion energy (10 fold return on energy!)

Integrate wide range of current device components and diagnostics

*SlidecourtesyofJ.Klabacha

USITER SevenWorldClassDiagnostics

11*SlidecourtesyofJ.Klabacha

EPP03EPP09

UPP11

UPP14

USITER FourintegratedPortPlugPackages

12*SlidecourtesyofJ.Klabacha

UpperPort14andATILLA

ATILLAneutronicsworknotacceptedwithoutATILLAQA,validation NoPPPLprocedureexistedtoprovevalidation Projectdeliveryhalted

StopgapmeasureenactedtouseUSITERprocess Parallelefforttocreatenewlabprocess

ExerciserevealedmissingverificationofANSYSaswell StructuralanalysisinANSYSalsodelayedawaitingQA

MustdevelopSQApolicy/proceduretodeliverITERdiagnostics!

13

Firstattempt,falsestart QAProgramRevamp,DOEOrderrequiresSoftwareQAfornuclearsafety

software

Nothingsafetyrelated,butmuchimportantsoftwarewithnoSQAprogram

QADepartmentaskedforaninitialauditto: Determineifcontrolsarecurrentlyinplaceandiftheyaresufficient Check,verifycompliancewithITER

Initialauditrevealedvirtuallynocontrols,outdatedlistofimportantsoftware

Auditcancelled,resultsclear:Roomtoimprove Actionplancreated StafftrainedonSQAatArgonne InitialSQAPolicydraftedandsentforreview

14

SQAProgramEvolutionOverview

15

IEEE730 Full

ImplementationMajorEffort

SimplifiedPolicySeparateSWDoc

UnifiedQAPD DeploymentCultureShock!

DOEO414.1DITERRequirementsBNL,ORNL,etal.

CorrectiveActionPlan

Timeline

Audit

Inventory

Identification

Categorization

Procedures

Outsourceddevelopment

Inhouseworkinggroup

SQAProgramhadseveralfalsestarts,neededinputfromotherlabs

QAPDcreationisidealforSQAlaunch Finalprogramperceivedasmajorculturechange,learningprocess

Followontasks,majordeliverablesidentifiedandinprogress

IEEE730 Full

ImplementationMajorEffort X

SimplifiedPolicyTooFragmented X

MergedQAPD Deployment CultureShock!

16

Whatissoftware?

Softwareincludescomputerprograms,firmware,procedures,operatingsystems,applications,rules,anddocumentation.[NITSLSQA2005]

Notableexample:anExcelspreadsheet*is*software

17

WhatissoftwareQA?

Activitiesthatdefineandassesstheadequacy ofsoftwareprocessestoestablishconfidencethattheprocessesareappropriate toproducesoftwareproductsofsuitablequalityfortheirintendedpurposes.AkeyattributeofSQAisobjectivitywithrespecttotheproject.TheSQAfunctionmayalsobeorganizationallyindependentoftheproject:freefromtechnical,managerial,andfinancialpressuresfromtheproject.[IEEE730]

18

WhatsoftwareneedsQA?

SoftwareusedatPPPLforthedesign,analysis,control,andoperationofresearchexperimentsandLaboratoryinfrastructure[QAPD]

19

IEEE730SQA

IEEE730describesthenecessarypartsforanSQAPlan Standards,practices,conventions,andmetrics Softwarereviews,Tests,Problemreporting,andcorrectiveactions Tools,techniques,andmethodologies Mediacontrol,Suppliercontrol,Recordscollection,maintenance,and

retention

Training

Riskmanagement

20

IEEE730informedthehighlevelroadmapfortheinitialpolicydraft

ExpandedIEEESQAStandardsList

21

IEEE828 SCM ConfigurationManagement

IEEE829 STD TestDocumentation

IEEE830 SRS RequirementsSpecification

IEEE1012 V&V VerificationandValidation

IEEE1016 SDD DesignDescription

IEEE1058 SPM ProjectManagement

IEEE1063 SUD UserDocumentation

MostofthesehavebeensupersededbyjointIEEE/IEC/ISOefforts

Theneweststandardsareexcessively complex

Olderstandardsdoprovidegoodguidanceastowhattoconsider

ExpandedIEEESQAStandardsList

22

SCM ConfigurationManagement

STD TestDocumentation

SRS RequirementsSpecification

V&V VerificationandValidation

SDD DesignDescription

SPM ProjectManagement

SUD UserDocumentation

IfwehideIEEE,thegeneralconceptsareusefulandapplicable

Mostprojectshavesomedegreeoftheseareasalready

buttheresalwaysroomforimprovement

InitialPolicyDraft:VeryIEEEcentric

High:Personnel,operationalhazardsMedium:Impactsefficient,effectiveoperationsLow:TertiarysoftwarenotusedforoperationsNone:Everythingelse

ClassificationImpact

ConfigurationManagementTestDocumentationRequirementsSpecificationVerification&ValidationDesignDescriptionProjectManagementUserDocumentation

AvailableControls

23

GreaterImpact Morecontrolsrequired Morestandardsfollowed

LesserImpact Fewercontrolsrequired Fewerstandardsfollowed

FocusonIEEEstandardsgenerallydifficulttoacceptinallbutmoststringentcases

Toomuchrelianceonstandardsexpertiseacrosswiderangeofusers

SQAMergedIntoOverallQAProgramDescription ExplicitreferencestoIEEEStandardsremoved GeneralcategoriesofQAcontrols(SCM,SRS,etc.)remain Classificationlevelschanged:

A1:Allcontrolsapply A2:Mostcontrolsapply A3:Nocontrolsapply A4:Levelremoved

Levelsbasedon7criteria:PersonnelHazard,MissionImpact,Cost,Risk,RadiologicalImpact,Safety,ProgramImpact

LabwidesoftwareinventorystartedinearnestasprecursortofullimplementationofofficialQAPD

24

SoftwareInventory Workinggroupformedtocatalogallsoftwareinuseinthelab InitialattemptsusedWin/Macadministrativefeaturestogatherlist Firstcutincludeddevelopmentenvironments

Newguidance:QAthespreadsheet,notExcel WhataboutANSYSvstheANSYSmodel?Determinationstillambiguous

Checklistaddedtodeterminenecessitytocategorize IndividualSMEsassignedtoreducebias CategorizationcriteriaforA1/2/3stemsfromoverallQAPD

Example:Safetyhazard Minor,Considerable,Serious Muchdebateastostrictvsloosedefinitionsofterms

Itsalearningprocess,andadifficultuphilljourney

25

SCM:Trackingandcontrollingchanges Bareminimum(andagoodpractice

regardless):UseyourfavoriteVersionControlSystemdujour

CMrequirementsshouldinclude: Procedures/Expectationsforbranches,

releases

Changelogentrystandards:requiredinformation,format,etc.

Mechanismsforensuringsoftwaredeploymentsareauthorizedandaccurate

Additionalprovisionsformanaging Acquiredvsdevelopedsoftware LocalWindows/MacvsclusterLinux

26

SCM ConfigurationManagement

STD TestDocumentation

SRS RequirementsSpecification

V&V VerificationandValidation

SDD DesignDescription

SPM ProjectManagement

SUD UserDocumentation

STD:Repeatabletestsrunnablebyothers

GeneraltestingatPPPLthoroughlydocumentedusingPreoperationalTestPlans,orPTP

Usedformostengineeringsystems Inconsistentlyappliedtosoftware Rarelyorneverappliedtophysics/

researchsoftware Veryeasytouseforacquired

softwarepackages Considertestbasedengineering

27

SCM ConfigurationManagement

STD TestDocumentation

SRS RequirementsSpecification

V&V VerificationandValidation

SDD DesignDescription

SPM ProjectManagement

S