software quality assurance program and rollout at pppl .Ł sqa plan should define standard format,

Download Software Quality Assurance Program and Rollout at PPPL .Ł SQA Plan should define standard format,

Post on 27-Aug-2018

212 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • SoftwareQualityAssuranceProgramandRolloutatPPPL

    KeithEricksonkerickso@pppl.gov

    NLIT2018Nashville,TNMay2124,2018

  • Agenda

    PPPLBackground,SQAMotivation InitialPlan,Assessment,Surveyofotherlabs Industrystandards,firstattempt Adjustedmethods,inventory,andculturechanges Applicablecontrols SummaryandLessonsLearned

    2

  • PrincetonUniversityPlasmaPhysicsLaboratory

    ADOENationalLaboperatedbyPrincetonUniversity

    FocusedonPlasmaFusionscienceandresearch

    Oneprimaryfusionexperiment(NSTXU)

    Severalsmallerprojects,including7USITERdiagnostics

    3

  • MultipleProjectsFueledNeedforSQA

    MotivatingProjects

    CatalystCorrectiveActions

    QualityAssuranceProgram

    RootCauseAnalysis

    NSTXURecoveryProject

    ITERAcceptanceRequirements

    ITERDiagnostics

    4

    NSTXURecoveryhighlightedopportunitytoimproveexistingSQA

    ITERacceptanceofdelivereddiagnosticsrequiresQAofanalysissoftwaresuchasATILLA

    DOEOrder414.1DreviewidentifiedneedforstrongerSQAcontrols

  • NationalSphericalTorusExperiment UpgradeOneofthetwolargestfacilitiesintheglobalSTresearchprogram

    TS3/4, JapanUTST, JapanTST-2,

    Japan

    LTX- / CDX-U, USA

    LATE,

    Japan

    QUEST/CPD, Japan

    MAST-U, UK

    HIST, Japan

    PEGASUS, USA

    NSTX-U, USA

    ST40, UKSUNIST,

    ChinaETE, Brazil KTM,

    Kazakhstan

    Proto Sphera,

    Italy

    VEST, Korea

    17internationalfacilitiesforSTresearchandbroaderfusionscience

    GLOBUS-M2, Russia

    *SlidecourtesyofJ.Menard5

  • NSTXUResearchisHighlyCollaborative

    362 data users

    40 international

    29 graduate students

    25 post-doctoral researchers

    54 collaborating institutions

    32 domestic, 22 international

    *SlidecourtesyofJ.Menard6

  • MajordamagetointernalcomponentsinFY16

    7

    Coppercoilinsulationdeterioratedovertime

    Section of failed PF1AU

    NSTX-U Machine

    NSTX-U Centerstack

    *SlidecourtesyofJ.Menard

  • AchievingNSTXUPerformanceGoalsrequiresRecoveryProject

    FY2017:ExtentofConditionreview,resultingCorrectiveActionPlancoversentirefacility,addressesproceduralissuesalongwithaQualityAssuranceProgram

    *SlidecourtesyofJ.Menard8

  • ITER TheWaytoNewEnergy

    9*SlidecourtesyofJ.Klabacha

  • ITER TheWaytoNewEnergy

    10

    Worlds largest tokamak being built in southern France

    Bringing the power of the sun down here to earth

    7 ITER Members: China, EU, India, Japan, Korea, Russia, USA

    Nothing on this scale has ever been done before

    Achieve a burning plasma

    Produce 500 [MW] fusion energy (10 fold return on energy!)

    Integrate wide range of current device components and diagnostics

    *SlidecourtesyofJ.Klabacha

  • USITER SevenWorldClassDiagnostics

    11*SlidecourtesyofJ.Klabacha

  • EPP03EPP09

    UPP11

    UPP14

    USITER FourintegratedPortPlugPackages

    12*SlidecourtesyofJ.Klabacha

  • UpperPort14andATILLA

    ATILLAneutronicsworknotacceptedwithoutATILLAQA,validation NoPPPLprocedureexistedtoprovevalidation Projectdeliveryhalted

    StopgapmeasureenactedtouseUSITERprocess Parallelefforttocreatenewlabprocess

    ExerciserevealedmissingverificationofANSYSaswell StructuralanalysisinANSYSalsodelayedawaitingQA

    MustdevelopSQApolicy/proceduretodeliverITERdiagnostics!

    13

  • Firstattempt,falsestart QAProgramRevamp,DOEOrderrequiresSoftwareQAfornuclearsafety

    software

    Nothingsafetyrelated,butmuchimportantsoftwarewithnoSQAprogram

    QADepartmentaskedforaninitialauditto: Determineifcontrolsarecurrentlyinplaceandiftheyaresufficient Check,verifycompliancewithITER

    Initialauditrevealedvirtuallynocontrols,outdatedlistofimportantsoftware

    Auditcancelled,resultsclear:Roomtoimprove Actionplancreated StafftrainedonSQAatArgonne InitialSQAPolicydraftedandsentforreview

    14

  • SQAProgramEvolutionOverview

    15

    IEEE730 Full

    ImplementationMajorEffort

    SimplifiedPolicySeparateSWDoc

    UnifiedQAPD DeploymentCultureShock!

    DOEO414.1DITERRequirementsBNL,ORNL,etal.

    CorrectiveActionPlan

    Timeline

    Audit

    Inventory

    Identification

    Categorization

    Procedures

    Outsourceddevelopment

    Inhouseworkinggroup

    SQAProgramhadseveralfalsestarts,neededinputfromotherlabs

    QAPDcreationisidealforSQAlaunch Finalprogramperceivedasmajorculturechange,learningprocess

    Followontasks,majordeliverablesidentifiedandinprogress

  • IEEE730 Full

    ImplementationMajorEffort X

    SimplifiedPolicyTooFragmented X

    MergedQAPD Deployment CultureShock!

    16

  • Whatissoftware?

    Softwareincludescomputerprograms,firmware,procedures,operatingsystems,applications,rules,anddocumentation.[NITSLSQA2005]

    Notableexample:anExcelspreadsheet*is*software

    17

  • WhatissoftwareQA?

    Activitiesthatdefineandassesstheadequacy ofsoftwareprocessestoestablishconfidencethattheprocessesareappropriate toproducesoftwareproductsofsuitablequalityfortheirintendedpurposes.AkeyattributeofSQAisobjectivitywithrespecttotheproject.TheSQAfunctionmayalsobeorganizationallyindependentoftheproject:freefromtechnical,managerial,andfinancialpressuresfromtheproject.[IEEE730]

    18

  • WhatsoftwareneedsQA?

    SoftwareusedatPPPLforthedesign,analysis,control,andoperationofresearchexperimentsandLaboratoryinfrastructure[QAPD]

    19

  • IEEE730SQA

    IEEE730describesthenecessarypartsforanSQAPlan Standards,practices,conventions,andmetrics Softwarereviews,Tests,Problemreporting,andcorrectiveactions Tools,techniques,andmethodologies Mediacontrol,Suppliercontrol,Recordscollection,maintenance,and

    retention

    Training

    Riskmanagement

    20

    IEEE730informedthehighlevelroadmapfortheinitialpolicydraft

  • ExpandedIEEESQAStandardsList

    21

    IEEE828 SCM ConfigurationManagement

    IEEE829 STD TestDocumentation

    IEEE830 SRS RequirementsSpecification

    IEEE1012 V&V VerificationandValidation

    IEEE1016 SDD DesignDescription

    IEEE1058 SPM ProjectManagement

    IEEE1063 SUD UserDocumentation

    MostofthesehavebeensupersededbyjointIEEE/IEC/ISOefforts

    Theneweststandardsareexcessively complex

    Olderstandardsdoprovidegoodguidanceastowhattoconsider

  • ExpandedIEEESQAStandardsList

    22

    SCM ConfigurationManagement

    STD TestDocumentation

    SRS RequirementsSpecification

    V&V VerificationandValidation

    SDD DesignDescription

    SPM ProjectManagement

    SUD UserDocumentation

    IfwehideIEEE,thegeneralconceptsareusefulandapplicable

    Mostprojectshavesomedegreeoftheseareasalready

    buttheresalwaysroomforimprovement

  • InitialPolicyDraft:VeryIEEEcentric

    High:Personnel,operationalhazardsMedium:Impactsefficient,effectiveoperationsLow:TertiarysoftwarenotusedforoperationsNone:Everythingelse

    ClassificationImpact

    ConfigurationManagementTestDocumentationRequirementsSpecificationVerification&ValidationDesignDescriptionProjectManagementUserDocumentation

    AvailableControls

    23

    GreaterImpact Morecontrolsrequired Morestandardsfollowed

    LesserImpact Fewercontrolsrequired Fewerstandardsfollowed

    FocusonIEEEstandardsgenerallydifficulttoacceptinallbutmoststringentcases

    Toomuchrelianceonstandardsexpertiseacrosswiderangeofusers

  • SQAMergedIntoOverallQAProgramDescription ExplicitreferencestoIEEEStandardsremoved GeneralcategoriesofQAcontrols(SCM,SRS,etc.)remain Classificationlevelschanged:

    A1:Allcontrolsapply A2:Mostcontrolsapply A3:Nocontrolsapply A4:Levelremoved

    Levelsbasedon7criteria:PersonnelHazard,MissionImpact,Cost,Risk,RadiologicalImpact,Safety,ProgramImpact

    LabwidesoftwareinventorystartedinearnestasprecursortofullimplementationofofficialQAPD

    24

  • SoftwareInventory Workinggroupformedtocatalogallsoftwareinuseinthelab InitialattemptsusedWin/Macadministrativefeaturestogatherlist Firstcutincludeddevelopmentenvironments

    Newguidance:QAthespreadsheet,notExcel WhataboutANSYSvstheANSYSmodel?Determinationstillambiguous

    Checklistaddedtodeterminenecessitytocategorize IndividualSMEsassignedtoreducebias CategorizationcriteriaforA1/2/3stemsfromoverallQAPD

    Example:Safetyhazard Minor,Considerable,Serious Muchdebateastostrictvsloosedefinitionsofterms

    Itsalearningprocess,andadifficultuphilljourney

    25

  • SCM:Trackingandcontrollingchanges Bareminimum(andagoodpractice

    regardless):UseyourfavoriteVersionControlSystemdujour

    CMrequirementsshouldinclude: Procedures/Expectationsforbranches,

    releases

    Changelogentrystandards:requiredinformation,format,etc.

    Mechanismsforensuringsoftwaredeploymentsareauthorizedandaccurate

    Additionalprovisionsformanaging Acquiredvsdevelopedsoftware LocalWindows/MacvsclusterLinux

    26

    SCM ConfigurationManagement

    STD TestDocumentation

    SRS RequirementsSpecification

    V&V VerificationandValidation

    SDD DesignDescription

    SPM ProjectManagement

    SUD UserDocumentation

  • STD:Repeatabletestsrunnablebyothers

    GeneraltestingatPPPLthoroughlydocumentedusingPreoperationalTestPlans,orPTP

    Usedformostengineeringsystems Inconsistentlyappliedtosoftware Rarelyorneverappliedtophysics/

    researchsoftware Veryeasytouseforacquired

    softwarepackages Considertestbasedengineering

    27

    SCM ConfigurationManagement

    STD TestDocumentation

    SRS RequirementsSpecification

    V&V VerificationandValidation

    SDD DesignDescription

    SPM ProjectManagement

    S