emi infso-ri-261611 emi quality assurance processes (ps06-4-499) alberto aimar (cern) cern it-gt-sl...

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

EMI Quality Assurance Processes (PS06-4-499)

Alberto Aimar (CERN)CERN IT-GT-SL Section Leader

EMI SA2 QA Activity Leader

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• EMI context and QA objectives

• Guidelines and Metrics• Tools and Testbeds• Reports and Reviews

• Conclusions

CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 2

Outline

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

EMI Mission Statement


The European Middleware Initiative (EMI) project represents a close collaboration of the major European middleware providers - ARC, gLite, UNICORE and dCache - to establish a sustainable model to support, harmonise and evolve the grid middleware for deployment in EGI, PRACE and other distributed e-Infrastructures

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

EMI Project Structure

NA1 - Administrative and Technical Management

NA2 – Outreach and Collaborations

SA1 - Maintenance and Support

JRA1 - Development, Integration and Evolution

4EMI QA Activities - A.Aimar (CERN)CHEP 2010, Taipei

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 5

The Product Teams

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Four large MW projects developing several products – Many development teams > 25 Product Teams

• Request from the grid infrastructures and the EU funding – Have more uniform releases and interoperable middleware distributions– Provide a common degree of verification & validation– Have metrics and objective criteria to compare product quality and

evolution over time

• No resources to have independent QA efforts– Dev teams focus on development (JRA1) and maintenance (SA1)– No time to set up report on metrics, install tools, maintain testbeds

• Provide homogeneous packaging, reporting and reviewing of the products – Have limited impact on the way development teams work– Let them still use their build systems, dev tools, bug trackers, etc


The Challenge

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Quality Assurance Process Definition and Monitoring – Standards-compliant software engineering process. – Continual activity of monitoring its application

• Metrics Definition and Reporting – Definition, collection and reporting of software quality metrics. – Reports information on the status of the software to take corrective actions

• QA Implementation Review and Support – Review activities of the QA, test and certification implementations. – Sample review of test plans, compliance, porting guidelines, documentation, etc– Supporting the Product Teams in implementation of tests and use of testing tools

• Tools and Repositories, Maintenance and Integration – Definition and maintenance of tools required to support QA process– Supporting activity to software providers to integrate external tools– Repositories for the EMI software packages, tests, build and reports

• Testbeds Setup, Maintenance and Coordination – Setup and maintenance of distributed testbeds for continuous integration testing– Coordination and provision of larger-scale testbeds from collaborating providers


EMI QA (SA2) Tasks and Objectives

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611


Software Quality Plan

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Software Quality Assurance Plan (SQAP) – Document that specifies the tasks needed to define and measure quality,

responsibilities for quality monitoring, documentation required and procedures – Plan that will be followed to manage the QA process

• SQAP specifies the manner in which the EMI project is going to achieve its software quality goals. – Metrics and measurements are associated in order to evaluate the quality of

the EMI software and lifecycle

• SQA tasks, roles and responsibilities– EMI technical activities (SA1, SA2 and JRA1) – EMI technical bodies (PTB and EMT)– Even of specific individuals/roles in EMI

• The SQAP covers documentation, reporting and reviewing tasks– Describes the metrics that will be used for the QA reporting and reviews

• Will be updated regularly, based on experience and needs• It is complemented by a set of guidelines (on dev and doc)


Software Quality Plan

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• A set of very practical Guidelines for QA and Software Development is available:– Configuration and Integration – Packaging and Releasing – Change Management (bugs, patches, etc) – Certification and Testing – Metrics Generation

• Move towards a uniform setup and common definitions and conventions in the project– Releases, patches, versions– Packaging, repositories, distributions– User support, documentation

• Advantages for e-infrastructures are obvious but it requires some work and little changes by the dev teams in EMI– The guidelines were defined taking the best practices from the 4

middlewares and are mostly agreed by all dev teams


Development Guidelines

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611


Development Guidelines

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Definition of the Minimum Required Documentation for each software component. – Installation Guide and Troubleshooting guide – User Documentation – Software Requirements Specifications – Software Design Description – Software Verification and Validation Plan and Report

• General Project-wide Required Documentation– Technical Development Plan– Software Release Plan and Schedule– Software Maintenance and Support Plan– QA Tools Documentation– Continuous Integration and Certification Test beds Documentation– Security Assessments


Documentation Guidelines

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Metrics are needed to quantify and qualify the status of the software components

• Use as much as possible numerical metrics – All automated and extracted in the same exact way , by the same tool(s)

• Starting with a selection of useful and simple metrics– Tools available give a common environment to extract metrics and test– Same metrics for all components, in order to have fair reports

• Types of SQA metrics– Metrics on code, process, support, documentation – Internal and external metrics (code and process)– Language dependent (Java, C++, Python)

• Examples of metrics– Reaction time to critical bugs, delays in releases– Complexity, bug density, test coverage

• We refer to QA standards, but use what is realistically applicable– ISO /IEC 9126-1,-2,-3,-4 and ISO/IEC 25000


Quality Metrics

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611


Metrics

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Metric, Testing and Packaging Tools– Compliance and interoperability of the software products– Integration builds of all middleware components– Same identical platforms for all builds, use standard packages on platforms– Automatic deployment and distributed testing of software products

• Integration of data coming from different sources and tools– Common report of metrics from different static and dynamic software QA tools– Collection of data from several req and bug trackers used by dev teams – Using same tool for packaging and testing tools– Use of an exchange format for different sources

• Support for repositories and distribution– Common software repository for all EMI middleware– Use the standard RHEL/SL and Debian repositories and formats

• Generation of QA reports– Metrics extraction, storage and archival– Graphs and reports at all levels of detail – Comparison tables and plot trends


Tools

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611


Tools

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• EMI SA2 provides a distributed testbed – Real hw resources (+ obviously using virtualization) – For integration testing in EMI project – For the product teams testing– Distributed over the sites of the middleware partners

• Testbed available to Dev teams for testing their software– The teams can easily test their components with what is in production

or will soon be (RC)– Production and RC available for all components, servers, etc– Product Team can configure the services needed for its specific tests

• Provide support for the typical scenarios– Integration testing within a minor release– Integration testing for a major release – Cross middlewares tests across the network– Large-scale tests, real usage scenarios


Testbeds

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611


Testbeds

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• QA reports objectively describe the quality of the product– Only clear metrics are included, e.g. number of bugs/SLOC, reaction

time, trends over time, successful/failed releases, etc – Reports both on the product but also on how the team works– Not the goal of the QA activity, a mean to see strengths and weaknesses

• The same type report for all components– Allows comparisons among components – Trends over time of the same components

• Fully automated. Dev teams can have their report weekly– See the results and execute corrective actions

• SA2 reviews of the QA reports and supports the teams– Provides the general reviewing every Month and formally every Quarter – Reports to the EMI mgmt in case of issues– Supports the dev team that need help with metrics, tools, testbeds


QA Reports and Reviews

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611


QA Reports

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Dedicated QA is a useful activity in large projects, but guidelines, procedures should not overload the developers

• Challenge of implementing QA in a distributed and heterogeneous environment – Different kind of sw products, different tools, distributed teams, etc

• Collected experience from the developers in order to define realistic and shared solutions– Never seen a top down or theoretical software approach working...

... so we try a (slightly) different one.

• Define metrics and automate report generation • Provide also practical services, not just procedures

– to developers: supported and automated tools, testbeds, packaging– to e-infrastructures: verified and homogeneous sw, doc, repositories

https://twiki.cern.ch/twiki/bin/view/EMI/SA2


Conclusions

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Thank you

CHEP 2010, Taipei 22EMI QA Activities - A.Aimar (CERN)

EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611

emi infso-ri-261611 emi quality assurance processes (ps06-4-499) alberto aimar (cern) cern it-gt-sl...

Documents