Download - SA2 – Quality Assurance
EMI is partially funded by the European Commission under Grant Agreement RI-261611
SA2 – Quality AssuranceAlberto AIMAR (CERN)SA2 Leader
EMI Second EC Review12 June 2012, Brussels
EMI I
NFSO
-RI-2
6161
1
2
ContextObjectives and Recommendations
Impact on EMI Quality • Build and Test
(ETICS)• Integration Infrastructure
(EMI Testbed)• Metrics• Quality Control
Conclusions
Outline
12/06/2012 EMI 2nd EC Review
https://bit.ly/emisa2
EMI I
NFSO
-RI-2
6161
1
EMI 2nd EC Review 3
Quality Assurance in EMI (SA2)
12/06/2012
EMI I
NFSO
-RI-2
6161
1
4
DoW Objectives
12/06/2012 EMI 2nd EC Review
DoW SO 1.4• Common software quality assurance policies and metrics
for all engineering activities• Monitoring metrics and trends quality control activities
value trends, reviews and related tests, providing support and consultancy in QA matters.
• Allow EMI to pass customer acceptance criteriacontinually improve software quality and process itself
DoW SO 1.5• Continuous building, integration and testing process
by selecting and maintaining tools and resources for building and testing either within the project or in collaboration with external resource providers
EMI I
NFSO
-RI-2
6161
1
5
SA2 Quality Assurance
12/06/2012 EMI 2nd EC Review
QA Policies
Build and Test Tools
Integration Infrastructure
QA Metrics
Quality Control
Reports & Dashboards
Development Trackers
NA2JRA1
SA1
JRA1
SA1
SA2Quality Assurance
EMI I
NFSO
-RI-2
6161
1
EMI 2nd EC Review 6
EMI merged 4 established Middleware projects• Each had its own QA practices, tools, reports, etc.
Working with a 3-years vision • Y1: explain, define and implement • Y2: review and automate• Y3: consolidate and optimize
Benefitted from existing QA practices • Used existing QA tools, resources and expertise• Involved PTs, SA1, JRA1 and also explained QA constraints• Associated policies with tools to support them for build, test,
release and automated reports
SA2 QA Strategy
12/06/2012
EMI I
NFSO
-RI-2
6161
1
EMI 2nd EC Review
1. Resubmit SQAP deliverable• Submitted in October 2011. Reorganized as a plan for the 3 years.2. Reduce correlated deliverables • Merged QA and QC deliverable (DSA2.3.3)3.Integrate Quality Control from SA1 and JRA1 into SA2 • Completed by June 2011• Solid QC integration with policies, metrics and reports4. Foster sustainable solutions after EMI • Added standard tools (Mock, Pbuilder)• Compel compliance to established distributions (EPEL, Debian)• Distribution of buildable sources in addition to binaries
Recommendations from 1st Review
12/06/2012 7
EMI I
NFSO
-RI-2
6161
1
8
All SA2 deliverables submitted
• DSA2.1 - Quality Assurance Planrewritten and resubmitted taking into account the review of Y1
• DSA2.2.3 - QA Tools Documentationupdate of previous deliverables DSA2.2.1 and DSA2.2.2
• DSA2.3.3 - Periodic QA and QC Reportsincludes DJRA1.7.3 - Software Development QC Report
SA2 Deliverables
12/06/2012 EMI 2nd EC Review
EMI I
NFSO
-RI-2
6161
1
9
Full support Fedora (SL5, SL6), Debian 6 • Standard build tools (Mock and PBuilder) • Standard packaging/repositories (EPEL and APT)Two ways of building• Use ETICS for build and packaging• Give sources to build with Mock/PbuilderAll EMI software builds for EMI platforms• Single project configuration• Used by PTs and for global Nightly Builds• Better workload management• Maven mirror, VM images for developers• Elastic build infrastructure
EMI Tools and ETICS in Y2
EMI 2nd EC Review12/06/2012
0
1000
2000
3000
4000
5000
6000
7000
Montlhy ETICS Builds
DoW SO 1.5 : Continuous integration, building and testing process
Build and Test Tools
EMI I
NFSO
-RI-2
6161
1
10
Fundamental for releases• 3 platforms x Releases x RCs x 32/64 bits • >200 installations of pre-EMI, EMI-1, EMI-2 products• CERN, CESNET, CNAF INFN, DESY, JUELICH, KOSICE, NIIF Everything has been deployed and tested on the QA Testbed• 15 EMI 1 Updates + EMI 2 Release• Provided feedback on documentation, and deployment issues to PTsFundamental for testing, training, demos, monitoring• Adopted common EMI deployment tests, integration tests across products• Used training and demos, supported “EGI TF 2011 best demo” awardCollaboration with EGI is operational and effective• Releases EMI project regularly pass the EGI validation you will see later• Deployed SAM-NAGIOS for automated testing and monitoring
Testing Infrastructure in Y2
12/06/2012
Integration Infrastructure
DoW SO 1.5 : Continuous integration, building and testing process
EMI I
NFSO
-RI-2
6161
1
EMI 2nd EC Review 11
Metrics on code, process, docs• Reaction to RfCs, delays in releases• Compliance to standards and templatesAutomated reports and live dashboards• Focus on metrics immediately useful • RfC management, certification and testingTotal control of Development Process• Status of the products, tests, trackers• Used by QC and Product TeamsFaster and compliant product releases • Submit all QA issues to PTs for prompt action• Products ready for passing the QC tests
QA Metrics and Reports in Y2
12/06/2012
DoW SO 1.4: Common software quality assurance policies and metrics
QA Metrics
Reports & Dashboards
EMI I
NFSO
-RI-2
6161
1
EMI 2nd EC Review 12
EMI RfC Dashboard
12/06/2012
Data from >30 trackers of all EMI Product Teams, status of EMI developmentAccess to all the RfC in each tracker, search and select, export to Excel
http://emi-rfc.cern.ch
EMI I
NFSO
-RI-2
6161
1
13
EMI Verification Dashboard
12/06/2012 EMI 2nd EC Review
Access to all reports of each product, complete view of the productshttp://emi-verification.cern.ch
EMI I
NFSO
-RI-2
6161
1
EMI 2nd EC Review 14
Software Quality Assurance Plan updated• Reorganised to match your recommendations• Added milestones and targets that were in other documentsAll Product comply with the platforms supported• Multiple platforms support (release, configuration, packaging, doc, etc.)Quality Control monitors compliance to policies• Uses the QA dashboards to view and extract the desired metrics• Metrics automatically extracted to tabular data (e.g. for SA2 and SA1)• Weekly reports and plots from the metrics and constant follow-upQuality Control has impact on work of Product Teams• Products passing are deployed, product not passing are
discussed with the PTs• RfCs are issued and the execute the changes
QA Policies and Control in Y2
12/06/2012
QA Policies
Quality Control
DoW SO 1.4: Common software quality assurance policies and metrics
EMI I
NFSO
-RI-2
6161
1
15
QC of SA2 Services (KPIs)
12/06/2012 EMI 2nd EC Review
KPI Target P1 Q5 Q6 Q7 Q8 P2 KSA2.2 - Services Availability
ETICS 97% 97.5 % 98.0 % 95.8 % 96.8 % 97.9 % 97.1 %Testbed 97% 98.3 % 99.0 % 98.9 % 99.1 % 99.3 % 99.1 %
KSA2.3 - Testbed Size 200 CPUs 73 97 112 154 204 204
KSA2.6 - No of RequestsETICS - 230 33 36 47 46 162
Testbed - 35 14 8 11 5 38 KSA2.7 - Average Response Time
ETICS - 6.4 h 3.2 h 2.45 h 0.95 h 0.73 h 1.68 hTestbed - 5.2 h 8.1 h 2.65 h 4.7 h 1.30 h 5.0 h
KSA2.8 - Average Solution TimeETICS - 21.9 h 96.6 h 133.0 h 84.2 h 57.9 h 90.0 h
Testbed - 20.3 h 45.5 h 46.6 h 18.5 h 39.6 h 37.1 h
EMI I
NFSO
-RI-2
6161
1
EMI 2nd EC Review 16
QC – Testing Improvements
12/06/2012
EMI 1
Updates 1-7 Updates 8-14
Quality Control
EMI I
NFSO
-RI-2
6161
1
17
QC – Regression Testing
12/06/2012 EMI 2nd EC Review
Report of Regression Tests for High/Immediate RfCs
Data not collected before
update 5
Policy enforced by Release Manager
after update 7
Quality Control
EMI I
NFSO
-RI-2
6161
1
EMI 2nd EC Review 18
Convergence to standards of EMI software
QC – Compliance to Fedora/EPEL
12/06/2012
Quality Control
90 %
60 %
EMI I
NFSO
-RI-2
6161
1
19
QC – Defect per Product
12/06/2012 EMI 2nd EC Review
Quality Control
Number of Defects per Products per Quarter
EMI I
NFSO
-RI-2
6161
1
20
From EGI TCB 11 – April 2012 DoW SO 1.4: Allow EMI to pass customer acceptance criteria
80%
70%
90%
100%
EMI I
NFSO
-RI-2
6161
1
EMI 2nd EC Review 21
Main Achievements• New platforms successfully supported• Convergence to standards, QA has an impact on the actual quality• Faster testing and successful deployment of products• Very high success rate in the acceptance of our releases to EGIExcellent collaboration and results across all EMI activities• SA1 for releases, updates and deployment. NA2 for training, documentation.
JRA1 quality control and dashboardsQuality Control merge in SA2 a very good decision• Quicker feedback to EMT and within SA2 (metrics, reports, reviews)•Someone external to PTs checks products (deployment, compliance) Y3: Consolidate and Optimize • Continue to support/improve current tools, testbed, dashboard and reports• QC on sustainability of middleware products beyond the EMI Y3
Conclusions
12/06/2012
EMI I
NFSO
-RI-2
6161
1
22
Thank You