Download - QA_EA and Certification Testing
EA and Certification TestingAndrew PritchardEA Certification Microsoft Platform Supervisor Electronic Arts Inc.
Agenda
• Who We Are• EA Certification Services• EA Certification Structure• Sharing Knowledge • Aiming to Align• Tips
Who We AreIntroducing EA Certification• EA Certification is responsible for the Final Certification process on all EA Xbox 360
products
• An audit function post QA Final through which all EA products must pass before submission to the Microsoft Certification Group
• EA Certification manages the submission process into Microsoft, as well as handling TCR Exceptions
• Main Certification liaison between EA and Microsoft Certification Group (via Release Management)
• Service Provider to EA Studios, Consultation, Pre-Final Reviews, Age Ratings, Knowledge Sharing
• To provide consistent level of service, support and quality to Microsoft despite disparate EA teams around the world
Who We Are Where We Are, Who We Work With
EAC – Burnaby ,Canada (Black Box, EA SPORTS +
QA)
Singapore(Cert and QA)
Bucharest, Romania
(QA)
EA Montreal, Canada
(Studio + QA)
EALA – Los Angeles (Danger Close + QA)
EA Tiburon, Florida
(Studio + QA)
Stockholm, Sweden(DICE+ QA)
Baton Rouge, Louisiana
(QA)
Buenos Aires, Argentina (QA)
Guildford, U.K.(Bright Light Studio, Criterion
+ QA)EA Certification (Canada)Microsoft DLC Specialist
EA Certification (Redwood Shores)
MS Submissions TeamMicrosoft DLC Specialist
EA Certification – Madrid, Spain
Microsoft Test GroupMS Submissions Team
Microsoft DLC SpecialistTester Pool
EA Redwood Shores (EARS)
(Visceral Games + QA)
Edmonton, Canada
(BioWare)
Who We AreWhat Certification Means at EA • Ensuring Release Readiness – become a key contributor to the drive for quality and
predictability, through effective audit, knowledge sharing and strong partnerships between Microsoft and Electronic Arts.
Any game whose quality is certified by EA Certification…
...is ready for release to Microsoft & EA’s Customers
• Xbox 360 Test Group• Submissions Group
• Project Management• Technical Operations
Software Quality
• Pre-Certification (Compliance)• Pre-Final Audit (PFA)• Submission Reviews
• Age RatingsOnline Performance +
CompatibilityCross-Continent Online Checks
Post Launch Support
Additional Services
Who We Are What EA Certification Offers• High Level Audit Function: Testing encompasses Xbox 360 Build Verification Tests, Technical Certification Requirements, Functional Test Cases and EA Standards, working towards alignment with Microsoft
• Focused Coverage: Test ALL EA titles. Aim for 1st Time Pass/Release Readiness/No Repeat Fails
• Balancing Turnaround Time With Coverage: More Streamlined Service Level Agreement. Expert testers on all Product types: Disc Game, Digital Download Preview (DPP), Xbox LIVE Arcade, Title Updates, Marketing Beta, Tech Beta, Games on Demand (with code changes), DLC.cab
• Separate from Studio Teams: Internal and EA Partners (EAP) studio teams are our customers, Worldwide QA are our partners
• Team of experts united in meeting Microsoft and consumer expectations
• By actively engaging our business partners early in development and through continued support and guidance, we ensure software and game quality and customer satisfaction
• An essential partner to our customers, the EA Certification team builds trust through timely interactions, adaptability and transparent relationships
Who We AreOur Customers• EA Certification interact with all EA Studios developing for Xbox 360, including:
• As well as certifying games developed by third-party developers via the EA Partners program,
• Harmonix (Rock Band® 3)• Paramount Digital Entertainment (Rango™)• Spicy Horse (ALICE: MADNESS RETURNS™)• Crytek (Crysis®2)• Epic Games/People Can Fly (BULLETSTORM™)• GRASSHOPPER MANUFACTURE INC.(SHADOWS OF THE DAMNED™)
Rock Band ®3: © 2010 Harmonix Music Systems, Inc. All rights reserved. Covered by the following U.S. Patents: 6,429,863, 6,482,087, 7,164,076, 7,459,624, 7,625,284 and patent applications pending. Harmonix, Rock Band, Rock Band 2, Rock Band 3, Rock Band Network and all related titles and logos are trademarks of Harmonix Music Systems, Inc., an MTV Networks company. Rock Band, Rock Band 2, Rock Band 3 and Rock Band Network developed by Harmonix Music Systems, Inc. MTV, MTVGames and all related titles and logos are trademarks of MTV Networks, a division of Viacom International Inc. EA and the EA logo are trademarks of Electronic Arts Inc. All rights reserved. All other trademarks are properties of their respective owners.Rango™ : © & ™ 2011 Paramount Pictures. All rights reserved. EA and EA logo are trademarks of Electronic Arts Inc. All other trademarks are the property of their respective owners.ALICE: MADNESS RETURNS™ : © 2011 Electronic Arts Inc. EA, the EA logo and Alice: Madness Returns are trademarks of Electronic Arts Inc. All other trademarks are the property of their respective owners.Crysis®2: © 2011 Crytek GmbH. All Rights Reserved. Crysis, Crytek and CryENGINE are trademarks of Crytek GmbH. EA and the EA logo are trademarks of Electronic Arts Inc. All other trademarks are the property of their respective owners.BULLETSTORM™: © 2011, People Can Fly Sp. z o.o. All rights reserved. People Can Fly, the People Can Fly logo, Bulletstorm, and the Bulletstorm logo are trademarks or registered trademarks of People Can Fly in the United States of America and elsewhere. Epic, Epic Games, Unreal, the Powered by Unreal logo and the Epic games logo are trademarks or registered trademarks of Epic Games Inc. in the United States of America and elsewhere. EA and the EA logo are trademarks of Electronic Arts Inc. All other trademarks are the property of their respective owners.SHADOWS OF THE DAMNED: © 2011 GRASSHOPPER MANUFACTURE INC. Shadows of the Damned is a trademark of GRASSHOPPER MANUFACTURE INC. EA and the EA logo are trademarks of Electronic Arts Inc. All other trademarks are the property of their respective owners.
Who We AreEA Certification Road To Release (generic timeline)
Proj
ect
Gre
en L
it
+8 Days-35 Days
EA Certification Final Review
(8 calendar days)- BVTs/TCRs/FTCs/
EA Standards- CertNet/ProdNet Server
Prep #2
- Pre-Certification (TCRs)
- Network Performance & Compatibility
-Cross-Continent Playtests with WW
QA
- Concept Submission Form
- XeMID/Title IDs Obtained
- PartnerNet Config
Project Manager/Submissions Stakeholder Communication & Planning• Confirm Scope, Cost Estimate, Pre-Plan• Testing Resources• Understand Game Team Needs via Face to Face/Calls• Milestone Dates Communicated• Risk Prediction
Microsoft Optional Pre-Certification
- Pre-Final Audit (TCRs/FTCs)
- Submission Reviews
- Packaging Reviews- CertNet/ProdNet
Server Prep #1
Age Rating Submissions
-15 Days 0 Days Final
Final Submission
to Microsoft
Certification
Microsoft Optional
Final Certification
Green = Reviews at MicrosoftBlack = Reviews at EA Certification
RTM
EA Certification ServicesTest
Pre-Certification ReviewCompliance Testing
TCR SweepNetwork Performance
Pre-Final AuditCompliance TestingFunctional Testing
Submission Reviews
Printed Material Reviews/In-game Manuals
Final Certification Review Compliance TestingFunctional TestingFull TCR/FTC Pass
EA Standards
EA Certification ServicesSubmissions
Technical ChecksAsset Verification
Worldwide Age RatingsSubmissions to Age Rating Bodies
Asset/Paperwork/Code Verifications
Microsoft POC via Release Manager
Co-Ordinate submission to Microsoft
Reduces issues on
submission assets and
details
Ensures that all
submissions are done
consistently
Highlights areas of
concern prior to submission
Consistent communicatio
n with MSNo Repeat
Fails
All Submissions to Microsoft are consolidated to minimal POCs
EA Certification ServicesLate Shift/Technical Operations
Late Shift
• Shift that overlaps with the regular Xbox 360 Test Hub: 2pm-11pm (CET) / 5am-2pm (Pacific)•Regression support•Out of hours point of contact for our North American studios
Technical Operations
•Develop Certification processes for EA wide adoption• Improve internal systems/tools, propose new technical solutions• Provide key Xbox 360 knowledge company wide•Align with Studios/QA, find out what’s important to them, provide continuous process improvements• Identify topical/top Hold, Rejection and Failure areas in Process and Code, direct internal process improvements
EA Certification StructureXbox 360 Test Hub Structure
Xbox 360 Platform Supervisor
Xbox 360 Project Lead
Xbox 360 Test Specialist
(Compliance)
Xbox 360 Senior Tester
Xbox 360 Project Lead
Xbox 360 Test Specialist (Online)
Xbox 360 Senior Tester
Xbox 360 Project Lead
Xbox 360 Test Specialist
(Functional)
Xbox 360 Submissions
Specialist
The Xbox 360 Test Hub is made up from the following positions:
• Platform Supervisor• Submissions Specialist• Project Lead• Test Specialist• Senior Tester
This team is complimented by a dynamic pool of testers which expands or decreases based on work demand
All members are focused on and highly knowledgeable in the certification of Xbox 360 titles
EA Certification Structure Xbox 360 Test Hub Roles
•High Knowledge of Xbox 360 guidelines•Leads the EA Certification Xbox 360 team
Xbox 360 Hub Supervisor
• Responsible for scheduling/managing testing on specific titles• Works closely with QA/Studio to ensure any concerns from our own testing are communicated and discussed (1x Lunch/1x EOD Meeting)
• Project accountability on a title by title case –Testing, Planning
Xbox 360 Project Lead
•Dedicated to specific Microsoft areas; Compliance, Online, Functional, Submissions
•Use of specialised Microsoft tools (APImon, XBWatson etc..)
Xbox 360 Specialist Compliance, Online,
Functional, Submissions
•Test all full products, Title Updates, DLC, Demos and more•Works closely with Test Specialists•Directs and mentors/oversees results from the Test Pool
Xbox 360 Senior Tester
All roles within the Xbox 360 Test Hub have different areas of expertise and accountability
Working together for the common goal of improving the quality of EA Xbox 360 titles and striving for passing at Microsoft 1st time, every time
Knowledge SharingCertification Excellence
Strong Partnerships Release Readiness
• Certification is a secure central hub all titles must pass before Microsoft Certification; providing quality and predictability for frontline EA products
• Established Compliance, Functional, Online and Submission teams with EA & Microsoft wide knowledge of process, best practice and guidelines
• Direct relationship with Microsoft Release Management; monitoring and providing real-time status updates across EU and NA time-zones• Dedicated POCs between EA Cert – EA Studios –
MS• Close relationship with Worldwide QA
• Our review is the final checkpoint and approval step within EA before release to Microsoft• Consolidate submissions process to minimise risks• Assessing key attributes of a release against quality
thresholds and standards, ensuring the highest quality of an EA title/brand
• Visibility of Microsoft test activity, allows Cert to analyse and report on common failure trends, helping provide company wide-knowledge sharing
• Certification Support Service, answering game queries within 24 hours
• Knowledge base for Xbox 360 requirements
EA Certification StructureBenefits
Ensure Release Readiness, Target 1st Time Pass, Plan Customer Satisfaction
Sharing KnowledgeResponding to a Rejection or CFR
Step 1.EA Cert receives the Fail Report
from MS
Step 2.Communicate CFR to QA and
Dev teams
Step 3.Enter CFR into
our online Xbox 360 fail tracker
database
Step 4.Investigate Root Cause of failure
Step 5. Create “Action Plan” within 24
hrs
Step 6.Update Test Cases for worldwide
QA/Cert adoption
This quick response action plan is put into place with immediate effect as soon as EA Certification is alerted to a failure at Microsoft
EA Certification will have a fully realised Action Plan in place within 24 hours of first receiving the failure
This plan leads to our goal of “No Repeat Fails”
Sharing KnowledgeCreating Worldwide StandardsEA Standards – an EA maintained database that
contains:• Microsoft, Sony, Nintendo, PC/Mac, Facebook
standards• Compliance Docs, Tools & Training Material• First Party News• First Party Failure Databases• TCR/Guideline Downloads• Xbox 360 Wikipedia
QA/Cert Forums/WW QA Committee - a shared forum for all platform related queries• Submit Compliance, Functional or Online queries
related to Microsoft Standards• 24 Hour Turnaround on responses – Guidance of
TCR implementation in titles • Review of in-game guideline messages/terms/in-
game manual
Aiming to AlignLearning from Microsoft PFA Service – Offered to our dev
teams as a companion to the MS Optional Final, during which EA Certification run a full TCR/FTC pass on a near Final build this
benefits us by ensuring we receive better quality Final builds,
which results in better submissions to MS
PreCert/Final – EA Certification run the same reviews as MS,
sometimes in tandem, sometimes with a degree of overlap
Issue Reporting – EA Certification use an issue
reporting template based on the MS format. This clear and concise format has resulted in simple to
read and easy to understand issues meaning dev teams can fix
issues quicker
Test Cases – All test cases are kept up to date in real time with all MS updates and fail examples
Submissions Checks – EA Certification run a formalised set of Submissions Checks based on
MS requirements
BVT Process – A series of checks designed to determine suitability for submission to MS. Detailed
on next slide
EA Certification strive to align with MS Certification in as many ways as possible, including the following:
Aiming to AlignBVT Process • To improve our Certification process and to align with MS testing practices we implemented a Build Verification Test (BVT) system to avoid any potential rejections
• This system is based directly on the 17 BVT tests run at MS Cert
•While we will not hold of testing if BVT items fail, a title will not be submitted to MS until all applicable BVT tests are in a “PASS” status
Title Submitted to
EA CertBVT Checks will
be run
BVT Pass/Fail notification will be sent to game
team via BVT Template email
Game team will respond to
notification with required
items/information
Aiming to AlignBVT Process• Example of BVT notification email:
• The implementation of this process has seen a decline in rejections at Microsoft Mastering Lab
TipsGeneral Advice on CertificationUnderstand Game Team Needs - Preparation
• Face to Face, Direct Calls to Pre-Plan your Test Plan, understand resource/hour coverage, and align your services to studio expectations
• Explain your own Internal Review processes and communicate Microsoft dependencies, pre-empt any risks• Partner with QA (if a separate entity) to build relations and provide certification consultation
Communication, Communication, Communication• Communicate with Studio and associated QA Teams (daily reports, daily/twice daily check-ins, Office/Windows IM, Live
Meeting)• Ensure Bugs are clearly written with steps to reproduce, link to TCR/FTC (if applicable), report issues as soon as they
are found• Create detailed Test Plans with clear test team assignments, consider Matrix’s for
Achievements/Options/Quadrant/Features testing
Submission, Submission, Submission• Verify all assets in your submission to avoid On Holds & Rejections, ensure no grey areas exist between testing
functions• Run Build Verification Tests (BVT’s) all the time, every time, ensure LSP is configured!• Have a dedicated POC in your organisation responsible for submission/communication with your Release Manager
Analyse and Share Failure/Rejection information company-wide • Learn from previous mistakes to avoid repeat occurrence – timely updates to processes and test cases to ensure No
Repeat Fails!• Know your organisation’s Top 10 Mastering, Compliance and Functional Failures for all submission types
and…
TipsKnow your Resources and Friends at Microsoft!
• 360cert for Certification Technical Support to help improve your processes and remove ambiguity
• ATG for general help on Xbox 360
• Your Release Manager for • General Enquires• Troubleshooting• TCR Exceptions• Submission Management• LSP Configuration• Publisher & Cert Liaison• RTM Management• Platinum Program Updates
• Game Developer Network Portal (GDNP) - Resources• Process Documentation and White Papers on GDNP – dissect, understand and filter relevant information to
your immediate teams and company-wide
Thank You from EA Certification!
Q&A
© 2011 Electronic Arts Inc. Trademarks belong to their respective owners. All rights reserved.
www.microsoft.com
© 2011 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.