testing euro changeover – too big to fail?' by raivo päts
DESCRIPTION
You are facing testing a project called “Do or die!”. The biggest testing event – Estonia’s biggest bank (Swedbank) is going over to EURO currency. You have six months to do it. Roll-back or postponement is not an option. You have 15 testing teams to coordinate and you have to test ONLY the whole IT system of a bank and Changeover to EURO process. What will you do? I was set as an IT Test Manager for the program. I’ll tell you what I did! The testing teams to be coordinated needed one “common thing” to rule them all. It turned out to be the common test environment. So was the environment driven testing plan born. Lesson learned: planning the common, align everyone – YES! No single test plan was agreed but test plans of teams were published to a commonly agreed and structured place. Teams aligned their activities against plans of others and in case of need re-planned their activities together with other teams. Lesson learned: Don’t over manage, believe in self-organization! Who’s afraid of EURO? The business is! Why not to let them ease their worries by playing it all through themselves? Over and over and over again. Lesson learned: those afraid the most, test the best. What happened, how we did it, what went well and not so well are the topics to be covered during my speech, flavored with examples, tips & tricks for the Test Manager of the next “Do or die!” project.TRANSCRIPT
© Swedbank
Testing EURO Changeover – too big to fail?
Raivo Päts
EURO IT Test Manager
© Swedbank
Swedbank home markets
SwedenPopulation 9.3mEmployees 8 401Private customers 4.1m of which Internet customers 2.9mCorporate customers 306 000 out of which Internet customers 280 000Organizations 113 000Branches 340ATMs 748Cards 3.8m
LatviaPopulation 2.2Employees 1 724Private customers 1.0m of which Internet customers 0.9mCorporate customers 65 000Branches 59ATMs 372Cards 0.9m
LithuaniaPopulation 3.3mEmployees 2 228Private customers 3.2m of which Internet customers 1.4mCorporate customers 98 000Branches 97ATMs 452Cards 1.7 m
EstoniaPopulation 1.3mEmployees 2 514Private customers 1.2m of which Internet customers 0.9mCorporate customers 106 000Branches 64ATMs 554Cards 1.1m
© Swedbank 3
About EURO IT program …
• Goal: Changeover according to Estonia's National Changeover Plan• Scope: applications/systems supporting Estonian business
– Pre EURO– Changeover
• More than 1000 activities in Changeover scenario across 15 areas• Convert ~2 million accounts within 5 minutes• No card payments downtime • All ATM-s up at 00:00 • Included “Plan B” 's
– post EURO
• Timeline: from 01.03.2010 till 01.04.2011– Main development effort starting from summer 2010– Final decision on going to EURO in the beginning of July– Ex. Slovakia had two years
• The FIRST:– First all-bank program in years– First fully combined business/IT testing
© Swedbank
It is said that we learn from failures, but ...
... we didn’t fail!
© Swedbank
The Testing Challenge …
© Swedbank 6
Challenges in planning
• Why – to still be a bank after 31.12.2010 – Do or die!
• What – The whole bank IT system!– Very interconnected system– A lot of applications– Legacy applications
• When – no later than 31.12.2010!– Challenging timeline, no postponement
• Who – IT and business together– Very large business engagement– All IT testers involved
• Where – Environments– Production like environment– External partners, other banks
• How – Different testing priorities– 15 departments = 15 sets priorities
I keep six honest serving-men(They taught me all I knew);
Their names are What and Why and WhenAnd How and Where and Who.
- Rudyard Kipling
© Swedbank
Environment driven testing plan– planning the common, aligns everyone!
© Swedbank
04.10.2010
01.01.2011
Changeover Readiness Testing
Bank Business Acceptance Testing
30.12.2010
Changeover
verification
Iteration 7Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5 Iteration 604.Oct
till
10.Oct
11.Oct
till
17.Oct
18.Oct
till
29.Oct
30.Oct
till
14.Nov
15. Nov
till
28.Nov
29. Nov
till
12.Dec
26. Dec
till
30.Dec
“Real” test“Real” test
04.01.2011
Area 1 - System TestingArea 2 - System Testing
…
Area 15 - System Testing
© Swedbank 9
Environment driven testing plan
1. Dedicated environment
2. Clear schedule – info mails– Monitoring site
1. Quick user right provisioning and installation of changes
2. Development not finished
3. System administration overload
+
+
–
Sunday
Preparing Test env. for next iteration
Monday
Installation of changes
EEK
Tuesday EUR
Wednesday
EUR
Thursday EUR
Friday EUR
Saturday EUR
Sunday
Preparing Test env. for next iteration–
+
© Swedbank
04.10.2010
01.01.2011
Changeover Readiness Testing
Bank Business Acceptance Testing
30.12.2010
Changeover
verification
Iteration 7Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5 Iteration 604.Oct
till
10.Oct
11.Oct
till
17.Oct
18.Oct
till
29.Oct
30.Oct
till
14.Nov
15. Nov
till
28.Nov
29. Nov
till
12.Dec
26. Dec
till
30.Dec
“Real” test“Real” test
04.01.2011
Area 1 - System TestingArea 2 - System Testing
…
Area 15 - System Testing
© Swedbank 11
Schedule – Good and bad
1. Testing by iterations
2. Simultaneous Changeover and Business Acceptance Testing across bank
3. Stable environment
4. No rest time between iterations
5. No possibility to run Conversion-only cycles
6. One single common environment may be not enough to meet conflicting needs of areas
++
+
––
© Swedbank
Biggest project = Biggest Test Management? Not!
© Swedbank
Not leading but enabling testing!
© Swedbank
Self organization over centralization
• Trainings to establish common framework• Test managers forums• Common Test Plan
• aligned with environment plan• Environment monitoring
• Changeover monitoring meetings
Title of presentation, Author, 2011-01-01Title of presentation, Author, 2011-01-01 14
© Swedbank
Divided responsibility
• Area teams responsible for planning and execution• No detailed involvement from IT Test Manager
• Project analysis found: – Testing should be a separate workgroup
+
–
+
© Swedbank
Business engagement – those afraid the most, test the best!
1st of January?
© Swedbank 17
Business Acceptance
• Business knew the impact to business – they use it every day!• Requirements not clear – business knows how it should work• Building confidence – business needs to be sure they will
survive!• Testing prioritization based on criticality classes of services• Changeover activities prioritized by the Management Board• From front-line and back office to heads of department
– Special testing classes
• Prioritization & scope• Responsibility in areas• Overloaded• Synchronization with IT
+–
+
–
© Swedbank
• Changeover test – measured metrics with targets• High level empowered decision body• Crisis Centre
18
Crisis & Plan B
© Swedbank
What would I do in similar projects?
© Swedbank
Same, but different
• One thing to rule them all, but let them rule a little bit as well• One testing team – more strict follow-up and control• Change management for test environment earlier• Environment management
• Earlier involvement of the “real doers” – How to implement – how to prepare changeover procedures
© Swedbank
THANK YOU!
Questions?