web application performance using six sigma

8/14/2019 Web Application Performance Using Six Sigma

1/48


2/48

2

We will talk about

What is Quality?

Types of Performance Measurement

How to Measure Web App Performance? Traps we fall into and how Six Sigma

showed us the right path

Overview of Six Sigma

Using Six sigma to Improve Performance

5 Steps to Improve Web App Performance Ensuring problems do not come back

Questions


3/48

3

Whats the biggest problem today

Todays Market demand best of everything

Quality

Cost

Schedule

Reliable, Responsive and Secure service 24X7is the demand of the global customer

Schedule


4/48

4

Defects are inevitable

Fact: No software can be guaranteed100% defect-free

Action: No action

Result: We make it horribly true

Ask: Why the defect happened?

Do: Analyze data and improve theprocess to prevent it

And Sustain it

Six Sigma


5/48

5

What is Quality?

Meets expectations Serves the Purpose / Needs

Intuitive / Usability Reliable

Responsive / Performance Security / Privacy

High Quality/Low Defect

Getting it right the first time, every time

Think Global


6/48

Why is Performance important?

More online activities

Competitors are faster

Users have more options

Attracting and retaining users

We cannot overcome speed of light

Release it and then fix it no longer

an option we may loose Mind-Shareand we may not get second chance

6


7/48

Solving Performance Problems

Your app is having performance issues

Traditional way

Put more people

Do more performance testing

But Performance Testing tools

Outcome/Results

We often march on a journey to solve a problem withoutunderstanding the problem

If we dont know what we are improving, how would weknow if we have improved?

Structured way of solving the problem

Use Six Sigma, we can define the problem, measure theright thing, analyze root cause and then solve the right

problem the right way

7


8/48

What is Six Sigma?

Structured Problem solving methodology

Solving the right problem the right way

Focus on finding and fixing the root cause Ensuring problem does not come back

Drive Continuous Improvement

DMAIC

8


9/48

1.DEFINE

2.MEASURE

3.ANALYZE4.IMPROVE

5.CONTROL

(SUSTAIN)

Identify major userscenarios & Perf

Goals

Use right set of toolsto measure & baseline

current Performance

Fix root cause of perfproblems in your code/architecture/deployment

Monitor to

ensure perf donot degrade withfuture changes tothe code

Measure the followingPage Load Time (1 & 2)Time to First Byte# of round tripsAvg Bytes Download

WebPagePerformance

ImprovementDMAIC cycle

Analyze data, find reasons/root-cause for slow perfConsult Perf Experts


10/48

Six Sigma - DMAIC

Define (D) : Zero in on specific problem with defined return on effort

Measure (M) : Determine current performance of process

Analyze (A) : Validate key drivers of performance (root cause of problem) Improve (I) : Improved performance and validated realized results

Control (C) : Implement controls to ensure continued performance

10

ControlDefine Measure Analyze Improve

Lock in the results(control plan)

Mistake proofingControl pointsMonitoring PlanPositive hand offof control plan

Financial impact of the

project

Project selection Project charter

Critical to Customer(CTQ) needs

High level process map

Key output variables(metrics or Ys)

Possible causes ofdefects (Xs)

Data collection andpresentation plan

Current Performance Internal/ external

benchmarking

Key causes (vital few)of defects (Xs)

Improvement strategy Prioritize solutions

Tested & measuredsolutions

Final solutions

Project Phases and Deliverables


11/48

11

The Six Sigma Project - challenges

Bottom-up approach

Difficult to get buy-in

Myth: Six Sigma cannot be used for software We dont need a Six Sigma project to

understand this problem and to fix it

Its common sense we can fix it

Do it if you have time on your personal time

Measuring the right things Avoiding temptation of jumping to solution


12/48

Six Sigma Define Phase

Project Selection The right project (Strategic, ROI, etc.)

Project Charter

Why are we doing this? Business Case

Goal, Scope, Time-line

Exec Sponsors, Team & Resources

Web App XYZ (which drives 40% of revenue to the company) haveperformance problems (based on survey). This is impacting significant

% of users and we are seeing decline in users to the site. Customer Focus

Voice of Customer (VOC) what do customer need

Critical to quality (CTQ)

Customers have repeatedly complained about the performance of theXYZ app. They are not able to complete transactions in timely manner.

Current Situation

Process Map

Current Performance

12


13/48

13

The Define Phase: Problem Statement

Testers did not do performance testing

Performance bugs were not fixed

Performance goals/expectation not clear

Performance of any product/featureshould managed in a better way


14/48

14

The Define Phase: Goal

Ship Zero Performance Bugs

Find twice # of performance bug

Performance goals should be clear

Improve tests ability to find 95% ofthe performance bugs before Beta1


15/48

Define Perf User Scenarios

Understand the User (Voice of the Customer) First time visitor / Guest user / authenticated user

Returning users with cache/no cache

User on the same site/session User from other MS sites/domains (w/ passport)

User Demographics (Geo, Home/Office, machine config,consumer/info worker/social, connection speed)

Typical user transactions, Back/Forward/Refresh usage

Perf goal

Regional competitor performance

What is acceptable performance

Do not use it should be fast, try JP broadband usersshould be able to get the page in 4 seconds (75th percentile), when

they visit for the first time (PLT1), 2 seconds for PLT2

15


16/48

16

The Define Phase: Voice of Customer

Performance at-least as good as last release

Comparable to the competitors

UI Responsiveness

Notification in case of delays

I f it appears slow it is slow , irrespective ofwhat the data says


17/48

17

The Define Phase: Critical To Quality (CTQ)

Examples:

For common scenarios, Improve performance by

atleast 10% (compared to last release)

Any action taking more than 3 seconds should havea progress bar/notification

Similar/predictable performance in any connectionmode (speed, latency)

Handle failures/major issues gracefully


18/48

InternationalInternetRoutes

NotesMinimum bandwidth to be seen on this map is 14 GigsDoes not report bandwidth within a country


19/48

Source data for timings is 75th percentile for country in question from: http://msncore/performance/netsmart/Netstats.asp

RT = Round TripsRed line is 34 RT

Green line is 21 RTDotted Green is 10

RT

Blue line is 3 RT

Latency and the impact to page loadtimes based on number of round trips

19Microsoft Confidential
http://msncore/performance/netsmart/Netstats.asphttp://msncore/performance/netsmart/Netstats.asp


20/48


Network Round trip delays


21/48

Performance/Load/Stress Testing

Performance testing User-Scenarios testing (typical case & best case) Establish Baseline & perform trend analysis Detect performance issues Tools: WebRunner, WANSim, HttpWatch, etc.

Load testing (volume/longevity/endurance)

Expected MAX # of concurrent users Volume of data Very Long active sessions

Stress testing (negative testing) What happens with the load exceeds significantly OR

system goes thru resource constraints/failures Does the system gracefully recovers from failure?

(e.g. Tiwan Earthquake in Dec 2006)


22/48

Six Sigma - Measure

Measurement Process Data Collection Plan Testing, Monitoring, Measurement system analysis, sampling

Clear, concise definition of variables, process involved

Identify Key measures/drivers of performance Y = F(X1,X2,X3,X4,)

Ishikawa (Fishbone) diagram Cause & Effect diagram

Internal / External Benchmarking

Baseline Current performance & Impact Identify and measure current performance and its impact on

customer, collect more data if requiredOn 300kbps connection & 300ms Round-Trip-Time, it takes 6 seconds to load the

page for the PLT1 case, 2.5 seconds for PLT2. 20% of our user abandon the pagebefore it loads.

Web Page X have 2 HTML files, 3 .js files, 3.css, 5 images, the web App opens 2parallel tcp ports to download them.

22


23/48

Types of Performance Measurements

Client UI Response Time Server Response Time

Load/Stress

Bytes over wire/Throughput Availability

Latency (anywhere in the world)

Browser Processing & Rendering time

User Machine - Resource utilization

Perceived Performance

If the user feels the product is slow, your product i s slow no matter what our data says

PLT1: The user visiting the site with NO CACHE PLT2: Returning user with CACHE

23


24/48

What is Performance

PLT1: The user visiting the site withNO CACHE

PLT2: Returning user with CACHE



25/48

Measure

Business Measurements # of unique users

# of page views

Click-Thru-Rate (CTR)

Revenue

Market Share Net Promoter

Other Measurements

% of people on PLT1 / PLT2

Errors

Abandon rate (Incomplete/Closed/Click-Away)

25


26/48

Measure: Key variables for Performance

Factors attributing to Web Page Performance # of Files, Static/Dynamic content

Page Load Time

Bytes download DNS Lookup time

Peak hours / Load & Stress

User spread / Global? Data Centers / CDN / Redirects

Multiple versions of the app

Web Page Architecture (parallel/sequential download)

Compression

Expiry dates

Keep-Alive

PLT1 / PLT2 (Caching?)

26


27/48

Performance Cause-Effect Diagram

27

End-UserPage Load Time

Server/ Infrastructure

Application / Web Page End-User

Latency

Bandwidth

Static/Dynamic content

Total Bytes/ Compression

GeoLocation DataCenter/CDN

# of Servers

Machine config# of Files

JS/CSS processing

Load/Stress Peak Hours

DNS Setup

Caching/Keep-Alive

# of TCP ports

Page Architecture

App versioning

Best Practices @ http://MSNPerf
http://msnperf/http://msnperf/


28/48

Six Sigma - Analyze

Find Root Cause of the problems Analyze the data from Measure phase

Identify vital few variables (x)

Perform correlation and regression analysis Data stratification

Use 5 Why techniques

Hypothesis testing Sources of variation

Use Cause-Effect diagram

Plot data on graphs (trend, releases)

Special Cause / Common Cause# of files on the site is 10, some of these files can be combined, 2 files are notcompressed. Majority of the users who abandon the site are from UK (Latency)and Dial-up users from US (slow connection). The problem started to happen fromMarch 01 (when release 2.2 went live)

28


29/48

AnalysisFrom the data it looks like 17% of the users are in PLT1 and 83% in PLT2There are several users in a middle category (16K to 21K)

PLT1/PLT2 User Distribution

29


30/48

Impact of slow performance on user

As it takes more time to display page users STOP the pageTypically after 5 seconds of wait, 15% user stops the page from loading and the % grows with thetime 30Microsoft Confidential


31/48

User count & Performance across states

31


32/48

Bytes & Page Load Time distribution

32


33/48


34/48

Six Sigma - Improve

Improvement strategy & Plan

Improvement solution selection

Generate ideas Involve diverse team Identify and rank solution alternatives

Pilot solutions and select final solution

FMEA Design of experiments

Test and implement final solution

Communication plan

ToBe Process Map

Track improvement, monitor trend Share success/failure stories Best Practices

34


35/48

Six Sigma Control (Sustain improvements)

Process Monitoring/Control Plan Mistake Proofing (Poka-Yoke)

Control Chart

Response Plan

Document standard process/procedure

Train resources Share Learning

Project closure Positive Handoff

Measure Benefits (Financial / Customer Sat)

Celebrate35


36/48

36

Sample Control Plan

We will get user feedback on Beta1 build and evaluate if weneed any more improvements

We will continue to have meeting (every other week) to

discuss any perf test related issues

Person ABC will do the measurements and monitor the appperformance until Beta1 and analyze/report user feedback

A monthly status mail will be sent to (10th of each month) theProject team, Product Manager and VP, reporting the statsrelated to the project (Perf for various category of users,

improvement over time, survey/feedback from users, etc.)

If we find that we are deviating from our goal, we will call ameeting (within 2 days) and analyze problem and develop

solution


37/48

37

Lessons Learned / Suggestions

Lessons Learned:

Six Sigma Process Measurements helped us uncover

problems in a structured way and come up withsolutions to eliminate them or minimize its impact.

Implementing improvements requires involvement

from everyone at all levels and all disciplines

Suggestions:

By getting all disciplines (dev/pm/test) involved wecan focus on preventing the problems rather thanrelying on finding and fixing them.

Integrate new practices into the development cycleonce the improvements have been validated


38/48

Results

Note: Specific % & # are purposely removed from this presentation

Product benefits:

Page size drastically reduced Site performance improved

User satisfaction improved

Increased click-thru-rate

Overall impact:

Increased focus on Performance More investment in performance

(expanding the performance group)

38


39/48

Visual Round Trip Analyzer

39

Vi l R d T i A l


40/48

40

Visual Round Trip Analyzer


41/48

Visual Round-Trip Analyzer (VRTA)

http://msncore/performance/netsmart/VRTA_animation/sample1/main.html

File Transfer Duration

Client Port Browser toServer Connection

Time in Seconds

BandwidthUtilization

Servers

Color codedBy File Type
http://msncore/performance/netsmart/VRTA_animation/sample1/main.htmlhttp://msncore/performance/netsmart/VRTA_animation/sample1/main.html


42/48

Bad Good

42

Use more parallel TCP ports


43/48

Unblock Java Script

43

Standard JS downloads in Serial andcreates bandwidth bottlenecks

Use a binding methodology to getaround this issue

bl k


44/48

Unblock JS

44

Solution IThis method has been

successfully used by the

Windows Live Hotmail Team==================function AsyncLoad(){var l = arguments.length;for (var i=0;i


45/48

Expiration dates

45

Max-Age: Supersedes Expiration

RelativeTime

URI

ContentLen Status Code

0.00http://groups.msn.com/people/ 27659 200 -- OK

0.70http://c.msn.com/c.gif 42 200 -- OK

4.53http://groups.msn.com/global/css.htm 0 304 -- Not Modified

4.88http://groups.msn.com/spacer.gif 0 304 -- Not Modified

4.89http://groups.msn.com/home_icons_chat_48x40.gif 0 304 -- Not Modified

4.90http://www.match.com/msnprofile/profile.aspx 2481 200 -- OK5.22http://groups.msn.com/home_icons_IM_48x40.gif 0 304 -- Not Modified

5.55http://groups.msn.com/msnmess_themes_65x60.gif 0 304 -- Not Modified

5.57http://groups.msn.com/home_icons_heart_42x39.gif 0 304 -- Not Modified

5.81http://www.match.com/lib.msnprofiles.master.style.css 383 200 -- OK

5.81http://www.match.com/lib.msnprofiles.style1.css 214 200 -- OK

5.89http://groups.msn.com/home_icons_MD_48x40.gif 0 304 -- Not Modified

5.91http://groups.msn.com/home_icons_groups_48x40.gif 0 304 -- Not Modified

6.20http://www.match.com/libraries/lib.template.globaljs.js 0 304 -- Not Modified6.22http://groups.msn.com/msn_phone3n_48x40.gif 0 304 -- Not Modified

6.24http://groups.msn.com/whitepages/msnwhitepages.htm 0 304 -- Not Modified

6.57http://groups.msn.com/home_icons_msnbutterfly_48x40_c.gif 0 304 -- Not Modified

6.61http://xml.eshop.msn.com/xmlbuddy/eShopOffer.aspx 1227 200 -- OK

6.86https://reader010.{domain}/reader010/html5/0625/5b30eaf68f5ad/5b30eb0ce01a1.jpg 8708 200 -- OK

7.35http://groups.msn.com/match_com_header_blue_matte.gif 0 304 -- Not Modified

7.35http://groups.msn.com/spacer.gif 0 304 -- Not Modified

7.36http://view.atdmt.com/MSN/iview/msnnkhac001300x250xWBCK4000109msn/direct;wi.300;hi.250/01 320 200 -- OK

7.43http://rad.msn.com/ADSAdClient31.dll 489 200 -- OK

7.69http://groups.msn.com/1195B_goBtn.gif 0 304 -- Not Modified

7.71http://groups.msn.com/gfol_180x150_survey_express_jan03_2.gif 0 304 -- Not Modified

8.91http://att.atdmt.com/b/MSMSNMATCVON/Harmonics_WorstNightmare_2499_300x250.gif 14852 200 -- OK

10.58 56375

304 = Bad!

U C i
http://groups.msn.com/people/http://c.msn.com/c.gifhttp://groups.msn.com/Editorial/en-us/Content/global/css.htmhttp://groups.msn.com/Editorial/en-us/images/spacer.gifhttp://groups.msn.com/Editorial/en-us/images/home_icons_chat_48x40.gifhttp://www.match.com/msnprofile/profile.aspxhttp://groups.msn.com/Editorial/en-us/images/home_icons_IM_48x40.gifhttp://groups.msn.com/editorial/en-us/images/msnmess_themes_65x60.gifhttp://groups.msn.com/Editorial/en-us/images/home_icons_heart_42x39.gifhttp://www.match.com/stylesheets/msnprofiles/lib.msnprofiles.master.style.csshttp://www.match.com/stylesheets/msnprofiles/lib.msnprofiles.style1.csshttp://groups.msn.com/Editorial/en-us/images/home_icons_MD_48x40.gifhttp://groups.msn.com/Editorial/en-us/images/home_icons_groups_48x40.gifhttp://www.match.com/libraries/lib.template.globaljs.jshttp://groups.msn.com/Editorial/en-us/images/msn_phone3n_48x40.gifhttp://groups.msn.com/Editorial/en-us/Content/whitepages/msnwhitepages.htmhttp://groups.msn.com/Editorial/en-us/images/home_icons_msnbutterfly_48x40_c.gifhttp://xml.eshop.msn.com/xmlbuddy/eShopOffer.aspxhttp://images.match.com/match/matchscene/articles/spotlight1617.jpghttp://groups.msn.com/Editorial/en-us/images/match_com_header_blue_matte.gifhttp://groups.msn.com/editorial/en-us/images/spacer.gifhttp://view.atdmt.com/MSN/iview/msnnkhac001300x250xWBCK4000109msn/direct;wi.300;hi.250/01http://rad.msn.com/ADSAdClient31.dllhttp://groups.msn.com/editorial/en-us/images/1195B_goBtn.gifhttp://groups.msn.com/editorial/en-us/images/gfol_180x150_survey_express_jan03_2.gifhttp://att.atdmt.com/b/MSMSNMATCVON/Harmonics_WorstNightmare_2499_300x250.gifhttp://att.atdmt.com/b/MSMSNMATCVON/Harmonics_WorstNightmare_2499_300x250.gifhttp://groups.msn.com/editorial/en-us/images/gfol_180x150_survey_express_jan03_2.gifhttp://groups.msn.com/editorial/en-us/images/1195B_goBtn.gifhttp://rad.msn.com/ADSAdClient31.dllhttp://view.atdmt.com/MSN/iview/msnnkhac001300x250xWBCK4000109msn/direct;wi.300;hi.250/01http://groups.msn.com/editorial/en-us/images/spacer.gifhttp://groups.msn.com/Editorial/en-us/images/match_com_header_blue_matte.gifhttp://images.match.com/match/matchscene/articles/spotlight1617.jpghttp://xml.eshop.msn.com/xmlbuddy/eShopOffer.aspxhttp://groups.msn.com/Editorial/en-us/images/home_icons_msnbutterfly_48x40_c.gifhttp://groups.msn.com/Editorial/en-us/Content/whitepages/msnwhitepages.htmhttp://groups.msn.com/Editorial/en-us/images/msn_phone3n_48x40.gifhttp://www.match.com/libraries/lib.template.globaljs.jshttp://groups.msn.com/Editorial/en-us/images/home_icons_groups_48x40.gifhttp://groups.msn.com/Editorial/en-us/images/home_icons_MD_48x40.gifhttp://www.match.com/stylesheets/msnprofiles/lib.msnprofiles.style1.csshttp://www.match.com/stylesheets/msnprofiles/lib.msnprofiles.master.style.csshttp://groups.msn.com/Editorial/en-us/images/home_icons_heart_42x39.gifhttp://groups.msn.com/editorial/en-us/images/msnmess_themes_65x60.gifhttp://groups.msn.com/Editorial/en-us/images/home_icons_IM_48x40.gifhttp://www.match.com/msnprofile/profile.aspxhttp://groups.msn.com/Editorial/en-us/images/home_icons_chat_48x40.gifhttp://groups.msn.com/Editorial/en-us/images/spacer.gifhttp://groups.msn.com/Editorial/en-us/Content/global/css.htmhttp://c.msn.com/c.gifhttp://groups.msn.com/people/


46/48

Use Compression

46

Types of Compression: Static, Dynamic

Levels 0 through 10

26KB/5.5 = 5KB; 20KB BW savings

K Ali TCP t


47/48

Keep-Alive TCP ports

47


48/48

Questions?

Mukesh [email protected]
mailto:[email protected]:[email protected]

web application performance using six sigma

Documents