Pros and Cons of Doing Performance Testing Along with Development Versus at the End of it

PhD. Federico Toledo@fltoledoSofía Palamarchuk@sopalamarchuk

• Introduction• Performance, testing, different approaches

• What you always have to take into account• People, tools and methodology

• Comparison: Waterfall vs. Agile• Pros and cons based on real world experiences

• Conclusion

Agenda

• Co-founder and CEO US at Abstracta (www.abstracta.us)

• PhD and author

• +10 years experience in performance testing

• +80 projects worldwide

Who we are

Agenda

• Introduction• Performance, testing, different approaches

• What you always have to take into account• People, tools and methodology

• Comparison: Waterfall vs. Agile• Pros and cons based on real world experiences

• Conclusion

What is computer

performance?

USEFUL WORK

TIME & RESOURCES

USED

VS

What is performanc

e testing about?

Simulation

Sour

ce: h

ttp://

www.

min

iatu

r-wun

derla

nd.d

e/

Measurement

Breaking pointBottlenecks

• At the end • Waterfall• Acceptance testing• Simulation of the expected load scenario

• Along with development• Agile• Continuous integration • Unit and component• Early feedback• Performance engineering

Performance Testing

Agenda

• Introduction• Performance, testing, different approaches

• What you always have to take into account• People, tools and methodology

• Comparison: Waterfall vs. Agile• Pros and cons based on real world experiences

• Conclusion

Agenda

• Introduction• Performance, testing, different approaches

• What you always have to take into account• People, tools and methodology

• Comparison: Waterfall vs. Agile• Pros and cons based on real world experiences

• Conclusion

What you always have to take into account

What are the main hard-skills a performance tester needs?

• Scripting and programming• Communication protocols• Architectures • Monitoring tools and metrics• Network and Operating Systems

administration• Business knowledge

People: hard-skills

What are the main soft-skills a performance tester needs?

• Abstraction• Communication • Analysis • Perception

People: soft-skills

As Scott Barber says:

A good performance tester should be a “mid-level” in everything.

http://www.perftestplus.com/resources/pt_for_managers_ppt.pdf

People

What you always have to take into account

Technology

• Load simulation• JMeter (jmeter.apache.org)• Taurus (gettaurus.org)• BlazeMeter (blazemeter.com)• Gatling (gatling.io)

• Measurement • Profilers and similar (JVisualVM, JProfiler,

dotTrace)• Operating system (NMON, perfmon, Netdata)• APMs (AppDynamics, NewRelic, DataDog)

Technology – Server Side

Technology

• Web• Page Speed• Yslow• monkeytest.it• APMs

• Desktop• Perfmon or NMON • Profilers Java or .NET • APM

• Mobile• Google Test Cloud / Testdroid / Perfecto• Xcode - Instruments / XCTest• APMs• Monkop

Technology – Client Side

Ebay ESPN Fantasy Instagram Rio 2016 VivaVideo

Monkop - Explorations

https://results.monkop.com/d8af6f27-7b0e-4573-b806-8303946269e3/report/report/index.html

What you always have to take into account

• Test design• Test automation• Test execution and

measurement

Processes

• Execution plan• Baseline (used as a reference)• Then, iterative and incremental approach

Processes

Server

Server

Agenda

• Introduction• Performance, testing, different approaches

• What you always have to take into account• People, tools and methodology

• Comparison: Waterfall vs. Agile • Pros and cons based on real world experiences

• Conclusion

Agenda

• Introduction• Performance, testing, different approaches

• What you always have to take into account• People, tools and methodology

• Comparison: Waterfall vs. Agile• Pros and cons based on real world experiences

• Conclusion

• Most of our customers ask for a load simulation before they go-live. Main motivations:• New version with new features• New architecture (win to web, web to mobile)• Marketing campaigns• One company acquires another (more users)

• Only some asked to intervene since the beginning• That’s how we decided to compare both

approaches

What is the most used one?

PROS1. It’s easier to plan 2.Focus in a specific context3.Environment similar to

production

CONS4. Infrastructure availability5. Costs of architectural changes

Performance Testing at the End

PROS1. Minimize risk2. Early and constant feedback3. Learning

• Monitoring tools and Analysis • Good and bad practices

CONS4. More automation effort

• Pay attention to which level you should automate

5. Fallacy: if you test the components then the entire system will work properly.

Performance Testing along with development

Agenda

• Introduction• Performance, testing, different approaches

• What you always have to take into account• People, tools and methodology

• Comparison: Waterfall vs. Agile• Pros and cons based on real world experiences

• Conclusion

Agenda

• Introduction• Performance, testing, different approaches

• What you always have to take into account• People, tools and methodology

• Comparison: Waterfall vs. Agile• Pros and cons based on real world experiences

• Conclusion

Which approach is better?

You should ask a different question

Conclusion

• Do you need to verify that your system supports a specific load? • Do you need evidence for your

customers?

Go with a LOAD SIMULATION.WATERFALL APPROACH.

Which approach is better?

• Do you want to optimize costs and reduce risks pertaining to the system’s performance? • Do you want to increase your team’s knowledge

in terms of monitoring and performance engineering? • Do you want to follow a real continuous

integration scheme?

Go with PERFORMANCE ENGINEERING along with development.

AGILE APPROACH.

Which approach is better?

•We need both approaches in different stages of our development cycle.

•We should start early by doing performance engineering and we also need to simulate load for acceptance testing.  

Which approach is better?

THANK YOU!Pros and Cons of Doing

Performance Testing Along with Development Versus at the End of

it

• Performance testing fallacieshttps://www.stickyminds.com/article/7-performance-testing-fallacies-undermining-your-test-strategy

• What’s more important, the tool or the tester?http://www.abstracta.us/2015/05/24/whats-more-important-the-tool-or-the-tester/

• Introduction to Taurus, an alternative to JMeterhttp://www.abstracta.us/2015/09/07/introduction-to-taurus-an-alternative-to-jmeter/

• Gatlinghttp://blog.abstracta.com.uy/search/label/Gatling

References

• Blog and resources (in English)• http://www.abstracta.us/knowledge-center/

• Blog (in Spanish) • http://blog.abstracta.com.uy

• eBook “A complete Introduction to Functional Test Automation” (in English)• http://www.abstracta.us/2016/05/31/ebook-complete-introdu

ction-to-functional-test-automation/

Free eBook and materials

If you like it

Certification in Performance Testing with JMeter and BlazeMeter

• Online course (one month)• www.abstracta.us/training

Cupon code: 25% OFF this month

IMPACT2016 http://bit.do/librodetesting

Silicon Valley office425 Broadway Street Redwood City, CA

HeadquartersJose Ellauri 1126 Montevideo, Uruguay

www.abstracta.us

PhD. Federico Toledo@fltoledo

Sofia Palamarchuk@sopalamarchuk