icse 2012: test confessions - a study of testing practices for plug-in systems
DESCRIPTION
TRANSCRIPT
Test Confessions: A Study of Testing Practices for Plug-In Systems
Michaela Greiler, Arie van Deursen and Margaret-Anne Storey
1
Testing Practices for Plug-In Systems?
2
3
loaded at runtime
extending without the need of change
assemble new products
4
5
integrating multiple plug-ins
complex compositions
different developers
6
Incompatibility
7
8
“The number one reason people give us for not upgrading to the latest version of
WordPress is fear that their plugins won’t be compatible.”
[WordPress]
9
“Thanks, but I think we have given up on Eclipse and Bugzilla integration.”
[Eclipse – Bug ID: 268207]
10
11
Research Questions 12
Plug-in Testing?
Challenges?
Practices?
Which test practices are prevalent in testing of
plug-in-based systems?
How are plug-in specific integration issues
(i.e. versioning, configurations) tested?
What are challenges experienced when testing
plug-in based systems?
LITERATURE STUDY?
Survey of existing approaches – over 200 resources
13
A Grounded Theory Study
14
Systematic procedure to discover a theory from (qualitative) data
Objective Increase understanding of
what testers and developers think and do when it comes
to testing plug-in-based systems.
15
Study
16
Interviews with 25 experienced Eclipse developers or testers representing well-known open and closed source projects.
Research Questions 17
Plug-in Testing?
Challenges? Are there compensation strategies to support the
testing of plug-ins?
Compensation?
Practices?
Triangulation
Resonance @ EclipseCon
Survey among 151 developers
18
1.
2.
TESTING PRACTICES Outcome
19
Focus on Unit Testing
20
Focus on Unit Testing
“Unit testing is where you find the most bugs”
“At least 70% of our test effort is spent on unit testing.”
“Ultimately, unit test are our best friends”
P14
P18
Focus on Unit Testing
21
Integration Tests
14 20
22
“We think that with a high test coverage through unit tests, integration tests are
not necessary.”
“QF-tests [were] too rigid when the system was evolving”
Other forms of testing are less popular
P14 P20
23
SURVEY
24
1. Finding: (Automated) unit testing is widely adopted; Integration, system, UI and acceptance testing are much less automated
PLUG-IN TESTING Outcome
25
26
Cross-plug-in integration?
Testing platform or plug-in versioning?
Cross plug-in testing is optional
“We handle problems between several plug-ins in a bug-driven way”
“We have no automated tests for cross plug-in testing, but we do manual testing.”
P19 P18
27
Version testing is minimal
“But I’m willing to bet that 99% of the people do not test that their stuff works.”
“A lot of people put version ranges […], and they say they can run with 3.3 up to version 4.0 […].”
P13
28
29
test for dependency versions
Testing combinations or versions?
55% don’t test for platform versions
63% don’t
only 4% test this thoroughly
only 10% test this thoroughly
only 3% test this thoroughly
43% don’t test integration of different products
30
Testing not needed?
31
“Simply fetch the latest and you'll end up in a mess!”
[Ian Bull on updating problems with plug-in systems]
Testing not needed?
Versioning, cross-plug-in testing not needed?
32
“Sometimes we update, but there is always the risk that it will break something and then you
have to do extensive [manual] testing.”
P12
P9
“I do not even have a chance to test [all possible combinations]. There are too many operating systems, there are too many Eclipse versions.”
BARRIERS Outcome
33
Plug-In Integration Testing Barriers
34
Findings: Barriers
• Responsibility for integration unclear
• Lack of ownership
• Insufficient plug-in testing knowledge
• Test execution too long
35
COMPENSATION Outcome
36
Self-Hosting
37
“Eating your own dog food”
Manual Testing
38
“Tests that I do are very simple manual tests, the
real tests are coming from the users, that are doing
all kind of different things with [x].“—P9
39
“Testing is done by the user-community and they are rigorous about it. We have
more than 10,000 installations per month. If there is a bug it gets reported
immediately.“ P12
Developer Involvement
“We’re a framework. If the user downloads a new version and lets his application run with it,
then this is already like a test.”
“Perhaps it is not our own product, but our product relies on this other product. So it is normal to improve [it].”
P20
P11
40
A Prerequisite
Openness
Communication Release Management Extensibility
Feedback Manual Testing Automated Testing
Requirements Alpha & Beta Tester Downstream Projects & Release Train
41
Summary: Findings
1. (Automated) unit testing is widely adopted; Integration, system, UI and acceptance testing are much less automated
2. The plug-in nature has little direct impact on test practices
3. Barriers to adopt techniques include unclear ownership, responsibilities, and test effort & execution time
4. Limited integration testing is compensated by community
42
43
Plug-in specific testing support
Provide a test modus
Rewarding community
Centralized compatibility information
More Details?
Michaela Greiler, Arie van Deursen & Margaret-Anne Storey. “Test Confessions: A Study of Testing Practices for Plug-in Systems”
Contact: [email protected] [email protected] [email protected]
44