Download - Test Design For Everyone
Test Design for Everyone
Alan PageMicrosoft
Here, Test This!
“Test Design” is what Testers Do
What is test design?
Simple:
Come up with some tests
Better(?):
Come up with a well thought out and broad set of tests for the given context (based on
application, schedule, etc.)
Another Take:
Create an optimal set of tests that finds the maximum amount of customer facing issues possible with a minimum of over-testing and
under-testing
“Over Testing” and
“Under Testing”
Application
Application Testing Effort
Application
(potential)
Over Testing
Under Testing
(stuff you forgot to
test
or didn’t know to test)
Testing Effort
bool Validate(string sideName) try { sideVal = Convert.ToUInt16(inputVal); if (sideVal < 1) { MessageBox.Show(sidename + “must be greater than 0”, ...); return false; } } catch (OverflowException) { MessageBox.Show(sideName + "must be less than 65536", ...); return false; } catch (FormatException) { MessageBox.Show(sideName + "must be an integer value, ...); return false; }
}
bool Validate(string sideName) try { sideVal = Convert.ToUInt16(inputVal); if (sideVal < 1) { MessageBox.Show(sidename + “must be greater than 0”, ...); return false; } } catch (OverflowException) { MessageBox.Show(sideName + "must be less than 65536", ...); return false; } catch (FormatException) { MessageBox.Show(sideName + "must be an integer value, ...); return false; }
}
In English
Three different types of errors:
1. Zero or less2. Bigger than 655363. Non-numbers
Any other input is testing the same thing
Application
Reduce
Over Testing
(don’t test the same
thing multiple times)Reduce Under Testing
(miss less)
Testing Effort
How do you become a better Test Designer?
You need ideas!
Ideas you have used beforeNew ideas to tryIdeas suited for this productIdeas suited to your release schedule
…and you need to know what to look for
Why Is it so Hard?
You Don’t Know What You Don’t Know
Read Philip Armour’s The Five Orders of Ignorance
All you can use is what you know
Hammers are useful…
…but not everything is a nail
What you’ve already heard about test design
http://www.satisfice.com/glossary.shtml
The “what” has been covered
Let’s focus on the “how”
You need a Toolbox
Bigger is Better
What kinds of tests do you write
New Information
False Negative
False Positive
No Information
Useful Tests?
Useful Test Provide High Levels of “New Information”
“Usefulness” depends on context
Some general guidelines:
The first time you run a test it (almost) always provides new information
If you run the same test the same way again, it may or may not provide new information
Some general guidelines (continued):
Varying your tests gives them a higher chance of providing new information (randomization, data driven tests, “smart monkeys”, model-based testing, etc. all help here)
Examples / Practice
Test This:Add or Remove Contacts
Mind Mapping
What about adding a contact with a blank name?
12
3 4
A/V and Web Conferencing Workload
Edge Servers
Externalfirewall
Internalfirewall
HTTPS:443
Reverse proxy
HTTPS:4443
SIP/MTLS:5061
SIP
/TL
S:5
06
1
A/V Edge - STUN/TCP:443, STUN/UDP:3478
A/V Edge – SRTP:443,3478,50,000-59,999
SR
TP
/UD
P:4
91
52
-65
53
5
Range of ports is configurable.
PS
OM
/TL
S:8
05
7
HT
TP
S:4
43
HTTPS:443 is used to download conferencing
content.
Traffic goes directly to Web Conferencing Service WITHOUT going through the pool’s hardware load balancer
Two inbound and two outbound unidirectional streams. Media codec varies on workload:- RTAudio for audio- RTVideo for video
Traffic goes directly to A/V Conferencing Service WITHOUT going through the pool’s hardware load balancer.
MRAS traffic.
SIP/MTLS:5062Web Conf Edge - PSOM/TLS:443
Access Edge - SIP/TLS:443
PSOM/MTLS:8057
STUN/TCP:443, STUN/UDP:3478
Directors
Monitoring
Server
SIP/MTLS
SIP/MTLS:5061
Port number to service traffic assignment:
5063 - A/V Conferencing Service
MSMQ
Codec varies per workload:- G.722 or Siren for audio- RTVideo for video
Enterprise
pool
Meeting content + metadata + compliance file share.
HTTPS traffic
SIP traffic: signaling
RTP/SRTP traffic: A/V Conferencing
PSOM traffic: Web Conferencing
If client connects on port 80, it gets redirected to port 443
Only traffic on ports 443, 4443 and 5061 must be load
balanced by HLB.
How to get better at Test Design
Practice, practice, practice
Practice testing – be curious
Collaborate
http://testing-challenges.org
Weekend Testing (http://weekendtesters.com)
Learn
How does your application work?
Find new testing ideas
Study critical thinking
Read…anything
Make your tests useful…and notice when they’re not useful
The Best Test Tool of All Time
Your Brain (of course)
Be A Great Test Designer
(and be a great tester too)
http://angryweasel.com/bloghttp://twitter.com/alanpage
Credits
• Opening slide pattern - some rights reserved by km6xo• “Scrap metal” – some rights reserved by
MAG (Mines Advisory Group)• Ideas - Some rights reserved by orkboi• Impossible triangle - some rights reserved by yui.kubo• Hammer – some rights reserved by Noel C. Hankamer• Nail - some rights reserved by crd!• Hammer / screw - some rights reserved by justinbaeder• Tomatoes - some rights reserved by lucianvenutian• Toolbox - some rights reserved by Austin ampersand Zak• Big Toolbox - some rights reserved by Fuschia Foot• Brain - Some rights reserved by biologycorner