detection theory applied to finding and fixing defects
TRANSCRIPT
BT6 ConcurrentSession11/12/1511:30am
“Detection Theory Applied to Finding and Fixing Defects”
Presented by:
Ru Cindrea
Altom Consulting
Broughttoyouby:
340CorporateWay,Suite300,OrangePark,FL32073888-268-8770·904-278-0524·[email protected]·www.techwell.com
Ru Cindrea Altom Consulting
Ru Cindrea (@ru_altom) is a senior test consultant and managing partner at Altom Consulting (altom.fi). With more than ten years of experience in software testing, she is particularly interested in mobile testing and test automation with a special interest in mobile games. Ru is an instructor for the Black Box Software Testing series offered by Altom in collaboration with KFA. She is an organizer of the Test Lab, an open-space concept organized at testing conferences where participants practice testing and learn from their peers.
'HWHFWLRQ�7KHRU\�$SSOLHG�WR�)LQGLQJ�DQG�)L[LQJ�'HIHFWV
5X�&LQGUHD���$OWRP�&RQVXOWLQJ
$ERXW�PH
Ɣ IURP�5RPDQLD��QRZ�OLYLQJ�LQ�)LQODQG
Ɣ VWDUWHG�DV�D�WHVWHU�LQ�����
Ɣ SDUWQHU�DQG�VRIWZDUH�WHVWHU�DW�$OWRP�VLQFH�����
Ɣ %%67�,QVWUXFWRU��ZRUNLQJ�ZLWK�&HP�.DQHU�DQG�%HFN\�)LHGOHU�RIIHULQJ�%%67�FODVVHV�RQOLQH
Ɣ LQWR�PRELOH�WHVWLQJ��PRELOH�DXWRPDWLRQ�DQG�PRELOH�DSS�GHYHORSPHQW
#UXBDOWRP
6SHDN�(DV\KWWS���VSHDNLQJ�HDV\�FRP
7KDQNV�WR�0DDUHW�3\K¦M¦UYL��$QQH�0DULH�&KDUUHWW�DQG�)LRQD�&KDUOHV�IRU�VXSSRUW�DQG�PHQWRULQJ
7KH�VWRU\�RI�WKH�ZRUVW�EXJ�,�HYHU�IRXQG
ƌ,I�\RX�FDQ�UHSURGXFH�LW��WKH�GHYHORSHU�LV�ILUHG��,I�\RX�FDQƊW��\RXƊUH�ILUHG�ƍ
$�ORW�RI�LQWHUHVWLQJ�FRQGLWLRQV�IRU�WKH�EXJ
Ɣ ,�ZDV�D�MXQLRU�WHVWHU
Ɣ &RPSOH[�HQYLURQPHQW�ZLWK�PDQ\�GLIIHUHQW�([FKDQJH�VHUYHUV
Ɣ 0RVW�RI�WKH�FRPSRQHQWV�ZHUH�VLPXODWHG
Ɣ &RPPXQLFDWLRQ�YLD�EXJ�GDWDEDVH
Ɣ %XJ�ZDV�KDUG�WR�UHSURGXFH
Ɣ 6XEFRQWUDFWLQJ�DJUHHPHQW���EXJV�RQ�SURYLGHUƊV�WLPH�DQG�H[SHQVHV
Ɣ 'HYHORSHU�GLGQƊW�KDYH�WLPH
Ɣ &(2�GLGQƊW�NQRZ���GLGQƊW�XQGHUVWDQG�WKH�UHDO�SUREOHP
Ɣ 'HYHORSHUV�ZHUH�WDXJKW�WKDW�WHVWHUV�DUH�QRW�WR�EH�WUXVWHG
Ɣ 'HYHORSHUVƊ�WLPH�FRQVLGHUHG�PRUH�YDOXDEOH�WKDQ�WHVWHUVƊ�WLPH
Ɣ 'HYHORSHUV�QRW�HQFRXUDJHG�WR�LQYHVWLJDWH�GLIILFXOW�EXJV
,�GHFLGHG�WR�
䚉 UHSRUW�WKH�EXJ
䚉 QRW�WDON�WR�WKH�GHYHORSHU�GLUHFWO\�EXW�FRPPXQLFDWH�YLD�WKH�EXJ�GDWDEDVH
䚉 QRW�LQVLVW�WKDW�WKH�EXJ�VKRXOG�EH�IL[HG
䚉 QRW�WR�VSHQG�PRUH�WLPH�LQYHVWLJDWLQJ�LW
7KH�'HYHORSHU�GHFLGHG�WR�
䚉 QRW�WDNH�WKH�EXJ�VHULRXVO\
䚉 QRW�VSHQG�PRUH�WLPH�LQYHVWLJDWLQJ�LW
7KH�&(2�GHFLGHG�WR�
䚉 QRW�IL[�WKH�EXJ
䚉 QRW�DVN�DERXW�LWV�LPSOLFDWLRQV
䚉 QRW�WUXVW�PH
$�ORW�RI�GHFLVLRQV�ZHUH�LQYROYHG�LQ�KDQGOLQJ�WKH�EXJ
7KHVH�GHFLVLRQV�ZHUH�PDGH�XQGHU�FRQGLWLRQV�RI�XQFHUWDLQW\
7KHVH�GHFLVLRQV�ZHUH�VXEMHFW�WR�RXU�RZQ�ELDVHV�
'HWHFWLRQ�7KHRU\
Ɣ XVHG�ZKHQ�SV\FKRORJLVWV�ZDQW�WR�PHDVXUH�WKH�ZD\�ZH�PDNH�GHFLVLRQV�XQGHU�FRQGLWLRQV�RI�XQFHUWDLQW\
Ɣ GHFLVLRQ�PDNHU�LV�127�D�SDVVLYH�UHFHLYHU�RI�LQIRUPDWLRQ��EXW�PDNHV�GLIILFXOW�SHUFHSWXDO�MXGJPHQWV�
$Q�([DPSOH��'ULYLQJ�LQ�IRJJ\�FRQGLWLRQV
'HWHFWLRQ�7KHRU\���'ULYLQJ�LQ�IRJJ\�FRQGLWLRQV
Ɣ ZH�KDYH�WR�GHFLGH�KRZ�IDU�DZD\�IURP�XV�DQ�REMHFW�LV
Ɣ EDVHG�RQ�YLVXDO�VWLPXOXV�LPSDLUHG�E\�WKH�IRJ� !�FRQGLWLRQV�RI�XQFHUWDLQW\
'ULYLQJ�LQ�IRJJ\�FRQGLWLRQV 'ULYHU�GRHVQƉW�EUDNH 'ULYHU�EUDNHV
5HLQGHHU�RQ�WKH�VLGH�RI�WKH�URDG 0,66 +,7
1RWKLQJ�RQ�WKH�VLGH�RI�WKH�URDG &255(&7�5(-(&7,21 )$/6(�$/$50
'HWHFWLRQ�7KHRU\���'ULYLQJ�LQ�IRJJ\�FRQGLWLRQV
$SSO\LQJ�'HWHFWLRQ�7KHRU\�WR�)LQGLQJ�'HIHFWV
'HWHFWLRQ�7KHRU\���)LQGLQJ�'HIHFWV
Ɣ ZH�KDYH�WR�GHFLGH�LI�D�FHUWDLQ�EHKDYLRU�LV�D�EXJ�RU�QRW
Ɣ EDVHG�RQ�RXU�XQGHUVWDQGLQJ�RI�KRZ�WKH�SURGXFW�VKRXOG�ZRUN�DQG�EDVHG�RQ�WKH�VSHFLILFDWLRQV�ZH�DUH�DZDUH�RI
Ɣ EHFDXVH�ZH�PLJKW�QRW�QRWLFH�D�FHUWDLQ�EHKDYLRU�RU�PLJKW�QRW�XQGHUVWDQG�KRZ�WKH�SURGXFW�VKRXOG�ZRUN��ZH�PLJKW�WKLQN�WKH�SURGXFW�ZRUNV�DV�H[SHFWHG�ZKHQ�D�EXJ�RFFXUUHG�
)LQGLQJ�'HIHFWV 7HVWHU�GRHVQƉW�UHSRUW�D�SUREOHP 7HVWHU�UHSRUWV�GHIHFW
0LVEHKDYLRU�LQ�WKH�SURGXFW 0,66 +,7
%HKDYLRU�DFFRUGLQJ�WR�
VSHFV�H[SHFWDWLRQV
&255(&7�5(-(&7,21 )$/6(�$/$50
'HWHFWLRQ�7KHRU\���)LQGLQJ�'HIHFWV
$SSO\LQJ�'HWHFWLRQ�7KHRU\�WR�)L[LQJ�'HIHFWV
'HWHFWLRQ�7KHRU\���)L[LQJ�'HIHFWV
Ɣ ZH�KDYH�WR�GHFLGH�LI�D�FHUWDLQ�EXJ�VKRXOG�EH�IL[HG�RU�QRW
Ɣ EDVHG�RQ�RXU�XQGHUVWDQGLQJ�RI�KRZ�WKLV�EXJ�ZLOO�LPSDFW�WKH�SURGXFW
Ɣ EHFDXVH�ZH�PLJKW�QRW�XQGHUVWDQG�WKH�IXOO�LPSOLFDWLRQV�RU�FRQVHTXHQFHV�RI�D�GHIHFW��ZH�PLJKW�WKLQN�ZH�GRQƊW�QHHG�WR�IL[�WKH�GHIHFW�ULJKW�DZD\
)L[LQJ�'HIHFWV 'HFLVLRQ�WR�127�IL[�WKH�GHIHFW�LV�PDGH 'HFLVLRQ�WR�IL[�LW�LV�PDGH
'HIHFW�KDV�KXJH�LPSDFW�WR�PRVW�RI�WKH�FXVWRPHUV
0,66 +,7
'HIHFW�LV�LQGHHG�PLQRU�DQG�QRW�DW�
DOO�XUJHQW&255(&7�5(-(&7,21 )$/6(�$/$50
'HWHFWLRQ�7KHRU\���)L[LQJ�'HIHFWV
%DFN�WR�WKH�ZRUVW�EXJ�,�
HYHU�IRXQG���
Ɣ ,�IRXQG�DQG�UHSRUWHG�D�GHIHFW��D�UHDO�SUREOHP�� !�+,7
Ɣ 7KH�&(2�WKRXJKW�LW�ZDV�D�)$/6(�$/$50�
Ɣ 7KH�&(2�GHFLGHG�WR�QRW�KDYH�WKH�EXJ�IL[HG� !�0,66
$�ELW�RQ�ELDV
:H�FDQƊW�FRQWURO�RXU�ELDV��ZH�GRQƊW�NQRZ�KRZ�WR�127�EH�ELDVHG
HJ��([SHULPHQWHU�%LDV
%XW�ZH�FDQ�ELDV�RXUVHOYHV��RU�RWKHUV��LQ�D�GLIIHUHQW�GLUHFWLRQ���EDODQFH�WKH�VFDOH�GLIIHUHQWO\
$�ELW�PRUH�RQ�ELDV
0DLQ�ELDVLQJ�YDULDEOHV
Ɣ 0RWLYDWLRQ���3UHIHUUHG�UHVXOW
Ɣ 3HUFHLYHG�3UREDELOLW\
Ɣ ([SHFWHG�FRQVHTXHQFHV
Ɣ 3HUFHLYHG�LPSRUWDQFH�RI�WKH�WDVN
KWWSV���ZZZ�IOLFNU�FRP�SKRWRV���������#1��������������
:H�FDQ�PDQLSXODWH�
ELDV
81&216&,286�35()(55('�5(68/7��ƌ,�+23(�7+(5(�$5(�
12�5(,1'((5ƍ
(;3(5,0(17(5�%,$6�
KWWSV���ZZZ�IOLFNU�FRP�SKRWRV���������#1��������������
:H�FDQ�PDQLSXODWH�
ELDV&+$1*(�3(5&(,9('�352%$%,/,7<���025(�
ƌ+,76ƍ
KWWSV���ZZZ�IOLFNU�FRP�SKRWRV���������#1��������������
:H�FDQ�PDQLSXODWH�
ELDV&+$1*(�3(5&(,9('�352%$%,/,7<���025(�
ƌ+,76ƍ
025(�)$/6(�$/$506�$6�:(//�
KWWSV���ZZZ�IOLFNU�FRP�SKRWRV���������#1��������������
:H�FDQ�PDQLSXODWH�
ELDV&+$1*(�3(5&(,9('�
352%$%,/,7<���/(66�)$/6(�$/$506
KWWSV���ZZZ�IOLFNU�FRP�SKRWRV���������#1��������������
:H�FDQ�PDQLSXODWH�
ELDV&+$1*(�3(5&(,9('�
352%$%,/,7<��/(66�)$/6(�$/$506
025(�Ƌ0,66(6ƌ�$6�:(//�
KWWSV���ZZZ�IOLFNU�FRP�SKRWRV���������#1��������������
:H�FDQ�PDQLSXODWH�
ELDV&+$1*(�3(5&(,9('�
,03$&7�&216(48(1&(6
$*$,1��025(�)$/6(�$/$506
%LDV�SHRSOH�ZKR�
ILQGEXJV
025(�%8*�5(32576�
Ɣ HQFRXUDJH�UHSRUWLQJ�DQ\WKLQJ�WKDW�ORRNV�VWUDQJHƔ SUDLVH�JRRG�ZRUNƔ FRQWHVWVƔ KHOS�WHVWHUV�XQGHUVWDQG�WKH�SRVVLEOH�
FRQVHTXHQFHV�FRVWV�RI�D�GHIHFWƔ SUDLVH�ZRUN�RQ�EXJV�WKDW�DUH�YDOXDEOH�HYHQ�LI�GHIHUUHG
DQG�NHHS�LQ�PLQG�WKDW�ELDVLQJ�WRZDUGV�PRUH�EXJ�UHSRUWV�DOVR�PHDQV�PRUH�GHIHUUHG�QRQ�YDOLG�RQHV
%LDV�SHRSOH�ZKR�
ILQGEXJV
025(�9$/,'�5(32576�/(66�)$/6(�$/$506�
Ɣ LPSRUWDQW�GXULQJ�GLIIHUHQW�SKDVHV�LQ�WKH�SURMHFW��HDUO\�RQ��!�UHSRUW�HYHU\WKLQJ�
Ɣ HQFRXUDJH�GLUHFW�LQIRUPDO�FRPPXQLFDWLRQ��ƉPHQWLRQ�LQ�SDVVLQJƊ�
Ɣ H[SODLQ�ZK\Ɣ WUDLQ�SHRSOH��JLYH�H[DPSOHVƔ DFFHSW�\RX�PLJKW�KDYH�PRUH�PLVVHG�EXJV�DV�ZHOO
25«��OLNH�LQ�P\�VWRU\�
%LDV�SHRSOH�ZKR�
ILQGEXJV
025(�9$/,'�5(32576�/(66�)$/6(�$/$506���7+(�5,6.<�:$<�
Ɣ HYDOXDWH�WHVWHUV�EDVHG�RQ�KRZ�PDQ\�LQYDOLG�EXJV�WKH\�UHSRUW
Ɣ HYDOXDWH�GHYHORSHUV�DQG�WHVWHUV�EDVHG�RQ�DPRXQW�RI�WLPH�VSHQW�LQYHVWLJDWLQJ�LVVXHV�WKDW�PLJKW�QRW�EH�YDOLG�DIWHU�DOO
Ɣ XVH�D�WHVWHUƊV�EXJ�UHSRUWV�WR�SRLQW�RXW�PLVWDNHVƔ GLVPLVV�EXJV�UHSRUWHG�E\�MXQLRUVƔ QRW�PRQLWRU�ODQJXDJH�WRQH�RI�EXJ�UHSRUWV�DQG�FRPPHQWV�
WR�WKHP
+RZ�FRXOG�,�KDYH�KDQGOHG�WKH�VLWXDWLRQ�EHWWHU�LQ�WKDW�VWRU\"
%LDV�SHRSOH�ZKR�GHFLGH�RQ�IL[LQJ�EXJV
$6�$�7(67(5��<28�&$1�
Ɣ EXLOG�\RXU�FUHGLELOLW\�DQG�NQRZ�ZKDW�\RXU�UHSXWDWLRQ�LV
Ɣ ZRUN�RQ�WKH�TXDOLW\�RI�\RXU�EXJ�UHSRUWV
Ɣ OHDUQ�WR�HIIHFWLYHO\�LQYHVWLJDWH�KDUG�WR�UHSURGXFH�EXJV
$6�$�7(67(5��<28�&$1�
Ɣ ZRUN�RQ�WKH�WRQH�RI�\RXU�EXJ�UHSRUWV
Ɣ DFFXUDWHO\�HVWLPDWH�VHYHULW\�LPSDFW�RI�EXJV
Ɣ DYRLG�H[DJJHUDWLRQ�
%LDV�SHRSOH�ZKR�GHFLGH�RQ�IL[LQJ�EXJV
$6�$�7(67(5��<28�&$1�
Ɣ OHDUQ�ZKHQ�WR�JLYH�XS��DQG�ZKHQ�QRW�
Ɣ OHDUQ�ZKR�FDQ�KHOS�\RX�RXW���PDNH�ƌIULHQGV�ZLWK�SLFN�XS�WUXFNVƍ�
%LDV�SHRSOH�ZKR�GHFLGH�RQ�IL[LQJ�EXJV
$6�$�7(67(5��<28�&$1�
ƌ7UHDW�WKH�SURJUDPPHUV�DQG�RWKHU�VWDNHKROGHUV�ZLWK�FRXUWHV\�DQG�UHVSHFW��HYHQ���HVSHFLDOO\���RQHV�ZKR�GRQƊW�GHVHUYH�LW�ƍ�
�&HP�.DQHU��%XJ�$GYRFDF\�&RXUVH��/HVVRQ����
%LDV�SHRSOH�ZKR�GHFLGH�RQ�IL[LQJ�EXJV
7KDQN�\RX�#UXBDOWRP
UX�FLQGUHD#DOWRP�IL
48(67,216"