detection theory applied to finding and fixing defects

21
BT6 Concurrent Session 11/12/15 11:30am “Detection Theory Applied to Finding and Fixing Defects” Presented by: Ru Cindrea Altom Consulting Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 · 904-278-0524 · [email protected] · www.techwell.com

Upload: techwell

Post on 21-Jan-2018

154 views

Category:

Software


1 download

TRANSCRIPT

Page 1: Detection Theory Applied to Finding and Fixing Defects

BT6 ConcurrentSession11/12/1511:30am

“Detection Theory Applied to Finding and Fixing Defects”

Presented by:

Ru Cindrea

Altom Consulting

Broughttoyouby:

340CorporateWay,Suite300,OrangePark,FL32073888-268-8770·904-278-0524·[email protected]·www.techwell.com

Page 2: Detection Theory Applied to Finding and Fixing Defects

Ru Cindrea Altom Consulting

Ru Cindrea (@ru_altom) is a senior test consultant and managing partner at Altom Consulting (altom.fi). With more than ten years of experience in software testing, she is particularly interested in mobile testing and test automation with a special interest in mobile games. Ru is an instructor for the Black Box Software Testing series offered by Altom in collaboration with KFA. She is an organizer of the Test Lab, an open-space concept organized at testing conferences where participants practice testing and learn from their peers.

Page 3: Detection Theory Applied to Finding and Fixing Defects

'HWHFWLRQ�7KHRU\�$SSOLHG�WR�)LQGLQJ�DQG�)L[LQJ�'HIHFWV

5X�&LQGUHD���$OWRP�&RQVXOWLQJ

$ERXW�PH

Ɣ IURP�5RPDQLD��QRZ�OLYLQJ�LQ�)LQODQG

Ɣ VWDUWHG�DV�D�WHVWHU�LQ�����

Ɣ SDUWQHU�DQG�VRIWZDUH�WHVWHU�DW�$OWRP�VLQFH�����

Ɣ %%67�,QVWUXFWRU��ZRUNLQJ�ZLWK�&HP�.DQHU�DQG�%HFN\�)LHGOHU�RIIHULQJ�%%67�FODVVHV�RQOLQH

Ɣ LQWR�PRELOH�WHVWLQJ��PRELOH�DXWRPDWLRQ�DQG�PRELOH�DSS�GHYHORSPHQW

#UXBDOWRP

Page 4: Detection Theory Applied to Finding and Fixing Defects

6SHDN�(DV\KWWS���VSHDNLQJ�HDV\�FRP

7KDQNV�WR�0DDUHW�3\K¦M¦UYL��$QQH�0DULH�&KDUUHWW�DQG�)LRQD�&KDUOHV�IRU�VXSSRUW�DQG�PHQWRULQJ

7KH�VWRU\�RI�WKH�ZRUVW�EXJ�,�HYHU�IRXQG

Page 5: Detection Theory Applied to Finding and Fixing Defects

ƌ,I�\RX�FDQ�UHSURGXFH�LW��WKH�GHYHORSHU�LV�ILUHG��,I�\RX�FDQƊW��\RXƊUH�ILUHG�ƍ

$�ORW�RI�LQWHUHVWLQJ�FRQGLWLRQV�IRU�WKH�EXJ

Ɣ ,�ZDV�D�MXQLRU�WHVWHU

Ɣ &RPSOH[�HQYLURQPHQW�ZLWK�PDQ\�GLIIHUHQW�([FKDQJH�VHUYHUV

Ɣ 0RVW�RI�WKH�FRPSRQHQWV�ZHUH�VLPXODWHG

Ɣ &RPPXQLFDWLRQ�YLD�EXJ�GDWDEDVH

Ɣ %XJ�ZDV�KDUG�WR�UHSURGXFH

Ɣ 6XEFRQWUDFWLQJ�DJUHHPHQW���EXJV�RQ�SURYLGHUƊV�WLPH�DQG�H[SHQVHV

Ɣ 'HYHORSHU�GLGQƊW�KDYH�WLPH

Ɣ &(2�GLGQƊW�NQRZ���GLGQƊW�XQGHUVWDQG�WKH�UHDO�SUREOHP

Ɣ 'HYHORSHUV�ZHUH�WDXJKW�WKDW�WHVWHUV�DUH�QRW�WR�EH�WUXVWHG

Ɣ 'HYHORSHUVƊ�WLPH�FRQVLGHUHG�PRUH�YDOXDEOH�WKDQ�WHVWHUVƊ�WLPH

Ɣ 'HYHORSHUV�QRW�HQFRXUDJHG�WR�LQYHVWLJDWH�GLIILFXOW�EXJV

Page 6: Detection Theory Applied to Finding and Fixing Defects

,�GHFLGHG�WR�

䚉 UHSRUW�WKH�EXJ

䚉 QRW�WDON�WR�WKH�GHYHORSHU�GLUHFWO\�EXW�FRPPXQLFDWH�YLD�WKH�EXJ�GDWDEDVH

䚉 QRW�LQVLVW�WKDW�WKH�EXJ�VKRXOG�EH�IL[HG

䚉 QRW�WR�VSHQG�PRUH�WLPH�LQYHVWLJDWLQJ�LW

7KH�'HYHORSHU�GHFLGHG�WR�

䚉 QRW�WDNH�WKH�EXJ�VHULRXVO\

䚉 QRW�VSHQG�PRUH�WLPH�LQYHVWLJDWLQJ�LW

7KH�&(2�GHFLGHG�WR�

䚉 QRW�IL[�WKH�EXJ

䚉 QRW�DVN�DERXW�LWV�LPSOLFDWLRQV

䚉 QRW�WUXVW�PH

$�ORW�RI�GHFLVLRQV�ZHUH�LQYROYHG�LQ�KDQGOLQJ�WKH�EXJ

7KHVH�GHFLVLRQV�ZHUH�PDGH�XQGHU�FRQGLWLRQV�RI�XQFHUWDLQW\

Page 7: Detection Theory Applied to Finding and Fixing Defects

7KHVH�GHFLVLRQV�ZHUH�VXEMHFW�WR�RXU�RZQ�ELDVHV�

'HWHFWLRQ�7KHRU\

Ɣ XVHG�ZKHQ�SV\FKRORJLVWV�ZDQW�WR�PHDVXUH�WKH�ZD\�ZH�PDNH�GHFLVLRQV�XQGHU�FRQGLWLRQV�RI�XQFHUWDLQW\

Ɣ GHFLVLRQ�PDNHU�LV�127�D�SDVVLYH�UHFHLYHU�RI�LQIRUPDWLRQ��EXW�PDNHV�GLIILFXOW�SHUFHSWXDO�MXGJPHQWV�

Page 8: Detection Theory Applied to Finding and Fixing Defects

$Q�([DPSOH��'ULYLQJ�LQ�IRJJ\�FRQGLWLRQV

'HWHFWLRQ�7KHRU\���'ULYLQJ�LQ�IRJJ\�FRQGLWLRQV

Ɣ ZH�KDYH�WR�GHFLGH�KRZ�IDU�DZD\�IURP�XV�DQ�REMHFW�LV

Ɣ EDVHG�RQ�YLVXDO�VWLPXOXV�LPSDLUHG�E\�WKH�IRJ� !�FRQGLWLRQV�RI�XQFHUWDLQW\

Page 9: Detection Theory Applied to Finding and Fixing Defects

'ULYLQJ�LQ�IRJJ\�FRQGLWLRQV 'ULYHU�GRHVQƉW�EUDNH 'ULYHU�EUDNHV

5HLQGHHU�RQ�WKH�VLGH�RI�WKH�URDG 0,66 +,7

1RWKLQJ�RQ�WKH�VLGH�RI�WKH�URDG &255(&7�5(-(&7,21 )$/6(�$/$50

'HWHFWLRQ�7KHRU\���'ULYLQJ�LQ�IRJJ\�FRQGLWLRQV

$SSO\LQJ�'HWHFWLRQ�7KHRU\�WR�)LQGLQJ�'HIHFWV

Page 10: Detection Theory Applied to Finding and Fixing Defects

'HWHFWLRQ�7KHRU\���)LQGLQJ�'HIHFWV

Ɣ ZH�KDYH�WR�GHFLGH�LI�D�FHUWDLQ�EHKDYLRU�LV�D�EXJ�RU�QRW

Ɣ EDVHG�RQ�RXU�XQGHUVWDQGLQJ�RI�KRZ�WKH�SURGXFW�VKRXOG�ZRUN�DQG�EDVHG�RQ�WKH�VSHFLILFDWLRQV�ZH�DUH�DZDUH�RI

Ɣ EHFDXVH�ZH�PLJKW�QRW�QRWLFH�D�FHUWDLQ�EHKDYLRU�RU�PLJKW�QRW�XQGHUVWDQG�KRZ�WKH�SURGXFW�VKRXOG�ZRUN��ZH�PLJKW�WKLQN�WKH�SURGXFW�ZRUNV�DV�H[SHFWHG�ZKHQ�D�EXJ�RFFXUUHG�

)LQGLQJ�'HIHFWV 7HVWHU�GRHVQƉW�UHSRUW�D�SUREOHP 7HVWHU�UHSRUWV�GHIHFW

0LVEHKDYLRU�LQ�WKH�SURGXFW 0,66 +,7

%HKDYLRU�DFFRUGLQJ�WR�

VSHFV�H[SHFWDWLRQV

&255(&7�5(-(&7,21 )$/6(�$/$50

'HWHFWLRQ�7KHRU\���)LQGLQJ�'HIHFWV

Page 11: Detection Theory Applied to Finding and Fixing Defects

$SSO\LQJ�'HWHFWLRQ�7KHRU\�WR�)L[LQJ�'HIHFWV

'HWHFWLRQ�7KHRU\���)L[LQJ�'HIHFWV

Ɣ ZH�KDYH�WR�GHFLGH�LI�D�FHUWDLQ�EXJ�VKRXOG�EH�IL[HG�RU�QRW

Ɣ EDVHG�RQ�RXU�XQGHUVWDQGLQJ�RI�KRZ�WKLV�EXJ�ZLOO�LPSDFW�WKH�SURGXFW

Ɣ EHFDXVH�ZH�PLJKW�QRW�XQGHUVWDQG�WKH�IXOO�LPSOLFDWLRQV�RU�FRQVHTXHQFHV�RI�D�GHIHFW��ZH�PLJKW�WKLQN�ZH�GRQƊW�QHHG�WR�IL[�WKH�GHIHFW�ULJKW�DZD\

Page 12: Detection Theory Applied to Finding and Fixing Defects

)L[LQJ�'HIHFWV 'HFLVLRQ�WR�127�IL[�WKH�GHIHFW�LV�PDGH 'HFLVLRQ�WR�IL[�LW�LV�PDGH

'HIHFW�KDV�KXJH�LPSDFW�WR�PRVW�RI�WKH�FXVWRPHUV

0,66 +,7

'HIHFW�LV�LQGHHG�PLQRU�DQG�QRW�DW�

DOO�XUJHQW&255(&7�5(-(&7,21 )$/6(�$/$50

'HWHFWLRQ�7KHRU\���)L[LQJ�'HIHFWV

%DFN�WR�WKH�ZRUVW�EXJ�,�

HYHU�IRXQG���

Ɣ ,�IRXQG�DQG�UHSRUWHG�D�GHIHFW��D�UHDO�SUREOHP�� !�+,7

Ɣ 7KH�&(2�WKRXJKW�LW�ZDV�D�)$/6(�$/$50�

Ɣ 7KH�&(2�GHFLGHG�WR�QRW�KDYH�WKH�EXJ�IL[HG� !�0,66

Page 13: Detection Theory Applied to Finding and Fixing Defects

$�ELW�RQ�ELDV

:H�FDQƊW�FRQWURO�RXU�ELDV��ZH�GRQƊW�NQRZ�KRZ�WR�127�EH�ELDVHG

HJ��([SHULPHQWHU�%LDV

%XW�ZH�FDQ�ELDV�RXUVHOYHV��RU�RWKHUV��LQ�D�GLIIHUHQW�GLUHFWLRQ���EDODQFH�WKH�VFDOH�GLIIHUHQWO\

$�ELW�PRUH�RQ�ELDV

0DLQ�ELDVLQJ�YDULDEOHV

Ɣ 0RWLYDWLRQ���3UHIHUUHG�UHVXOW

Ɣ 3HUFHLYHG�3UREDELOLW\

Ɣ ([SHFWHG�FRQVHTXHQFHV

Ɣ 3HUFHLYHG�LPSRUWDQFH�RI�WKH�WDVN

Page 14: Detection Theory Applied to Finding and Fixing Defects

KWWSV���ZZZ�IOLFNU�FRP�SKRWRV���������#1��������������

:H�FDQ�PDQLSXODWH�

ELDV

81&216&,286�35()(55('�5(68/7��ƌ,�+23(�7+(5(�$5(�

12�5(,1'((5ƍ

(;3(5,0(17(5�%,$6�

KWWSV���ZZZ�IOLFNU�FRP�SKRWRV���������#1��������������

:H�FDQ�PDQLSXODWH�

ELDV&+$1*(�3(5&(,9('�352%$%,/,7<���025(�

ƌ+,76ƍ

Page 15: Detection Theory Applied to Finding and Fixing Defects

KWWSV���ZZZ�IOLFNU�FRP�SKRWRV���������#1��������������

:H�FDQ�PDQLSXODWH�

ELDV&+$1*(�3(5&(,9('�352%$%,/,7<���025(�

ƌ+,76ƍ

025(�)$/6(�$/$506�$6�:(//�

KWWSV���ZZZ�IOLFNU�FRP�SKRWRV���������#1��������������

:H�FDQ�PDQLSXODWH�

ELDV&+$1*(�3(5&(,9('�

352%$%,/,7<���/(66�)$/6(�$/$506

Page 16: Detection Theory Applied to Finding and Fixing Defects

KWWSV���ZZZ�IOLFNU�FRP�SKRWRV���������#1��������������

:H�FDQ�PDQLSXODWH�

ELDV&+$1*(�3(5&(,9('�

352%$%,/,7<��/(66�)$/6(�$/$506

025(�Ƌ0,66(6ƌ�$6�:(//�

KWWSV���ZZZ�IOLFNU�FRP�SKRWRV���������#1��������������

:H�FDQ�PDQLSXODWH�

ELDV&+$1*(�3(5&(,9('�

,03$&7�&216(48(1&(6

$*$,1��025(�)$/6(�$/$506

Page 17: Detection Theory Applied to Finding and Fixing Defects

%LDV�SHRSOH�ZKR�

ILQGEXJV

025(�%8*�5(32576�

Ɣ HQFRXUDJH�UHSRUWLQJ�DQ\WKLQJ�WKDW�ORRNV�VWUDQJHƔ SUDLVH�JRRG�ZRUNƔ FRQWHVWVƔ KHOS�WHVWHUV�XQGHUVWDQG�WKH�SRVVLEOH�

FRQVHTXHQFHV�FRVWV�RI�D�GHIHFWƔ SUDLVH�ZRUN�RQ�EXJV�WKDW�DUH�YDOXDEOH�HYHQ�LI�GHIHUUHG

DQG�NHHS�LQ�PLQG�WKDW�ELDVLQJ�WRZDUGV�PRUH�EXJ�UHSRUWV�DOVR�PHDQV�PRUH�GHIHUUHG�QRQ�YDOLG�RQHV

%LDV�SHRSOH�ZKR�

ILQGEXJV

025(�9$/,'�5(32576�/(66�)$/6(�$/$506�

Ɣ LPSRUWDQW�GXULQJ�GLIIHUHQW�SKDVHV�LQ�WKH�SURMHFW��HDUO\�RQ��!�UHSRUW�HYHU\WKLQJ�

Ɣ HQFRXUDJH�GLUHFW�LQIRUPDO�FRPPXQLFDWLRQ��ƉPHQWLRQ�LQ�SDVVLQJƊ�

Ɣ H[SODLQ�ZK\Ɣ WUDLQ�SHRSOH��JLYH�H[DPSOHVƔ DFFHSW�\RX�PLJKW�KDYH�PRUH�PLVVHG�EXJV�DV�ZHOO

Page 18: Detection Theory Applied to Finding and Fixing Defects

25«��OLNH�LQ�P\�VWRU\�

%LDV�SHRSOH�ZKR�

ILQGEXJV

025(�9$/,'�5(32576�/(66�)$/6(�$/$506���7+(�5,6.<�:$<�

Ɣ HYDOXDWH�WHVWHUV�EDVHG�RQ�KRZ�PDQ\�LQYDOLG�EXJV�WKH\�UHSRUW

Ɣ HYDOXDWH�GHYHORSHUV�DQG�WHVWHUV�EDVHG�RQ�DPRXQW�RI�WLPH�VSHQW�LQYHVWLJDWLQJ�LVVXHV�WKDW�PLJKW�QRW�EH�YDOLG�DIWHU�DOO

Ɣ XVH�D�WHVWHUƊV�EXJ�UHSRUWV�WR�SRLQW�RXW�PLVWDNHVƔ GLVPLVV�EXJV�UHSRUWHG�E\�MXQLRUVƔ QRW�PRQLWRU�ODQJXDJH�WRQH�RI�EXJ�UHSRUWV�DQG�FRPPHQWV�

WR�WKHP

Page 19: Detection Theory Applied to Finding and Fixing Defects

+RZ�FRXOG�,�KDYH�KDQGOHG�WKH�VLWXDWLRQ�EHWWHU�LQ�WKDW�VWRU\"

%LDV�SHRSOH�ZKR�GHFLGH�RQ�IL[LQJ�EXJV

$6�$�7(67(5��<28�&$1�

Ɣ EXLOG�\RXU�FUHGLELOLW\�DQG�NQRZ�ZKDW�\RXU�UHSXWDWLRQ�LV

Ɣ ZRUN�RQ�WKH�TXDOLW\�RI�\RXU�EXJ�UHSRUWV

Ɣ OHDUQ�WR�HIIHFWLYHO\�LQYHVWLJDWH�KDUG�WR�UHSURGXFH�EXJV

Page 20: Detection Theory Applied to Finding and Fixing Defects

$6�$�7(67(5��<28�&$1�

Ɣ ZRUN�RQ�WKH�WRQH�RI�\RXU�EXJ�UHSRUWV

Ɣ DFFXUDWHO\�HVWLPDWH�VHYHULW\�LPSDFW�RI�EXJV

Ɣ DYRLG�H[DJJHUDWLRQ�

%LDV�SHRSOH�ZKR�GHFLGH�RQ�IL[LQJ�EXJV

$6�$�7(67(5��<28�&$1�

Ɣ OHDUQ�ZKHQ�WR�JLYH�XS��DQG�ZKHQ�QRW�

Ɣ OHDUQ�ZKR�FDQ�KHOS�\RX�RXW���PDNH�ƌIULHQGV�ZLWK�SLFN�XS�WUXFNVƍ�

%LDV�SHRSOH�ZKR�GHFLGH�RQ�IL[LQJ�EXJV

Page 21: Detection Theory Applied to Finding and Fixing Defects

$6�$�7(67(5��<28�&$1�

ƌ7UHDW�WKH�SURJUDPPHUV�DQG�RWKHU�VWDNHKROGHUV�ZLWK�FRXUWHV\�DQG�UHVSHFW��HYHQ���HVSHFLDOO\���RQHV�ZKR�GRQƊW�GHVHUYH�LW�ƍ�

�&HP�.DQHU��%XJ�$GYRFDF\�&RXUVH��/HVVRQ����

%LDV�SHRSOH�ZKR�GHFLGH�RQ�IL[LQJ�EXJV

7KDQN�\RX�#UXBDOWRP

UX�FLQGUHD#DOWRP�IL

48(67,216"