How to Become A Testing Expert Liang Gao

Typical Interview Question

• Do you know programming• Answer: Yes, a little• Which language do you know the best• Answer: I know C a little• How many lines of code have you written• Answer : Umm.., about 300 lines• Do you know Java or .NET, • Answer : No• Do you know Linux• Answer : No• Do you know networking• Answer : I installed Windows XP on my PC….• Ok, you can work for us as testing engineer

Tester’s Dilemma

• Testing is not cool• Tester is second class when compared with

developer• Developer using me as servant• I don’t see I can make difference in our

company• Testers are keep coming, they are young and

better than me. • 我在吃青春饭， I don’t know where my career

path is• OK ， I need to be a developer and write

code!

Top Management's Dilemma

你也知道， 现在招到好的测试工程师是有多难 - 华为北京研究所副所长 2008

It took Juniper 6 month plus to find a new QA director

Career Story of Ting

• Graduate at 1990, only job found is the tester at Sun

• Join Cisco as tester in 1993, then test automation engineer (design Cisco’s automation framework)

• Cisco grows from 2000 to 40000 from 1993 to 1998

• Join Netscreen as the 1st testing engineer at 1998• NetScreen went to Nasdaq on 2001• Grow with NetScreen as testing lead, testing

manager, testing senior manager and testing director

• Found Sigma at 2004, served as CEO as today

Lesson

• You may have to stay long enough to be competitive.

What is a testing expert?

• Ability to find critical bugs in given time frame (hot gun)

• Ability to build a comprehensive testing strategy in given time frame (expert)

• Ability to manage the release process (top expert)

What is a testing expert Looks like?

• Play the video of James Bach

Find Bug that could make News

Bugs in the News

• A Cisco Secure Access Control Server (ACS) that is configured to use Extensible Authentication Protocol-Transport Layer Security (EAP-TLS) to authenticate users to the network will allow access to any user that uses a cryptographically correct certificate which can be expired, or come from an untrusted Certificate Authority (CA) and still be cryptographically correct.

• CSCse58195. The WLC contains a bug when processing WLAN ACLs that causes the WLANvACL configuration to be saved with an invalid checksum. When the configuration is subsequently reloaded at boot time, the checksum fails and the WLAN ACLs are not installed. 

Bugs in the News

• CSCdv24925 It is possible to read stored configuration file from the Storage Router without any authorization.

• CSCdu45417 It is possible to halt the Storage Router by sending a fragmented packet over the Gigabit interface.

• CSCdv24925 An unauthorized person may read the configuration of the Storage Router. That may lead to unauthorized access of a storage space.

Bugs in the News

• Versions of the Cisco ACE 4710 Application Control Engine appliance prior to software version A1(8a) use default administrator, web management, and device management account credentials. The appliance and module do not prompt users to modify system account passwords during the initial configurationprocess.

• Crafted SSH Packet Vulnerability • Crafted SNMPv2c Packet Vulnerability

2 Factors of a hot gun

• Technical Expertise• Thinking methodology

Hot Gun’s Bug Percentage

P4

P2

P1

P3

Cosmetic

Functionality

Major Functionality

Critical Functionality, Crash, Hang

10%

30%

40%

20%

Thinking Methodology

Thinking Methodology

• Customers• Information• Developer

revelations• Team• Equipment

& Tools• Schedule• Test Items• Deliverable

s

Structures Functions

Data

Platform

Operation

Time

Capability

Reliability

Usability

Security

Scalability

Performance

Installability

Compatiabilty

Supportability

Testabilty

Measurability

Portability

Localizations

Function testing

Domain testing

Stress testing

Flow testing

Scenario testing

Claims testing

User testing

Automatic testing

Thinking Methodology

• Customers• 客服• 售前• 售后• 市场

Thinking Methodology

发布

升级 / 安装

配置

入网

运营

维护

发布流程测试

Installation/Upgrade 测试 HA

你知道客户典型配置么？你知道客户跑什么业务？你知道客户跑什么业务？

你知道客户典型应用场景么？

客户 1-2 年运营的可能遇到情况

可维护性测试

兼容性？

Thinking Methodology

Negative Stress Boundary Features Interaction

Security

Feature Point 1

Feature Point 2

Feature Point 3

Feature Point 4

Feature Point 5

Develop Testing Strategy

• Be the review person• 知其然也知其所以然

Develop Testing Strategy

• What is your goal first?– Find more bugs?– Find more critical bugs?– Ensure product or feature has no critical

defects– Ensure customers will be ok after the release?

• The strategy– Bug oriented?– Coverage oriented?– Customer oriented?

What is Coverage Strategy?

• How to thoroughly test OSPF Hello protocol?

– Function points? (tester)– User Scenarios? (test expert)– Scalability? (test expert)– Performance?– Security? (test expert)– ………

What is Coverage Strategy?

• 7 platforms• 6 different line cards• 2 modes (main/aggressive)• AH/ESP• CA/No CA• HA/No HA• Hub Spoke/Partial Mesh/Full Mesh

7x6x2x2x2x3 = 1088

Types of Testing Covered on Different Release

Major Release

Minor Release

Patch Release

Platform Release

SFR CSP

New Feature Test Full Full TBD Full

Regression Test Full Partial Partial Partial Partial

System Test Full Full Full Full

Interoperability Test Full TBD TBD TBD

Performance Test Full TBD Full TBD

Capacity Test Full Full TBD

Security Test Full Full Full Full Full

Automation Test(partial regression)

Full Full Full Full Full Full

SFR – Special Feature Release CSP – Customer Specific Patch

Advise to the New Expert

• Practice, Practice, Practice• Don’t confuse experience with Expertise• Don’t trust the folklore – but learn it anyway• Take nothing on faith, own your methodology• Drive your own education, no one else will• Reputation = Money: Build and protect your

reputation• Relentlessly gather resources, materials and

tools. • Associate with demanding colleagues. • Write, speak

An Expert’s Vision

• An Expert’s Vision– I can test anything– Under any condition– On any given time frame.

An Expert’s Vision

Top Expert

• Release process• Testing organization management

Release process

Release Process

• Start from the MRD• End with the FCS (first customer ship)

敏捷开发与持续集成 – 迭代中的持续集成

Test Organization management

Thank you!