enabling better software through better testing jason anderson, microsoft [email protected]

Enabling Better Software through Better Testing

Jason Anderson, [email protected]

Agenda

• Team System Overview• Test Authoring & Execution

● Manual Testing● Web Testing● Automated \ Unit Testing● Load Testing

• Test Case Management• Code Coverage• Test Extensibility• Team Tester and Team Foundation Server• Questions

Development Teams

InfrastructureInfrastructureArchitectArchitect

SolutionSolutionArchitectArchitect

Project ManagerProject Manager

DeveloperDeveloper

TesterTester

End UserEnd User

Visual Studio Team SystemPro

cess

an

d A

rch

itect

ure

Pro

cess

an

d A

rch

itect

ure

G

uid

an

ceG

uid

an

ce

Vis

ual S

tud

io In

du

stry

V

isu

al S

tud

io In

du

stry

Part

ners

Part

ners

Dynamic Code Analyzer

Visual Studio

Team Architect

Static Code Analyzer

Code Profiler

Unit Testing

Code Coverage

Visio and UML Modeling

Team Foundation Client (includes CAL)

Visual Studio Professional Edition

Load Testing

Manual Testing

Test Case Management

Application Designer

Logical Infra. Designer

Deployment Designer

Visual Studio

Team DeveloperVisual Studio

Team TestLoad Testing

Manual Testing


Change Management

Work Item Tracking

Reporting

Project Site

Visual Studio

Team Foundation Integration Services

Project ManagementBig Build

Class Designer

Test AuthoringTest Authoring

QA SKU UI: Authoring Tests

• Design Principal: Enable authoring features specific to testing without impeding standard development experience

• Primary UI Elements● Solution Explorer● Test View

• Largest Real-estate reserved for code● Result details can share space via tab selection

with explicit user gesture

Types Explained

Visual Visual Studio Studio IDEIDE

EDT QA ToolsEDT QA Tools

ManualManual

UnitUnit

GenericGeneric

WebWeb

LoadLoad

ASPASP

Ordered Ordered 33rdrd Party Party

Test ExecutionTest Execution

QA SKU UI: Execution

• Design Principal: Provide rich environment for management and control of test cases

• Primary UI Elements● Test Manager● Command Line Parser● Results Window

• Primary user scenario is not test author, rather to manage and execute tests

• Local or Remote Execution● Distributed Across Machines● With Machine Constraints

Writing and Writing and Executing TestsExecuting Tests

Test Case ManagementTest Case Management


• Fully Integrated with Visual Studio• Uses a local file that can be checked into source code control

and shared● Can be used without the project system● Versioning

• Test Lists to managed Test Cases● Tests in one or more Test Lists● Test Lists can be hierarchical● Can be used from the Command Line

Test CaseTest CaseManagementManagement

Code CoverageCode Coverage

Code Coverage

• Test Authoring isn’t about quantity, it’s about quality

• Code coverage helps you monitor your tests effectiveness● Team Members can analyze results at a

high-level● Can also analyze source for specific

missed methods and branches

Code Coverage

void PurchaseItem(int itemID)

{

if (itemID == 0) {

throw new Exception();

} else

{ ProcessOrder(itemID);

}

}

UnitTest()

{

PurchaseItem(1);

}

Code CoverageCode Coverage

Test ExtensibilityTest Extensibility

Why Test Extensibility

• Must not lose investment in current infrastructure

• Testing requirements are diverse• Integration beyond Microsoft

Extensibility Story

• Generic Tests● Requires minimal integration work● Limitations to Integration with VS

• Extended Test Types● Custom type for full Test Type extensibility● Means of adding authoring, execution, and

reporting extensibility● Custom types are 1st class test citizens

Generic TestsHow execution works

GenericGenericTestTestTypeType Existing TestExisting Test

MetadataMetadata


ExecutionExecutionProcessProcess

YourYourTestTestProcessProcess

GenericGenericTestTestTypeType Existing TestExisting Test

MetadataMetadata

Generic TestsHow execution works



YourYourTestTestProcessProcess

YourYourTestTest

YourYourTestTest

YourYourTestTest

Test Type ExtensibilityExtended Types

• Test Environment can be extended to include your test tools

• Authoring, Execution, Results plug-ins• Offers first class test type• Used internally to create test types

Test Type Extensibility FrameworkTest Type Extensibility Framework

MS Test TypesMS Test Types VSIP Test TypesVSIP Test Types

Test Type Extensibility How execution works



Test AdaptorTest Adaptor

TestTest

Test UpdatesTest Updates

ResultsResults

Details ViewerDetails Viewer

Run ConfigRun ConfigSettingsSettings

Test View

Your Test Your Test SourceSource

Your Test Your Test SourceSource

Your Test Your Test CaseCase


Your Test Your Test Result Result

SummarySummary

Your Test Your Test Result Result

SummarySummary

Your Your DetailsDetailsYour Your

DetailsDetails



Test Explorer

Custom Custom Editor / WizardEditor / Wizard

Test ItemsTest ItemsManagement UIManagement UI

GeneralGeneralResults ViewResults View

Custom DetailedCustom DetailedResults ViewResults View

GenericsGenerics Test TypeTest Type

Hook into RunHook into Run Config SettingsConfig Settings

Context-sensitive Context-sensitive Right-click CommandsRight-click Commands

Complete control ofComplete control ofTest item executionTest item execution

Team Tester and Team Tester and Team Foundation ServerTeam Foundation Server


cess

an

d A

rch

itect

ure

Pro

cess

an

d A

rch

itect

ure

G

uid

an

ceG

uid

an

ce

Vis

ual S

tud

io In

du

stry

V

isu

al S

tud

io In

du

stry

Part

ners

Part

ners


Visual Studio

Team Architect


Code Profiler

Unit Testing

Code Coverage




Load Testing

Manual Testing




Deployment Designer

Visual Studio


Team TestLoad Testing

Manual Testing


Change Management

Work Item Tracking

Reporting

Project Site

Visual Studio

Team Foundation Integration Services

Project ManagementBig Build

Class Designer


cess

an

d A

rch

itect

ure

Pro

cess

an

d A

rch

itect

ure

G

uid

an

ceG

uid

an

ce

Vis

ual S

tud

io In

du

stry

V

isu

al S

tud

io In

du

stry

Part

ners

Part

ners

Change Management

Work Item Tracking

Reporting

Project Site

Integration Services

Project Management


Visual Studio

Team Architect


Code Profiler

Unit Testing

Code Coverage




Load Testing

Manual Testing




Deployment Designer

Visual Studio


Team Test

Change Management

Work Item Tracking

ReportingVisual Studio

Team Foundation

Class Designer

Big Build

Roles Addressed by Team System

Project Manager

Architect Developer Tester

Change Management

Work Item Tracking

Reporting

Project Portal

Visual StudioTeam Foundation

Integration Services

Project Management


Work Item Integration

• Project Management can assign work items and track across lifecycle of the project

• Testers and Developers can assign defects and these integrate with PM work items

• Artifacts (tests, product modules, source code) can be associated with Work Items

Reporting Integration

• Information aggregated from Team System services used in reporting

• Stored in central database for trending• Drill-down into operational stores for

more detail

Reviews Find / Fix Rates…

..and Daily Trends

…and Bugs Against Code Churn and Testing Activity

Visual Studio Team System Team Test• Write your Tests with or without Code• Manage and Execute Test Cases• Extend the Visual Studio Test Types• Integrated with TFS

● Source Code Control● Project Management● Bug Tracking● Reports

• All fully functional from the Visual Studio Team System 2005

enabling better software through better testing jason anderson, microsoft [email protected]

Documents