introduction to visual studio team system – test edition

Testing Best Practices with VSTS

Shahidul Islam, MPH

QA Manager

Emerging Health Information Technology12/4/08

Agenda

Visual Studio Test Edition for Software Testers (VSTEST) Manual Tests Web Tests Load Tests TFS Work Item Tracking TFS Reporting

Sources for Test Cases by Test Type

Business ReqsUser Reqs

System ReqsTechnical Architecture

Hardware SpecsSoftware Specs

Detailed DesignDev Standards

Produce Code

Unit Testing

System Testing

Integration Testing

Acceptance Testing

Validates

Validates

ValidatesBusiness

Focus

Technical Focus

Regression Testing

Reference: Learning Tree International Course 316 “Software Testing and Inspection Methods

Validates

VSTSWebTests

Test PlanDocuments project-specific strategy for all test types.

Master Test Plan

Acceptance Test Plan

System Test Plan

Integration Test Plan

Unit Test Plan

Manual test case scenarios

Automated test case scenarios

Load test scenarios

GENERATES

Defects log

Test logs

VSTEST TFS

VSTEST

Visual Studio Test Edition for Software Testers

Author Tests

Run Tests

Publish and Share the results

Reporting with TFS

VSTEST Basics

Test Projects are Visual Studio Projects Compile/deploy like any assembly in .NET Visual Studio options under “Test Tools” The solution should be stored within source control

(like any other piece of code)

Tool set integrated in VS allows for easy adoption from developers.

Testers are more integrated with the development process

Requirements Management

Manual Tests

VSTS Manual Tests—Excel

File>Properties

Custom TestType: “Manual Test” TestID: “Get a GUID from VSTS>Tools>Create a GUID” Save As .mht file

Web Testing

Uses a recorder for test creation: Records web requests using Internet Explorer Analyzes the HTML traffic– each response can be validated

for expected behavior Ordered series of HTTP requests against a target web

applications Integrates with Visual Studio Available only via VSTS Team Edition for Software Testers Runs with Test Manager Integrated with Team Build

Web Testing

Web Tests are of two types: Recorded Tests

Launch the browser Record session steps Playback recorded session Convertible to (Coded Tests) C# or VB code to handle complex

scenarios and flow control Coded Tests

Scripted Better, adapted from recorded tests Playback

Supports: ASP.NET, HTTP based, Web Page, Web Services Compatible with HTTPS

Demo then proceed to slide 43

Web Test Demo

Recording a Web Test

Record a browser session using the Web Test Recorder

Analyze the http requests being recorded

Can pause in between requests

Recording can be a good baseline for coded Web tests

Recording a Web Test

Steps: Test..New Test Choose Web Test Choose Default Visual C#

test project Create

A new Internet Explorer instance opens up

Recording a Web Test

Two windows open: A new Internet Explorer

instance Web Test Recorder on the

left pane of the IE instance Each request or page navigated

is displayed Options available to stop or

pause recording Can add comments during

recording

Recording a Web Test

Once stopped, recorded test can be run from Test Window

Editing a Web Test

Web Test Editor available once recording is stopped

Features: Toolbar for basic commands Expandable Request Tree

view of requests in the execution order

Can expand to see Query String parameters or Form Post parameters

Properties Window Properties and Settings of

the Web Test

Editing a Web Test

Adding a New Request: Right click on request to

add new request

Clean up unnecessary recorded steps

Remove extraneous or wrong user actions

Modify the order of requests Be conscious of

dependent requests

Editing A Web Test: Properties

Properties: Description: Text for describing the test Pre-authenticate:

True: Every request includes authentication header (default)

False: Authentication header sent when requested

User Name and Password: Values set using the Set Credentials button on Web Test Editor for sites using

Basic authentication or Integrated Windows authentication Credentials can be edited directly on

properties window or Can bind to data source for multiple runs

with different credentials – (recommended)

Editing A Web Test: Requests

Requests: HTTP request steps which are executed in given order and whose response is processed

Request Tree in Web Test Editor consolidates child requests to be executed in order for main request

Properties of a Request: Cache Control Encoding Follow Redirects Method Parse Dependent Request Record Results Response Time Goal Think Time Timeout URL Version

Editing A Web Test: Requests

Cache Control: True/False directive to simulate browser caching

Encoding: Text format for requests - usually UTF-8 for HTTP traffic

Follow Redirects: True/False to honor status codes 300-307 (redirect) to final page or stop and treat the redirect as final page

Method: GET or POST

Parse Dependent Request: True/False to process dependent requests such as loading image files. True will get standard behavior (load images)

Editing A Web Test: Requests

Record Results: Used only with Load Testing to determine if timing and performance data need to be included

Response Time Goal: Used with Load Test reporting to record how long the request took to process

Think Time: Delay (in seconds) to pause on page before processing next request

Timeout: Maximum time (in seconds) to wait for response before timing out and logging a failure

URL: URL of the request

Version: Versions of HTTP to use – default 1.1

Editing a Web Test: Request Sub Items Request Sub-items: Simple requests

can be enhanced with sub-items like validation rules etc.

Dependent Requests: A normal HTTP request which

has a parent request Example: Load image on a page Handled by Parse Dependent

Requests property Can have own dependent

requests Can be added explicitly –

example, extraction or validate rule

Editing a Web Test: Request Sub Items

Query String Parameters: Name=Value pairs set in the Property window Show Separate Request Results:

Used to group results in Load Testing Result data is grouped if False (default); if True,

displays separate grouping for data (response time etc)

URL Encode: Determines if Name/Value will be URL encoded

Form Post Parameters: If the request method is POST, enables two form POST parameters:

Common Form Post Parameter File Upload Parameter

Headers: Name=Value pair to be included in HTTP header. Ex: Referer=www.microsoft.com

Validation Rules: Add validation rules to verify if response from the request is expected one

Extraction Rules: Capture data from response

Editing a Web Test : Transactions

Allow grouping of requests as a unit to

Document Move/Copy Capture timing information

To insert: Right click at the request

where the transaction is to be inserted

Complete the Add Transaction dialog

Requests can be moved between transactions using drag and drop.

Editing a Web Test: Comments, Context Parameters

Comments: Tests can be documented with comments

Context Parameters: Replace static values with

dynamic values for property, value or parameter

Variables expand to real values at runtime to make the test flexible and dynamic

Usually done through Parameterizing Web Servers

Editing a Web Test: Parameterizing Web Servers

Select web test, and right-click

Select Parameterize Web Servers… Allows changing URLs from static

to context-parameterized variables to re-target the test

Parameterize Web Servers dialog lists web servers used with a default Context Parameter Name

Can be changed using the Change button to change

Web Server Local ASP.NET Development

Web Server

Editing a Web Test: Parameterizing Web Servers

After changing, the Context Parameters node appears in the web test

Values can be edited via right click on Properties of newly added

parametername=value

Context Parameter can then be used in requests using

{{..}} notation

Example: {{WebServer6}}/live/search/search.xml

Editing Web Tests: Extraction Rules Data from POST/GET requests can be extracted for

Subsequent requests Providing debugging information

Extraction rules verify that a Web application is working correctly by

extracting data from the responses to Web requests

storing results in the test context as name value pairs.

Extraction rules can extract form fields Text Attributes Headers regular expressions hidden fields.

Editing Web Tests: Extraction Rules

To Extract, right-click on a request and choose Add Extraction Rule

Extraction rules define what data is to be extracted from an HTTP response and store the data in context parameters

Dynamic data generated via requests can be passed down to later requests.

Editing Web Tests: Extraction Rules

Editing Web Tests: Extraction Rules

To add Extraction Rule: Right-click on request Choose Add Extraction

Rule Modify/add parameters Set Context Parameter to

a unique name

Annoying Error Messages!!

RequestFailed: Context parameter '$HIDDEN1.__VIEWSTATE' not found in test context”.

Add Extract Hidden Fields

Annoying Error Messages!!

Web Test ran flawlessly right after recording but Next Morning?

500 Server Error

Search for parameters you think may be dynamic (Session Variable)

Add Extract Text Rule

Example: Context Parameter Name = ControlID, Ends With = &, HTMLDecode = True, Ignore Case = False, Index = 0, Required = True, Starts With = ControlID =, Use Regular Expression = false.

Editing Web Tests: Validation Rules

Validation rules validate existence of

Text tags, or attributes

on the page returned by a request.

Validation rules also verify the amount of time it takes a request to finish, and the existence of form fields and their values.

Editing Web Tests: Validation Rules

Editing Web Tests: Validation Rules To add a validation rule:

Select a request Right-click Add Validation rules in

dialog Validation Level:

Used only during a load test

Higher validation levels effect speed, hence slower tests

Does not mean priority Determines when the

validation rule is used in a load test.

Setting to High means that the rule is executed only when the load test validation level is set to high.

Editing Web Tests: Validation Rules Validation Levels:

Used only during a load test Higher validation levels

means slower tests Does not mean priority Determines when the

validation rule is used in a load test.

Setting to High means that the rule is executed only when the load test validation level is set to high.

Running Web Tests

Settings can be configured from .testrunconfig file

Choose Web Test from dialog: Number of iterations Browser Type Network Type Simulate Think Times

These settings are overridden when run under a Load Test

Running Web Tests:

From the tool bar: Run Test

From the command line Invoke MSTEST

Interpreting Web Test Results Results Viewer:

Results Panel: Request: The full URL requested HTTP Status: HTTP status returned from the server – typically

200 OK Response Time: Time web server took to respond Size: Size of response

Interpreting Web Test Results Details Panel:

Web Browser Tab: Response rendered as browser page

Request: Contents of the selected request

Response: Contents of the response to selected request

Context: Current context parameters, useful for debugging extraction rules

Details: Results of extraction and validation rules for selected request.

Coded Web Tests

Written in C# or Visual Basic

Incorporates flow of control as opposed to sequential recorded tests

Use Recorded tests for baseline and then convert:

Select web test Right-click Generate Code

Why Coded Web Tests?

Can read expected results from a data source

Flexibility of conditionals and looping

Copy set of request from one test to another

Factor out sections of requests and other operation into methods that any test can call

AJAX in Web Tests

VSTS 2008 now records AJAX requests

Freeware Fiddler need not be used, although it’s still useful for diagnosing

See: http://www.fiddlertool.com/fiddler/

Test Case Management

Test List Editor New Test List per release/project/cycle Properties>Associated Work Items

Run Test Run one or multiple tests Debug tests

Test Results Window

Run/Debug

Pause/Stop Run

Import/Export Results

Group results by

LOAD TESTS

Step 1: Identify Objectives.

Step 2: Identify Key Scenarios.

Step 3: Identify Workload.

Step 4: Identify Metrics.

Step 5: Create Test Cases.

Step 6: Simulate Load.

Step 7: Analyze the Results.

Understanding Load Test

Primary goal of a Load Test is to simulate many users accessing server at the same time

By adding Web Tests to a Load Test you can,

Simulate multiple users opening simultaneous connections to a server

Make multiple HTTP requests

Set other properties that apply to individual Web Tests

Understanding Load Test Continues..

Load Tests are used in several different types of testing:

Smoke: How your application performs under light loads for short duration

Stress: To determine if the application will run successfully for a sustained duration under heavy load

Performance: To determine how responsive your application is

Capacity: To determine how many users and/or transactions a given system will support and still meet performance goals

Identify Objectives

Write down the performance objectives for your application. Response time. For example, the product catalog must be

displayed in less than 3 seconds. Throughput. For example, the system must support 100 requests

per second. Resource utilization. A frequently overlooked aspect is how

much resource your application is consuming, in terms of CPU, memory, disk I/O, and network I/O.

Maximum User Load. Determining how many maximum users can run on a specific hardware configuration

Identify Key Scenarios

What are scenarios? Scenarios are anticipated user paths that generally incorporate

multiple application activities.

How do you identify scenarios? Key scenarios are those for which you have specific performance

goals or those that have a significant performance impact.

For example: Login to the site. Browse albums. Register to the site.

Identify the workload

Identify the distribution IIS Logs (Exist). Research (Not exist).

Identify the user loads What are the max expected

concurrent users?

53

Identify Metrics

Metrics provide information about how close your application is to your performance objectives.

Metrics help you identify problem areas and bottlenecks.

Network-specific metrics.

System-related metrics.

Platform-specific metrics.

Application-specific metrics.

Create Load Test Cases

What is a Load test case? An group of activities involved in a scenario. For example: Place an Order.

The test cases are created based on the scenarios and the profile mix identified in the previous steps.

Each test case should mention the expected results in such a way that each test case can be marked as a pass or fail after execution.

Performance Test Cases Load Test Case example:

Expected Results example:

VS Performance Test Load patterns and Real world test analogy:

Constant pattern Load test, Stress test

Step pattern Compatibility test, Performance test, Benchmarks

Goal based pattern Stress test , Capacity test

Load Test Life Cycle http://blogs.vertigosoftware.com/teamsystem/archive/2006/02/06/VSTS_Load_Testing_Lab_Setup.aspx

Demo

1. Add a blank web test

2. Call web test from another web test (Login and LinkToLos)

3. Data binding-csv-LoginPage

4. Run WebTest once then edit run settings to run per row

5. Validation rule passing

6. Show Load Test setup and run

7. Analyze LoadTest Result

Control Load Modeling

Load Tests now offer additional load-modeling options that enable us to create load tests that more accurately model the expected real-world usage of an application:

Number of tests run Determines which test is run when a virtual user starts a

iteration Follow this model when you are basing test mix on

transaction percentages in an IIS log or production data

Control Load Modeling continues..

The amount of time spent on each test Determines the percentage of virtual users who will run a

particular test Follow this model when you are basing the test mix on the

percentage of users running particular test

The pace/speed at which the users run the tests Each test is run the specified number of times per user per

hour Follow this model when you want virtual users to run tests at

a certain pace throughout the load test

How should the test mix be modeled

Improved Load Test Analyzer Views

New Summary View with key indicators in a single page

Four new built-in graphs display key information at the same time

Easily accessed table view

Load Tests Summary

Test Run Information Name, Description, Start/End times, Controller

Overall Results Number of request/sec Number of failed requests Average response time Average page time

Load Test Summary Continues…

Key Statistics: Top 5 Slowest Pages URL (Can click the link to see details)

and average page load time

Key Statistics: Top 5 Slowest Tests Test name (Can click the link to see details)

Average test time

Key Statistics: Top 5 Slowest SQL operation Name of the operation (Can click the link to see details)

Duration in Microseconds

Load Tests Summary Continues..

Test Results List of all tests and scenarios Number of time each scenario ran Number of times it failed Average test time

Load Tests Summary Continues..

Page Results List of all web pages in the load test Total count of requests made for the web page Average page time

Load Test Summary Continues..

Transaction Results

NOT A DATABASE TRANSACTION Web Test Transaction List of all the transactions in the Load Test Response Time Elapsed Time Total Counts

Load Test Summary Continues…

Controller and Agent Resources Contains list of computers that are used to run the test % processor time Available memory at the end of test run

Load Test Summary Continues…

System Under Test Resources List of computers that are the set of target computers for

which load is being generated Includes any computer from which you collect counter sets

other than Agent or Controller.

Load Test Summary Continues…

Errors List of all errors

Type i.e. HttpEerror Sub Type i.e. LoadTestException

Error count Number of each error type

Error message i.e. 404-NotFound

Graphs View

Table View

Errors-Displays a list of errors that occurred during the load test run

Pages-Displays a list of pages accessed during a load test run

Requests-Displays details for individual requests issued during a load test

SQL Trace-Displays the results of SQL tracing only if SQL tracing was enabled with right connection string

Tests-Displays details for individual tests run during a load test

Thresholds-Displays a list of threshold rule violations that occurred during a load

Transactions-Displays a list of transactions that occurred during a load test run

Load Test Result Repository Management

Information gathered during a Load Test run is stored in the Result Repository

Repository contains performance counter data and error data

You can manage your load result from the Load test editor Open Import Export Remove You do not have to go the dbase to clean up the repository

Defects Tracking Using TFS

Defects Tracking Using TFS Continues..

Reporting Using TFS—Remaining Work Items

Reporting Using TFS—Bug Rates

References

http://blogs.msdn.com/slumley/pages/how-to-debug-a-web-test.aspx

http://blogs.msdn.com/slumley/pages/web-test-correlation-helper-feature-in-orcas.aspx

http://msdn.microsoft.com/en-us/library/aa730850(vs.80).aspx