mtlm visual studio 2010 alm workshop - day1
DESCRIPTION
MTLM Visual Studio 2010 ALM workshop - day1 deckTRANSCRIPT
DAY 1 MTLM
Testing Practices with ALM tools
Tune the value of MTLM, its capabilities and the processes it supports…
GOAL deep dive:
Know the value of MTLM and know how to use its capabilities …
GOAL general course:
Agenda Day 1 Start Topics
9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration 10:45 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM
Lot of discussions and hands-on lab’s
Environments setup TFS supported operating system Installed: • Team Foundation Server with Basic configuration • Visual Studio 2010 with SP1 • Microsoft Test Manager 2010 (comes with VS10 Ultimate)
You get an USB with: • Demo applications (we create a Team Project for them, see lab 1)
• Microsoft Test Agents.msi (we install this together)
• Hands-on lab documents • This deck, talk through and useful information
Lab 1
Create one team projects for the demo applications
Timebox: 15 minuts
People Introductions
Clemens Reijnen Sogeti Netherlands | Management Consultant http://www.ClemensReijnen.nl @ClemensReijnen
Start Topics
9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration 10:45 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM
Application Lifecycle
Management
Understand the term ALM, know it’s value and Visual Studio’s ALM support
GOAL:
Suite of Application Lifecycle Management tools
http://msdn.microsoft.com/en-us/library/fda2bad5.aspx
application lifecycle management
http://www.clemensreijnen.nl/post/2008/02/18/ALM-Definitions.aspx
work together
__with work packages __on alm artifacts
Testing Lifecycle
See also on www.tmap.net: - The Test Lifecycle Model - The 4 essentials
http://msdn.microsoft.com/en-us/library/bb385832.aspx
Start Topics
9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration 10:45 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM
Work items
Test Activities Planning…
Testing Lifecycle
See also on www.tmap.net: Planning Phase
http://msdn.microsoft.com/en-us/library/bb385832.aspx
VS2010 Process Support with Process Templates M
ult
iple
pro
cess
tem
pla
te
Artifact types
Work items
Reports
Workbooks
Dashboards
Metrics
User Story Work Item Type
Work Item Type Flow
User Story
Testing processes in the Lifecycle
Testing processes in the Lifecycle
Process template with work items, flows, links, queries and more
support | drive | enforce a way of work
Lab 2
Create user stories (2 or 3) and development and test tasks (optional) for the employee application.
Timebox: 5 minuts
Workitem Queries
and Reports
Queries
create your own
Create your own – advanced –
Query Types
Default Reports
Excel Reports
create your own
Excel basic
create your own
Report Wizard Excel Pivot
Test Activities Planning…
See also on www.tmap.net: Planning Phase
Microsoft Test Manager
Microsoft Test Manager 2010 The UX
Start Topics
9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration 10:45 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM
Test Case Management
See also on www.tmap.net: Control Phase
Test Plans Test Suites Test Configurations
Organize the testing effort in a Test Execution Plan Create dynamic test suites for changing selection criteria Define various test configurations and share automation across those
Test (Execution) Plans
Test (Execution) Plans
Test Plan Organization Naming of test plans for agile environments: Iteration number and name and one extra test plan for end to end tests.
Test Configurations
Test Configurations
Test Suites
Test Suites Organization Naming of test suites for agile environments: Suite per user story (requirement based) and one additional test suite for regression tests.(query based?) (test cases are copied to the regression test suite)
Proposed organization
Iteration 1 Create test plan for iteration and a ‘end to end’ test plan. Create requirement based test suites which hold test cases for that suite.
Iteration 1 Create two additional suites, one for regression sets, which will be taken along during the project and one specific for automation.
Copy Test Cases
ID 21 ID 21
ID 22
Copy a Test Case
ID 21
Copy the reference to a Test Case
End iteration 1, some important test case are promoted to the ‘end to end ’ test plan.
Iteration 2, test suites are created for the requirements and the regression test suit is copied (link or hard?)
During the execution of iteration 2 the regression test suit grows and an automation suit is added.
Again important ones are promoted to the ‘end to end’ test plan
TCM.exe
TCM plan /list
TCM suites /lists planid:16
Lab 3
Create Test Plan, Test Configurations and make a test suite organization. Optional: use TCM.exe to list the lab results
Timebox: 10 minuts
Share MTM Artifacts
mtm://clemenes-pc:8080/tfs/defaultcollection/p:MSF/Testing/testplan/connect?id=19
mtm://server:8080/tfs/ProjectCollection/p:Project/testing/testcase/open?id=67
Open Test results Open Test Cases (or any work item) Open Test Runs Open Test Plans Connect to test plans
mtm://<server name>:<port>/<tfs vdir>/<Collection name>/p:<project name>/<center group>/<group specific>
http://blogs.msdn.com/b/vstsqualitytools/archive/2010/02/09/sharing-content-in-microsoft-test-manager.aspx
Start Topics
9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration 10:45 BREAK 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM
BREAK TILL:
11:00
(extra time to finish the labs :)
Start Topics
9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration 10:45 BREAK 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM
Test Case Specification
See also on www.tmap.net: Specification Phase
Test Case Shared Steps Test Data Iterations
Link test cases and requirements to provide requirement traceability Data drive test cases with different parameters Refactor common shared steps in test cases
Test Case
Test Case Naming of test case: Purpose: • Finding the right test case • Common understanding • Intention from the title • Scan test plans more effectively and efficiently
Proposed solution #1 make use of linked work items #2 maximal length of the title 20 characters #3 format <functionality identifier> <action> <additional information> #4 make useful queries
Test Case from Excel
Test step types
Test steps Make them descriptive Not every step is a validation step Use the step attachment
Test steps copy-past
Test Data Iterations Data drive test cases with different parameters
Test Data Iterations
Test data copy-past
Shared Steps Refactor common shared steps in test cases Reduce test maintenance by sharing test steps across test cases
Shared Steps Shared Step candidates?
• REUSE – Many steps, reused for many test cases . Benefit: time, retyping, boring work.
• EXPECTED CHANGES – for example environment variables change [URL]. Benefit: Time, test cases
which need this step are updated automatically, also action recording.
• PRE- CONDITIONS – bring the system under test to a certain state and execute the different
verifications. Benefit: beside time, you are certain the application under test is in the same state for every test case
• BACK TO INITIAL STATE - after test execution bring the environment back to the initial state for
new test cases. Benefit: beside time, you are certain the application under test is always in a clean state for new test cases.
Shared Steps what about parameters and expected result in shared steps?
Lab 4
1. Specify multiple test cases. 2. Add some test data iterations. 3. Create shared step and reuse it.
Optional: play with the parameters and expected result steps in the shared steps…
Timebox: 15 minuts
Start Topics
9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration 10:45 BREAK 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM
Test Execution
Test Runner
A test that you will want to run multiple times. Different manual tests that contain common steps. Verifying a bug is fixed by using a manual test.
Test Case Execution
See also on www.tmap.net: Execution Phase
Test Case Execution
Action recording
TCM run /create
Test Case Execution Results
View Test Results Using Microsoft Test Manager 1: from the ‘Run Tests’ 2: from the ‘Analyze Test Runs’
From Analyze Test Runs Test Results for multiple Test Cases
From Run Tests Test Result for a Test Case
Test Plan
From TCM run /export id:239 /resultsfile:”c:\runresult”
Share with mtm://
mtm://clemens-pc:8080/tfs/defaultcollection/p:MSF/Testing/testrun/open?id=292
Lab 5
Execute the created test cases, view the results Optional: run them from TCM.exe
Timebox: 10 minuts
Test queries
Test reports
Excel Reports for Tests
• POINT COUNT TREND - I want to see the results for test in a particular test run
• RESULT COUNT - I want to see trend report for my tests regardless of build or plan
• RESULT COUNT TREND - I want to see trend report for my tests regardless of build or plan
• TEST CASE COUNT - I want to see the number of test cases that have been planned for automation
Microsoft Test Manager Test Artifact organization
Test Plans and TFS items
Server
TCM Server Plan “Iteration 2” Suite US1 Suite US2
Test Case A Test Case B Test Case C Test Case D
MTLM Client
Plan “Iter 1”
Plan “Iter 3”
TFS
User Story 2
Test Case A
Test Case B
Test Case C
Task X
Task Y
Task Z Test Case C
Lab 6
Create excel pivot table to list test points per Tester.
Timebox: 10 minuts
Start Topics
9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration 10:45 BREAK 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 Lunch 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 Break 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM
LUNCH TILL:
12:45
(extra time to finish the labs :)
Start Topics
9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration 10:45 BREAK 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 Lunch 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 Break 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM
Controlling the testing effort BUG’s
File a BUG
Diagnostic Data Adapters File a Bug File an Exploratory Bug Verify a Bug
Diagnostic Data Adapters
File a Bug Submit a Bug using Test Runner
Ways to file a Bug
Submit a Bug from the Test Runner Submit a Bug from the Verify Bugs View Submit a Bug from Test Results for a Test
Verify a Bug Verify a Bug Fix Using Microsoft Test Manager
The Bug Flow
T
E
S
T
D
E
V
Link build 2 test plan. Configure Data collectors.
Setup build and deploy
Build definition is
the intergration
Execute Manual Tests
File Bug
Resolve Bug
Checkin Build Deploy
Verify fix by using playback
Close or reopen Bug
Lab 7
Create a build File a bug Resolve a bug Verify a bug
Timebox: 15 minuts
Bug Reports
Bug Status Report Bug Trends Report Reactivations Report
Bug Status Report
Bug Trends Report
Reactivations Report
Start Topics
9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration 10:45 BREAK 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 Lunch 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 Break 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM
Test Impact
T
E
S
T
Execute ‘passed’ Tests
Execute ‘passed’ Tests
Test Impact
D
E
V
Assign build 2 test plan. Configure Data collectors.
Setup build and deploy
Build definition is
the intergration
Execute ‘passed’ Tests
Make code changes
Checkin Build Deploy
Assign ‘NEW’ build 2 Test Plan
Evaluated and reset tests impacted by code churn
Lab 8
Attach build to test plan Execute tests Make code change Kickoff build Attach new build to test plan Evaluate impacted tests Timebox: 15 minuts
E2E Scenario
Iteration N
T
E
S
T
Sprint plan
Write tests for US1
Write tests for US2
D
E
V
Sprint plan
Build 1 Build 2 Build 3
Implement User Story ‘1’
(US1)
Test US1 & file bugs
Implement US2
Build 4
Fix bugs
Build 6 Build 7
Regress impacted
tests
Build 5
Implement US2
Fix bugs
Test US2 & file bugs
Verify fixes
Verify fixes
T
E
S
T
Sprint plan
Write tests for US1
Write tests for US2
D
E
V
Sprint plan
Build 1 Build 2 Build 3
Implement User Story ‘1’
(US1)
Test US1 & file bugs
Build 4
Fix bugs
Build 6 Build 7
Regress impacted
tests
Build 5
Implement US2
Fix bugs
Test US2 & file bugs
Verify fixes
Verify fixes
Sprint plan
Write tests for US1
Sprint plan
Build 1 Build 2 Build 3
Implement User Story ‘1’
(US1)
Test US1 & file bugs
Sprint plan
Write tests for US1
Key Value: Requirements Planning
Build 3
Implement User Story ‘1’
(US1)
Test US1 & file bugs
User Stories and corresponding done dev tasks. Create test tasks and assign to someone else.
T
E
S
T
Sprint plan
Write tests for US1
Write tests for US2
D
E
V
Sprint plan
Build 1 Build 2 Build 3
Implement User Story ‘1’
(US1)
Test US1 & file bugs
Build 4
Fix bugs
Build 6 Build 7
Regress impacted
tests
Build 5
Implement US2
Fix bugs
Test US2 & file bugs
Verify fixes
Verify fixes
Sprint plan
Write tests for US1
Sprint plan
Build 1 Build 2 Build 3
Implement User Story ‘1’
(US1)
Test US1 & file bugs
Sprint plan
Write tests for US1
Key Value: Requirements Planning
Build 3
Implement User Story ‘1’
(US1)
Test US1 & file bugs
Execute the Test Specification Tasks and other test tasks
T
E
S
T
Sprint plan
Write tests for US1
Write tests for US2
D
E
V
Sprint plan
Build 1 Build 2 Build 3
Implement User Story ‘1’
(US1)
Test US1 & file bugs
Build 4
Fix bugs
Build 6 Build 7
Regress impacted
tests
Build 5
Implement US2
Fix bugs
Test US2 & file bugs
Verify fixes
Verify fixes
Sprint plan
Write tests for US1
Sprint plan
Build 1 Build 2 Build 3
Implement User Story ‘1’
(US1)
Test US1 & file bugs
Sprint plan
Write tests for US1
Key Value: Requirements Planning
Build 3
Implement User Story ‘1’
(US1)
Test US1 & file bugs
Execute the Test Execution Tasks and file bugs (if any)
T
E
S
T
Sprint plan
Write tests for US1
Write tests for US2
D
E
V
Sprint plan
Build 1 Build 2 Build 3
Implement User Story ‘1’
(US1)
Test US1 & file bugs
Build 4
Fix bugs
Build 6 Build 7
Regress impacted
tests
Build 5
Implement US2
Fix bugs
Test US2 & file bugs
Verify fixes
Verify fixes
Test US1 & file bugs
Fix bugs
Verify fixes
Build 3 Build 5
Fix bugs
Test US1 & file bugs
Build 5
Verify fixes
Key Value: Virtuous Bug Cycle Dev solves the bugs. Test Verifies tests
T
E
S
T
Sprint plan
Write tests for US1
Write tests for US2
D
E
V
Sprint plan
Build 1 Build 2 Build 3
Implement User Story ‘1’
(US1)
Test US1 & file bugs
Build 4
Fix bugs
Build 6 Build 7
Regress impacted
tests
Build 5
Implement US2
Fix bugs
Test US2 & file bugs
Verify fixes
Verify fixes
Key Value: Virtuous Bug Cycle Check Results
Start Topics
9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration 10:45 BREAK 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 Lunch 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 Break 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM
BREAK TILL:
14:30
(extra time to finish the labs :)
Start Topics
9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration 10:45 BREAK 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 Lunch 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 Break 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM
Create test from Bug
File an Exploratory Bug Submit an Exploratory Bug using Test Runner
Record scenario by business user with MTM on application under test. Action recording is captured.
1
Create test steps (with parameters) in MTM by using the action recording.
2
Save test case and execute it by a tester on application under test, by using MTM.
3
CON: - Manual actions and tests. - Test steps text are technical names - Action recording (step 1) must be clean. - Unknown what is validated, during step 2, need
additional knowledge. - The business user must think of the important
scenarios The action recording during step 1 can have unnecessary steps, these can be removed during the execution (1) or during the specification (2).
PRO: - Reuse scenario knowledge business user - No typing detailed test steps - Easy to parameterize - Writing standards for test steps - MTM capabilities in step 3. (bug, codedUI,
recommended test etc …) Most benefit during step 2 (test case writing), benefit in step 1: run once in common situation. Benefit in step 3: common MTM benefits (record playback, fast forward, parameters etc) and benefit of well written steps.
Lab 9
Timebox: 10 minuts
Create an exploratory bug and create a test case from it
Start Topics
9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration 10:45 BREAK 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 Lunch 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 Break 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM
Test Infrastructure
Where and how to run the automated test?
Test Environments
See also on www.TMap.net: - Checklist "Intake Test Environment“ - Checklist "Test Environment“ - Infrastructure Phase
Test lab Visual Studio 2010 Lab Management
physical test environments virtualized test environments
Where and how to run the automated test?
TFS 2010
Physical Test Client
MTLM Client
Physical Test Client
Hyper-V Host
VM
System Center Virtual Machine Manager (SCVMM)
TFS 2010
VM
Library Shares
share
share
MTLM Client
physical test environments
virtualized test environments
Build application
Provision ‘clean’ env
Deploy application
Run tests
physical test environments versus virtualized test environments
Start Topics
9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration 10:45 BREAK 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 Lunch 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 Break 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM
Visual Studio
Test Manager
Team Foundation Server
Build Server
Test Agent
Test Agent
Test Agent
Lab for Physical Environments, the different flavors…
A
Flavor A: Execution from VS2010…
Configurations for automated test execution:
Real developer test / test automation dry run (Triggered by: right mouse click – run tests, in test project) http://msdn.microsoft.com/en-us/library/dd286580.aspx No additional configuration needed* * Data Diagnostic Settings in the Local.Testsettings file are configurable.
Pro: Easy setup
Debug-able test automation
Con: No collection of test results in TFS Run on developer environment
B
Flavor B: Execution during build with Build controller…
Configurations for automated test execution:
Part of Build Verification Tests (triggered by: build) Set the Build Service to run in interactive mode http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx Configure the build to run the tests http://msdn.microsoft.com/en-us/library/ms182465.aspx#CreateBuildType * Data Diagnostic Settings in the *.Testsettings file are configurable.
Pro: Easy setup
Test results in build result
Con: No collection of test results in TFS Build Controller needs to run in interactive mode Tests are executed on build environment Run on build environment
not recommended, strange to run UI tests on build
C
Flavor C: Execution during build with Test controller…
Configurations for automated test execution:
Part of Build Verification Tests on multiple test agents (triggered by: build) Configure Test Controller (don’t register it with a project collection ) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers Configure Test Agents on clients (interactive mode) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents Configure *.Testsettings file in solution to use Test Controller and Test Agents Configure the build to run the tests (see B)
Pro: Test run on test environments
Tests run on multiple environments Test Results in Build result
Con: No collection of test results in TFS Harder to configure Need for specific test client environments Test Settings from VS
Preferred configuration above flavor B
D
Flavor D: Execution from Microsoft Test Manager…
Configurations for automated test execution:
Part of Regression Tests (triggered by: MTM user, right mouse click on test case, run) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. http://msdn.microsoft.com/en-us/library/ee390842.aspx http://msdn.microsoft.com/en-us/library/dd293551.aspx Associate CodedUI test with WI Test Case from VS. http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html
Pro: Test run on test environments
Tests run on multiple environments Test Result in MTM and TFS
Test Settings from MTM
Con: Manually triggered by Tester Hard to configure Need for specific test client (same as MTM?)
Other type of test than BVT
E
Flavor E: Execution from MTM during Build…
Configurations for automated test execution:
Part of BVT (triggered by: Build) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. Associate CodedUI test with WI Test Case from VS. Create Build task to run TCM or MSTEST task for Test Plan http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx
Pro: Test run on test environments
Tests run on multiple environments Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con: Hard to configure Need for specific test client (same as MTM?)
Preferred configuration above flavor C Flavor D and E can be configured together
Test Controller
Test Agent
Lab 10
Install and configure the test controller and test agent
Timebox: 20 minuts
http://msdn.microsoft.com/en-us/library/dd648127.aspx http://www.microsoft.com/downloads/en/details.aspx?FamilyID=a3216d2a-0530-4f6c-a7c9-0df37c54a902
Start Topics
9:00 Introduction, environments setup 9:15 introduction by the students 9:45 Application Lifecycle Management and Visual Studio General 9:55 Test Planning 10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration 10:45 BREAK 11:00 Specify test cases and corresponding capabilities of MTM 11:45 Test execution, fast forward, reporting, olap cub 12:00 Lunch 12:45 Bug workflow, infrastructure configuration for data diagnostic adapters 13:30 Test impact and recommended tests 14:15 Break 14:30 Exploratory testing and create test from bug 14:45 Test Controllers and Agents 15:45 Test Environments in MTLM
New physical environment
Test Activities Planning…
RECAP
Test Case Management RECAP
Test Case Specification
RECAP
Test Case Execution
RECAP
Test Case Management RECAP
Start Topics
9:00 Recap
9:15 Create Basic CodedUI, from Test Case and Manual
9:45 UIMap and customize the code for optimization
10:15 Data driven Tests and assertions
10:45 break
11:00 Troubleshoot CodedUI, Common Practices and questions
11:30 Configure the build to execute the CodedUI tests, test settings from VS2010, Execute from VS2010 with test agents and analyze the results
11:30 Associate test automation with test case, execute automated test case from MTM, configure the build to run from the build within MTM and analyze the results
12:30 break
13:15 MTLM, Scrum and other project Methodologies
13:45 Lab Management - Virtualized Test Environment and execution Demo
14:15 Test info in the OLAP Cube
14:30 Extensibility - test Scribe, TCMIMport
14:45 Create your own Data Diagnostic Adapter
15:15 MTLM usages with Azure projects
16:00 Closing
DAY 2