too dependent on shared test environments? kick start local workstation testing!

© ThoughtWorks, 2009

Too Dependent on Shared Test Environments? Kick Start Local Workstation Testing!

Stelios Pantazopoulos

Eric Liu


Expectations

• What would you like to get out of this workshop?

• Please write your expectation on a sticky note and place them on the board…


Agenda

• Challenges with shared test environments

• Local test builds

– Impact to overall test strategy

– Types of testing suitable for local test builds

– Requirements for local test builds

• Sample application walkthrough

– Local test examples with live application

• Technical

– Application architecture

– Build support


Local Dev Sandbox

IsolatedFull Access

Very Few Controls

Integrated Test Environment

SharedRestricted Access

Controlled

Test Workflow


Challenges Faced in Shared Test EnvironmentsHas the calculation fix been deployed to the QA environment? I’d like to re-

test it.

Sorry, the deployer is out for the rest of the

afternoon, so we’ll have to wait till tomorrow.


Challenges Faced in Shared Test Environments (2)I’d like to re-test the screen which an administrator should be able to get to,

but couldn’t.

The fix to that screen has been deployed, but we’re

still waiting for the administrator role to be added to your user in

Active Directory…


Challenges Faced in Shared Test Environments (3)How can I test the behaviour of the

system during the last hour of the day?

We’ll have to setup data and wait till 11pm…


Discussion

• What other challenges have you faced with a shared QA environment?



Local Test Sandbox

Local Dev Sandbox


Test Workflow



Local Test Sandbox

and


Impact on Overall Test Strategy

• Story Acceptance• User Interface and Usability• System Integration• Security Roles Based Access• Security Penetration• Exploratory• Data Migration• End-to-End Functional Integrated

• Performance/Load/Stress• Failover/Recoverability• Configuration• Production Release Dry Run• Compliance/SOX


Story Acceptance

Local Test Sandbox

• Run through each acceptance test scenario to ensure its correctness.

• Ensure that the data set and simulated integration points are complete enough for story acceptance to be done.

Integrated Environment

• Run through each acceptance test scenario to ensure it works in an integrated environment.

• Subject Matter Experts and the Business validate that a story implementation meet their needs.


User Interface and Usability

Local Test Sandbox

• Ensure that the look and feel is consistent.

• Ensure that the user interface meets the standards: for example: font size, positioning of screens.


• Smoke test to ensure that release did not affect the look and feel.

• Test conformity to user interface standards and that the end to end business processes are intuitive


System Integration

Local Test Sandbox

• Simulate flows of data into and out of the system via a simulated version of the integration point.


• Run the same test with the actual integration point.

• Test that the system’s interfaces to other internal and external systems work properly in isolation.


Security Roles Based Access

Local Test Sandbox

• For each role and capability, ensure system access to functionality and data is correct.


• Test the system’s interface to the security infrastructure where end user, their roles and their capabilities are stored and managed.

• Ensure internal and external end users have proper access to the system given their assigned roles and capabilities.


Security Penetration

Local Test Sandbox

• URL manipulation, manipulate post parameters, SQL injection attacks and cross site scripting.


• Smoke test which includes subset of full test done on local build.

• Ensure the system is safe from an attack to the system from an external party.


Exploratory

Local Test Sandbox

• Take advantage of having full control of the local test sandbox (application and data) to do unorthodox tests.


• The scope of Exploratory testing is limited due to the controlled nature of the environment.

• Unscripted testing where Business Users, Analysts and Expert Testers with a strong knowledge of the business, exercise the system.


Data Migration

Local Test Sandbox

• Focus on the relationships between the data.

• Focus on historical data.

• Witness how the application handles the data.


• Most of the testing can be done locally, with a production dry run in this environment

• Validate that data is equivalent between the legacy system and its replacement after the data migration has been performed..


End-to-End Functional Integrated

Local Test Sandbox

• Run an end of end business scenario where flows of data between internal and external integration points are simulated.


• Same test as local, except internal and external integration points are not simulated, and as close to production as possible.

• Business cycle test run in parallel with external systems. Tests ensure that operations with a cross system impact work properly.


Testing that cannot move to Local Test Sandbox

• Performance/Load/Stress

• Failover/Recoverability

• Configuration

• Production Release Dry Run

• Compliance/SOX


Requirements for Local Workstation Testing

• Fully-Functional and Self-Contained

• Allow Integration Points to be Locally Controlled

• Allow Complex Aspects to be Locally Controlled

• Example: CSV Files

• Example: Test Support Screens

• Rich Data Set

• Include Realistic Subset of Production Data

• Easy to Install


Sample Application Walkthrough

• Application: Movie Event Planner

• Summary: Plan movie events and allow interested parties to sign up. Integrates with external movie showtimes provider.

• Sample Requirements:

– As an organizer, I want to create an event so that others can sign up and go watch a movie together.

– As an attendee, I want to join an event so that I can go watch a movie with others.

– As an attendee, I want to be notified via SMS text message about an upcoming event 2 hours in advance.

– As an administrator, I want to retrieve upcoming movie locations and showtimes from a reliable source so that organizers can plan accurate events.


Discussion

• What are some challenges you foresee in testing these requirements?


Fire up Local Workstation Test Build!


Local Test Example: Story Acceptance

• Verify that I can’t join an event where movie has already started.


Local Test Example: Security Roles Based Access

• Verify that an attendee can join an event.

• Verify that an attendee cannot create events.

• Verify that an administrator can retrieve upcoming movie locations and showtimes.


Local Test Example: System Integration

• Verify that an attendee gets notified via SMS text message about an upcoming event 2 hours in advance.

• Verify that an administrator can retrieve upcoming movie locations and showtimes from a reliable source so that organizers can plan accurate events.


Application

Database

Application Architecture to Support Local Testing

CorporateDirectory

ExternalService


Application Architecture to Support Local Testing

• Local database

• Swappable external service implementation

• Swappable security implementation

• Swappable implementations for other complex aspects of system

• Domain layer should be implementation agnostic

– This may already be the case especially if developers are practicing unit testing

• Implementation choices controlled by configuration


Application

Database

Application Architecture to Support Testing

CorporateDirectory

ExternalService

Clock

Configuration

System


Application

Database

Application Configured to Support Local Testing

Clock

Security Groups

Local Store

Configuration

RichDataSet

External Data


Build Support for Local Test Builds

• Identify Good Build Candidates

– Compiles and Passes All Automated Tests

• Configure Application with Local Implementations

• Populate Rich Data Set

• Bundle for Installation and Stamp with Build #

• Publish Build

• Tip: Easily Automated through Continuous Integration


Process Improvement Analysis


Review Expectations


Questions?


Thank you!

• Stelios Pantazopoulos, [email protected]

• Eric Liu, [email protected]

mailto:[email protected]

mailto:[email protected]

too dependent on shared test environments? kick start local workstation testing!

Technology