a software tool to assist scenario testing of...
TRANSCRIPT
SCHOOL OF COMPUTER SCIENCE
ANU COLLEGE OF ENGINEERING & COMPUTER SCIENCE
A Software Tool
to Assist Scenario Testing of
Websites
Student: Siqi Fan (u4761704)
Supervisor: Assoc Professor Chris Johnson
Course Code: COMP8790 – Software Engineering Project
31th October, 2010
A Software Tool to Assist Scenario Testing of Websites
Pag
e1
Pag
e1
Acknowledgement
First and foremost I would like to offer my sincerest gratitude to my supervisor,
Assoc Professor Chris Johnson, who has supported me throughout of my thesis with
his patience and knowledge. In the various research works and laboratories along the
entire project, I have been also aided in setting the research direction by him. Without
Chris Johnson‟s kind support and encouragement this thesis would not have been
possible.
Secondly, I would like to express my thankfulness to all my previous colleagues
in the Office International Publishing Group (OIPG), Microsoft Ireland. It was them
who created the heuristic work environment and offered me the opportunity to receive
the fundamental knowledge and experience of a large scale business oriented website
production.
Finally, I would like to thank my beloved parents, wife and friends for their
understanding and endless love throughout all my studies.
A Software Tool to Assist Scenario Testing of Websites
Pag
e2
Pag
e2
Abstract
Effective prioritization on tremendous amount of defects can ease the pain for
large scale business oriented websites‟ production. The severity of defects plays a
critical role in assisting defect prioritization. However, the traditional way of ranking
severity lessens the significance of the original purpose, especially facing a large
amount of defects with limited developing and testing resources. A thorough study has
been done for discovering the way to represent severity in a more distinguishable
form and the way to estimate severity in a more systematic manner. The analysis
result shows that three factors (impact, importance and likelihood) are identified as
having significant referential value in evaluating severity but these factors are
measured according to different perspectives (scenario, quality and usage). Each
perspective has been further studied respectively, especially the two aspects of
website quality and their implicit gap. To consider these views while assessing
severity in a systematic manner, a theoretical framework is proposed to estimate
defect severity by utilizing the structural view of predefined scenarios and deriving
testing tasks from scenario related quality dimensions. The three sub factors‟
classification scheme and the severity representation are defined to assist the
framework to represent severity in a more distinguishable form. A prototype is built to
prove the feasibility to adopt the framework into the practical website scenario testing
tools.
A Software Tool to Assist Scenario Testing of Websites
Pag
e3
Pag
e3
Table of Contents
Acknowledgement ........................................................................................................ 1
Abstract ......................................................................................................................... 2
1. Introduction .......................................................................................................... 4
2. Severity and its three sub factors ........................................................................ 7
2.1. Impact on Scenario ..................................................................................................... 8
2.1.1. Case Study .......................................................................................................... 8
2.1.2. Scenario ............................................................................................................ 11
2.2. Importance ................................................................................................................ 13
2.2.1. Quality .............................................................................................................. 13
2.2.2. Case Study ........................................................................................................ 22
2.3. Likelihood ................................................................................................................ 24
2.3.1. Case Study ........................................................................................................ 26
3. Theoretical Framework and Severity Classification Scheme ......................... 26
3.1. Building the Framework........................................................................................... 27
3.2. Severity Classification Scheme ................................................................................ 31
3.2.1. Impact ............................................................................................................... 32
3.2.2. Importance ........................................................................................................ 33
3.2.3. Likelihood ......................................................................................................... 34
3.2.4. Representation of severity ................................................................................ 34
3.3. Case Study ................................................................................................................ 34
4. Prototype ............................................................................................................. 40
5. Results and Discussion ....................................................................................... 43
6. Related Works..................................................................................................... 51
7. Conclusion and Future work ............................................................................. 61
8. Bibliography ........................................................................................................ 62
A Software Tool to Assist Scenario Testing of Websites
Pag
e4
Pag
e4
1. Introduction
Nowadays business oriented websites are becoming increasingly complex in three
dimensions, quantity of content, functionality of features, and diversity of language,
while the production of this kind of website is experiencing painfully. The content of
the website needs to be frequently added, removed, and updated for providing
up-to-date information. The features of the website need to be enriched for enhancing
interaction with site users. The languages supported by the website need to be
increased for expanding the global business. The trend can be driven by various
business objectives not mentioned above.
However, this paper is not meant to evaluate or to predict the trend but to focus
on what is happening behind the scene. The production of this type of websites in
industry is becoming to be an endless cycle. The business analysts add and track the
customer requirements aligning to the business strategies. When a new requirement is
added, the authors create the new content or update the existing content. Meanwhile
the product manager and website designers design the new features and refine the
website layout and theme. The developers then develop and maintain the site features
and content publishing tools. The testers launch testing when the new version website
is assembled. They assign severity to each defect caught during testing and report the
defects to the project managers. The project managers then prioritize the defects and
assign the defects to the developers. The developers pick up the defects to investigate
and to fix according to the priority level set by the project managers. When a defect is
assumed to be fixed, the developer will assign the defect back to the testers for
verifying. The testers will close the defect if it is fixed or otherwise reassign back to
the developer. When the scheduled releasing date is approaching, the project
managers will triage the open defects and reprioritize them to avoid delivering critical
defects to the users. Although all seems to be blameless on the surface so far, it is
worth having a closer look at what they are suffering from.
A Software Tool to Assist Scenario Testing of Websites
Pag
e5
Pag
e5
During the production, defects are often captured and counted in a significantly
high volume. The previous version Office Online of Microsoft Office.com [19]
reported more than 15,000 defects during 3 years production. Generally the reason is
not just because of the poor production activities, unavoidable human errors, or low
quality production tools but the accumulatively nature (If the defects cannot be fixed
by developers quick enough before the testers discovering new defects, the total
amount of defect count will be accumulated into a very high number.) and the fast
change of the business needs. High volume of defects‟ occurrence cannot help but
leave panic to the developers. Many high priority defects flowing into developers‟ bug
account soon bewilder the developers. “They are all high priority defects and which
one should I choose to fix at first?” Hearing such a voice, the questions need to be
answered that why the priority loses its meaning and how to get the defects truly
prioritized.
To answer the question, it is necessary to go back to see where the priority is
coming from. Priority is set with main consideration in severity in addition to taking
some other project factors into consideration such as impact on business value,
available developing and testing resources, business decisions and so on. Priority is
ranked subjectively to describe the relative order the defect should be investigated and
resolved. While severity is set according to certain guidelines defined in the
organizations after evaluating the defect usually from website quality perspective.
Severity is ranked to describe how serious the impact of the defect is on the site users.
There is no standard to instruct who should set severity and priority. In industry,
severity is usually ranked by testers and priority is set by project managers.
Although the priority is assigned to the defects, the situation occurs often that
many defects cannot be fixed when scheduled release date is approaching. Commonly
the cause is not inefficiency of the development team or the immature technical skill
level but the facts that defect fixing rate is lower than the defect reporting rate and
ineffective prioritization.
A Software Tool to Assist Scenario Testing of Websites
Pag
e6
Pag
e6
If there is no way to avoid the situation, there is at least some way to ease the pain.
Effective prioritization can assist developers to tackle the most important defects
within limited resources rather than placing the triage team in a dilemma. The severity
of defects is the key in deciding priority. Although it is not the only factor in
determining priority, severity should provide the greatest reference value to other
factors. In order to make it be fuller in meaning, severity should be represented in a
more distinguishable form and set in a more systematic manner.
Following the intuitive idea, this paper adventures an opinion to decompose
severity into three sub factors impact, importance, and likelihood based on three
circumstances observed. The three factors and cases are considered in detail in section
2. Impact in this context is the consequence brought from the defect upon the process.
The process is one particular users‟ activity on the website, to fetch information (for
example, read news), to receive services (for example, order product), and so on.
Fortunately, using a scenario is probably the best way to describe the process.
Scenario is an ordered set of interactions between the website user and the
information and the functionalities (features) of the website. Scenario becomes the
author‟s research focus. Its importance is that scenario can provide a systematic
overview of the website, which offers great support to judge the impact objectively.
On the other side, importance is to describe the relative order of quality factors. This
relationship is emphasized and applied to single scenario after researching website
quality. It is discovered that website quality has two aspects, the customers' quality
expectation and the website providers' quality expectation. These two aspects and the
gap between them are often blurred. This paper proposes a way to narrow the gap by
defining the deliverables according to quality factors and associating the testing tasks
with the deliverables. It assumes the failure of certain testing affect the corresponding
quality factor directly. Hence, the relative quality impact of the defect can be
objectively measured. Last but not least, likelihood is inspired by risk analysis to
reflect the end user impact scope of defects. Existing website users‟ behavior
A Software Tool to Assist Scenario Testing of Websites
Pag
e7
Pag
e7
statistical analysis is feasible to deliver the objective justice. It is suggested to
consider the factor for justifying the severity level. A theoretical framework is
proposed to conduct three sub factors to be considered into scenarios. The framework
includes predefining the testing tasks for each step of the scenario, evaluating the
three sub factors based on the classification scheme before testing, and representing
defect severity in a more distinguishable form. A prototype is implemented and the
result proves the feasibility to adopt the framework into practical use.
The rest of paper is structured as follows. The theoretical framework, the details
of the classification scheme of three sub factors, and the severity representation are
discussed in section 3. The entire framework is adopted and implemented in a
prototype introduced in section 4. The implementation result of the prototype is
discussed in section 5. The related work from other researchers is commented in
section 6. The paper is summarized and the future work is pointed out in section 7.
The scenario cases in this paper are selected from a specific website Microsoft
Office.com [19].
The objective of the present research is to build a theoretical framework for
taking good advantage of the systematicness of scenario to evaluate severity of defect
objectively. It is meant to light up the direction for further investigation on the areas
such as identifying the sub factors of severity, narrowing the gap between customers‟
quality expectation and website providers‟ quality expectation, identifying the
deliverables and the testing tasks, and adopting the framework into practical tools.
2. Severity and its three sub factors
IEEE standard 1044-2009 defines “Severity is the highest failure impact that the
defect could (or did) cause, as determined by (from the perspective of) the
organization responsible for software engineering” [0]. Different organizations have
different understanding of severity. It is easy to understand as different projects and
A Software Tool to Assist Scenario Testing of Websites
Pag
e8
Pag
e8
products have different objectives and orientations. In a project building a human life
critical system, severity of defects is to measure the impact on the end users. In a
project building a complicated workflow product, severity of defects is to measure the
impact on the operational system. Whereas in a project that building a large scale
business oriented website, severity of defects is suggested to measure the impact on
the scenarios, the importance of the website quality dimensions, and the usage of the
pages and their elements. This section will guide you to understand the points listed
below:
Why these three factors need to be considered?
How is impact measured?
Why scenario plays a critical role in measuring severity?
What is the relationship between scenario and three sub factors?
Why scenario testing is the potential testing activity to measure severity?
What are the two aspects of website quality?
What is the order relationship of website quality factors?
How to narrow the gap between the two aspects of website quality?
How is importance measured?
How is likelihood measured?
2.1. Impact on Scenario
Impact is an ambiguous term. Before determining what object the impact should
be upon, let us consider a situation.
2.1.1. Case Study
A defect is captured during testing that the “Download” button as pointed by the
black arrow in Figure 1 does not behave properly when user clicking the button. The
image selected by the user cannot be downloaded. Another defect is captured that the
“Download” button as pointed by the black arrow in Figure 2 is out of function as
A Software Tool to Assist Scenario Testing of Websites
Pag
e9
Pag
e9
well. The PowerPoint template selected by the user cannot be downloaded. The
traditional way of evaluating severity will rank both defects in the same level that
describe the defect stop the downloading operation. However, if the defects are
considered in a scenario context, the relative order can be distinguished.
Let us briefly describe the scenarios for the both cases. The first scenario is a user
would like to find an image from the website and to use the image in his PowerPoint
slides. He browses the image pages and selects the image he likes. (1) He clicks the
“Copy to Clipboard” button on the page and to manually paste the image in his slides.
(2) He clicks the “Download” button on the page, saves the image on the local disk,
and inserts the image manually into his slides. (3) He clicks the “Add to Basket”
button on the page and decides to search for more similar images. He is not happy
with other images and clicks the “download” link in the pop up dialog illustrated in
Figure 3, saves the image on the local disk, and inserts the image manually into his
slides. The second scenario is a user would like to find a PowerPoint template and to
use the template in his PowerPoint application. (1) He clicks the “Download” button
on the page, saves the template on the local disk, and the template is automatically
loaded in his PowerPoint application. As you can see from the first scenario, there are
three paths to achieve the user‟s goal. In another word, there are two ways to walk
around if the “download” button does not work. However, in the second case, the
broken “download” button will block the scenario that the user will have no way to
achieve his goal. Therefore, should it be reasonable to differentiate the relative order
of two defects that the broken “download” button in the “templates” page should be
investigated and resolved earlier than the one in the “images” page.
A Software Tool to Assist Scenario Testing of Websites
Pag
e10
P
age1
0
Figure 1 the "Download" button is reported as a defect in the "images" sub site.
Figure 2 the "Download" button is reported as a defect in the "templates" sub site.
A Software Tool to Assist Scenario Testing of Websites
Pag
e11
P
age1
1
Figure 3 Dialog is popped up after click “Add to Basket” button
2.1.2. Scenario
From the case study above, it is noticeable that the sub factor impact is measuring
upon the scenario. Considering impact on scenario gives a big lift on differentiate the
relative order of defects.
IEEE Standard 1044-1993 [1], the ancient version of IEEE Standard 1044-2009,
once mentions “The impact of an anomaly shall be considered at each step of the
anomaly process.” Process is the word describing a formal path in a limited scope for
the large scale business oriented website, a derivative of traditional software or web
application. The large scale business oriented website is a platform of information
exchanging and services providing. The user or the customer has freedom to use the
website (for examples, jumping between links, querying the searching engines,
interacting with the controls and so on) to achieve his purposes or leave alternatively
to visit other websites. Like a customer in a supermarket, he has many ways (for
example, following the promotion advertisements, searching by category signs, asking
for assistance, wandering around and so on) to find the products he want or to leave
for other shops. Focusing on a single path (process) is not enough to justify the impact
A Software Tool to Assist Scenario Testing of Websites
Pag
e12
P
age1
2
from a systematic view.
From the other hand, using scenarios is probably the best way to describe how the
user will access the information and the functionalities (features) of the website to
illustrate the entire picture. Scenario is reviewed thoroughly in the section 7 related
works. There are two key points in Ryser‟s and Glinz‟s work [8] need to be
emphasized. One is the predefined scenarios during requirement elicitation and
analysis phase have the critical value for requirement validation and verification.
Another is scenarios can help to choose appropriate testing strategy and testing
techniques to make decisions in allocating resource in testing plan.
In the first point, the customer requirements include not only the functional but
also the non-functional requirements. And the non-functional requirements in natures
are the customer quality expectations which will be discussed in the section 2.2.
Hence, ideally, a testing activity should cover both functional and non-functional
testing. The suitable testing phases [2] are System Testing, User Acceptance Testing,
and Pilot Testing. In this paper, System Testing phase is only considered as the phase
is the earliest stage which is capable of launching both functional and non-functional
testing. The typical testing activity in System Testing is the scenario testing. Literately
speaking, the scenario testing is tests based on scenarios, the hypothetical stories [4].
The test cases are built based on the scenarios. Each step in the test case can be
matched with the user behavior on the website (for example, click a link, click a
button, and many other mouse events and control interactions). Details of scenario
testing will be discussed in section 3.
In the second point, it implicitly emphasizes that scenarios provide a systematic
and structured view so that the right testing strategy can be selected, the proper testing
techniques can be applied, and the reasonable effort can be allocated. Regarding the
effort allocation, it is related to another sub factor likelihood which will be discussed
in section 2.3.
To sum up, this section analyzes a real life example to reveal the relative order for
A Software Tool to Assist Scenario Testing of Websites
Pag
e13
P
age1
3
defect handling through considering the defect impact on the scenario. Meanwhile,
scenario testing and the interrelationship between scenario and the other two sub
factors are introduced with further research into scenario.
2.2. Importance
Defects have implicit relationships with quality. In industry, it is a common
understanding that defects with higher severity level have deeper impact on quality.
Although the statement is not wrong, it lacks transparency of the inside view of the
relationship between severity and quality. This section will review the two aspects of
quality, the website quality dimensions, the ordering relationship of website quality
dimensions, and the approach to associate the ordering relationship with severity
measuring.
2.2.1. Quality
Business oriented websites play center role to the business success. High quality
website leads to customer satisfaction which is essential for customer to open a door
to commit to the company or organization. Therefore the quality of the website needs
to be seriously concerned.
After researching on website quality, it is discovered that website quality has two
aspects, the customers' quality expectation and the website providers' quality
expectation, which are not often distinguished and discussed. The customers‟ quality
expectation is subjective to the website design, the content and features on the website,
and the personally experiences of using the website. Loiacono, Watson and Goodhue
identified 13 quality dimensions listed in the Table 1 in their study [14]. The business
analysts are responsible for gathering the customer quality expectation (for example,
via surveys or interviews to conduct marketing research) and integrating the quality
expectation into the requirements.
A Software Tool to Assist Scenario Testing of Websites
Pag
e14
P
age1
4
Quality
Category
Quality
Dimension Description
Ease of use Ease of
understanding
Easy to read and understand.
Intuitive
operations
Easy to operate and navigate.
Usefulness in
gathering
information
Information
Quality
The concern that information provided is accurate,
updated, and appropriate.
Tailored
Communications
Communications can be tailored to meet the user‟s
needs.
Usefulness in
carrying out
transactions
Informational
fit-to-task
The extent to which users believe that the Web site
meets their needs.
Response time Time to get a response after a request or an
interaction with a Web site.
Trust Secure communication and observance of
information privacy.
On-line
completeness
Allowing all or most necessary transactions to be
completed on-line (e.g., purchasing over the Web
site).
Relative
Advantage
Equivalent or better than other means of interacting
with the company.
Consistent image The Web site does not create dissonance for the user
by an image in compatible with that projected by the
firm through other media.
Entertainment
Value
Visual appeal The aesthetics Web site.
Innovativeness The creativity and uniqueness of a Web site.
Emotional appeal The emotional effect of using the Website and
intensity of involvement.
Table 1 Quality Dimensions of Website [14]
From the other hand, the website providers' quality expectation is normally
defined by the website producer who assumes the minimum acceptable quality for the
customers according to the requirements. As you can see here, the requirements are
the common understanding between the customer and the producer. To satisfy the
A Software Tool to Assist Scenario Testing of Websites
Pag
e15
P
age1
5
customer quality expectation, the producer need to define the testing strategy against
the quality documented in the requirements. However, the quality dimensions are still
too abstract to identify the exact testing tasks. How can those testing activities such as
the unit testing, the functional testing and the integration testing guarantee to fulfill
the customer perception quality? Before answering this question, let us have another
interesting point of view found during research.
Zhang and Dran in their studies identify the importance order of the website
features and interfaces [11]. More information regarding Zhang‟s and Dran‟s studies is
detailed in the section 7 related works. They considered the forms of quality factors as
the groups of website features and interfaces listed in Table 2, namely the quality
feature cluster listed in Table 3, which might not be a bad idea to give a concrete
representation of the customer perception quality. Their studies have two important
conclusions strongly relating to the topic of this paper, (1) the quality feature clusters
are unlikely to have the same importance in different website domains; for instance,
the five most important quality factors in its order are D03, D02, D07, D01 and D10
in a Finance domain website; while the five most important quality factors in its order
are D08, D04, D07, D06 and D13 in an Entertainment domain website; (2) the
importance order of the quality features are likely to change as time goes on; for
instance, when the business requires an online ordering feature to be added into a
product demonstration website, D12 Security/Privacy surely will increase its
importance position in its original order list.
Category
Feature
ID (FID) Features
C1 Information
content
F1-1 Information on Web site stays for a reasonable period of
time before it disappears.
F1-2 Absence of improper materials.
F1-3 Accurate information.
F1-4 Appropriate detail level of information.
F1-5 Up-to-date information.
F1-6 Relevant information.
A Software Tool to Assist Scenario Testing of Websites
Pag
e16
P
age1
6
F1-7 Complete coverage of information.
F1-8 Content that supports Web site‟s intended purpose.
F1-9 Controversial materials.
F1-10 Novel (new) information.
C2 Cognitive
outcomes
F2-1 Learned new knowledge and/or skills by using Web site.
C3 Enjoyment F3-1 Use of humor.
F3-2 Multimedia.
F3-3 Fun to explore.
C4 Privacy F4-1 Access requirements (e.g., pay a fee, sign on, enter a
password, and provide some private info before one can
access info).
F4-2 Authorized use of user‟s data for unanticipated purposes.
F4-3 Authorized collection of user data.
F4-4 Assurance that user-entered data is encrypted.
C5 User
empowerment
F5-1 User controls order or sequence of information access.
F5-2 User controls how fast to go through Web site.
F5-3 User controls opportunities for interaction.
F5-4 User controls complexity of mechanisms for accessing
information.
F5-5 User controls difficulty level of information accessed.
C6 Visual
appearance
F6-1 Attractive overall color use.
F6-2 Sharp displays.
F6-3 Visually attractive screen layout.
F6-4 Attractive screen background and pattern.
F6-5 Adequate brightness of screens/pages.
F6-6 Eye-catching images or title on homepage.
C7 Technical
support
F7-1 Indication of system loading/responding time.
F7-2 Support for different platforms and/or browsers.
F7-3 Stability of Web site availability (can always access Web
site).
C8 Navigation F8-1 Indication of user‟s location within Web site.
F8-2 Navigation aids.
F8-3 Directions for navigating Web site.
C9 Organization
of information
content
F9-1 Presence of overview, table of contents, and/or
summaries/ headings.
F9-2 Structure of information presentation is logical.
C10 Credibility F10-1 Reputation of Web site owner.
F10-2 External recognition of Web site (e.g., site has won
awards, number of times it has been visited).
F10-3 Identification of site owners/designers.
C11 Impartiality F11-1 Unbiased information.
A Software Tool to Assist Scenario Testing of Websites
Pag
e17
P
age1
7
F11-2 Absence of gender or racial/ethnic biases and stereotypes.
Table 2 Web Site Features [11]
ID Family Definition
D01 Accuracy No errors, correct, exact, precise, right, true
D02 Completeness
/Comprehensiveness
of Information
Large in scope or content, contains variety of information
or sources
D03 Currency /timeliness
/Update
Information is current, up to the moment, real-time, timely
D04 Engaging Cognitive advancement, emotional connections, personal
expressions
D05 Information reliability
/reputation
Information dependable, condition of being held in high
esteem, authoritative, good reputation of information
source
D06 Information
representation
The way information is presented, maybe in different
format/media, customized displays
D07 Navigation Features to make navigation possible, site maps
D08 Visual design Visual appearance
D09 Product and service
concerns
Features concerned with products/services offered/sold
through Website, not about site itself; price and
availability of products/services
D10 Readability
/Comprehension
/Clarity
Ability to comprehend meaning of written or printed
words or symbols, to perceive or receive well
D11 Relevant Information Information that directs to the point, having to do with
matter at hand
D12 Security /Privacy Confidentiality of information, things that give or assure
safety and guarantee
D13 Site accessibility
/responsiveness
Being able to access website, responsiveness of site to
user's request in terms of time
D14 Site technical features Such features as search tools, downloadable
(printer-friendliness), chat rooms.
Table 3 Quality Feature Clusters [11]
The interpretation of the first conclusion is the interface elements on certain
webpage (for example, links, text, images, media, controls and so on) and the
functions (invoked by the elements‟ events) as the essential parts of the website
features are not equally important to the customers. In another word, the defects
A Software Tool to Assist Scenario Testing of Websites
Pag
e18
P
age1
8
associated with the interface elements or the functions are not equally important to the
customers.
Similarly, it can be stated that website quality dimensions might not be equally
important. This can be proved by building the linkage between quality factors and
quality dimensions. The general approach proposed in this paper is mapping the
quality feature clusters (introduced in Zhang„s and Dran‟s studies and listed in Table 3)
to the website quality dimensions (introduced in Loiacono‟s, Watson‟s and Goodhue‟s
studies listed in Table 1). For example, the feature cluster “Navigation” is able to
deliver the quality dimension “Intuitive operations” to the website users. The mapping
result is shown in Table 4(a) and 4(b). Table 4(a) and 4(b) show what feature clusters
can directly deliver the quality dimension, although there is no feature cluster
identified for the quality dimensions “Consistent image” and “Innovativeness”. This
linkage is important as the abstract customer quality expectation has a relative
concrete form. Since the relative order of feature clusters is identified by Zhang and
Dran, the relative order of quality dimensions can be determined by the linked feature
clusters.
A Software Tool to Assist Scenario Testing of Websites
Pag
e19
P
age1
9
Table 4(a) Mapping Quality Feature Clusters to Quality Dimensions
A Software Tool to Assist Scenario Testing of Websites
Pag
e20
P
age2
0
Table 4(b) Mapping Quality Feature Clusters to Quality Dimensions
A Software Tool to Assist Scenario Testing of Websites
Pag
e21
P
age2
1
Another key point embedded in the first conclusion is the importance order does
not simply depend on the website domain as the large scale business oriented website
may have many sub domains such as product demonstration, product promotion,
product online shopping, product support, corporation introduction, business
cooperation and so on. And this can further breakdown even to a single page or a
response to an event (For example, AJAX technology [3] allows a part of page to be
updated.). This paper suggests considering the scale as a single step in the context of
scenarios as scenarios describe the use of the website and the customer‟s purpose of
using the website. For example, an online shopping scenario might involve a couple
of steps. The first few steps are about viewing products and adding products into the
basket (for instance, by clicking “add” button). The last few steps are about
processing order (for instance, by clicking “confirm” button) and getting confirmation
message. The tolerance of response time (D13) between earlier and later steps is
different. The customer expects to view product information with a quick response
(for example, click a product image and receive the production information page
within a second). Otherwise the longer loading time will be likely to lose the
customers‟ patience. To the later steps, security (D12) is more expected than response
time (D13). Hence, as long as the step in the scenario changes, the relatively
importance order of the quality dimension will change.
Now, let us come back to the previous question that how to guarantee the testing
activities would fulfill the customer perception quality. There might be no way to
guarantee at all because some quality dimensions such as Consistent Image and
Innovativeness in Table 4(b) are purely subjective to the customers and some quality
dimensions are not well covered although some feature clusters have been mapped to
the dimensions in Table 4(a) and Table 4(b). The gaps exist that further research work
is potentially needed on these areas. However, this paper proposes a naïve approach to
build the linkage between quality dimensions and testing tasks and to narrow the gap.
This approach is aimed to identify the deliverable corresponding to certain quality
A Software Tool to Assist Scenario Testing of Websites
Pag
e22
P
age2
2
dimension, to determine whether the deliverable can be tested and measured, to define
the test actions if the deliverable is testable, and to consider the way to test and
whether the testing tasks will be included in the scenario testing. Table 5 illustrates the
result of applying this approach to link the quality dimension “Intuitive operation” to
its testing tasks. The general step is to extend Table 4(a) and Table 4(b) by mapping
the features in Table 2 to the corresponding feature cluster. In Table 2, F8-1, F8-2, and
F8-3 are the features found belong to the feature cluster D07 Navigation. Hence, these
features are treated as deliverable in Table 5. Then each deliverable is analyzed what
associated interface elements or functions are testable. In Table 5, it shows that the
breadcrumb is the testable interface element associated with the feature F8-1 and the
test action is to verify breadcrumb is indicating the correct location on the webpage.
The last four columns are to label whether this testing task can be executed
automatically, or manually, or not sure and whether this testing task need to be
included in the scenario testing. Again, the feature items in Table 2, the testing task,
the testing mode are open to discuss and redefine.
Table 5 Mapping Deliverable and Testing Task onto Quality Dimension
2.2.2. Case Study
Let us have a real case study to consolidate the idea of considering the
importance of quality factors during measuring severity. The defects are captured on
the Visio article page that the side navigation bar links are not translated in Arabic and
A Software Tool to Assist Scenario Testing of Websites
Pag
e23
P
age2
3
their links are invalid. The defects are pointed by black arrow in Figure 4 and its
original U.S. English version is shown in Figure 5. The purpose of the scenario is to
provide product information to Arabian website users. Should the engineers launch
content re-localization process first or correct the hyperlinks first? As it is easy to see,
the quality dimension D06 information representation is more important than D07
navigation at this step when the user browse to the article page. The untranslated sub
navigation bar links will disable the sub navigation functionality. Similarly, what if
the article itself is not translated, which content should be re-localized first?
Figure 4 untranslated side navigation bar links on Visio article in Arabic
A Software Tool to Assist Scenario Testing of Websites
Pag
e24
P
age2
4
Figure 5 the original version in U.S. English
To sum up, this section explains that the importance of quality should also be
considered when measuring the defects and the considering should be based on the
scenarios.
2.3. Likelihood
Likelihood is inspired from risk analysis and found it usefulness to assist to
prioritize, especially at the testing and developing resource shortage time. There is a
concept here illustrated in Figure 6 needed to introduce first. It is called the visit trip
of the website. The visit trip is counted when a user enters the website (regardless
whether the entrance is the homepage of the website or any other page of the website)
till the user leaves the website (regardless whether the user turns off the browser or
jumps to the other site). A visit trip could be a single page trip which only one page of
the website is visited or a multiple pages trip which at least two pages of the website
is visited. Likelihood in the context of measuring severity is equivalent to the
percentage of the page (on which the deliverables are provided) occurring in the visit
A Software Tool to Assist Scenario Testing of Websites
Pag
e25
P
age2
5
trip (page trip rate) multiply the percentage of the deliverable used on that page (hit
rate). Likelihood can be represented in the following formula where P(page) is the
probability of the page occurring in all visit trips (page trip rate) and P(deliverable) is
the probability of the deliverable (the associated interface element or function) is used
on the page (hit rate*):
Likelihood = P(page) × P(deliverable)
or
Likelihood = Page Trip Rate × Hit Rate
* The hit rate of the layout, the strings, and the images are 100%.
Figure 6 single and multiple trips
The information of P(page) and P(deliverable) can be gathered from the website
statistical analysis service provided by third party such as Webtrends [5]. The
statistical analysis is able to tell not only the percentage of usage of certain elements
(for example, links and buttons) on a single page, but also how visitors move through
the website (This information can help to validate the scenarios defined during
requirements and refine the scenarios timely.), where the user tends to drop out. There
is no reason to fix the defects that unlikely to be hit by the user. It is no surprise to
know that the usage rate of the scenarios is changing from time to time (For example,
when a new product is going to be release, the usage of seeking for product
information will increase.). It is even interesting to know that the usage rate of the
elements on a single page is changing from time to time. The typical case will be
discussed in the following section 2.3.1 Case Study. Therefore, the usage of the
A Software Tool to Assist Scenario Testing of Websites
Pag
e26
P
age2
6
elements or functions or the usage of the scenarios can differentiate the relatively
order of the defects.
2.3.1. Case Study
A typical case is the link “Calendars” pointed by the black arrow in Figure 7 start
to receive much higher hit rate than usual when the New Year is approaching.
Meanwhile the page trip rate is increasing. This phenomenon may occur when some
special events such as season, festival, social, news, and even fashion are triggered.
Defects caught during the time critical period should be seriously considered to fix
with high priority.
Figure 7 the link "Calendars" on the sub homepage "Template"
3. Theoretical Framework and Severity Classification Scheme
In section 2, the three sub factors of severity are discussed and their related cases
A Software Tool to Assist Scenario Testing of Websites
Pag
e27
P
age2
7
are reviewed. During the discussion, a series of extended topics such as scenario,
scenario testing, website quality, deliverable and testing tasks are elicited but they are
disconnected. This section is going to walk you through to connect all of them to
build a comprehensive framework which turn severity to be in a more distinguishable
figure and be set in a more systematic and objective manner. This framework is the
principle of the prototype discussed in section 4.
3.1. Building the Framework
The two aspects of website quality are revealed that the requirements are the
common understanding between the customer and the website provider. The model is
illustrated in Figure 8.
Figure 8 requirements and quality expectations
However, the gap between customer‟s quality expectation and website provider‟s
quality expectation exists because the customer‟s quality expectation is too abstract,
shown in Figure 9.
A Software Tool to Assist Scenario Testing of Websites
Pag
e28
P
age2
8
Figure 9 the gap between different roles’ quality expectations
.
Scenarios can be derived from the requirements and are found its usefulness in
differentiating the impact of defects, shown in Figure 10.
Figure 10 Scenarios are derived from Requirements
The relatively importance order of quality dimensions is found by mapping the
quality feature clusters to the quality dimensions, shown in Figure 11. Meanwhile the
quality feature clusters offer a more detailed view of customer‟s quality expectation.
A Software Tool to Assist Scenario Testing of Websites
Pag
e29
P
age2
9
Figure 11 Mapping Feature Cluster on Quality Dimensions
Nevertheless, the feature clusters are still too abstract to be measured objectively.
The deliverable is suggested to be identified, which is shown in Figure 12.
Figure 12 Identify deliverables from the feature clusters
Scenario testing is adding testing tasks into scenarios derived from requirements.
Scenario testing facilitates the objective measuring of the importance of quality
dimensions by identifying the testing tasks for the deliverables, which is shown in
Figure 13.
A Software Tool to Assist Scenario Testing of Websites
Pag
e30
P
age3
0
Figure 13 identify the testing tasks for deliverables and integrate into scenarios
Scenario testing is a typical testing activity of system testing which can validate
both functional and non-functional requirements to ensure the quality deliverables
produced to satisfy the users. When adopting this framework, it does not mean that
scenario testing is the only testing activity can deliver quality to the users. Scenario
testing is the earliest testing activity that can measure website quality. The functions
of a feature have no flaws, which do not mean the feature has fine quality. The feature
might not be easy to operate. However, to deliver a fine quality feature, the feature
should avoid any function flaw. Therefore, Unit testing, Functional testing, Integration
testing and other testing activities not mentioned at earlier testing phases [2] are the
fundamental upstream activities to ensure the high quality website to be tested and
measured at the scenario testing stage.
Finally, the three sub factors of the defects can be measured objectively after
scenario testing. The overview of the framework is shown in Figure 14.
A Software Tool to Assist Scenario Testing of Websites
Pag
e31
P
age3
1
Figure 14 the overview of the framework
This framework gives the capability of setting the three sub factors beforehand
when creating the scenario test cases. As long as the scenario is defined, the impact of
the scenario and the importance of the quality dimension are known before testing.
The likelihood can be gathered from the historical data of the statistical analysis of
usage. When a defect is exposed by certain testing task, the severity will be measured
by these three predefined sub factors. The benefit of having these three factors set
earlier than testing is because it avoids the subjective setting by the testers who
execute the testing and often lack the big picture.
3.2. Severity Classification Scheme
The only thing left is to explain the severity classification scheme. This paper
applies a naïve approach to classify these three sub factors, impact, importance and
likelihood and to convert three values into one value to represent the severity. The
A Software Tool to Assist Scenario Testing of Websites
Pag
e32
P
age3
2
entry criteria for testing is all level settings of three sub factors are not 0.
3.2.1. Impact
As discussed in section 2.1, impact is to measure how the defect affects the
scenario. The following list is suggested for ranking impact:
Level 0: Not set
Level 1: The defect impairs at least the third level scenario and can be skipped
without worries in the entire scenario. There is at least one easy work around. The
user can go further with no difference in the scenario. The issue is low and
cosmetic.
Level 2: The defect impairs at least the third level scenario and can be skipped
with notice by the user. There is at least one work around. The user can go further
but it will be better to have the defect fixed. The issue is enhancement.
Level 3: The defect impairs the dependent (secondary) scenario and can be
skipped with worries in the entire scenario. There is at least one work around. The
user can go further with mind in the scenario. The issue is minor.
Level 4: The defect impairs the key (primary) scenario and can be skipped with a
lot of troubles in the entire scenario. There is at least one unexpected work around.
The user can hardly go further in the scenario. The issue is major.
Level 5: The defect blocks the entire scenario. There is no work around available.
The user cannot go further in the scenario. The issue is fatal.
As been noticed from the above the list, there is a scenario level concept involved.
The scenario level relationship is illustrated in Figure 15. To reach the goal of the
scenario, the shortest path is the primary scenario (the first level scenario) denoted in
red in Figure 15. The secondary scenario (the second level scenario) is either the
alternative path (longer than the primary path) or the control interaction (using the
control will not jump pages but get more information for supporting) on the page
which belong to the primary scenario (Both cases are denoted in yellow.). The third
A Software Tool to Assist Scenario Testing of Websites
Pag
e33
P
age3
3
level scenario is either the loop path from a page which belong to the secondary
scenario or the control interaction on the page which belong to the secondary scenario.
The level of scenario from design perspective is to provide the comprehensive
information to the user.
Figure 15 Scenario Level
3.2.2. Importance
Importance is to measure how important the quality feature cluster or quality
dimension in the scenario. The following list is suggested for ranking importance:
Level 0: Not set
Level 1: Negligible. The quality factor or the quality feature cluster or the quality
dimension will not affect the overall website quality.
Level 2: Acceptable. The quality factor or the quality feature cluster or the quality
dimension will slightly affect the overall website quality.
Level 3: Remarkable. The quality factor or the quality feature cluster or the
quality dimension will moderately affect the overall website quality.
Level 4: Significant. The quality factor or the quality feature cluster or the quality
dimension will strongly affect the overall website quality.
A Software Tool to Assist Scenario Testing of Websites
Pag
e34
P
age3
4
Level 5: Critical. The quality factor or the quality feature cluster or the quality
dimension will dominantly affect the overall website quality.
3.2.3. Likelihood
Likelihood is to measure the product value of page trip rate and hit rate. The
following list is suggested for ranking likelihood:
Level 0: Not set
Level 1: 0% - 20%
Level 2: 20% - 40%
Level 3: 40% - 60%
Level 4: 60% - 80%
Level 5: 80% - 100%
3.2.4. Representation of severity
The severity is measured by normalizing the product of the rank value of three
sub factors, impact, importance and likelihood. The representation of severity is
formulated in the following equation:
𝐒𝐞𝐯𝐞𝐫𝐢𝐭𝐲 = Impact + Importance + Likelihood
ImpactMax + ImportanceMax + LikelihoodMax
where ImpactMax is the max level of impact, ImportanceMax is the max level of
importance and LikelihoodMax is the max level of likelihood.
The representation of severity becomes to be a double number which is more
distinguishable, comparing to an integer ranking value in the traditional way. This
formula is not proved and is open to further research for refining.
3.3. Case Study
This section is going to walk you through how to set three sub factors of severity
at the key step in the buying Office product “Home and Student 2010” scenario in
A Software Tool to Assist Scenario Testing of Websites
Pag
e35
P
age3
5
Australian English version shown in Figure 16. The blue line guides the primary
scenario and the green lines guide the secondary scenarios in Figure 16. The dashed
boxes are the triggers (elements) in Figure 17. The key step, “Buy Office Home and
Student 2010” page is snapshotted in Figure 18.
Figure 16 snapshot of simplified version of the buying Office product scenario
Figure 17 snapshot of simplified version of the buying Office product scenario with triggers
A Software Tool to Assist Scenario Testing of Websites
Pag
e36
P
age3
6
Figure 18 “Buy Office Home and Student 2010” page
Since there is no design document for reference, there are some assumptions
needed. Firstly, let us assume the following quality features and quality dimensions
are considered at this step:
D10 Readability/Comprehension/Clarity (Ease of understanding)
D07 Navigation (Intuitive operations)
D01 Accuracy (Information Quality)
D02 Completeness/Comprehensiveness of Information (Information Quality)
D03 Currency/timeliness/Update (Information Quality)
D04 Visual design (Visual appeal)
D11 Relevant Information (Information fit-to-task)
Then let us breakdown the page into deliverables, shown in Figure 17.
A Software Tool to Assist Scenario Testing of Websites
Pag
e37
P
age3
7
Figure 19 the key step breakdown by deliverables
Now let us pick up two quality features to see their associated deliverables and testing
tasks, and to see how to set three sub factors.
For D10 Readability/Comprehension/Clarity (Ease of understanding), it is meant
to make the user understand the meaning of the interactive elements such as links and
controls. Therefore the associated deliverables A (breadcrumb), G (button), H
(clickable string), I (tab control), and J (clickable strings) are identified. The testing
task is verifying all the strings of the interactive elements are readable and meaningful.
This testing task should involve manual effort. Taking the deliverable G for example,
the importance of deliverable G should be set as level 5 because the button is unlikely
to be clicked if the string on the button is not readable and meaningless (for example,
a strange sign). The impact should be set as level 5 because the entire scenario is
going to be blocked (the button “Buy Now” is the trigger for jumping to the next page,
A Software Tool to Assist Scenario Testing of Websites
Pag
e38
P
age3
8
shown in Figure 16). Calculating the likelihood should notice that the hit rate should
be counted as 100% but not the percentage of page jumping caused by clicking the
button. If assuming the page trip rate is 47%, likelihood is equal to 47% from the
formula:
Likelihood = 47% × 100% = 47%
Likelihood should be set as level 3. Therefore the severity can be calculated from its
formula:
𝐒𝐞𝐯𝐞𝐫𝐢𝐭𝐲 = 5 + 5 + 3
5 + 5 + 5 ≈ 0.867
Taking the deliverable I for another example, the importance of deliverable I
should be set as level 3. That is because the user can still read the article in the tab
control to recovery the meaning if the strings “Product Overview” and “System
Requirements” are not readable and meaningful. The impact should be set as level 3
because clicking the tabs “Product Overview” and “System Requirements” will only
update the article area on the same page which can be treated as the second level
scenario. The consequence caused by such defect might be that the tab “System
Requirements” will not be clicked by the user and the article in the “System
Requirements” tab will not be read by the user. However, this does not bother the user
to click the “Buy Now” button. Let us assume the page trip rate is 47% and the hit
rate is 100% (two strings of the tabs are obviously presented on the page.). Hence,
likelihood is equal to 47% from the formula:
Likelihood = 47% × 100% = 47%
Likelihood should be set as level 3. Therefore the severity can be calculated from its
formula:
𝐒𝐞𝐯𝐞𝐫𝐢𝐭𝐲 = 3 + 3 + 3
5 + 5 + 5 ≈ 0.6
Taking the deliverable H for the last example, the importance of deliverable H
can be set as level 1 that the user might guess the meaning of the link but will not care
A Software Tool to Assist Scenario Testing of Websites
Pag
e39
P
age3
9
too much about it in this scenario. The impact on the scenario can be set as level 1
that the link does not affect much on this scenario (It will link to the other scenario).
The page trip rate and the hit rate are calculated like the previous ones. The severity
can be calculated from its formula:
𝐒𝐞𝐯𝐞𝐫𝐢𝐭𝐲 = 1 + 1 + 3
5 + 5 + 5 ≈ 0.267
For D07 Navigation (Intuitive operations), the deliverables must have hyperlinks.
Such deliverables are identified as A, G, H, and J. The testing task is verifying the
links of the deliverables are valid and this testing task can be executed automatically.
Let us have a look at deliverable G again. The impact and the importance settings
should be in level 5 as the button is the key in this scenario. The hit rate here is not
100% as the testing task is to verify whether the link is valid. The real usage of the
link should be measured. The hit rate and the page trip rate are assumed as 50% and
47% respectively in this case. Hence likelihood can be calculated by its formula:
Likelihood = 47% × 50% = 23.5%
Base on the result, likelihood should be set as level 2. Therefore, the severity can be
calculated from its formula:
𝐒𝐞𝐯𝐞𝐫𝐢𝐭𝐲 = 5 + 5 + 2
5 + 5 + 5 ≈ 0.8
So far, we have estimated severity for three deliverables (In this example, G, I
and H are all interface elements.) for feature cluster D10 and one deliverable for
feature cluster D07. Now if assuming all the elements tested above fail the testing and
the defects are reported with severity assigned according to the predefined settings in
a table like Table 7, the personnel who is responsible for assigning priority can easily
make decision to assign high priority to Bug 1and 4 (with relatively higher severity
score), to assign middle priority to Bug 2 and to assign low priority to Bug 3
(Assigning priority is just an example to show the meaning of the severity score. The
meaning of the severity score is to reveal the relative order between defects). Even if a
A Software Tool to Assist Scenario Testing of Websites
Pag
e40
P
age4
0
developer receive two high priority Bug 1 and 4, he can still easily make decision to
work on Bug 1 at first as Bug 1 has relatively higher severity score.
Bug ID Description Severity D(F) …
1 Unable to read the string on the button… 0.867 G(D10) …
2 Unable to read the string on the tab control… 0.6 I(D10) …
3 Unable to read the string of the link… 0.267 H(D10) …
4 Broken link of the button… 0.800 G(D07) …
… … … … …
Table 6 bug report is simulated in a table
4. Prototype
The prototype is built to prove the concept that the framework can be feasibly
adopted into a practical website scenario testing tool. Existing commercial tools [15,
16] have mature capabilities such as creating and managing scenario testing cases and
arranging various testing tasks into the steps of the test cases. The prototype is not
intended to copy the mode and to provide an advanced automation solution of any
testing tasks but (1) to assign and to evaluate impact, importance and likelihood to the
elements which will be tested; (2) to represent the severity scores of the defects
caught during testing. The basic requirements of the prototype are listed in the
following:
The system shall allow the user to load the predefined scenario.
The system shall provide basic user interface for configuring the scenario and executing
testing.
The system shall allow the user to configure the scenario to assign the testing tasks to
any step in the scenario according to the quality dimensions
The system shall allow the user to select the elements to be the target of the testing tasks.
A Software Tool to Assist Scenario Testing of Websites
Pag
e41
P
age4
1
The system shall allow the user to assign the level value of impact, importance and
likelihood onto the testing elements.
The system shall display the severity score for each defect after testing.
The prototype is built as a Windows Presentation Foundation (WPF) application
[20] coded in C#. The motivation is from the author‟s personal learning purpose. WPF
is a new concept to the author. The difficulties during implementation are to
understand the WPF data binding concept [21], to integrate Ribbon Control [22], and
to use WatiN library [18] which will be mentioned in section 6.
For instance, WPF data binding concept is to facilitate a fast, effortless and
codeless way to tie data instances and user interfaces and to have a clean business
logic and user interfaces separation. The WPF data binding concept is applied in a
couple of places in the prototype. For example, binding the sub factors setting user
interfaces which is shown on Panel F in Figure 21 (combobox [25], a drop down list)
with the severity XML file shown in Figure 27 and the WebPart object shown in
Figure 20. The former binding is to enable the combobox to display the level values in
the severity XML file, which could avoid the XML file parsing codes. The later
binding is to enable the change of the user‟s selection of the combobox to be reflected
in the data object that the impact, importance, and likelihood attribute values will be
updated automatically if the user changes the selection of the comboboxes. In order to
achieve the data binding, it is necessary to understand the binding target, the binding
source, the direction of the data flow and the triggers of the source updates. And the
knowledge is gained from self-learning through [21, 25].
During the implementation, the author had some unsuccessful experience. The
memorable one was to integrate W3C link validation [23] to be the logic of the testing
task for deliverable “Directions for navigating Web site”. It was planned to let W3C
link validation valid all links of a page and to let the system match the user selected
elements and the running results. It was found that the incompatibility of W3C
A Software Tool to Assist Scenario Testing of Websites
Pag
e42
P
age4
2
validation service provided could hardly achieve the goal. Due to the time constraint,
W3C validation library was not studied thoroughly. Further research work may
discover the possibility of utilizing the W3C validation services. Alternatively, the
prototype uses a WatiN‟s web browser object to valid the links.
The design of the prototype is quite straightforward and described by the class
diagram shown in Figure 20. The WindowMain class is responsible for user interface
logic. The inherited classes from the Verification class contain the specified logic of
the testing tasks. The interface INotifyPropertyChanged is used for data binding
between the application user interface and the data. The Verification class is an
abstract class that contains the logic of the testing task. In this prototype, it is designed
to override the “run” function in the inherited classes to specify the logic. The
WebPart class is to describe the deliverable elements mentioned before. Different
Verifications (testing tasks) will have the different WebParts (elements). The Result is
to keep the testing result for the element and the TestResult class is to describe all the
testing results and their severity scores. This prototype is designed for meet the needs
of the requirements only and the architecture can be redesigned in a more
sophisticated way.
A Software Tool to Assist Scenario Testing of Websites
Pag
e43
P
age4
3
Figure 20 class diagram of the prototype
5. Results and Discussion
In order to satisfy the requirements mentioned in section 4, especially to
demonstrate how to assign testing tasks to a single step of the scenario, how to set
three sub factors to an individual interface element, and how to represent the testing
A Software Tool to Assist Scenario Testing of Websites
Pag
e44
P
age4
4
result, a vertical prototype [24] is implemented to provide the basic user interface to
facilitate the scenario configuration functionalities and the severity score illustration.
This section will present the prototype and discuss the implementation results.
The prototype has seven panels ranged from A to G shown in Figure 21 and
Figure 29. Panel A is the command bar (Ribbon Control) which replaces the
traditional drop down menu and toolbars for increasing the discoverability of features
and functions.
Figure 21 prototype user interface layout and the relationships between panels
Panel B is the project panel which contains a tree view control to describe the
relationship between scenarios and projects. The project and scenario information is
stored in the XML file (Project File) shown in Figure 22.
A Software Tool to Assist Scenario Testing of Websites
Pag
e45
P
age4
5
Figure 22 the project and scenario information is stored in the XML file
After clicking the “Open” button highlighted in Figure 23, the project and
scenario information will be represented in a tree view control in Panel B. The tree
view control will display the dependent relationship between projects and scenarios.
When a particular scenario is selected by the user, the scenario-and-step relationship
will be represented in a tree view control in the scenario panel (Panel C). The tree
view control in Panel C will only display the dependent relationship between
scenarios and steps at this stage. When a specific step is selected from the tree view
control in Panel C by the user, the target web page of the step (the attribute
“actionValue” of the “Step” tag in the project XML file shown in Figure 22) will be
displayed in a web browser control in Panel D. The selecting and displaying
relationships, from B to C and from C to D, are arrowed as 1 and 2 respectively in
Figure 24.
Figure 23 Open button is highlighted
A Software Tool to Assist Scenario Testing of Websites
Pag
e46
P
age4
6
Figure 24 the interaction relationships 1 and 2, from B to C and from C to D
After loading the webpage, Panel E shown in Figure 26 will appear to list all the
quality dimensions, the feature clusters, the deliverables and their associated testing
tasks in a tree view control. The information is stored in a XML file shown in Figure
25. The structure of the XML file just matches the structure of Table 6 discussed in
section 2.2.1. In the prototype, only the testing task “Verify the link is valid” of the
deliverable “Direction for navigating website” and the testing task “Verify the image
is visible with proper content” of the deliverable “Attractive screen background and
pattern” are implemented. However, the testing tasks, the deliverables, and the feature
clusters can be customized and refined according to the real features designed on the
website. When the user select a specific deliverable/testing task, Panel F shown in
Figure 26 as well will list all related elements and allow the user to decide whether the
element needs to be tested and which level the three sub factors should be set. To
assist the user to make decision, the element will be highlighted on the webpage in
Panel D when the user selects an element. The interaction relationships, from E to F
and from F to D, are arrowed as 3 and 4 respectively in Figure 26.
A Software Tool to Assist Scenario Testing of Websites
Pag
e47
P
age4
7
Figure 25 quality dimensions, feature clusters, and deliverable structured in XML file
Figure 26 interaction relationships 3 and 4, from E to F and from F to D
The levels of the three sub factors are defined according to the classification
scheme proposed in section 3.2 and the information is kept in a XML file with the
structure shown in Figure 27. This file can be modified as long as the classification
scheme is customized and refined.
A Software Tool to Assist Scenario Testing of Websites
Pag
e48
P
age4
8
Figure 27 severity classificaion scheme structured in XML file
After selecting the testing elements and setting the sub factors, the user can
simply click the “Add” button in Panel F. The series of information, the testing task,
the deliverable, the feature clusters, and the quality dimension will be grouped and
added as a child node under the step in the tree view control in Panel C. This allows
the user to see what quality dimensions are measured by what specific testing tasks at
what step. Such information can explain whether the quality of the step is well
covered. Such information can indirectly explain whether the quality of the scenario is
well covered and whether the quality of the website is well covered. The interaction
relationship from F to C is arrowed as 5 in Figure 28.
A Software Tool to Assist Scenario Testing of Websites
Pag
e49
P
age4
9
Figure 28 interaction relashionship 5
The testing result is displayed in Figure 29. All defects caught during testing are
listed in the result table. The items in Group 1 bracketed in Figure 29 are the elements
tested by one particular testing task. From Figure 29, it is shown four defects
(elements) are caught by the testing task “Verify the link is valid” of the deliverable
“Direction for navigating website”. The severity scores are calculated based on the
formula discussed in section 3.2.4 and displayed on the next column to three
predefined sub factors for each defect. If the single factor is at its maximum level
(level 5 in the defined classification scheme), the factor is highlighted in red in the
table. The absolute severity score does not have much meaning but provide the
relative order to distinguish defects and give a strong reference for prioritization.
Interestingly, if summing up the severity scores for Group 1, the severity score can be
used to represent the deliverable score. In Figure 29, 2.667 is the severity scores
summed for the deliverable “directions for navigating website”. Similarly, the severity
scores for Group 2 (deliverables and testing tasks) can be aggregated to represent the
step score and the severity scores for Group 3 (steps) can be aggregated to represent
the scenario score. These scores can reveal the relative order at different level
A Software Tool to Assist Scenario Testing of Websites
Pag
e50
P
age5
0
(deliverable level, step level, scenario level, and even project level) which could be
the strong reference for prioritization.
Figure 29 testing result and severity score aggregated in different level
There is another interesting found that some severity scores are same (for
example, 0.6 and 0.533). Although the three sub factors‟ settings are different, the
total values of their settings are same. This phenomena gives a hint to go back to
refine the equation of the severity representation. What if three different weights are
added into the equation, should the severity score be more distinguishable:
𝐒𝐞𝐯𝐞𝐫𝐢𝐭𝐲 = ω1 × Impact + ω2 × Importance + ω3 × Likelihood
ImpactMax + ImportanceMax + LikelihoodMax
A Software Tool to Assist Scenario Testing of Websites
Pag
e51
P
age5
1
However, it is just an immature thought that both the weights and the equation need
further research for discussion.
As mentioned at the beginning of the section, this is a vertical prototype to prove
the concept that (1) Severity can be set in a systematical way; (2) Severity can be
represented in a more distinguishable shape. The implementation result is pretty
positive to demonstrate the feasibility to adopt the framework into the practical
website scenario testing tools.
6. Related Works
The definition of scenario in software engineering discipline was firstly
conceptualized by Jacobson in [6] in early 90‟s. Scenario has been popularized to
apply in requirement elicitation and analysis to capture the system functionality and
behavior from a user-centered perspective. However there was neither any proper
procedure defined to derive the scenarios from the requirements nor any concept
mentioned to apply scenarios in any other phases of software development life cycle.
With the rapidly development of the maturity of software engineering in 90‟s, the
value of using scenario for system design has been explored by many researchers such
as Carroll [13]. Carroll concludes five reasons for using scenario in design. (1)
Scenarios expose not only the functionalities of the system but specific claim about
how the user will access the functionalities and what the user will experience in doing
so. Hence, scenarios assist the designer to coordinate design action and reflection.
Scenarios provide a concrete envisionment of a design solution. (2) Scenarios are
flexible to be revised and be elaborated so that the designer can manage the instable
design situations. (3) Single scenario can describe designs at multiple levels of detail
and with respect to multiple perspectives to help designers move toward various
consequences. (4) Scenarios can be abstracted and categorized to help designers to
recognize, capture and reuse generalizations to mitigate the risk of gaps between
technical knowledge and technical design. (5) Scenarios are efficient and effective
A Software Tool to Assist Scenario Testing of Websites
Pag
e52
P
age5
2
media for both stakeholders and designers to avoid the communication risk.
In the meantime, the use of scenario has been gradually appreciated in software
testing. Ryser and Glinz [8] extend Hsia et al. s‟ work [17] to disseminate to bring
scenario use into software testing field for requirement validation and verification.
They have done significant work in this area. Ryser and Glinz state that testing is
usually done in an ad-hoc manner and test cases are usually developed in an
unstructured and unsystematic way. They analysis the existing testing problems
beneath the surface: (1) Testing is done in the last phase of the development only; (2)
Testing methods are not integrated with development methods; (3) Test cases are not
created/generated in a systematic manner. They also discuss the drawbacks of some
proposed solutions such as expensiveness and complicatedness of testing automation,
specialized test languages and formal specifications. Ryser and Glinz suggest reusing
the predefined scenarios created during requirement elicitation and analysis phase to
formalize the natural languages of scenarios; then transform the natural languages into
statecharts and dependency chart; next generate test cases based on the charts in order
to address the issues. They define this method as a scenario-based approach to support
systematic test case development for validation and test, named SCENT (standing for
SCENario-based validation and Test of software). Ryser and Glinz believe that (1)
Scenarios can validate the system under development; (2) The statecharts of scenarios
can uncover ambiguities, contradictions, omissions, impreciseness and vagueness in
natural language descriptions; (3) Scenarios contains the information such as pre- and
post-conditions, data ranges and data values, and non-functional requirements (for
example, performance) which are extremely valuable to testing; (4) The statecharts of
scenarios can help to choose appropriate testing strategy and testing techniques to
make decisions in allocating resource in testing plan.
Ryser and Glinz refine the terminology of scenario, use case and their
relationships: Scenario is an ordered set of interactions between partners, usually
between a system and a set of actors external to the system. It may comprise a
A Software Tool to Assist Scenario Testing of Websites
Pag
e53
P
age5
3
concrete sequence of interaction steps (instance scenario) or a set of possible
interaction steps (type scenario); Use case is a sequence of interactions between an
actor (or actors) and a system triggered by a specific actor, which produces a result for
an actor, being a type scenario. They also propose a series of steps for scenario
creation listed in the Table 8 and the ideal practical steps are demonstrated in Figure
30.
Steps Step Description Outcome
1 Find all actors interacting with the system List of actors
2 Find all (relevant system external) events List of events (triggers)
3 Determine results and system in- and output System in- & outputs
4 Determine system boundaries Context diagram
5 Create coarse overview scenarios (instance or type
scenarios on business process or task level), name,
short description
[First high level statecharts are created: Few states,
no fully specified behavior]
List of scenarios
6 Prioritize scenarios according to their importance
and assure that the scenarios cover all system
functionality (completeness)
List of prioritized scenarios.
Links scenarios – actors
7 Transform instance to type scenarios. Create a
step-by-step description of events and actions for
each scenario (task level), structuring scenarios
according to a given template
[Statecharts created in step 5 are modified and
extended to describe fundamental behavior]
Flow of actions in scenarios
8 Create a dependency chart and an overview
diagram
Dependency chart, overview
diagram
9 Have users review and comment on the scenarios
and diagrams
Comments to scenarios
10 Extend the scenarios by refining the description of
the normal flow of actions, break down tasks to
single working steps
[Statecharts are refined along with scenarios]
Description of normal flow of
actions
Hints on test case derivation
A Software Tool to Assist Scenario Testing of Websites
Pag
e54
P
age5
4
11 Model alternative flows of actions, specify
exceptions and how to react to exceptions
Alternative flows of
actions, exceptions and their
handling in scenarios
Hints on test case derivation
12 Factor out abstract scenarios Abstract scenarios
13 Include qualities, non-functional and performance
requirements in scenarios
Scenarios with non-functional
requirements
14 Revise the dependency charts and overview
diagram
Revised diagrams
15 Have users check and validate the scenarios
(Formal reviews)
Validated scenarios
16 Structure scenarios according to given template;
create preliminary test cases / write test plan and
specification
Scenario specification paper
Table 7 steps to Create Scenario [8]
A Software Tool to Assist Scenario Testing of Websites
Pag
e55
P
age5
5
Figure 30 Generate Scenario Workflow [8] Figure 30 Generate Scenario Workflow [8]
A Software Tool to Assist Scenario Testing of Websites
Pag
e56
P
age5
6
Ryser and Glinz also define the textual representation in a template format which
consists of scenario identifier, name, a short description to explain the goal, the actors,
the preconditions and postconditions, trigger, normal flow, alternative flows,
non-functional requirements, version/change history, owner, type, diagrams, open
questions/known problems, and its associated test cases. They also distinguish three
major kinds of dependencies of scenarios, abstraction dependencies (hierarchy
pattern), temporal dependencies (time-variant execution order), and causal
dependencies (conditional pattern). Ryser and Glinz extend Jackson-like notations to
represent the scenario and their dependencies in comprehensive Dependency Chart
and statecharts in order to not only deliver a clear understanding of the dependencies
and relationships between scenarios but also help to derive additional test cases to
enhance the test suites generated from the state model of the system.
Finally, Ryser and Glinz light the path to derive the test cases from the state and
dependency charts by traversing all paths in the statecharts and the dependency chart.
They also suggest prioritizing the scenarios to refine the test suite.
Ryser and Glinz emphasize the dependency of scenarios in [9] and the practical value
in [7]. However, Ryser and Glinz neither define the concept of scenario testing nor
propagate the scenario testing for websites. In addition, their objects have not been
developed further to include the evaluation of testing results that how the failure of
scenario tests impact the entire quality of the system.
Ricca and Tonella categorize the empirical reported web faults to build a fault
model and evaluate the available web testing techniques against the model in [10].
They group the existing web testing techniques into three sets which are functional
testing, structural testing and model-based testing. Ricca and Tonella illustrate the
differences among these three categories of techniques by running a simple sample.
The difference is exhibited in finding various types of defects. Functional testing is
targeted on verifying the functionalities using black box method. Structural testing
and model-based testing is focused on the code coverage in either source code level or
A Software Tool to Assist Scenario Testing of Websites
Pag
e57
P
age5
7
page level. Ricca and Tonella select those faults reported for publicly available open
source web applications to construct a fault model. Those five web applications are in
from middle to large scale by measuring the lines of code. The general approach of
building such a fault model is in the following steps:
1) Review the randomly selected bugs and their fixing solutions and comments from
repository
2) Produce a description for each bug
3) Map the bug descriptions to different categories (create a new one if necessary)
4) Revise and merge/split categories.
5) Build a matrix to check what web defects can be detected by any testing technical
category.
By using this approach, Ricca and Tonella convincingly prove that there are still
demands for novel techniques to tackle these types of defects which cannot be
detected by the categorized testing technologies. Ricca and Tonella bring a heuristic
idea to think backward but in a pity that they miss one further step to relate the fault
categories and quality attributes of website.
Zhang and Dran study the website quality from user perceptions by investigating
the importance of website features and interfaces [11]. They did two studies. One is to
apply Kano‟s model to observe customer quality expectations for a specific type of
site. One is to extend the model to perceive the various domains. The Kano model, an
original marketing model defines that the product and service quality from customer
perspective have three levels: basic, performance and exciting. The quality
expectation is measured by the features. Basic quality features are the minimum
acceptable to customer. They are often ignored by customer and complained for not
having. Performance quality features are consciously noticed by customer but
disappointed feeling is raised if missing. Exciting quality features will surprise and
please the customer. The customer will be greatly motivated to increase loyalty if
receiving the features but will have no effect if missing. The Kano model also reveals
A Software Tool to Assist Scenario Testing of Websites
Pag
e58
P
age5
8
the time factor to these three quality features. With the passing time, the exciting
quality features will degrade to performance expectations, and the performance
quality features will turn into basic expectations. Zhang and Dran refine a feature list
of CNN website base on existing website and ask a group of frequent users to judge
the type of quality features and quality features transition. They prove: (1) the Kano
model is applicable to reasonably categorize the quality features of website; (2) the
features unlikely have the same importance to the user; (3) some quality features take
different length of transition time.
Zhang and Dran do the second study to address the limitations of the first study.
They extend to weight the importance of each feature across multiple domains and
take some domain-specific features into account. They leave the limited diversity of
participants to their future work. The result of the second study reveals that: (1) the
customer has different expectation of quality factors according to different web
domains. In another words, the importance of quality factors to different domains are
not equal. (2) the most important factors in most domains are shown as being
dominated by the basic and performance features, which is consistent to the Kano
theory. To sum up, Zhang and Dran provide solid empirical evidence to illustrate the
unequal importance of quality factors regarding the various domains from customer
perspective. Furthermore, they build a theoretical framework for evaluating website
quality from the customer satisfaction. However, their framework does not meet the
gap of the complexity of the modern websites having multiple sub domains and site
features. They also leave the blank of measurability of those user quality factors from
website testing point of view.
Loiacono, Watson and Goodhue develop a measure of website quality in order to
predict consumer reuse of the website [14]. They start from combining the Theory of
Reasoned Action (TRA) established by Ajzen and Fishbein in 1980 and Technology
Acceptance Model (TAM) extended base on TRA by Davis in 1989 to bridge the user
beliefs about a website and the behavior of reusing the website. They found TRA miss
A Software Tool to Assist Scenario Testing of Websites
Pag
e59
P
age5
9
specifying appropriate beliefs to technology user behaviours and TAM only identify
two very general beliefs, ease of use and usefulness. They believe it exist some other
relevant and distinct categories of beliefs such as entertainment which will also
encourage to reuse of website. Their general approach is to define the possible
dimensions of quality and identify the corresponding aspects firstly by extensively
reviewing the management information system and marketing literatures, examining
popular press publications, soliciting criteria from Web surfers, interviewing Web
designers, and studying a large organization‟s standards for website design. They
define four major categories: (A) ease of use, (B) usefulness in gathering information,
(C) usefulness in carrying out transactions and (D) entertainment value and identify
the aspects of each category. They detail that ease of use including (1) being easy to
read and understand, and (2) being easy to operate and navigate through. Usefulness
in gathering information means a website should provide (3) quality information and
(4) tailored communications. Usefulness in carrying out transactions includes (5)
functional fit-to-task, (6) response time, (7) trust, (8) online completeness, (9) relative
advantage and (10) consistent company image. The entertainment value contains (10)
visually appealing, (11) creative or innovative, and (12) emotionally appealing. Next,
they develop three questions for each aspect to create an initial instrument. Then they
refine the instrument and empirically confirm the measurement validity of the
instrument.
Although Loiacono, Watson and Goodhue theoretically present the natural of the
quality of website, they leave several areas for future researching such as concept of
total quality, cultural issues, the relationship between quality dimensions and website
types (for instance, business-to-business and non-commercial website) and so on. The
quality dimensions are not further discussed regarding evaluating from web testing
point of view that how to quantify the quality, and whether the quality could be
assured through testing , and how the quality dimensions influence the total quality of
website, and so on.
A Software Tool to Assist Scenario Testing of Websites
Pag
e60
P
age6
0
Huang and Chen [12] prototype an automated scenario testing tool for functional
scenario testing of web applications. They extend the semantics of UML activity
diagram to capture and refine description of use cases for covering both success
scenarios and failure or exception scenarios. They construct a scheme and breakdown
the web applications into five testable stereotypes: web pages, navigators, forms, links,
and frames. Each concept has at least one element for modelling testing. Then they
develop an algorithm named Distilling Testing Scenario to split the branches of the
activity diagram into a standalone scenario and export the reconstructed activity
diagram from XMI standard into XML file in a format of defined scheme. Then they
propose an algorithm TTSITC (Translating Testing Scenario Into Testing Code) to
transform each XML file (a scenario) into testing code. TTSITC applies XML DOM
(a XML and XSLT parser) to parse the XML file and coordinates HttpUnit (An open
source API for building executable test cases) to generate the testing code. However,
Huang and Chen miss the linkage between the scenario testing results and quality
measurement.
Existing tools in the market such as Microsoft Visual Studio 2010 Testing Edition
(VS10TE) [15] and Telerik WebUI Test Studio 2010 (TWTS) [16] provide essential
features for Website testing such as creating and managing test cases, running test
cases in automatic or semi-automatic mode, capturing and replaying user behavior on
the website, testing UI control, configuring testing environments and managing
testing effort. These features can be combined to fit the purpose to assist website
scenario testing. However, they do not have such a feature to bridge the quality
analysis and testing results. Often, in practical use, only the number of pass and failed
test cases are shown in the testing report. It is not efficient to provide such
information to the tester as one has to go back to review each failure and to determine
the impact of the failure. It is also not effective for the engineer to spend limited effort
on the proper defects which should be prioritized to fix. Contrasting to these existing
tools, the proposed prototype shifts the quality analysis into upstream, test case
A Software Tool to Assist Scenario Testing of Websites
Pag
e61
P
age6
1
creation phase. The quality analysis includes extracting quality factors from customer
requirements, prioritize the quality factors according to the scenario, and select the
corresponding scenario testing methods base on the quality factors. Since the quality
factors are ordered for each scenario, the associated testing results can be weighted
and summed to represent the scenario quality, the severity of the defects can be
differentiated. Hence, an overall quality of the website can be described. The testing
result will be provided with a figure to assist the engineer to prioritize the fixing
targets.
WatiN [13] is an open source library for web application testing in .Net, being
similar with WatiR (in Ruby scripting language). WatiN is developed in C# to ease the
website testing using .Net. It supports all major HTML elements testing, AJAX
website testing, major control testing, multiple web browsers testing such as Internet
Explorer (version 6, 7, 8) and Firefox testing. This library can facilitate the
implementation of various testing tasks in the prototype.
7. Conclusion and Future work
This paper analyses three sub factors (impact, importance and likelihood) of
severity and discusses their significant referential value in evaluating severity. This
paper also investigates the different views (scenario, quality and usage) from which
the three factors are measured. Considering the different views of these factors, this
paper proposes a framework to estimate defect severity by utilizing the structural view
of predefined scenarios and deriving testing tasks from scenario related quality
dimensions. This paper also defines the three sub factors‟ classification scheme and
the severity representation to support the framework. A prototype is built to prove the
feasibility to adopt the framework into the practical website scenario testing tools.
This study is relevant because it enables the severity of defects to be set in a
systematic manner and be represented in a more distinguishable form so that severity
can provides the greatest reference value for effective defect prioritization. This study
A Software Tool to Assist Scenario Testing of Websites
Pag
e62
P
age6
2
also can contribute to the website design and website testing research regarding
quality delivery and quality assurance. It is also foreseeable that an improved
framework will be widely adopted in the industry.
However, this paper just lights up a new direction and further research is needed
on the following areas: (1) although the quality dimensions, the feature clusters and
the deliverables are identified by the previous research work, extensions to these
contents are necessary in order to enhance the importance of scenario testing; (2) this
framework needs to be empirically proved whether it will improve the effectiveness
of defect prioritization; (3) the severity classification scheme and the representation
equation should be refined, especially likelihood which needs to be verified by
practice; (4) the way to aggregate the severity scores needs to be refined to make it
meaningful.
8. Bibliography
[0] IEEE Standard Classification for Software Anomalies, IEEE Std 1044-2009
[1] IEEE Standard Classification for Software Anomalies, IEEE Std 1044-1993
[2] Managing the Testing Process: Practical tools and Techniques for Managing Hardware and
Software Testing, 3rd
edition, page 4-8, Rex Black, ISBN: 978-0-470-40415-7
[3] Building Rich Web Applications with Ajax, Linda Dailey Paulson, Computer, vol. 38, no.
10, pp. 14-17, Oct. 2005.
[4] An Introduction to Scenario Testing, Cem Kaner, Florida tech, June 2003
[5] Webtrends Website: http://worldwide.webtrends.com/
[6] Object Oriented Software Engineering: A Use Case Driven Approach, I. Jacobson, M.
Christerson, P. Jonsson, G. Övergaard, Amsterdam: Addison-Wesley, 1992.
[7] A Practical Approach to Validating and Testing Software Systems Using Scenarios,
Johannes Ryser, Martin Glinz, Quality Week Europe '99, Brussels, 1999
A Software Tool to Assist Scenario Testing of Websites
Pag
e63
P
age6
3
[8] SCENT: A Method Employing Scenarios to Systematically Derive Test Cases for System
Test, J. Ryser, M. Glinz, University of Zurich, Institution of Information, Zurich, 2000/03,
2000
[9] Using Dependency Charts to Improve Scenario-Based Testing, J. Ryser, M. Glinz, the
17th International Conference on Testing Computer Software TCS„2000 in Washington, D.C.
[10] Web Testing: a Roadmap for the Empirical Research, Filippo Ricca and Paolo Tonella,
Proceedings of the 2005 Seventh IEEE International Symposium on Web Site Evolution
(WSE‟05)
[11] User Expectations and Rankings of Quality Factors in Different Web Site Domains, Ping
Zhang, Gisela M. von Dran, International Journal of Electronic Commerce / Winter 2001–
2002, Vol. 6, No. 2, pp. 9–33
[12] A Tool to Support Automated Testing for Web Application Scenario, Cheng-hui Huang,
Huo Yan Chen, 2006 IEEE International Conference on Systems, Man, and Cybernetics,
October 8-11, 2006, Taipei, Taiwan
[13] Carroll, J., "Five reasons for scenario-based design", Proc. of the IEEE Annual Hawaii
International Conference on System Sciences (HICSS). Hawaii, USA, 1999.
[14] WebQual™: A Measure of Web Site Quality, Eleanor T. Loiacono, Richard Watson,
Dale L. Goodhue
[15] Visual Studio Test Professional 2010
http://www.microsoft.com/visualstudio/en-us/products/2010-editions/test-professional
[16] Telerik Website: http://www.telerik.com
[17] Formal Approach to Scenario Analysis, P. Hsia, J. Samuel, J. Gao, D. Kung, Y.
Toyoshima, C. Chen, IEEE Software, vol. 11, # 2, pp. 33-41, 1994.
[18] WatiN Website: http://watin.sourceforge.net
[19] Microsoft Office.com Website: http://www.office.com/
[20] Microsoft Windows Presentation Foundation Website (WPF) Website:
http://msdn.microsoft.com/en-us/library/ms754130.aspx
[21] Microsoft Windows Presentation Foundation Website (WPF) Data Binding Website:
A Software Tool to Assist Scenario Testing of Websites
Pag
e64
P
age6
4
http://msdn.microsoft.com/en-us/library/ms752347.aspx
[22] Microsoft WPF Ribbon Control Website:
http://msdn.microsoft.com/en-us/library/cc872782.aspx
[23] W3C validation Website: http://www.w3.org/Status.html
[24] Software Requirements, 2nd
Edition, page 236, Karl E. Wiegers, ISBN: 0-7356-1879-8
[25] WPF 4 Unleashed, 1st Edition, page 282, Adam Nathan, ISBN: 0-672-33119-5