A Methodology for Evaluating Quality and Functional Size of Operative WebApps SILVIA ABRAHÃO Valencia University of Technology. Spain and LUIS OLSINA National University of La Pampa. Argentine and

OSCAR PASTOR Valencia University of Technology. Spain ________________________________________________________________________ The building of Web-based systems has often been ad hoc, lacking systematic development approaches, and quality control and assurance processes. In order to avoid the Web crisis, urge Web Engineering approaches for eliciting, developing and maintaining Web sites and Applications (WebApps) of required quality within budgetary constrains. To bridge this gap, one of our current concerns is assessing the quality of operative WebApps considering both functional and nonfunctional requirements. We claim that an operative site should be assessed not only from the quality perspective of the nonfunctional requirements, but also from the functional viewpoint like informational and navigational architecture. WebFP_QEM is a methodology that allows capturing business and application goals for different audiences, specifying informational and navigational models, specifying quality models and assessment criteria, and analyzing outcomes in order to give recommendations for improvements. Furthermore, if a change is suggested based on a recommendation document, the estimation of the maintenance cost can be done using a Web-centered Function Points measure. Categories and Subject Descriptors: D2.1 [Software Engineering]: Requirements Specification – Methodologies; D2.2 [Software Engineering]: Design Tools and Techniques – OO Design Methods; D.28 [Software Engineering]: Metrics– Product Metrics. General Terms: Design, Measurement, and Reliability Additional Key Words and Phrases: WebApps Quality, Web Conceptual Modeling, Quantitative Evaluation, and Web Functional Size. ________________________________________________________________________

1. INTRODUCTION With the impressive evolution of the WWW, Web sites and Applications (WebApps) have unceasingly become more and more complex. Nowadays is widely accepted that a Web site is not merely a matter of contents and aesthetics: demanding functionality is required, combining navigation with complex services and transactions. Because of the increasing size, complexity, quality needs and market demands for WebApps, several problems have frequently been reported [Cutter Consortium 2000; Zelnick 1998] such as exceeding budgets, delivered systems didn’t meet business needs, unknown or bad product quality, in addition to a lack of requirements and architectural documentation.

Besides, the quality of Web applications has often been assessed in an ad-hoc way, primarily based on the common sense, intuition and expertise of developers. Studies for quality of products and processes for the Web are very recent and still there are no widely spread evaluation methods and techniques for different assessment purposes [Fleming 2001; Mendes et al. 2001; Olsina and Rossi 2001; Pooley et al. 2002].

This research was partially supported by the CYTED Program, in the VII.18 research project, WEST; the CICYT project with reference TIC 2001-3530-C02-01 (Spain), and by the UNLPam-09/F022 research project (Argentina). Authors' addresses: Silvia Abrahão and Oscar Pastor, Department of Information Systems and Computation, Valencia University of Technology, Camino de Vera s/n, P.O. Box: 22012, E-46022, Valencia, Spain; Luis Olsina, Software Engineering R&D Group (GIDIS), Department of Informatics, Engineering School, UNLPam, Calle 110 esq. 9, 6360 General Pico, La Pampa, Argentine.

Although some techniques currently exist for evaluating some quality characteristics such as usability, accessibility [Kirakowski et al. 1998; Nielsen 2000; WWW Consortium 2002], etc., and testing techniques for measuring stress, performance and faults ([Dustin et al. 2002; Fenton and Pfleeger 1997], among others), in practice, they provide only partial solutions because they separately focus either on nonfunctional requirements or on functional requirements. Indeed, flexible and integral solutions are needed; the Web environment presents many new challenges, and requires enhanced assessment methods and techniques. Consequently, a systematic approach that helps dealing with complexity, and that allows eliciting, assessing and maintaining WebApps of required quality within budgetary constrains is increasingly becoming necessary.

WebFP_QEM (Web Function Points and Quality Evaluation Method) is a methodology that was intended to perform an engineering contribution by proposing a systematic and flexible approach aimed to analyze the information architecture and to evaluate the quality of complex WebApps in the operative phase [Abrahão et al. 2001]. It allows capturing business and application goals for different audiences, specifying informational and navigational models (the functional requirements), specifying quality models and assessment criteria (the nonfunctional requirements), analyzing outcomes in order to give recommendations for improvements, in addition to drawing cost prediction reports.

During the assessment process of an operative WebApp, the sequence of events that happens as result of a restructuring (re-engineering) must be initiated with an analysis of impact of changes of the requested reorganization. This analysis must consider: the type of maintenance (corrective, adaptive, perfective or preventive), the scope of modification (the size of modification, the required cost and time), the involved audiences, and the consequences of the modification (impact in the navigational structure, performance, usability, etc.).

Thus, we claim that an operative site should be assessed not only from the quality perspective of the nonfunctional requirements, but also from the functional viewpoint like informational and navigational architecture. According to [Rosenfeld and Morville 1998], the main tasks of the information architect are: • Clarifying the mission and vision for the site, balancing the needs of its sponsoring

organization and the needs of its audiences, • Determining what content and functionality the site will contain, • Specifying how users will find information in the site by defining its organization,

navigation, labeling, and searching systems, • Mapping out how the site will accommodate change and growth over time.

Well-planned information architectures greatly benefit not only developers and maintainers but also different final audiences. In fact, a healthy navigational structure, an appropriate setting of services and contents, in addition to planned quality attributes are key factors for the success of WebApps. For instance, while modeling the navigational structure of a WebApp, we should take into account several aspects such as: a) what is the underlying structure of navigation (i.e., how the navigation space is organized); b) which objects will be navigated; c) which kind of composition structures exist among navigation objects; and also, d) what are the effects of a navigation. In order to represent functional requirements, the proposed methodology uses conceptual modeling techniques to capture the informational and navigational structure of operative WebApps [Pastor et al. 2001a; Pastor et al. 2001b].

Besides, to modeling nonfunctional requirements, a quality model is elaborated. In this model, quality characteristics have to be specified in an operational way. This means that for each quality characteristic and sub-characteristic not only a sound specification

should be given, but also associated attributes, metrics and criteria should be designed [Olsina and Rossi 2001]. The aim is that quality characteristics and attributes can be quantified and interpreted in a consistent and objective way.

Ultimately, by using the underlying process of WebFP_QEM, evaluators can give recommendations both by assessing functional and nonfunctional requirements of a WebApp in the operative phase. Moreover, if a change is suggested, the estimation of the maintenance cost can be done using a Web-centered Function Points measure [Abrahão et al. 2001]. Function Point Analysis (FPA) [IFPUG 1999] has become one of the most popular software functional sizing metrics. The functional points metric is an approach for the early evaluation of software characteristics such as size, productivity and cost. In addition, it provides a normalization factor that allows products to be compared independently of implementation technologies.

The rest of this paper is structured as follows. Section 2 provides an overview of the WebFP_QEM methodology with descriptions of its main steps; additionally, how to integrate these steps seamlessly is highlighted. We discuss a comprehensible example in the field of e-commerce in order to illustrate the methodology. Section 3 explains how to assess the current quality of a WebApp. Section 4 discusses how to analyze the results and give recommendations. Section 5 describes the web functional size measurement for the cost prediction report generation. Finally, in Section 6, a concluding discussion and further works are presented.

2. OVERVIEW OF THE WEBFP_QEM PHASES The WebFP_QEM methodology covers the major process steps in the evaluation process [ISO/IEC 1998] underlying in the modeling and analysis of the information and navigation architecture, and in the quality assessment of complex WebApps in the operative phase. The final goal is obtaining a document with strategic recommendations for decision-makers. These recommendations are based on the strengths and weaknesses of a WebApp obtained from quality measurements, conceptual modeling, and analysis activities. Thus, changes over or new functional and nonfunctional requirements can be identified.

As empirically demonstrated complex web application structures can increase maintenance costs considerably. Even worse, for continually changing web applications (considering their innate evolving nature), the cost can be even higher. In order to diminish risks, the maintenance cost can be predicted through a functional size measurement.

Therefore, to properly obtain a WebApp that addresses required functionalities and qualities within budgetary constrains, the appropriate actions to fulfill these requirements must be selected. These actions concern to the current quality modeling, to the conceptual modeling of the current and new functional requirements, in addition to the functional size measurement of the new obtained conceptual model. The functional size measurement is a model-based approach that takes into account the new nonfunctional requirements using adjustment factors.

Thus, the main process steps of the WebFP_QEM methodology are grouped in the following seven major technical phases: 1. Goal-Oriented Restructuring Analysis

a) WebApp goals determination (business, web-application) b) Audience determination c) Functional and nonfunctional requirements determination

2. Conceptual Modeling of an Operative WebApp (functional requirements) a) Construction of an Object Model

b) Construction of an Navigation Model i) Construction of a navigational map for each audience (user profile).

3. Quality Requirements Modeling (nonfunctional requirements) a) Construction of a Quality Requirements Tree for each audience b) Identification of characteristics and quantifiable web attributes c) Identification of relative importance of characteristics and attributes

4. Assessing the Quality a) Design and execution of elementary evaluation (for each attribute) b) Design and execution of global evaluation

5. Analyzing the Results (considering the established goals for each audience) a) Yielding a Recommendation Report with strategic recommendations for

decision-makers. i) Description of the strengths and weaknesses of a WebApp based on quality

measurements and conceptual modeling (1) Identification of changes over or new functional and nonfunctional

requirements. 6. Making the Desired Quality Explicit

a) Goal-Oriented Restructuring Analysis of new functional and nonfunctional requirements

b) Conceptual Modeling of new functional requirements 7. Functional Size Measurement of new requirements

a) Yielding a Cost Prediction Report of the WebApp maintenance.

Figure 1 shows the evaluation process model underlying the WebFP_QEM methodology, which includes the main steps, used models and tools, inputs and outputs. In order to discuss the major technical phases, they are thoroughly described in next sections.

3. ASSESSING THE CURRENT QUALITY OF THE WEBAPP 3.1 Goal-Oriented Restructuring Analysis The central aim of this process is to establish meetings with stakeholders in order to elicit and analyze functional and nonfunctional aspects of an operative site in its current state (targeting also implicit and explicit user needs). However, the restructuring requirements elicitation varies depending on users’ needs and the WebApp domain considered. As part of this process, the following activities are made in a rather simultaneous way: • WebApp goals determination (business, web-application), • Audience determination, • Functional and nonfunctional requirement determination.

The identification of users goals is interrelated with the business goals. The fulfillment of users’ needs for a software product could be justified by the impact a software product can have in reaching the different business and applications goals.

Two kinds (or levels) of goals can be determined, namely: business and web-application goals. The former embraces the business (e.g. fortifying the brand, adapting or extending the market segment, etc.), while the latter directly affects the components of the web application (e.g. the extension or incorporation of a new service for an existing audience, with certain level of performance or accessibility, etc.). The objective of a business goal is to improve the effectiveness of a Web site from the user's point of view, whereas the objective of an application goal is to generally extend the functionality of the application within quality and cost constrains. (It is important to remark that in the

context of our methodology, the goal determination will be mapped into the potential alternatives of maintenance, as indicated in Section 1).

On the other hand, one aspect that makes Web applications complex is that they usually support different audiences (user profiles). Differences between users can occur related for example to the played responsibility or role, and the specific application usage made by them. In addition, individual preferences can also exist based on uneven backgrounds or interests. Then, for instance, the quality could often be evaluated differently among different users.

Goal-OrientedRestructuring

Analysis

ConceptualModeling

QualityRequirements

Modeling

Assessing theQuality

Analyzing theResults

Measuring theFunctional Size

WebAppDomain

Current Conceptual Model

OOWSApproach

FunctionalRequirementsSpecification

Non-FunctionalRequirementsSpecification

Quality Models/WebQEM_Tool

Stakeholders

MetricsCatalog

EvaluationCriteria/ ScoringModel

Outcomes

RecommendationReport

QualityRequirementsSpecification

OOmFP-Web

Cost PredictionReport

Current / New Conceptual Model

Audience

Fig. 1. The evaluation processes underlying in the WebFP_QEM methodology. Regarding the audience, three abstract evaluation views of quality may be defined,

i.e., visitors, developers and managers views. For example, the visitor category can be decomposed in anonymous user and registered user subcategories. The former represents a casual or intentional audience maybe having a general interest and/or minimum domain knowledge; the latter represents an intentional or an expert in the domain (it is rather an habitual user that already knows the application).

One of the essential factors the user values is the WebApp’s functionality. The functional requirements defined for a web site is embedded in the application logic. Each component of the application logic has specific functions that must be performed in order to fulfill one or more system requirements. However, nonfunctional requirements (such

as usability, performance, scalability, etc.) are other aspects that influence the final quality of WebApps. In order to identify in a structured way different goals (and quality needs) related with the different business and application aspects, the involved audiences, and associated functional and nonfunctional requirements have to be modeled. Based on such a model the meaning and the importance of quality needs can be identified and priorities can be determined. Regarding the modeling of quality characteristics in relation to business characteristics there are no well-structured approaches yet. Nevertheless, as a way to capture the relevant information about restructuring goals we propose the use of the template as shown in Table1.

Table I. Template to specify goals, audiences, functional, and nonfunctional requirements.

Goal: To incorporate a “wish list” functionality to the current web application.

Scope: Application

Audiences: Registered User Functional Requirement 1: Objective: To place an item into the wish list to possibly be

purchased at a later time. Preconditions: The user is currently viewing an item Category: navigation and interface Main Steps:

Registered user selects to add an item to the wish list. System displays the current contents, if any, of the list with the addition of the new item Registered user selects to modify the Quantity of an item in the list. System updates the quantity and displays the contents of the list.

Alternative Steps: System detects that the wish list is at its maximum capacity.

Priority: high Functional Requirement 2: … Nonfunctional Requirement 1: Objective: To provide feedback (e.g. by means of a linked

control) to return to the previous context. Category / Attributes: usability / availability of feedback to the

previous context Priority: high Nonfunctional Requirement 2: …

Specific information about goal definition, scope and audiences as well as functional

and nonfunctional requirements is recorded. For instance, to a typical e-commerce application a goal can be “incorporates a wish list functionality to the current web application”. This goal concern to the application scope and the selected audience is a registered user. Besides, for each functional requirement the objective, preconditions, category, main steps, alternative steps and priority are described. The objective for a functional requirement can be “to place an item into the wish list to possibly be purchased at a later time”. Preconditions express constrains under which the functional requirement will work properly. For the above example, the precondition is

that the user should be viewing an item. The category indicates the level of the web application (information, navigation or interface, etc.) involved in the functional requirement specification; in the example of table 1, interface objects that are responsible for mediating user interaction with navigation objects. 3.2. Conceptual Modeling of WebApps The conceptual modeling process incorporates two parallel activities: information design and navigation design. The former results in the acquisition of content and functionally within the WebApp, while navigation design establishes the structure of the WebApp and the flow of user interactions. In fact, the following aspects must be stated when introducing a conceptual modeling approach: a) which are the specific conceptual constructs that should be used when dealing with a web application; b) where new features as navigation and presentation are definitely relevant; and c) which notation is provided to properly capture those conceptual modeling constructs.

We argue that conceptual modeling is needed for developing correct web applications, to support complex applications, and to deal with the flexible nature of the Web. Moreover, we consider navigation as a critical feature in the conceptual modeling of WebApps.

Thus, the key aspects analysed in this phase are how to model the web, and in particular, which are the conceptual patterns that must be captured in the conceptual modeling of an operative WebApp. In order to accomplish this objective, we introduce the conceptual patterns proposed by an object-oriented web-solutions modeling approach – the OOWS approach1 [Pastor et al. 2001b], justifying their necessity from the point of view of what is required by a web conceptual schema in terms of information and navigation structure. This approach uses well-known UML-compliant diagrams to represent the required conceptual information.

In this phase, problem domain peculiarities and the behaviour that a WebApp offers to satisfy the user functional requirements are captured based on the functional requirements specification and the determined Audiences. We are representing this information in the following models: the Object Model and the Navigation Model. When dealing with the conceptual modeling phase, the abstractions derived from the problem domain are specified in terms of classes, their structure, behaviour and functionality.

The Object Model is a graphical model where system classes including attributes, services and class relationships (aggregation and inheritance) are defined. In an aggregation hierarchy is necessary to define if we have an association or a composition. In an inheritance hierarchy is necessary to specify if a specialization is permanent or temporal. In the former case, the corresponding condition on constant attributes must characterize the relationship; in the latter, a condition on variable attributes or the carrier service that activates the child role must be specified [Pastor et al. 2001a]. Additionally, agent relationships are specified to state the services that objects of a class are allowed to activate.

Figure 2 shows a simple schema for an online magazine (only the part related to the wish list functionality is showed). As mentioned in Section 3.1, two kinds of users are represented, viz. anonymous and registered. An anonymous user can register itself performing the signIn service, passing to the registered user category. Thus, the registered user is a temporal specialization -stereotyped with the reserved word «temporal spec.», of the anonymous, inheriting their properties and behaviour. Each registered user 1 The OOWS approach was proposed as an extension of the object-oriented method for automatic code generation based on conceptual models called OO-Method [Pastor et al. 2001a].

has a wish list to add, modify and delete items (references to products) to possibly be purchased at a later time. These products can be CDs or DVDs; the prodType attribute is used as a specialization condition.

According to OOWS approach [Pastor et al. 2001b], two complementary conceptual models can be specified: the Dynamic Model and the Functional Model. In the former, valid object life cycles and inter-object communications can be represented. In the latter, the semantics associated to any change of an object state is captured as a consequence of a service occurrence. However, we consider the Object and Navigation models sufficient to modeling the underlying semantics of an operative WebApp.

Fig. 2. The Object Model

A Navigation Model is obtained by adding a navigational view over the Object Model. The navigational semantics of a web application is captured based on the point of view of each agent (audience) identified in the Object Model. This model is essentially composed of a navigational map that represents the global view of the application for an audience. It is represented by a directed graph where nodes are navigational contexts and arcs are navigational links. A navigational context represents the perspective a user has on a subset of the object model, while a navigational link allows navigating from one context to another. Navigational contexts can be classified into two types: exploration contexts (E) and sequence contexts (S). The former can be reached at any moment independently of the current context (e.g. landmarks), whereas the latter can only be reached following a predefined sequence of navigation paths.

Figure 3 shows a part of the navigational map for the anonymous or registered users of the Amazon WebApp. There are four exploration contexts for organizing the navigation space. These contexts are related with the landmarks that appear in the Amazon main navigational menu.

Fig. 3. A Navigational Map

Anonymous

sigIn()

CD DVD

Productdescription : Stringtype : StringshipTime : datelistPrice : floatourPrice : float

type = "cd"<<temporal spec.>>

type = "dvd"<<temporal spec.>>

Registeredname : String

Anonymous.sigIn<<temporal spec.>>

Itemquantity : intdesired : intpurchased : int

<<new>> new_item()<<destroy>> destroy_item()modifyQuantity()add_to_Cart()

1..10..* 1..10..*

Wish_ListtotalItens : int

0..11..1

0..*

1..1

0..*

1..1

0..11..1

<<agent>>

<<Context>> View Cart

EE <<Context>>

Wish List

E <<Context>>

Your Account list

E <<Context>>

Help

E

Registered

<<Context>> Home

A navigational context is composed by a set of classes -stereotyped with the reserved word «view», connected by navigational relationships. These classes are called navigational classes and they represent projections (views) over specific classes in the object model. These views make a restriction to a given subset of attributes and services of the involved classes.

For each navigational context there is a main class that is called manager class from where navigation starts. The others classes are called complementary classes and they contribute to giving additional information to instances of the manager class. These classes are connected using navigational relationships, providing related information between them. Service links can be defined associated to class services. A service link connects with a target navigational context when the execution of the associated service is completed.

A navigational relationship is a unidirectional binary relationship between two navigational classes that must be defined over an existing aggregation or inheritance class relationship in the Object Model. There exist two types of navigational relationships: context relationship and contextual dependency relationship.

A context relationship is a navigational relationship that also defines a directed navigation from this navigational context towards the target one, where the target class of the context relationship acts as the manager class. Graphically they are represented using solid arrows. Four kinds of attributes can be specified to this type of relationship: • Context attributes: specifies the target navigational context of a navigational link. • Link attributes: specifies which attribute of the target class is involved in the

connection. • Role attributes: indicates the referenced relationship-role when two classes have

more than one relationship. • Filter attributes: introduces selection criteria on the population of the target class,

based on the value of the involved attribute. In this way, the search for specific objects is made easier. For a filter, three basic behaviours can be specified: exact (E), approximated (A) or range (R).

In a contextual dependency relationship, the navigational semantics towards the target class is not defined: this kind of relationship is used to provide the required additional information in the current context without denoting any further navigation. Thus, it is not necessary to specify any kind of related information. Graphically it is represented using dashed arrows.

Finally, in the advanced features area, the following information can be specified during the design phase: population filters and order. A population filter established a selection of objects over the manager class. It is represented by a logical expression evaluated on the state of these classes. The order primitive indicates a traversal order of the context elements.

A brief example that includes the above comments is shown in Figure 4. This example shows how a Wish List of an e-commerce application can be specified using the OOWS approach. The Wish_List context describes all the products selected by a Registered user.

The navigation starts in the Wish_List class that acts as the manager class. The totalItems attribute from the Wish_List class and the name of the connected user will be shown. The Registered and Item classes are complementary classes and they provide additional information related to the manager class. In order to provide this information, the Whish_list class has a contextual dependency relationship with them.

For instance, the relationship from Wish_List class to Item class presents the quantity, desired and purchased attributes of a wish list. Also, the user can modify the quantity of an item or add to a shopping cart, beginning the sale process. The service link associated to the add_to_Cart service indicates the target navigation context after the service

execution. Additionally, a context relationship appears: from an Item, users can obtain more details of a Product (Product_Details context) by selecting its description. In addition, a population filter over the wish_list class is accomplished in order to restrict the retrieved information about the interactive registered user. The items can be arranged by its description or price.

Fig. 4. A Navigational Context

We claim that using these few presented conceptual patterns it is possible to specify

the semantics attached to the information architecture and navigation requirements for web-based applications. In the end of this phase, a Current Conceptual Model representing the functional structure of a WebApp is produced.

3.3 Quality Requirements Modeling From the Nonfunctional Requirements Specification document built in the Goal-Oriented Restructuring Analysis phase, a quality model should be selected (e.g., a fixed model such as ISO 9126-1 [ISO/IEC 2001], or a user-defined model –without adhering to standards, or a mix of both) in order to specify systematically characteristics, sub-characteristics and attributes.

In previous case studies [Olsina et al. 2000] the ISO-prescribed characteristics as well as sub-characteristics and attributes customized to the Web domain were used. The relative importance of these components (mapped from the priority item of nonfunctional requirements as shown in Table 1) should be identified considering the audience and the extent of the required coverage. For instance the relative importance of attributes and characteristics are modeled by means of weights, as we explain in Section 3.4. Thus, taking into account the product descriptions (in our case, an WebApp or parts of it), the agreed goals, the level of coverage, and the selected audience, characteristics, sub-characteristics, and attributes should be specified in a form of a quality requirement tree. In the end of this process, a Quality Requirements Specification document is produced. WebQEM_Tool allows reusing (from previous evaluation projects), managing, and visualizing quality (or cost) requirements.

For instance, in the quoted case study, the Usability characteristic was split up in sub-characteristics such as Global Site Understandability, Feedback and Help Features, and Interface and Aesthetic Features. In turn, these sub-characteristics were hierarchically

Wish_List<<context>>

Population Filter: wish_list.registered = #registered# Order: {Product.description OR Product.ourPrice}

Product.description

If (Product.type = “cd”) [CD] elseif (Product.type = “dvd”) [DVD]

Wish_ListtotalItens

<<view>>

#Registered#name

<<view>>

ProductdescriptiontypeshipTimeourPrice

<<view>>

Itemquantitydesiredpurchased

modifyQuantity()add_to_Cart()

<<view>>

*[View Cart]

E

decomposed in sub-characteristics and attributes, as can be observed in the right pane of figure 5. In [Olsina et al. 2000] the reader can see a detailed description of a quality requirement tree customized to e-bookstores.

Fig. 5: One of the WebQEM_Tool windows where characteristics, sub-characteristics and attributes can be related (for the quality factor) in order to form

a requirement tree. 3.4 Assessing the Quality In this phase, two major stages are defined: the design and the implementation (execution) of the elementary and the global evaluation. Inputs to this process are the Quality Requirements Specification, the Metrics Catalog, Elementary Evaluation Criteria and an Aggregation and Scoring Model. This is analyzed in the next two sub-sections.

3.4.1 The Elementary Evaluation Stage. The elementary evaluation should be

designed and implemented. For each measurable attribute Ai from the requirement tree, a dependent variable Xi can be associated, which will take a numerical value from a direct or indirect metric. However, the value of this metric will not represent the level of satisfaction of this elementary requirement at all. For that reason, it is necessary to define an elementary criterion function that will result afterwards in an elementary indicator or preference value.

For instance, let us consider the Broken Links attribute, which measure (count) links that lead to missing destination pages. A possible indirect metric is: X = #Broken_Links / #Total_Links_of_Site. Now, how do we interpret the measured value? Or, what are the best, worst, and intermediate preferred values? The next formula represents a possible criterion function to determine the elementary quality preference EP:

EP = 1 (or 100%) if X = 0; EP = 0 (or 0%) if X >= X max ; otherwise EP = (X max – X) / X max if 0 < X < X max where X max is some agreed upper threshold such as 0.03

Editing Menu

Requirement tree

Attributes List

Delete Button

Add Button

So the elementary quality preference EP is frequently interpreted as the percentage of satisfied requirement for a given attribute, and it is defined in the range between 0, and 100% (so the scale type and the unit of metrics become normalized). Furthermore, to ease the interpretation of preferences, we could group them in three acceptability levels, namely: unsatisfactory (from 0 to 40%), marginal (from 40 to 60%), and satisfactory (from 60 to 100%).

On the other hand, in the implementation stage, the selected attributes’ metrics are applied possible taken from a repository of catalogued metrics. Some values can be measured observationally, while others can be obtained automatically by using computerized tools. In order to record the information that is needed during the evaluation processes, we have defined a descriptive specification framework [Olsina and Rossi 2001] that is implemented in the WebQEM_Tool.

Once evaluators have designed and implemented the elementary evaluation process they should be able to model attributes, sub-characteristics, and characteristics relationships. They should consider not only the relative importance of each attribute in the group but also if the attribute (or sub-characteristic) is a mandatory, an alternative or a neutral one. For this purpose, a robust aggregation and scoring model may be needed.

3.4.2 The Global Evaluation Stage. Again, two major steps are defined: the design

and the implementation of the partial/global quality evaluation. In the design stage, aggregation criteria and a scoring model should be selected. The goal of quantitative aggregation and scoring models is to make the evaluation process well structured, accurate, and comprehensible by evaluators. There are at least two types of models: for example those based on linear additive scoring models, and those based on nonlinear multi-criteria scoring models [Gilb 1976; Olsina and Rossi 2001] where different attributes and characteristics relationships can be designed. In both cases, the relative importance of indicators is considered by means of weights.

For example, if our procedure is based on a linear additive scoring model the aggregation and computing of partial/global indicators or preferences (P/GP), considering relatives weights (W) is based on the following formula:

P/GP = (W1 EP1 + W2 EP2+ ... + Wm EPm); (1)

such that if the elementary preference (EP) is in the unitary interval range the following is held: 0 <= EPi <= 1 ; or given a percentage scale, 0 <= EPi <= 100 ; and the sum of weights must fulfill that

(W1 + W2 + ... + Wm ) = 1; if Wi > 0 ; to i = 1 ... m;

The basic arithmetic aggregation operator for inputs is the plus (+) connector. The above (1) expression cannot be used to model simultaneity or replaceability of inputs, among other limitations. However, the nonlinear multi-criteria scoring model can fill the gap (this model is not discussed here for space reasons –see [Olsina and Rossi 2001] for more details).

Therefore, once a scoring model has been selected, the aggregation process follows the hierarchical structure as defined in the non-functional requirement tree, from bottom to top. Applying a stepwise aggregation mechanism, a global schema can be obtained in the end. This model allows us to compute partial and global indicators in the implementation stage. The global quality preference represents ultimately the global degree of satisfaction in meeting the stated requirements. Lastly, in the end of this process, an Outcomes report is produced.

4. ANALYZING THE RESULTS AND GIVING RECOMMENDATIONS

Once the conceptual model of a current WebApp and the final execution of the quality evaluation were performed and agreed, decision-makers are ready to analyze the results and draw recommendations (see this phase in Fig. 1). Besides in this phase, the documentation of Web product components, the current conceptual model, the specification of quality requirements, metrics, criteria, elementary, partial, and final results should be recorded.

Currently, from the technological point of view, WebQEM_Tool supports just the quality evaluation of nonfunctional requirements generating a recommendation report. The report system does not integrate yet the recommendation document for the conceptual modeling (this is a work in progress).

The recommending system of WebQEM_Tool shows in a Web browser the textual, tabular and graphical results through dynamically generated and linked pages. The pages include information about the evaluation project, the linked requirement tree, the tables with attributes, sub-characteristics, and characteristics indicators, the diverse analyzes shown in tabular and chart form, as well as recommendations.

Figure 6, shows a partial view of the recommendation report for the Amazon e-bookstore. This screenshot not only shows a statistic of the number of characteristics, sub-characteristics and attributes falling in the different acceptability levels, but also for example what attributes urge changes. As commented in Section 3.4.1, to ease the interpretation of preferences, three basic categories or acceptability levels are modeled, i.e. unsatisfactory (from 0 to 40%), marginal (from 40 to 60%), and satisfactory (from 60 to 100%). So a scoring within a marginal rating level can be observed as though improvement actions should be considered, as long as an unsatisfactory rating level can be interpreted as though necessary and urgent change actions must be taken.

Figure 6. A partial view of a Recommendation Report generated by WebQEM_Tool

Finally, a scoring within a satisfactory rating level can be interpreted, as no change is required for the analyzed feature. Thus, the strengths and weaknesses of the assessed product with regard to established goals and audience could be analyzed and understood by requesters and evaluators. Recommendations can be suggested and justified because the underlying mechanism of WebQEM_Tool favors the backward and forward traceability model.

Regarding the current conceptual model, key questions are, what primitives and structures in the conceptual modeling affect the quality of a WebApp?

The main issues related to the information architecture deal with analyzing if the conceptual model (Object and Navigation Models) of the current WebApp represents all the information and functionality required for the business and application goals. From the information point of view, we carry out the analysis of Object Model according to the following dimensions: • Content: does the content of a current WebApp matches what is supposed to be

there? Thus, classes are verified with regard to goals determination. • Service: do the services of a current WebApp matches what is supposed to be there?

This can be traced from the functional requirements. • Structure: How well do all classes of the object model hold together? Are there

parts of the WebApp that are not connected? The structure of classes and relationships are verified.

From the navigational architecture point of view, we consider among other features

the followings: • Cohesion level of a navigational context: the core idea is the same as in the

traditional O-O concept of cohesion (i.e. for methods or services), however, adding also the aspect of cohesiveness of classes in a navigational context. The higher the degree of “similarity” of classes in a context, the greater the cohesiveness and the higher the degree of encapsulation of the context.

• Coupling level among navigational contexts: Traditionally, coupling is when one object uses services or attributes of other. This concept is enlarged to contexts: the larger the number of couples, the greater the degree of interdependence and difficulty of maintenance, and the lower the potential for reuse.

Features as Depth level of a navigation context, and Breadth and depth of a

navigation map for a given Agent, where also considered. It is worthwhile to notice that we are following a heuristic-based approach in order to guide this analysis process. For instance, Rosenfeld and Morville suggest in p.38 [Rosenfeld and Morville 1998], that it is important to consider the balance between breadth and depth. “Breath refers to the number of options at each level of the hierarchy. Depth refers to the number of levels in the hierarchy. If a hierarchy is too narrow and deep, users have to click through an inordinate number of levels to find what they are looking for. If a hierarchy is too broad and shallow, users are faced with too many options…”

They propose a heuristic such that Web designers should minimize the number of options on the home page to ten and minimize depth to less than five levels. One of our lines of research is how heuristics can be mapped to metrics for conceptual models. In addition, how service patterns can be quantified regarding for instance the consistency of use in a design.

During the assessment process of an operative WebApp, the sequence of events that happens as result of a restructuring must be initiated with an analysis of impact of changes of the requested reorganization. This analysis must consider aspects such as the

type of maintenance (corrective, adaptive, perfective or preventive), the scope of modification (the size of modification, the required cost and time), the involved audiences, and the consequences of the modification (impact in the navigational structure, performance, usability etc). As final step of the analysis process, the Goal-Oriented Restructuring Analysis task should be carried out again with the purpose of contrasting the requirements described initially with the result of the evaluation. The new functional and not functional requirements could also be specified. 5. WEB FUNCTIONAL SIZE MEASUREMENT As a previous step to the functional size measurement, the conceptual modeling of new functional requirements should be accomplished as mentioned in subsection 3.2. In this phase, we propose an approach to predict the size of changes of evolving WebApps based on the current / new conceptual models.

With the current conceptual model measurement, the functional size of an on-line WebApp in its current state (before the quality evaluation) can be determined. Measuring the new conceptual model, the functional size of a WebApp after the quality evaluation can be estimated. By comparing both measurements, it is possible to predict the functional size of changes, and consequently, the maintenance cost of the web application.

However, as reported in [Reifer 2000; Cleary 2000; Symons 2000] the current estimation models must be adapted to address the estimation challenges for controlling project for the development activities involved in WebApps production. To achieve this goal, we extend a measure for object-oriented systems from conceptual models [Abrahão and Pastor 2001; Pastor et al. 2001] to the web environment. The functional size measurement process of a web application is accomplished by mapping Function Points concepts to the Conceptual Model primitives. The mapping rules are based on the standard FPA defined in the IFPUG Count Practice Manual [IFPUG 1999]. This mapping is conducted in the following steps: 1. The conceptual models are analyzed to determine the application boundary, and the

measuring scope. 2. The conceptual models are analyzed to identify the data and transactional candidate

function types. 3. The complexity of each identified function is determined. The logical functions are

mapped for low, average and high complexity levels in accordance with IFPUG tables [IFPUG 1999].

4. The complexity levels are mapped into values. The total value results in unadjusted OOmFP-Web (OO-Method Function Points for Web) [Abrahão et al. 2001].

5. The OOmFP-Web value is adjusted according to the general characteristics of the WebApp taking into account the nonfunctional requirement specification.

5.1 Defining the Application Boundary and the Measuring Scope The application boundary indicates the border between the project being measured, and the external applications or user domain. It defines what is external and what is internal to the application according to the user’s point of view. The measuring scope limits which functionality will be measured in a particular measurement (the (sub)set to be measured).

The scope of current web applications varies widely: from small web services to large enterprise applications distributed across the Internet. For instance, in a large e-commerce application can have many sub-stores that correspond to the different subsystems of a web application. Thus, a measuring scope can include one application and its subsystems,

all functions of a subsystem, some web services, or the functionality provided to a specific user, etc. In our approach, we define the application boundary from the navigational map of each audience. The proposed mapping rule is to accept each agent as a user of the WebApp. 5.2 Identifying Data and Transactional Function Types In the OO paradigm, an object is a collection of data (attributes and properties) and functional logic (methods). The data defines the state of the object and the functional logic defines the behavior of the object. In accordance to the FPA concepts, data is a logical group maintained or modified through an elementary process (method), and a transaction must have processing logic that is unique and represents the smallest unit meaningful to the user. Based on these considerations we have identified the following functions (see [Abrahão et al. 2001] for a more detailed discussion):

• Navigational Classes: Select every navigational class (manager or complementary)

of a navigation context as a candidate for an Internal Logical File function (ILF). In Function Point terms the parts of an object corresponds to the logical structure of the file concept. Optional and mandatory subgroups of files are called Record Element Types (RETs). An object that is aggregated into (part of) another object constitutes a subgroup.

• Method: It is a unit of functional logic contained within an object. Then, select every

method of a navigational class as a candidate for an External Input function (EI). • Context Relationship: The relationships between navigational classes express

inherits or aggregation relationships in the Object Model. This kind of relationship provides the ability to access a unique navigational context as the set of data, retrieved from the navigational classes (ILFs). A context relationship expresses navigation to another context (the target context). In this kind of relationship, two behaviors can occur: a) when the primary user input is navigating without change the state of application, no information crosses the boundary; b) when a hyperlink that sends a parameter that is used to search could be an example of an EQ. The current applications require high user interaction where the navigation may influence the content of a navigational context. That is, the hyperlink follows the rules required for an EQ: there is an input side (the parameter) and there is an output side the results of the search. In this case the output side is dynamic and changes.

• Navigational Context: Select each navigational context as a candidate for an

External Output (EO) or External Inquiry (EQ) function. An EO or EQ must reference at least one internal logical file and / or one external interface file. Both an internal logical file and an external interface file must be a logical group of related information. Each navigational context has a main class (manager class) from where navigation starts, and others classes (complementary classes) to giving additional information to instances of the manager class.

5.3 Establishing the Complexity for Data and Transactional Function Types As in FPA, key to developing repeatable predictor counts is a well-defined set of counting rules. Thus, to each navigational class a functional complexity is assigned according to the number of Data Elements Types (DETs) and the number of Record

Element Types (RETs). The proposed counting rules for DETs and RETs identification of a navigational class are: • Count a DET for each attribute of class • If exists a filter associated to the class, count a DET by each attribute that appears in

the formula of the filter • Count a RET for the navigational class • Count a RET for each context or contextual dependence relationship that has the

source the class that is being considered. Each one of these relations is a group of identifiable data by the user (target navigational class)

• If exists a filter associated to the class, count a RET by each navigational class that is referenced in the formula of the filter. Whenever this supposes the reference to a class that has still not been counted

The transaction functions (EOs and EQs) will be identified from each navigational

context (exploration or sequence). If in some of these contexts the attributes of the participant navigation classes indicates some calculation (derived attributes, complex filters, etc.) we will consider an EO. In the opposite case it is considered an EQ.

To each navigational context a functional complexity is assigned based on the number of Data Element Types (DETs) that processes the elementary process, and the number of File Type Referenced (FTRs) to which such a process accedes. Each EO or EQ function has two parts: the input side where the user provides the information and the output side where the result is presented to the user. We are considering only the DETs unique that cross both sides.

Then, the proposed counting rules for DETs and FTRs identification of a navigational context are: • Count a DET for each attribute of all navigational classes that appears in the context. • Count a DET for each attribute (context, link, filter or role) of a context relationship. • Count a DET for each service that can be performed. • Count a DET for the traversal order of the elements of the context. • Count a FTR for each navigational class that appears in the context. • Count a FTR for each navigational class referenced in the definition of a filter

attribute (in a context relationship). • Count a FTR for each class referenced in the population filter formula.

Finally, each service of a navigational class will be considerate an External Input (EI).

Nevertheless, if a same service can be executed from different contexts by the same agent, it is counted only once. The arguments of a certain service will be those that are defined for that service in the Object Model. As well as previously mentioned, to each identified service a functional complexity based on the number of DETs and FTRs is assigned. The dynamics for a service and the rules for determining the functional complexity are the same that those described in [Pastor et al. 2001c].

Using the counting rules above described, we have been able to predict the partial size of a WebApp, i.e., the size of each navigational map according to the functionality offered to the agents of the system. (the functionality provided by the Navigation Model). Integrating this partial result with the functional size obtained from applying the OOmFP metric (functionality provided by the others OO-Method conceptual models) the functional size of whole WebApp is obtained.

5.4 Achieving the Cost Prediction The definition of a metric for size measurement is a first step in developing a model that accurately predicts web development costs. Software size is a normalizing reference measure that can be successfully used as the base for a range of planning and control indicators, including: studies for the software production (function points per person-month), financial (cost analysis) [Cost Xpert Group 2002], software quality (defects per function points) [Cowderoy 1999], and to measure the functionality and productivity of web applications [Morisio 1999].

Most organizations use measures based on function points to assist in estimating project costs, effort and duration. Next, we discuss how the function points measure can be used in conjunction with financial indicators. These indicators can include value of the software, project costs, IT costs, and maintenance costs. The followings are examples of financial indicators: • Asset Value: (Cost /FP) x Portfolio Size • Project Costs: Total Project Costs /FP • Enterprise Costs: Total IT Costs /FP • Maintenance Costs: Total Cost of Maintenance / FP (including overheads)

The asset value places a value on the installed WebApp. The cost can be replacement cost, total cost, etc.; an organization has to establish which of these it will use. The project cost is the unit cost per Function Point that includes all costs the development project occurs, and the total project cost is derived from:

Total Project Cost = (work effort hours x hourly cost) + other expenses The enterprise cost is the cost equivalent to the enterprise productivity defined earlier.

It measures the total cost to the enterprise, including overhead cost, to deliver the new or enhanced functionality to the user. Finally, the maintenance cost is the cost equivalent to the WebApp support rate. It measures the all cost of application maintenance and support. The total cost of maintenance is obtained by:

Total Cost of Maintenance = (Hours of Work Effort x Hourly Costs) + other expenses Thus, size-based measures have traditionally been the basis for estimation models to

predict costs of software activities. In the context of our project research, efforts have been oriented towards the development of a new Cost Prediction Model that address the issues discussed here, supporting the proposed web functional size metric in order to produce a Cost Prediction Report. 6. DISCUSSIONS AND FURTHER WORK Because of the increasing size, complexity, and quick time-to-market cycles for WebApps, together with an unsystematic use of Web Engineering approaches, several pitfalls have been sadly neglected such as exceeding budgets, delivered products do not having the required functionality and quality, and generally lacking of documentation for requirements, information and navigation architecture, amongst others. As said elsewhere [Baskerville et al. 2001], “ In the Internet speed development developers often defer certain aspects of product quality such as scalability and maintainability during the original development. If the software doesn’t catch the market, no time was lost adding quality”.

As a consequence, there are increasing concerns about the ways in which WebApps are developed, maintained, and their quality delivered. There are compelling reasons for a

systematic and disciplined use of engineering methods and tools for developing, assessing and maintaining Web sites and applications.

Although different types of methods and techniques exist for evaluating quality and functionality such as testing, inspection, inquiry, simulation, etc, in the current practice, many of the used methods and techniques provide only partial solutions for the actual state of WebApps, because they separately focus either on nonfunctional requirements or on functional requirements. In fact, flexible and integral solutions are needed.

In this paper, we have claimed that an operative site should be assessed not only from the quality perspective of the nonfunctional requirements, but also from the functional viewpoint like the informational and navigational architecture. In addition, we have shown that WebFP_QEM (Web Function Points and Quality Evaluation Method) can be a useful approach for analyzing, assessing and potentially restructuring a WebApp in the operative phase. WebFP_QEM allows capturing business and application goals for different audiences, specifying informational and navigational models (the functional requirements), specifying quality models and assessment criteria (the nonfunctional requirements), analyzing outcomes in order to give recommendations for improvements, in addition to drawing cost prediction reports.

So far, we have used this integral approach in a real case study. Specifically, the CYTED (Programa Iberoamericano de Ciencia Y Tecnología para El Desarrollo) web site was assessed in order to determine the current functional size and quality. The informational and navigational architecture for a specific audience was obtained from scratch due to the documentation was missing (see [Abrahão et al. 2001] for more details). In the end, a recommendation report was delivered, however, the restructuring of the site is still an ongoing project.

It is worthwhile to remark that effective and efficient assessment and restructuring processes require both methodological and technological support.

There are many issues we need to address in the future. From the methodological point of view, we need to address: • A smooth integration of the Recommendation Report, including not only specific

sections for the quality report (as currently is in WebQEM_Tool), but also the sections for information and navigation architecture,

• A deeper research how heuristics can be mapped to metrics for conceptual models. In addition, how service patterns can be quantified regarding for instance the consistency of use in a design,

• A cost prediction model that integrates all the steps of methodology in a disciplined and repeatable manner, and

• A more robust adjusted model for the function points metric for the Web. From the technological point of view, we have developed different supporting tools

(WebQEM_Tool, Case OO-Method) however they need to be integrated. This technological support for the WebFP_QEM methodology is a future endeavor.

Finally, our contribution (the WebFP_QEM methodology) is the result of a productive effort by integrating pre-existing methodologies, techniques, models, and tools developed in research groups of two countries. This an amazing starting point with open end.

REFERENCES ABRAHÃO, S. M., AND PASTOR, O. 2001. Estimating the Applications Functional Size from Object-

Oriented Conceptual Models. In Proceedings of IFPUG Annual Conference, Las Vegas, USA, September 2001.

ABRAHÃO, S. M., PASTOR, O., OLSINA L., AND FONS, J. J. 2001. Un Método para Medir el Tamaño Funcional y Evaluar la Calidad de Sitios Web. In Proceedings of VI Jornadas de Ingeniería de Software y Base de Datos, Almagro, Spain, November 2001, 478-491. (In Spanish)

BASKERVILLE, R., LEVINE, L., PRIES-HEJE, J., BALASUBRAMANIAM, R., AND SLAUGHTER, S. 2001. How Internet Software Companies Negotiate Quality. IEEE Computer 34(5), 51-56.

CLEARY, D. 2000. Web-Based Development and Functional Size Measurement. In Proceedings of IFPUG Annual Conference, San Diego, USA, September 2000.

COST XPERT GROUP, Inc. 2002. Estimating Internet Development, http://www.costxpert.com/Reviews_Articles/SoftDev/

COWDEROY, A. J. C. 1999. Size and Quality Measures for Multimedia and Web-Site Production. In Proceedings of 14th International COCOMO/SCM Forum, Los Angeles, USA.

CUTTER CONSORTIUM. 2000. Poor Project Management – Problem of E-Projects, October 2000, http://www.cutter.com/consortium/press/001019.html

DUSTIN, E., RASHKA, J., AND MCDIARMID, D. 2002. Quality Web Systems: Performance, Security, and Usability. Addison-Wesley.

FENTON, N.E., AND PFLEEGER, S.L. 1997. Software Metrics: a Rigorous and Practical Approach, 2nd Ed., PWS Publishing Company.

FLEMING, J. 2001. Web Navigation: Designing the User Experience, O’Reilly & Associates. GILB, T. 1976. Software Metrics, Chartwell-Bratt, Cambridge, MA. IFPUG. 1999. Function Point Counting Practices Manual, Release 4.1, International Function Points Users

Group, Mequon, Wisconsin, USA, April 1999. ISO/IEC 9126-1. 2001. Software engineering -- Product quality -- Part 1: Quality model. ISO/IEC 14598-5.1998. International Standard, Information technology -- Software product evaluation -- Part

5: Process for evaluators. KIRAKOWSKI, J., WHITEHEAD, R., CLARIDGE N. 1998. Human Centered Measures of Success in Web

Site Design. In Proceedings of 4th Conference on Human Factors & the Web, Basking Ridge, New Jersey, USA.

MENDES, E., MOSLEY, N., AND COUNSELL S. 2001. Web Metrics – Estimating Design and Authoring Effort. IEEE Multimedia, January-March 2001, 50-57.

MORISIO, M., STAMELOS, I., SPAHOS, V., AND ROMANO, D. 1999. Measuring Functionality and Productivity in Web-based Applications: a Case Study. In Proceedings of METRICS'99, Florida, USA, November 1999, 111-118.

NIELSEN, J. 2000. Designing Web Usability: The Practice of Simplicity, New Riders Publishing. OLSINA, L., LAFUENTE, G. J., AND ROSSI, G. 2000. E-commerce Site Evaluation: a Case Study. In

Proceedings of 1st International Conference on Electronic Commerce and Web Technologies, London, UK, Springer Verlag, 239-252.

OLSINA, L., ROSSI, G. 2001. A Quantitative Method for Quality Evaluation of Web Sites and Applications, To appear in IEEE Multimedia Magazine, (accepted in mid-2001).

OLSINA L., PAPA, M.F., SOUTO, M.E., ROSSI, G. 2001. Providing Automated Support for the Web Quality Evaluation Methodology. In Proceedings of Fourth Workshop on Web Engineering, at the 10th International WWW Conference, Hong Kong, 1-11.

PASTOR, O., GÓMEZ, J., INSFRÁN, E., AND PELECHANO, V. 2001. The OO-Method Approach for Information Systems Modeling: From Object-Oriented Conceptual Modeling to Automated Programming. Information Systems 26(7), 507–534.

PASTOR, O., ABRAHÃO, S. M., AND FONS, J. J. 2001. Object-Oriented Approach to Automate Web Applications Development, In Proceedings of 2nd International Conference on Electronic Commerce and Web Technologies, Munich, Germany, Springer Verlag, 16-28.

PASTOR, O., ABRAHÃO, S. M., MOLINA, J. C., AND TORRES, I. 2001. A FPA-like Measure for Object-Oriented Systems from Conceptual Models. In Proceedings of 11th International Workshop on Software Measurement, Canada, Shaker Verlag, 51-69.

POOLEY, R., SENIOR, D. AND CHRISTIE, D. 2002. Collecting and Analyzing Web-Based Project Metrics. IEEE Software 19 (1), January/February 2002, 52-58.

REIFER, D. J. 2000. Web Development: Estimating Quick-to-Market Software. IEEE Software, November/December 2000, 57-64.

ROSENFELD, L., AND MORVILLE P. 1998. Information Architecture for the WWW, O’Reilly & Associates. SYMONS C. 2000. E-Commerce Changes the Economics of Software Production”, Newsletter of Software

Measurement Services, Issue 6, Spring 2000. WWW CONSORTIUM. 2002. WAI Accessibility Guidelines: Page Authoring, W3C Working Draft. WAI

Accessibility Guidelines: Page Authoring, http://www.w3c.org/TR/WD-WAI-PAGEAUTH/, 2002. ZELNICK, N. 1998. Nifty Technology and Nonconformance: The Web in Crisis, Computer, October 1998, 115

–116 and 119. Received April 2002; Revised May 2002; accepted May 2002.