Internet and Higher Education 9 (2006) 63–75

The Web as a source of unconventional research materials in secondlanguage academic writing

Paul Stapleton a,⁎, Rena Helms-Park b,1, Pavlina Radia c,2

a Institute of Language and Culture Studies, Hokkaido University, Kita 17 Nishi 8, S 318 Gengobunkabu,Hokkaido University, Sapporo 060-0817, Japan

b Linguistics, Division of Humanities, University of Toronto at Scarborough, H525, 1265 Military Trail, Toronto, Ontario, Canada, M1C 1A4c English for Academic Purposes, Division of Humanities, University of Toronto at Scarborough, H427, 1265 Military Trail,

Toronto, Ontario, Canada, M1C 1A4

Accepted 13 December 2005

Abstract

This study examined 68 Web sources selected by 19 second-language (L2) students while preparing to write research papers.Students submitted an annotated bibliography consisting of ten sources from print or electronic media. Each Web source wasclassified according to type (e.g., news or advocacy). Of the 68 sites, 29 were considered conventional, i.e., similar to materialshoused in libraries (e.g., books and journal articles), and eight were “dead” links. The remaining 33 were rated unconventional,consisting of materials from interest groups, commercial enterprises, and informal academic materials. This paper presents anassessment of the 33 unconventional sites based on WATCH, a rating scale consisting of four broad criteria: author's reputation; thesite's objectivity; its academic rigor; and the transparency of its publishing information (Stapleton, P., and Helms-Park, R. (inpress). Evaluating Web sources in an EAP course: Introducing a multi-trait instrument for feedback and assessment. English forSpecific Purposes). In addition, the paper analyzes seven unconventional sources to illustrate the need to focus on Websiteevaluation skills in academic contexts.© 2005 Elsevier Inc. All rights reserved.

Keywords: Web-based sources; Website evaluation; Critical thinking and electronic literacy; L2 academic writing

1. Introduction: the challenge of rating students' Web sources

Until recently most citations in academic papers referred to print sources found in a library, for example, books,journals, and, sporadically, unpublished materials such as dissertations and conference papers. Although other media,such as newspapers, television, and radio broadcasts also appeared in citations on occasion, the primary informationsource came in a paper-and-ink format and was housed in a library in some sort of bound form. With the advent of theInternet, however, Web pages are increasingly being cited in scholarly published papers of as well as students' essays.

⁎ Corresponding author.E-mail addresses: paulstapleton@gmail.com (P. Stapleton), rhelms@utsc.utoronto.ca (R. Helms-Park), pradia@chass.utoronto.ca (P. Radia).

1 Tel.: +1 416 287 7142, +1 416 324 8492(Home); fax: +1 416 287 7116.2 Tel.: +1 416 322 3165; fax: +1 416 287 7116.

1096-7516/$ - see front matter © 2005 Elsevier Inc. All rights reserved.doi:10.1016/j.iheduc.2005.12.003

64 P. Stapleton et al. / Internet and Higher Education 9 (2006) 63–75

The fact remains, however, that material found on the Web often originates in sources that are fundamentally differentfrom the ones found in print. Many pages on the Web come from mostly unscreened sources that are peculiar to thismedium, for instance, companies, NGOs, individuals, and discussion groups. Moreover, despite surface similarities,the standard method of arriving at a site is through the search engine, which is considerably different from a librarycatalogue. Rather than looking for matches to key words, as a library catalogue search does, search engines usealgorithms that largely produce results based on key words in the opening paragraphs of a text, as well as the relativerate at which a source page has been linked to other pages (i.e., a function of the site's popularity) (Sullivan, 2003).

While this new era of “electronic literacies” has undeniably broadened the scope and possibilities of academicresearch (Shetzer & Warschauer, 2000), the unfettered ability to publish Websites also means that a significantproportion of sites on the Internet are not suitable for citation in academic papers (Barker, 2004; Kirk, 2002). Thefundamentally different nature of the Web as a source of information compared with conventional sourcing has beenexplored at length, mostly by librarians, who have highlighted the need for greater critical appraisal of Web sources (seeBarker (2004) and Kirk (2002) for evaluative schemes and Auer (2002) for a list of references discussing this topic). Inan L1 context, Sorapure, Inglesby, and Yatchsin (1998) underscored the need to augment critical evaluation skills anddevelop new strategies for assessing Web sources. In a survey of 543 American college students, Burton and Chadwick(2000) found that quick access to a source, as well as ease of use were more important to students than whether thesource was conventional or Web-based. In a somewhat different vein, Slaouti's (2002) questionnaire-based studyuncovered an uncertainty among students and faculty about the legitimacy of Web sources as references in academicpapers.

While most of the research on Web use has focused on native-speaker writing, the issue has also begun to be ofconcern in second-language (L2) writing (see Jarvis, 2001). In light of the Web's unfiltered nature, Stapleton (2003)found that although his L2 students had an informed awareness of the variability in quality among both Web andconventional sources, this knowledge was not always utilized in their research practices. Results of a later study(Helms-Park & Stapleton, 2004) revealed that Web sources are being cited at a greater rate by L2 EAP (English forAcademic Purposes) students than their counterparts from just a few years earlier. Helms-Park and Stapleton (2004)found that Web and conventional sources followed a distinct pattern based on typology: Web citations largely camefrom government, news, and interest group sources, while conventional sources were mostly academic books andjournals. Extending the discussion on Web sourcing in academic writing, this paper strives to raise awareness aboutthe increasing number of Web citations used by L2 students, focusing particularly on non-conventional sourceswith ideological and pecuniary agendas (e.g., e-zines, think tanks, interest group Websites, and so on). Its mainpurpose is to typify new Web genres that, in the past, would rarely be cited in academic papers (if at all), to assesstheir quality in a consistent manner, and to discuss the implications of the growing use of such sources for L2writing pedagogy.

Although Web-sourcing presents concerns for academic writing in general, it raises particular difficulties for L2students, often hindering their ability to determine whether a Website contains reliable information or is relativelyobjective in its claims. Since the vast majority of Internet sites are in English, (68.4% compared to second placeJapanese at 6% (Global Reach, 2004)), native speakers of English have had plentiful exposure to a wide range ofWebsites, both in terms of quality and ideological agendas. Furthermore, given the pivotal role that English has had inthe formation of the Web, speakers of English have enjoyed a head start in both viewing and developing Web pages.Some of the specific obstacles L2 students of English may face when assessing Web materials could be theirunfamiliarity with a content area or with culturally determined interpretations of certain texts; their inability to perceivecriticism, bias, and other nuances such as sarcasm and irony in the target language; or students' culture-specificapproach to judging expertise in a subject area.

While the studies mentioned above go some distance towards raising awareness about using objectionable sourcesin L2 writing, an in-depth analysis and evaluation of the Websites referred to in students' academic writing is seldomfound in the L2 literature. Such analysis is important for various reasons. First, these newWeb sources, by now a majorsource of information for student writers, vary widely in quality, from sub-standard to excellent. Furthermore, in spiteof the numerous guidelines and cursory examinations of unsuitable sites that universities post on their Websites, L2students often experience difficulties when it comes to applying such instructions to specific research contexts(Stapleton, 2005a).

In one of the few studies that address this need for Web-source assessment, Helms-Park and Stapleton (in press)recommend the use of a program-specific rating instrument consisting of evaluation criteria based on a three-part

65P. Stapleton et al. / Internet and Higher Education 9 (2006) 63–75

questionnaire put to 31 Humanities faculty members. Four criteria were isolated from a list of nine possibilities as beingparamount in assessing the quality of electronic sources for citation in academic papers: (i) the authority and reputationof the author(s); (ii) the accuracy and objectivity of the contents; (iii) evidence of academic rigor in the text; and (iv)transparency, that is, the currency and maintenance of the site as well as the clarity of bibliographical information. Theactual rating tool that emerged from this study, called WATCH (Website Acceptability Tiered Checklist) (Stapleton &Helms-Park, in press), consists of hierarchically ordered descriptors under each of the four criteria. The instrument wastested via 84 Web-based sources selected by first-year EAP students (N=36) to support their points in a research-basedargumentative essay. Each Web source was scored individually by two trained raters. Importantly, inter-rater reliability(using the Cronbach-Alpha test) was found to be between 0.77 and 0.87 for the individual bands and 0.89 for theoverall scores (Stapleton & Helms-Park, in press).

In light of the uncertainty regarding the acceptability of Web sources for academic citation, this paper has three mainpurposes. First, via a set of case studies, the paper illustrates the types of non-conventional sources students arecurrently citing in their Web-based research. Second, it assesses the quality of non-conventional Websites used by L2students in a first-year EAP context. Finally, the paper re-considers the value of the guidelines given to these studentsfor evaluating sources, and proposes a set of steps that students and faculty can take to bring better overall rigor to Websourcing.

2. Method

2.1. Participants

The participants consisted of 19 students enrolled in an EAP writing course designed as a research-basedcomplement to an introductory first-year academic writing course for non-native speakers of English. These studentscame from a variety of study programs including management, economics, humanities and international studies.Seventeen of the 19 students were native Chinese speakers and two were speakers of Hindi. Twelve of the participantswere female.

2.2. Instrument and procedure

TheWeb sources that constituted the data for this study were collected from the second assignment given in the EAPcourse in which the participants were enrolled. The assignment, an annotated bibliography of 10 print and electronicsources on a research topic chosen from a list of recommended topics (see Appendix A), was designed to assist theparticipants in writing their research papers. Through illustrative examples of annotated bibliographies, the instructordemonstrated in a 90 minute lesson how to compile and annotate selected bibliographical entries and provided basicguidelines for assessing the quality of print and electronic sources (as per Troyka, 2002).

2.3. Classifying the Websites

The references in each student's annotated bibliography were first classified as Web and non-Web references. EachWebsite was then visited by two raters in order to determine its genre and to provide a general description of its content.To determine genre (see Fig. 1) the following categories were used in this study.

1. News sites were divided into (i) sources of news (e.g., daily newspapers or weekly magazines that also appeared inprint, or on-line news outlets with similar coverage of current events); and (ii) opinions and editorials on news-related issues in the same publications as in (i).

2. Academic sites were those written by academics or aspiring academics on topics requiring research and professionalexpertise.

3. Interest groups included sites with identifiable political, religious, ideological, or information-distribution agendas.4. Entertainment sites were those largely consisting of fashion and entertainment online magazines.5. Personal sites were written by individuals without stated credentials or institutional affiliations, and included texts

written in an academic style by these individuals for ostensibly personal reasons.6. Commercial sites were those with primarily pecuniary interests.

Web Genres Visited in This Study

NewsAcademicInterest GroupEntertainmentPersonalCommercialGovernment Search

Fig. 1. Web genres visited in this study.

66 P. Stapleton et al. / Internet and Higher Education 9 (2006) 63–75

7. Government Websites included materials produced by any government or quasi-governmental agency, at any level(international, national, or local).

8. Search sites were small-scale electronic databases leading to miscellaneous articles on issues of current interest.

Since one of the main purposes of this study was to determine what types of Web sources students consideredsuitable for a research-based assignment and to investigate the problems faced by students when evaluating suchsources, only those types which were deemed non-conventional for citation in academic writing were isolated fordetailed analysis. These included materials sponsored by interest and commercial groups, texts in personal Websites,and articles in entertainment magazines. Such materials, termed “gray literature,” are rarely housed in the librarysystem operated by the university in which the participants were enrolled. Interviews with the university librarians inJuly and August 2004 revealed that pamphlets are rarely accepted since materials need to be at least fifty pages longbefore being added to the university collection. Furthermore, gray literature is also screened carefully by those incharge of collections development. In some instances, materials falling outside the criteria “profile” (e.g., those from anNGO or a think tank) are accepted in order to present an alternative opinion. Accordingly, this study sought to examinesources that:

(i) came from sponsors who, prior to the Internet, would not have appeared as references in students' essays;(ii) had agendas extending beyond the bounds of objectivity normally expected of academic references; or(iii) were reprinted in unconventional sites.

These sites were then evaluated by two raters using WATCH (the multi-trait assessment tool discussed earlier) in amanner similar to that laid out in Stapleton and Helms-Park (in press). This instrument was used to determine thedegree to which:

1) the author and the sponsoring site could be considered authoritative or reputable;2) the site's aims may have compromised the accuracy of the information contained within, that is, the site's objectivity;3) the site demonstrated academic rigor;4) the site was current as well as transparent in identifying key features such as authorship, sponsorship, and purpose.

In WATCH, each band is further divided into four levels to provide as fine-tuned an evaluation as possible. Thefirst level indicates that the site is completely acceptable in this band while the last indicates that the site isunacceptable. The second level suggests that the site is partly acceptable, but questionable in some respect(s), andthe third and fourth levels indicate that the site is mostly unacceptable and unacceptable, respectively. Needless tosay, the acceptability scale works best for those sources that a student uses to support his or her own arguments orfactual information. Where the scale is used for some other purpose (a less usual occurrence in our experience), forexample, to expose a weak argument, the rating scale needs to be relaxed accordingly. In cases where an article isreprinted, the new venue is rated as well (see Appendix B for WATCH and Appendix C for descriptions of eachcriterion).

67P. Stapleton et al. / Internet and Higher Education 9 (2006) 63–75

In this study two trained raters conducted the front-line ratings of acceptability; a third rater was used as a tie-breakerwherever there was a discrepancy in scores. The genre classification process was based on several tests. In each case,the student's topic and rationale for including the site were taken into consideration. Where the student's purpose wasnot transparent, the instructor interviewed the student for clarification. The main domain in the URL often providedsome indication of the genre, as did the mission statement. Further delineations were sometimes required, such asdistinguishing between a journal and an “e-zine” (i.e., a magazine that appeared only in electronic form). In this case,genre-specific tests were applied (e.g., searching for an editorial board). As is inevitable in most classifications, therewas overlap between categories in some cases. However, when there was a difference of opinion about theclassification of a site, a third rater served as a tie-breaker. From this subset of unconventional sites, a final group ofseven was chosen as illustrative cases, reflecting a cross-section of various types of new sources of Web-basedinformation.

3. Results

Of the 190 entries cited by the participants, 104 were conventional print sources, and 78 were Websites. Of the latter,eight sites were deemed to be “dead” on the grounds that they were inaccessible on three separate occasions, and twowere specifically for small-scale searches. Searches using Internet archive engines, such as waybackmachine.org, wereconsidered unconventional and therefore were not used in this study. Approximately two-thirds of the remaining 68sites fell into the “news,” “academic” and “interest group” categories (see Table 1). Most of the academic sites consistedof relatively informal sources of information, such as professors' course outlines and articles in non-refereed journals.The remaining one-third of sites were from fashion and entertainment magazines, statistical information issued by thegovernment, research sites for locatingWeb-based publications, and information sites sponsored by interest groups (seeTable 1).

Of the 68 sites, 39 sites were considered unconventional by traditional print standards; these sources would probablynot be found within the hard-copy holdings of a university library, or, if they were present there (e.g., advice forstudents put out by the university), they would not be catalogued as reference materials. Four of these sites were nolonger accessible, that is, they were active when the sites were classified, but had disappeared by the time they wererated; In addition, another two sites were eliminated because they were used intentionally as exemplars of viewpointsthat the students were questioning rather than for bolstering their own stance. Of the remaining 33 sites, only one wasfound to be completely acceptable by the raters, that is, it was top-ranked in all four categories of WATCH; furtherdetails about this site appear in case 1 below. Three sites were mostly acceptable (i.e., they had an equal number ofacceptable and partly acceptable ratings). Sixteen sites were unacceptable or mostly unacceptable in various bands,

Table 1Web genres

General type Sub-type “Conventional” “Unconventional”

Academic (N=14) Books 2Refereed journals 2Unrefereed journals 3Other (professors' course outlines and lecturenotes, and university information sites)

7

Personal (N=7) “Academic” and advice 4Search (N=2) 2Government (N=4) Statistics 4News (N=21) News reports: electronic sources with print

equivalents15

News reports: electronic sources without printcounterparts

1

Opinion (editorials and special columns) 6Interest group (N=11) 11Commercial (N=4) 4Fashion/entertainment magazine

articles (advice and information) (N=7)7

68 P. Stapleton et al. / Internet and Higher Education 9 (2006) 63–75

typified by cases 4 through 7. The remaining 13 were in between the two extremes; they had an assortment of ratingsbetween “acceptable” and “mostly unacceptable”; examples of these appear as cases 2 and 3. A full breakdown of theratings of the 33 Websites appears in Table 2.

3.1. Description and rating of sites

The following section presents seven case studies of sources that reflect a small sample of sites that could be termednew and unconventional sources of information. These seven were chosen because they proportionally represent across-section of the non-conventional Web-genres received for the assignment in question. These include twocommercial sites, two interest groups, two personal sites, as well as “unpublished” academic materials that are notcreated for the purposes of being cited (e.g., a professor's lecture notes posted on the Web). In each case, the students'rationale for selecting a site was determined through their own annotation of the entry. The assessment of the sevenWeb sources aims to shed light on: (i) the great variation in the nature of materials available on the Web; (ii) students'frequent inability to assess Websites in terms of academic rigor, objectivity, authority, and transparency, that is, thecriteria in WATCH; and (iii) some of the perplexing problems faced by EAP instructors when evaluating their students'source materials. The seven sites below start with the best-rated site and end with the one with the lowest rating, the firstone being an example of an acceptable e-source.

3.1.1. Site 1 (Student 5): scholarly article at a family foundation siteType: interest groupStudent's topic: the impact of violence in media on children and adolescentsDescription: This is a properly referenced academic paper in which the author reviews recent research studies on the

effects of violence in the media.Authors and/or sponsors: The sponsor is a non-profit family foundation whose focus is on health issues.

Foundations are generally established by the very rich as an alternative to paying tax. Their purpose is typicallyphilanthropic, but is subject to certain laws, e.g., a minimum of five percent of the total foundation assets must be spentannually. This site's activities appear to be primarily for the purposes of gathering and analyzing sociological data in aneffort to improve health care, particularly for the disadvantaged.

Rating: This site satisfies all four criteria (authority, objectivity, rigor, and transparency) for inclusion in anacademic paper. Although without an author's name, the authority of the site, as established by other indicators issufficient.

3.1.2. Site 2 (Student 20): PowerPoint lecture slides from a university professor's siteType: unpublished academicStudent's topic: Web advertising and the question of profitDescription: This Website consists of 22 PowerPoint slides that outline a brief history of consumerism for an

undergraduate course. The opening slide has only a title without any mention of the author's name or institutionalaffiliation.

Authors and/or sponsors: A further search within the professor's site, running on the server of a U.S. state university,reveals that he has teaching experience and publications that demonstrate his expertise in the area of consumerism.

Rating: While the site seems to have objectivity and authority, it has little academic rigor and transparency since itwas obviously created as a framework for a lecture and not for formal citation. A similar site in the database containinga professor's lecture notes on gender roles (Student 17) had parallel weaknesses.

Table 2Acceptability ratings of the unconventional sites (N=33)

Authority of the writer(s) of the text Objectivity Academic rigor Transparency

Acceptable 3 6 2 6Partly acceptable 20 18 8 22Mostly unacceptable 12 5 19 5Unacceptable 0 4 14 0

69P. Stapleton et al. / Internet and Higher Education 9 (2006) 63–75

3.1.3. Site 3 (Student 11): editorial piece in a Vanity on-line magazineType: commercialStudent's topic: the effect of media on women's body imageDescription: This is an on-line magazine “editorial” lamenting women's obsession with thinness. The article

liberally quotes several female celebrities as well as a celebrity trainer. It argues that modern young women are beingpresented with unrealistic role-models with respect to physical appearance. Numerical data regarding women's self-image are presented from other media such as magazines and television. However, as is to be expected in a fashionmagazine, no references are provided either within the article or at the end.

Authors and/or sponsors: The article is written by the editor-in-chief and founder of this on-line magazine, self-described as “North America's oldest and most popular on-line women's magazine.”

Rating: This site was found to be partially acceptable in terms of objectivity and transparency, but mostlyunacceptable in the areas of authority and rigor because the site's agenda, the authority of the editor, as well as the citedsources were not credible. Six other sites in the database were similar in quality and kind as this case. Three of these(Students 8, 11, and 12) very closely mirrored this e-zine.

3.1.4. Site 4 (Student 18): encyclopedia piece reprinted in a popularized on-line information sourceStudent's topic: gender roles across culturesType: commercialDescription: This is a one-page encyclopedia-like article on the role of women in Afghanistan, ostensibly reprinted

in abridged form from a Library of Congress source.Authors and/or sponsors: The provider of this site disperses information offering expertise in hundreds of areas to a

self-proclaimed number of “over 20 million” visitors a month. Revenue is generated by topic-related advertisementsthat appear on the information sites.

Rating: As an abridged reprint, the article seems equivalent to a conventional encyclopedia source andrepresents a new genre of sourcing. However, there are serious concerns about the authority of the informationprovider, that is, the so-called “Guides” upon whose contributions the site depends. A closer examination of one ofthe “Guides” (at a higher level sub-directory of the same site) revealed that the “Guide” was an active advocate ofatheism and humanism on the Internet (i.e., the main page did not make such matters transparent). Such aninformation source with a religiously charged agenda (including one that promotes atheism) cannot be consideredobjective or appropriate for citation in an academic paper unless, of course, it is being deliberately used as anexemplar of a viewpoint that clearly diverges from the writer's. While there was only one other encyclopedia site inthe database, there were various other sites promoting religious beliefs, two of which are described in cases 6 and 7below.

3.1.5. Site 5 (Student 18): academic-like paper on a personal siteType: personalStudent's topic: gender roles across culturesDescription: Through argumentation, this article maintains that gender is a cultural construct and not a genetic one.

It includes a bibliography, termed a “Reference List,” at the end; however, no citations appear in the text.Authors and/or sponsors: The main domain of this site is “geocities,” a provider of free server space for individuals,

identifying the site as “personal.” A search of the site reveals that the author has written several opinion pieces, allessentially advocating empowerment of the disadvantaged.

Rating: The site does not have the rigor or the objectivity that would make it appropriate for citation in an academicpaper because it advances views that have received no peer-review and draws on gender theories without including anysources. As many as half a dozen other sites in the database fit the description of a “blog,” a personal opinion pieceposted on a server, presumably without either authority or screening.

3.1.6. Site 6 (Student 10): personal essay at a religious siteStudent's topic: culture shock and its impact on the acculturation processDescription: This site leads to a PDF document of 12 chapters (186 pages) that has every appearance of being a self-

published academic paper with 21 references, mostly to Christian organizations, in a randomly ordered annotatedbibliography. Four of the references are to the author's own works, including one published by a vanity press.

70 P. Stapleton et al. / Internet and Higher Education 9 (2006) 63–75

Authors and/or sponsors: A search of the “Who are we?” statement at the main domain and various other internallinks revealed that the home site at which the PDF document appeared was a fundamentalist Christian organizationbased in Singapore.

Rating: This site is not suitable for citation in an academic paper for several reasons. First, it is motivated by ahighly-charged evangelical agenda and therefore lacks objectivity and the authority expected for publications onculture shock. Although using this site to point out that religion can transcend culture and act as a cohesive force (whichin itself is an innocuous claim), the student fails to acknowledge that such a supporting source does not meet thestandard of rigor expected in an academic paper.

3.1.7. Site 7 (Student 12): marriage advice in an on-line women's Christian magazineType: interest groupStudent's topic: the future of marriageDescription: This 1700-word narrative begins with a childhood story serving as a metaphorical lead into advice for

preserving one's marriage by invoking eight principles, the last two heavily imbued with Christian values. There are noreferences to works other than the Bible.

Author and/or sponsors: The passage is included in a self-described Christian women's on-line magazine,which has the appearance of a typical, mass-market women's magazine with links to other sections in themagazine, such as articles on parenting, fashion, and beauty, as well as prayer and spiritual teachings. Many ofthe external links lead to Christian sites. The “Statement of Faith” makes it clear that this site is managed by afundamentalist Christian organization with a list of 17 doctrines of belief based on a literal interpretation of theBible.

Rating: This source is clearly unacceptable for academic citation. The authors, named “consultants in leadershipeducation,” lack professional credentials that would make them authorities on marriage and divorce. Moreover, thesponsoring site, a fundamentalist, proselytizing Christian organization, does not even faintly achieve the objectivity oracademic rigor required for academic citation.

4. Discussion

The rating of the 33 Websites identified as “unconventional” sources reveals that L2 learners frequently experiencedifficulties discerning agenda-driven sources of information, often because they are misled by simple keywordsearches, or are swayed towards more polarized positions through such research materials. Interest group sites are amajor concern, particularly those provided by so-called non-profit social organizations that promote information underthe guise of the “public good.”

While libraries do carry some publications issued by various institutes and foundations, collections experts inlibraries are bound by academic standards when assessing a source's authority, objectivity, rigor, and transparency.Since the Web is mostly free of such restrictions, the above-mentioned sites exemplify new, yet potentiallycontentious, sources of information. Unlike conventional academic papers, on-line materials are mostly not onlynon-refereed, but also frequently commissioned from those sympathetic to the ideological agendas of theirbenefactors. While the academic rigor of some of the on-line sources, as well as their authority and reputation mayreach an acceptable level for citation in an academic paper, their objectivity is often being compromised by theopaque agendas of their sponsors.

The above choice of sites, indicative of many other sites analyzed by raters, reveals that detecting such hidden biasescan be particularly challenging for many L2 writers whose cultural assumptions about ideological agendas may vary orwho might not fully understand the connotations of certain key words and phrases (e.g., crusade) or words whoseconnotations vary according to regional, religious, or political persuasion (e.g., liberal). In fact, the raters themselveshad to delve fairly deep into many sites to discover their weaknesses.

Such difficulties are reflected in some students' choice of religiously inflected sources, as in Sites 6 and 7. These twosources, sponsored by fundamentalist Christian organizations, were cited by the students as legitimate researchmaterials and not as exemplars of viewpoints that opposed their own stance. Although these two sites have some of theaccoutrements of legitimate academic sources, their authority and objectivity are highly questionable. Site 6 discussesthe complex process of acculturation through a strictly religious lens. In a similar manner, Site 7 advocates a Christianway of life as a solution to marital discord and divorce.

71P. Stapleton et al. / Internet and Higher Education 9 (2006) 63–75

It is also worth noting that two other sites among the 39 unconventional sources had equally religious overtones.Amounting to roughly 10% of the total Web references, this high number of religion-inflected citations raises concernand may indeed signify a lack of awareness among certain L2 learners regarding the questionable status of sourcematerials emanating from religious or ultra-religious groups and proponents of public morality.

It is possible that the perceptions of some L2 learners who have been socialized in cultures and academicenvironments where religion is either accorded greater reverence or, conversely, has little influence will differ from thestudents who are schooled to believe that personal religious beliefs generally fall outside the realm of academia.Specifically, learners new to a North American university may also not fully recognize the evangelical nature of somewell-funded religious groups (with a highly committed membership) whose proselytizing now pervades the Web.

Unlike traditional print media, whose need to justify bottom lines means that extreme, or poorly supported,viewpoints generally never make it to press, on-line magazines raise further concerns since they have few fiscalrestrictions to act as filtering agents. On-line magazines can be launched with few production or distribution costs and,as Site 3 reveals, perhaps without any sales staff. The minimal start-up costs and effort within new electronic mediahave led to a myriad of outlets of information with a legitimate appearance. As the considerable number of e-zinecitations uncovered by this study reveals, students may be overestimating the authority of these sources.

Furthermore, Sites 2 4 and 5 demonstrate that even academic and educational sites can also constitute a problematicarea of L2 Web-sourcing. For example, while the article in Site 4, an abridged reprint from an encyclopedia, is itselfacceptable, the sponsoring site is not. Although providing information under the guise of “public good,” this site isdriven by a hidden commercial agenda. Such contradictions are not readily apparent upon a cursory glance at the URL.Similarly, Site 5 gives the illusion of being a rigorous academic article since it draws on socio-constructivist theories ofgender that are compatible with academic writing. However, because it fails to provide proper references to scholarship,it cannot be considered suitable for academic citation. Another type of problematic academic source is Site 2, aPowerPoint presentation posted by a professor specializing in consumerism. Written in point form, the purpose of thissite is to serve strictly as a pedagogical or mnemonic tool, not as a source of information or academic citation. Theselection of such sites indicates that some L2 learners seem not to be familiar with North American expectations ofacademic rigor.

While much of this discussion has focused onWeb sources of dubious merit, there is the obvious positive dimensionto the new information era. As the case study of Site 1 indicates, some of the unconventional sources examined in thisstudy do indeed have merit. Site 1 provides cutting-edge information that is largely free of ideological agendas. It offersa rigorous review of studies on violence in the media and its effect on children and adolescents, providing pertinentreferences to scholarly works. Before the existence of the Web, such a rich source would most likely have beenunavailable to students in a general library search.

5. Pedagogical implications

The ambiguities of e-sourcing, highlighted by this study, point to a pressing need to bring critical Internetliteracy strategies into the classroom. While access to on-line materials has undeniably broadened the scope ofacademic research, it has also presented university students with a variety of challenges. Before the emergenceof the Internet, students would have had difficulty finding print-based information sources such as thosereviewed above, and this appears especially true for materials with proselytizing, commercial, and bias-inflectedagendas.

As discussed above, L2 learners may not only be unfamiliar with some of the ideologically charged agendas ofWebsites, but may also have their own culturally prescribed ideas as to what constitutes evidence (Kachru, 1999).This was reflected, for example, in some of the participants' undiscriminating use of religious sites, includingfundamentalist ones. In addition, language-related barriers may prevent them from recognizing the subtle nuancesof various texts (Stapleton, 2005b), often expressed through the connotations of lexical items and culturalallusions (e.g., “bleeding-heart liberals,” “vigilante justice,” or “axis of evil”) or through an awareness of theculture-specific history of an issue (e.g., gun-control issues in the United States). For these reasons, L2 writersmay need supplementary help in developing a critical awareness of how Web-sources can vary greatly in terms ofquality and agendas.

The borderline acceptability of some interest groups and the ambiguous status of articles (or decontextualizedexcerpts of these) reprinted in dubious venues uncovered by this study strongly calls for encouraging critical Web-

72 P. Stapleton et al. / Internet and Higher Education 9 (2006) 63–75

source evaluation in academic contexts, not only through a series of lecture-format guidelines, but also through student-centered workshops, and, where necessary, one-on-one conferencing. As indicated by the rating process, assessing thesuitability of electronic materials as academic sources in a non-interactive fashion can be labor-intensive for instructors;furthermore, such a process amounts to testing rather than teaching.

Since many students actively participate in the consumption of electronic information (Burton & Chadwick, 2000),it is not surprising that the instructor's initial notes of caution about using the Internet for academic purposes are oftenmet with students' general resistance. Therefore, lecture-format instruction is not always successful in debunking thestudents' assumptions about the “unlimited possibilities” of the Web. For instructors, this has specific pedagogicalimplications that include:

1. Instruction in familiarizing students with Web-genres.2. Identification of non-conventional Web-genres for the purposes of academic citation.3. Demonstration of actual examples of the features of these Web sources which both conform and do not conform to

the conventions normally associated with academic citations.

Accordingly, a strongly interventionist approach to guiding EAP students' Internet-based research may be advisablewhile allotting a sufficient amount of time for this purpose in any introductory EAP course. This component (organizedby the instructor, a librarian, or a teaching centre) needs to provide practice in: (i) using appropriate search engines; (ii)scrutinizing URLs, mission statements, and authors' biographies; (iii) examining whether the article is reprinted orabridged; (iv) checking out the various links on the page to determine what the site's real agenda might be; (v)ascertaining whether a keyword search leads to relevant materials (e.g., Dolly the cloned sheep rather than the countrysinger); and (vi) assessing whether the materials are suitable for academic purposes.

The use of WATCH, then, or a similar course-specific instrument, is recommended as a screening tool thatencourages students to apply more rigor to their Web research practices. Any whole-class instruction needs to befollowed up with small-group or individual assignments in which, for example, ten pre-selected Websites on a specifictopic are given to the students, which they rate using WATCH or any counterpart. Furthermore, students need toidentify who they think the sponsors, writers, and targeted audience of these sites are.

Experience with the EAP participants in this study reveals, one useful practice is to get students to submit theirreferences for a research paper, or better yet, an annotated bibliography early in the writing process, with aWATCH-typechecklist completed for each Web-based source. These need to be examined by the instructor (or, where available, ateaching assistant), who also conferences with those students who included questionable sources, preferably with anInternet-linked computer on site. The final submission would be checked to see whether the unacceptable references hadbeen dealt with. In fact, in the current study, it was found that, as a result of written feedback and special interviewsconducted by the instructor, participants eliminated many of these dubious sources, and also produced better researchpapers.

Naturally, the present study, conducted in a restricted academic context, needs more rigorous validation before it canbe fully generalized to a larger population. In addition to verifying the validity of WATCH, future studies may wish toinvestigate how unconventional Web sources are influencing the opinion-forming process of higher education students.Another area of possible research is the effect that the latest specialized search engines such as Google Scholar andScirus may be having on the research process.

A further limitation of the present study concerns the considerable amount of time and effort required toeffectively use the evaluation scale, WATCH, or other versions of this scale. In an era in which instructors areresponsible for large numbers of students with limited resources, a thorough checking of all a student's references isnot always realistic. Still, there are arguably few issues in the academic world more important than the integrity ofone's source of research information. Since the future of academic writing may very well be inextricably linkedwith the unlimited, and at times questionable, riches of cyberspace, a combination of Web-specific instruction andpractice may provide the necessary and much-needed signposts for students eager to explore this vast, but mostlyuncensored universe of information.

Acknowledgements

We would like to thank Laura Anne Gibson and Ashley Sankowski for their important contribution to this project.

73P. Stapleton et al. / Internet and Higher Education 9 (2006) 63–75

Appendix A. Research paper bibliography assignment (value: 15%)

Write an annotated bibliography of 10 entries that are relevant to the research topic you have selected. Yourentries should include a wide range of sources (i.e., newspaper articles, journal articles, books, electronic sources,etc.). The individual summaries should be no longer than 150 words. You have four weeks to finish thisassignment.

Research paper topics

1. Multiculturalism2. Women in business3. The future of marriage4. Culture shock and its effects5. Media and their impact on how we view the world6. Consumerism7. Gender differences across cultures8. Education across cultures9. Cultural tourism

10. Advertising.

Appendix B. Web site acceptability tiered checklist (WATCH)

Criteria

Authority and reputation As determined through the reputation of the publishing body and/or stated, linked or implied biographical information,

the author:□ can definitely be considered authoritative and reliable.□ appears to be authoritative and reliable in some respects but not others.□ appears to be only marginally authoritative and reliable.□ is clearly not authoritative and reliable.

Objectivity

As determined by the URL, “Mission” statement, “About Us” statement, visual features, or other textual means (e.g.,lexical choice), the contents of the Web source can be considered:□ to be unmotivated by ideological agendas, objective, and fair.□ to be somewhat objective and fair, but ideological agendas may exist.□ likely to be influenced by ideological agendas to the point of being unfair.□ to be clearly biased and unfair.

Academic rigor

As determined by factors such as research methodology, familiarity with current trends and leaders in the field, links tocredible sources, sufficiency of the text, or, in the case of an established expert, how well supported the ideas andarguments are:□ there is clear evidence of academic rigor throughout the source.□ there is some evidence of academic rigor, but only in some aspects of the source.□ most aspects of the site can be generally considered to be lacking in academic rigor.□ there is no attempt at achieving academic rigor.

Transparency

□ Site has been recently created/updated with authors, dates, and purpose clearly and systematically identified.□ Some aspects of who, when, and why the site was created are evident, but not completely and systematically.□ Site's date, purpose, and creators are not clearly evident.□ Site is without any particulars about the date, author, and purpose.

Appendix C. The four bands of WATCH

Authority and reputationSome key indicators:

• whether or not the Web source is an organization, as ascertained through distinctive seals or logos that confirmmembership in the organization;

74 P. Stapleton et al. / Internet and Higher Education 9 (2006) 63–75

• whether or not the Web source can be identified as a producer of primary-level knowledge or at least a source ofreliable information;

• whether or not the authors' biographical information identifies them as authorities in the field.

ObjectivityNumerous methods exist to detect bias in Web sources, most of which tend to center on who or what publisher

supplies the information being dispersed (Kirk, 2002). Although neutral in theory, on-line newspapers (indeed, likemost conventional newspapers), government Web sources, and academic Web sources encompassing sites rangingfrom assignments written by students to peer-reviewed scholarly papers often vary in objectivity. Among the mostproblematic, however, are sites with revenue-generating or ideologically inflected agendas run by enterprises, politicalparties, and interest groups that tend to promote their own products and ideas by depicting the competition in a negativelight. Personal Web pages, too, have a high likelihood of being biased.

Academic rigorAcademic rigor is associated with the research methods used to gather information along with the display of

knowledge and familiarity with leaders in the field through references or a bibliography (Barker, 2004), inter-textualreferences (de Beaugrande, 1980), and verifiability. In the case of experts in their field or institutions, references toother people's works often become less of an issue because the experts' reputation sufficiently infuses the text withcredibility.

TransparencyKey indicators:

• clearly identified authors;• specifics about sponsoring agencies or institutions, including mission statements;• date of original publication and/or time of updating.

References

Auer, N. J. (2002). Bibliography on evaluating Web information. Retrieved Sept. 22, 2003, from. http://Web.lib.vt.edu/research/evaluate/evalbiblio.html.

Barker, J. (2004). Evaluating Web page: Techniques to apply and questions to ask. Retrieved Jan. 25, 2004 from. http://Web.lib.berkeley.edu/TeachingLib/Guides/Internet/Evaluate.html.

Burton, V. T., & Chadwick, S. A. (2000). Investigating the practices of student researchers: Patterns of use and criteria for use of Internet and librarysources. Computers and Composition, 17, 309−328.

de Beaugrande, R. (1980). Text, discourse, and process: Toward a multidisciplinary science of texts. Norwood, NJ: Ablex.Global Reach (2004). Global internet statistics (by language). Retrieved Oct. 21. 2005 from. http://global-reach.biz/globstats/index.php3.Helms-Park, R., and Stapleton, P. (in press). How the views of faculty can inform undergraduate Web-based research: Implications for academic

writing. Computers and Composition.Helms-Park, R. and Stapleton, P. (2004). Rating first-year EAP students’ Web-based research materials. Paper presented at the meeting of the

Conference of the Canadian Society for the Study of Rhetoric, May 31, 2004, Winnipeg, Canada.Jarvis, H. (2001). Internet usage of English for academic purposes course. ReCall, 2, 206−212.Kachru, Y. (1999). Culture, context, and writing. In Eli Hinkel (Ed.), Culture in second language teaching and learning (pp. 75−89). Cambridge:

Cambridge University Press.Kirk, E. E. (2002). Evaluating information found on the Internet. Retrieved Sept. 22, 2003 from. http://Web.library.jhu.edu/elp/useit/

evaluate.Shetzer, H., & Warschauer, M. (2000). An electronic literacy approach to network-based language learning. In M. Warschauer, & R. Kern (Eds.),

Network-based language teaching: Concepts and practice (pp. 171−185). Cambridge: Cambridge University Press.Slaouti, D. (2002). The World Wide Web for academic purposes: Old study skills for new? English for Specific Purposes, 21, 105−124.Sorapure, M., Inglesby, P., & Yatchsin, G. (1998). Web literacy: Challenges and opportunities for research in a new medium. Computers and

Composition, 15, 409−424.Stapleton, P. (2003). Assessing the quality and bias of Web-based sources: Implications for academic writing. Journal of English for Academic

Purposes, 2, 229−245.Stapleton, P. (2005a). Using the Web as a research source: Implications for L2 academic writing. Modern Language Journal, 89, 177−189.

75P. Stapleton et al. / Internet and Higher Education 9 (2006) 63–75

Stapleton, P. (2005b). Evaluating Web-sources: Internet literacy and L2 academic writing. ELT Journal, 59, 135−143.Stapleton, P., and Helms-Park, R. (in press). Evaluating Web sources in an EAP course: Introducing a multi-trait instrument for feedback and

assessment. English for Specific Purposes.Sullivan, D. (2003). How search engines rank Web pages. Retrieved Aug. 22, 2004, from. http://searchenginewatch.com/webmasters/article.php/

2167961.Troyka, L. (2002). Simon and Schuster handbook for writers. 3rd Canadian edition. Scarborough, Ontario: Prentice-Hall Canada.