Tracking the Progress ofE-Mail Vs. Snail-Mail

Gap narrows on response rates, butapplications still limited.

By Duane Bachmann, John Elfrink, and Gary Vazzana

Advances in computer technology and the increased popularity of electronic-mail applications have enhanced the potential for conducting survey researchthrough e-mail. In an experiment comparing mail and e-mail data collection, e-mail fared well with respect to response rates, item omission, response time,and data quality. The authors make a case for using e-mail to conduct research,but only under specific circumstances.

The widespread use of personal comput-ers in tbe workplace and at home, theopening of the Internet to tbe generalpublic, and tbe dispersion of softwareand hardware that allows linking of

computers all are contributing to the feasibility ofelectronic mail as a survey mode,

A 1995 Gallup poll found that uine out of 10Eoriune 1,000 companies and balf of smaller busi-nesses now run e-mail applications on networks.The study also projects the number of e-mail usersto reaeh 38.9 million by 1998.

Although increased use of e-mail in tbe work-place seems apparent, tbe New York Times reportstbat home access and usage is growing even morerapidly. A 1995 survey of Internet users found thatmost people are tapping into tbe World Wide Weband commercial on-line services from bome ratbertban from tbe office or school. And, according totbe "Information and Interactive Services Report,"

approximately 14,000 new consumers are linkingup to the Internet daily.

E-MAIL RESEARCH

E-mail questionnaires appeared almost simulta-neously witb tbe introduction of e-mail and tbeopening of tbe Internet. Intra-company networkingallows managers to gather data from fellowemployees for in-house purposes through e-mail.And participants of on-line discussion groups reg-ularly receive requests for survey responses on top-ics of interest to the groups. However, despite tbeincreasing use of e-mail surveys, very fewresearchers bave investigated tbis new form of datacollection.

Tbe earliest studies of e-mail surveys wererestricted to populations sampled from within asingle company or university. For example, AT&Tconducted an e-mail survey of its international

MARKETING RESEARCH: b, Vol, 8 No. 2 3 1

employees in 1990 and obtained a 60% responserate, and Lotus Development Corp, achieved a56% response rate to a 1994 employee survey.

In 1986, Sara Kiesler and Lee Sproull surveyedstudents and employees in a university setting whoworked in a computer-intensive environment.Tbese researcbers obtained a 67% response rate andfound that respondents were more likely to give lesssocially desirable responses tban were thoseresponding on paper, Tbe e-mail subjects also leftfewer questions blank.

In 1992, Barbara Scbuldt and Jeff Totten surveyeda national sample of marketing and MIS facultymembers, comparing mail and e-mail metbods ofdelivery. The mail response rate was 56.5% while thee-mail rate was 19.3%. Tbe e-mail respondents wereprimarily MIS professors wbo bad a greater interest inand familiarity witb the technology involved.

The amount and quality of e-mail survey research,especially outside of a corporate or university setting,is very limited. Trying to generalize from studies thatuse sucb restricted sample frames would be unwise.

BREAKING NEW GROUND

We wanted to expand the investigation of e-mailas a mode of gathering data for research to includeuot only response rates, but also specific itemresponse, response time, cost, item omission, andtendency to respond to open-ended questions. Tbeoverall goal of the study we recently conducted wasto determine if e-mail survey techniques can beused in place of traditional mail metbods.

Due to newness of tbis area of study, rather tbantest specific bypotbeses, we attempted to answer aseries of researcb questions:

• Will e-mail surveys yield results similar tomailed surveys?

• Do distinct advantages and disadvantages existfor tbe e-mail metbod relative tbe mail method?

• Do different attitudes exist between e-mail andmail questionnaire respondents and uonre-spondents concerning surveys? Is there a dif-ference in tbeir likelihood of responding?

Our analysis provided evidence that tbe twomodes are comparable in tbe data gathered andresponse rates obtained. Tbe e-mail mode, bowev-er, bas some distinct advantages and some materi-al disadvantages. Respondents gave comparableresponses regardless of tbe mode of delivery.Althougb tbe mail survey response rate was superi-or to the e-mail group, botb rates compared favor-ably to bistorical results from mail surveys.

Altbough results do not support tbe widespread

use of e-mail to conduct surveys, its use is appro-priate for certain specialized populations.

Research MethodologyWe developed a survey requesting information

from business scbool deans and division cbairper-sons on tbe use of total quality management(TQM)-type processes in higher education. We alsoasked tbe subjects to judge tbe usefulness of TQMprocesses in the university and business environ-ment and tbe extent of its usage in tbe future.

The topic is an important and timely issue ineducation, business, and industry, so we anticipat-ed a response rate tbat would provide a good test oftbe effectiveness of eacb of tbe two differentresearch methodologies.

We used a combination of several samplingframes—including tbe 1994-95 Membership Directoryfor the American A.ssembly of Collegiate Schools ofBminess (AACSB), the 1995 Hasselback AccountingFaculty Directory, and tbe ]994 Hasselback Directoryof Management Faculty-—to obtain a representativesample of business leaders in academic institutions ofhigher education throughout tbe United States.

All 224 individuals with e-mail addresses pub-lished in the sample sources were sent question-naires via the Internet. From tbe remaining deausand divisions cbairs, we randomly selected another224 to receive the mail survey.

The survey was two typewritten pages in lengthand dealt with various TQM issues. The questionswere primarily category scaled and related to the pos-sible use of 28 different tools, concepts, or applica-tions. One demographic question and tbree semanticdifferential questions relating to either tbe value orexpected use of TQM were included in the survey.Those in the mail sample were also asked if they hadan e-mail address.

A pretest demonstrated tbat respondents shouldbave few problems understanding tbe questions.However, it became very clear tbat many did notunderstand bow to return tbe survey by e-mail, sothe instructions were carefully reworked.

An important objective of tbis study was to deter-mine why tbose who were sent tbe TQM survey eitherdid or did not return the questionnaire. Approximatelyfive weeks after sending the initial survey, tboseresponding to eitber tbe mail or tbe e-mail surveyreceived a mail survey designed to measure tbe Ukeli-bood of tbe respondent returning: (1) a mail survey onsometbing in whicb tbey were interested, (2) a mailsurvey on sometbing in which tbey were not interest-ed, (3) an e-mail survey on sometbing in wbich tbeywere interested, or (4) an e-mail survey on somethingin which they were not interested.

Tbis follow-up survey also asked respondents toreport the number of mail and e-mail sitrveys tbey badreceived over tbe past year and wbat percentage tbey

3 2 SUMMER 19%, Vol. 8 No, 2 MARKETING RESEARCH:

bad returned. Additional issues explored were tbe indi-vidual's level of knowledge about and use of e-tnail.

Individuals wbo did not respond to eitber themail or e-mail TQM sur\'ey received a follow-upmail questionnaire similar to tbe one sent to respon-dents. However, tbey also were asked to evaluate tbeextent to wbicb seven different factors contributed totbeir decision not to return tbe TQM survey.

ResultsResearch question 1: To determine if an e-mail

survey could be used as a surrogate for traditionalmail surveys, we tested for significant differencesin the responses of tbe two groups, T-tests were runon three Likert scaled questions to determine if adifference in the means existed, and no differenceswere found at tbe ,05 level.

Cbi-square tests were run on 28 yes-or-no-typequestions to test for differences in cell assignments.One question sbowed a significant differencebetween tbe mail and e-mail respondents at tbe ,05level. Given tbe number of items (28) included intbe analysis, a significant difference in one ques-tion can be expected due to chance.

Tests of differences were also run for accredita-tion status, number of faculty, and size of tbe uni-versities for tbe two groups (see Exbibit 1).Althougb the two groups of respondents appear tobave come from institutions and business areas ofsimilar size, the ratio of AACSB-accreditedschools was higher among the e-mail group(64.6%) than among the mail group (48.3%). Tbisdifference can largely be explained by tbe differ-ence in original samples; 50% of the deans select-ed for tbe regular mail survey and 63.4% of thoseselected for e-mail were from accredited schools.

Survey response rates are an itnportant factor injudging the quality of a questionnaire-based researcbproject. A strong response rate lessens a researcber'sconcern about possible nonresponse bias,

Altbougb tbe mail response rate (65.6%) in ourstudy was higher tban tbe e-mail rate (52.5%), botbmetbods of delivery resulted in very respectableresponse rates for survey researcb. The reportedresponse rates bave been adjusted for non-deliverableaddresses. Of tbe original 244 surveys sent, one of tbetnail surveys was returned because of a bad address,but 41 of tbe e-mail sur\eys were undeliverable.

Also, considering tbe results of tbe follow-upsurvey, a furtber upward adjustment in tbe e-mailresponse rate migbt be warranted. Wben nonre-spondents were asked wby tbey did not return oursurvey. 13 nonresponding e-mail subjects said theydid not remember receiving tbe survey. Althougbthe technology indicated to us that tbose e-mailquestionnaires bad been delivered, apparently tbeywent astray. Assuming tbat tbe 13 subjects wouldbave returned the questionnaire (they did return the

follow-up) would bike tbe e-mail response rate to59.6%, mucb closer to tbe mail response rate.

Tbe above analysis strongly supports our con-tention tbat, within certain populations, e-mail surveyscan be substituted for mail questionnaires. Witb onlyone question of the 31 nondemographic items from tbesurvey having significantly different responsesbetween the two groups, we conclude tbat virtually nodifference exists in the data gathered by the two metb-ods. Tbe demograpbic data concerning tbe size of tbeinstitution and number of faculty also indicate tbatrespondents fi'om the two groups were similar.

Research question 2: Exbibit 1 also contains sur-vey items tbat indicate advantages of tbe e-mailmode of data collection. Costs, willingness torespond to open-ended questions, and response timeswere all superior in tbe e-tiiail group. The respon-dents' tendency to leave an item blank or comment toquestions not requesting a comment were about tbesame wben comparing tbe two groups.

In assessing the cost of obtaining a response, weomitted the labor and computer costs. Labor costswere difficult to define and track accurately. Manyindividuals were involved in developing and pretest-ing tbe questionnaires, assembling tbe samples,stuffing tbe envelopes, locating e-mail addresses,etc. Many of these tasks overlapped tbe two groups,causing cost allocations concerns. The cost ofaccessing the Internet is a capital expenditure thatwas not passed on to us. Tbe only incremental costsclearly assignable to a specific sample group weretbe mailing and stationery costs for tbe mail group.

One open-ended question requesting additional

Exhibit 1

Mean results of selected survey items

E-mail Mail

EnrollmentNumber of facultyAACSB accreditedResponse rates"Cost/respondentResponded to open-

ended question

Days to respondAverage items omittedCommented to multiple-

choice questions

Non-deliverable addressesHave e-mail address

11,64155.664.6%52.5%

021.9%

4,68 days3.1%

22.9%

19.1%100%

10,63151.648,3%*65,6%'$1,53

4,8%-

11,18 days.7%

25.2%

,4%'85,6%

signiticant at the .05 level' excludes labor costs and capital expenditures for internet access

MARKETING RESEARCH: SUMMER 1996, Vol. 8 No, 2 3 3

Exhibit 2

E-mail vs. mail cumulative percent return

% Returned70

E-mail Mail

comments was included in the questionnaire. Tbee-mail respondents were much more willing to passon additional comments tban were mail respon-dents (21.9% vs. 4.8%). Tbe ability to type a quickresponse migbt be an advantage over a mail surveyrequiring a band-written response.

Exhibits 1 and 2 clearly indicate tbe response-time advantage of tbe e-mail survey: 6.5 days fastertban tbe mail on average. Tbis obviously reflectsthe in-transit time differences between tbe twomodes. Exhibit 2 also shows a very rapid rise in tbecumulative percent of e-mail responses received.More than balf of tbe total responses (not cumula-tive responses) iu tbe e-mail group arrived witbintwo days, and more than 80% witbin one week.

Two areas of concern to survey researcbers areitem nonresponse and additional informationpassed on tbrough comments to limited responsequestions. Very few items were left blank in eitberthe e-mail or mail surveys, making an inferencefrom our study difficult relative to item nonre-sponse. Approximately 25% of tbe respondents inbotb groups added additional comments to at leastone question for wbicb none was requested.

Our project revealed one major disadvantage fortbe e-mail mode of survey delivery: Non-deliverableaddresses were mucb more common. Of the originale-mail satnple, 43 out of 224 addresses were not use-able. The cuixent technology requires exact address-es for delivery, and tbe smallest error will result innondelivery, hnprovements in e-mail software willovercome this problem to some extent.

Researeh question 3: To determine reasons forresponse or nonresponse to eitber mode of data col-lection, a second survey was mailed to all the sub-

jects wbo received the TQM que,stionnaire. The fol-low-up surveys were sent via regular mail, and tbeoverall response rate was 34.5%. The respon.se ratesfor e-mail and mail respondents were 56.3% and44.2%, respectively. The e-mail and tnail nonre-spondent rates were low, 18.8% and 6.6%, respec-tively. Tbis was not unexpected, but we could notinclude their responses in our analysis. The problemwitb the nonrespon,se group was further complicat-ed by the fact that only tbree e-mail nonrespondeutsremembered receiving tbe questionnaire.

The results of botb tbe TQM and follow-up stud-ies support tbe contention that the availability anduse of e-mail is widespread among business scbooldeans and chairpersons. More tban 85% of the mailsample subjects bad e-mail addresses. Exbibit 3sbows tbat both sample groups' respondents believetbey are knowledgeable about the use of e-mail.

Subjects also are more likely to respond to surveysconcerning topics in wbicb tbey bave interest, a resulttbat agrees witb past survey researcb. However, ourfollow-up survey found significant differencesbetween tbe attitudes of tbe e-mail and mail groupwitb respect to tbeir wilbngness to respond to an e-mail survey. The number of e-mail surveys received(after adjusting for our survey) and the percentagereturned by the two groups were also significandy dif-ferent. This migbt be due to limited availability of tbemail groups' e-mail addresses to researcbers.

STRENGTHS AND WEAKNESSES

Our project supported several potential advan-tages of tbe e-mail survey, including cost savingsin postage and stationery, reduction iu retum leadtime, and willingness of subjects to respond toopen-ended questions. A major disadvantage wasnoted aud measured dealing witb the bigh percent-age of non-deliverable surveys iu the e-mail group.

As a result of our experience in tbis e-mail pro-ject, we observed but did not measure some addi-tional advantages, such as:

• The ability of subjects to ask questions abouttbe survey easily by using the reply functionavailable in e-mail.

• Tbe virtually instant feedback on problem e-mail addresses.

• Respondent flexibility in returning the ques-tionnaire, i.e., via e-mail, mail, or fax.

Included in tbe observed disadvantages aresome unanswered questions: Why did 13 of tbe e-mail respondents not remember receiving tbeir sur-veys? Wby did the mail respondents indicate on thefollow-up survey that tbey were reluctant to retum

3 4 SUMMER 1996, Vol. 8 No, 2 MARKETING RESEARCH:

Exhibit 3

Eollow-up survey results

E-mailrespondents Mail respondents

Response rates (number)Likely to respond to mail survey of interest**Likely to respond to mail survey not of interestLikely to respond to e-mail survey of interestLikely to respond to e-maii survey not of interestPercentage of mail sun/eys completedE-mail surveys received in past yearPercentage of e-mail survey completedKnowledgabie regarding e-mail '"

56,3%(54)1,63,41.5

3.145%

1,54%49.7%

1,85

44.2%(65)1.53.32.V

3,5*49%

,63%*10,3%'

1.64

'significant at ,05 level" I very likely and 4 very uniikely' " 1 very knowledgeable and 4 not very knowledgabie

e-mail surveys despite tbeir apparent access to andknowledge of e-mail? Could tbe data from the e-mail survey be read directly into a database forimmediate presentation or statistical analysis?

LIMITED USE

Despite tbe disadvantages and unanswered ques-tions, e-mail does appear to be a viable alternativeto regular mail in certain circumstances. At tbispoint in time, e-mail surveys would only be repre-sentative of groups such as infonnation system pro-fessionals, CPAs, engineers, architects, college fac-ulty, or auy select population tbat includes individ-uals wbo have access to the Internet and whose e-mail addresses are readily available.

E-mail surveys can be especially belpful inpretesting a survey instrument. In addition to cost

and speed advantages, tbe pretest subject can offersuggestions and corrections witb ease by using thereply function. The researcher can subsequentlyincorporate recommendations from tbe subjectsand retest tbe questions. E-mail can greatlyenbance tbe dialog during survey construction and,consequently, strengtben the final product.

Tbe novelty of this area opens tbe door to a vari-ety of additional researcb topics. Tbe feasibility ofusing e-mail for consumer research in marketsrelated to computer hardware and software seemsappropriate. Tbe viability of e-mail surveys tosolicit response from discussion groups withoutsending the test instruments to specific addressesalso warrants investigation. Aud, as tbe use of e-mail increases and users become more diversified,it will be worthwhile to follow its progress byduplicating tbis type of study periodically. EID

Dickson, John P, and Douglas L,

MacLachlan (1992), 'Tax Surveys?"

Markeiing Research: A Magazine of

Management & AppUcatinn.<i, 4(3),

26-30.

Dickson. John P, and Douglas L, Mac-

Lachlan (1996). "Fax Surveys:

Retum Patterns and Comparison

with Mail Surveys," Journal of

Marketing Research, forthcoming.

Kiesier, Sara and Ue S. Sproull (1986).

"Response EITccts in the Electronic

Survey." Public Opinion Quarterly.

50,402-13,

ADDITIONAL READING

Lewis, Peter H, (1995), '"Home Users

Biggest Presence on Computer

Networks," A'PH- York Times. 23

(October) D-6,

Lyons, Dan (1994), "The E-Mail

Advantage," Computer Reseller

News, (Septemtwr), 43-52.

Opperman. Martin (1995). "E-mail

Surveys—Potentials and Pitfalls,"

Marketing Research: A Magazine of

Management & Appiications, 7

(Summer), 28-33,

Parker, Loraine (1992), "Collecting Data

the E-Mail Way." Training & De-

velopment, (July), 52-54,

Schuldt, Barbara and Jeff E, Totten

(1994). "Electronic Mail vs. Mail

Survey Response Rates." Marketing

Research: A Magazine of Manage-

ment & Applications. 6(t), 36-9.

Va77ijna. Garj' A, and Duane Bachmann

(1994), "Eax Attracts," Marketing

Research: A Magazine of Manage-

ment & Appiications, 6, 19-25.

Duane Bachmann is aProfessoi" of Marketing atCentral Missouri StateUniversitv, Warrensburg, Mo,

John Elfrink is Chairmanof the AccountingDepartment at CentralMissouri State tJniversity.

Gary Vazzana is an AssociateProfessor of Management atCentral Missouri StateUniversity,

MARKETING RESEARCH: SUMMER 1996, Vol. 8 No, 2 3 5