innovation in research software

Innovation in Research SoftwareIs technology providing the answer?Tim Macer, meaning limited London, UKPresented at the Marketing Research Association Fall Technology & Training Event, Orlando, 12-14 November 2003

meaning limited. All rights reserved.

AgendaThe challenges facing research technologyTechnology roundup: whats newAn open futureMixing modes: does it work?


Too much of a good thing?MR software now a $200+ million industry, worldwide1Over 100 specialist software manufacturers2Over 230 commercial MR software products266 web interviewing products50+ CATI products45 Cross-tab productsResearch companies unwilling to pay realistic amounts for their softwareSources: 1Research Software Guide 2003; 2Research Software Central website


How much do MR firms spend?Typically less than 1 per cent!Does not include hardware spend (not estimated)Sources: survey of 35 international research companies in 2003 by MeaningAll revenuesQuantitative research revenuesRevenues from interview-based researchProportion of revenues spent on research software0.580.891.12Mean%0.400.470.47Median%


The legacy of under-investmentPunch card systems still aboundRelational databases the exceptionComplex scripting interfaces when GUIs can and should complement themHard to move data or survey instruments between different packageLabor intensiveHow much do you think research companies spend on specialist IT and DP staff?


Spend on technical staff v. technologyAround twenty times the amount!Spend on IT and DP related staffProportion of total revenues spent11%8%


The demandsLess intrusiveMore rewardingRelevantInteresting & enjoyableFaster More flexibleLess labor-intensiveLess scope for errorReduced training needsMore responsiveIntegrated with other dataLess data, more insightBetter value


Dilemmas for research todayDelivering dataNeeding insightUsing outdated tools, technology and techniquesStill researching the what-why-how-where-whenProviding advice as a value add on top of rigorous methodsNever enough timeGetting answers from CRM and information management expertsKnowing what but not understanding whySeeing value in advice but to hell with the methodologyThat report is so yesterdayThey see usWe see them


Getting a decent ROI on ITWith such a high spend on labor, the scope for rapid total ROI on IT is very goodThere are many candidates for improvementQuestion librariesAuto-assisted codingUse of style and appearance templatesAutomated data entry: OCR and ICR scanningAvoidance of data conversion or re-entry of text labels with full metadata import/export


Innovation in research technologyRelational databasesOpen interfacesWeb enablementProductivity and automationEase of useMultiple modesIntegration with other data sources


Technology to bridge the chasmThere is more to the web than cheap data collectionThe Internet has the potential to transform the way research is done in the futureClientAgencySupplierOutworkers


Web-based innovationsCATI VCC: The virtual call center, from VOXCOVirtual CATI with ASP predictive dialing over the Internet, from NEBUAlways-on mobile CAPI using GPRS cellular data services, from Open AirWeb-based coding of open endsAscribeWeb-based scanningBeyond QuestionWeb-based conjoint surveys and analysisInformation portals and ASP portal builders


Technology for QualitativeOnline focus groups, e.g. ItracksTranscript coding systems, e.g. QSR NVivoLexical analysis software for transcripts, e.g. Sphinx, Clementine LexiquestQual/Quant fusion, e.g. MR Stream, GMI Qualiquant


Online conjointSeveral solutions nowInteresting and enjoyable for respondentCan give feedback at the endThis is what your ideal product would look likeExample productsIdeamap.net (Moskowitz Jacobs)ConjAna (NIPO Software)Engaging for clients tooGood valuemultiple studies provide a rich insights database for re-analysis


Technology showcaseSome selective examples of technology solving todays research problems


CAVI visual interviewing from OpinionOne


ConjAna user-friendly conjoint analysis from NIPO Software


Reportal GUI portal creation tool from FIRM


Web-based scanning from forms defined in Excel


Integrating research and other dataCombining research data with other data sources does not mean throwing away research ethicsData can be combined in many ways:To provide sample for targeted or event driven researchAnonymously, combining findings on a case by case basisThrough a separate, research only data warehouseThrough data ascription, on similar donor casesTo create segmentation models, applying cluster solutions from the research data to the larger operational database


CRM and researchCRM is not a threat but an opportunityCRM is failing to deliver the what next insightCRM is perfect for the what; you need research to understand whyIt is not in the long term interest of any organization to confuse research with marketing in the view of its customers


Example: Combining research and CRM dataA1A2A3A4B1B2B32654187171174410928771019916117311781148Attitudinal SegmentsBehavioral SegmentsSegmentation AnalysisSource: Mike Page, Cognicient Consulting


Open standards for researchEnabling cooperation and best-of-breed solutions


Proprietary imports and exports are not good enoughToo many competing packages; you may be in a universe of one, in using your combinationProprietary data formats can change without noticeToo much effort wasted converting from one format to anotherData are useless without the definitions and texts - the metadata


XML is not a standard languageAccording to W3C, The Extensible Markup Language (XML) is the universal format for structured documents and data on the WebClose relative of HTML: both originated from SGMLLets you create your own definition languagesIncreasingly used within MR software for data definition and execution logicEveryones XML is differentEasier to translate but not mutually intelligible


Initiatives from around the worldTriple-s www.triple-s.org First published 1994 Originated in the UK but now implemented by 30 vendors worldwideExchange data and metadata via exports and imports in a generalized formatVersion 1.1 introduced XML supportNew version 1.2 adds filters, weighting and multi-language support


Example triple-s code Q4 Which sights have you visited? MULTIPLE 41 TO 42 2 Statue of Liberty Empire State Building

Initiatives from around the worldOpen Survey www.opensurvey.org Not-for-profit organization that promotes open standards for softwareEncourages production of open source software for market researchAskML is its proposed XML standard for exchanging survey instruments TabSML is its working standard for exchange of cross-tabular reports in a generalized formatAlso endorses triple-S


Initiatives from around the worldSPSS Dimensions Data Model www.spssmr.comA new open (though proprietary) metadata model for survey data Can be licensed independently of all SPSS MR products (dont have to use SPSS software)Comes with a developers library of tools for building applications that will read or write data via the SPSS Data ModelMany other software companies now providing support for the SPSS Data Model


Initiatives from around the worldQEDMLwww.philology.com.auNew multi-platform survey authoring toolExports scripting languages for several packages, including Quancept, Surveycraft and In2formXML based open system, allows other language translators to be added


Mixed Mode Research


OMR scanningFace-to-faceTelephoneCATITCASI (IVR)MCAPICAPICASIOCR scanningWAPEvolution of todays survey modesTechnology independentTechnology basedDisk by mailCAWI


The rise of multiple modesWeb now seen as an undisputed replacement for paper-based mail surveys*Response rates falling globallyLess than 20% is becoming normalSampling may only reach 20% of the universeMultiple modes seen as a way to redress thisTechnology needed to facilitate thisCase studies showing that mixing modes canAchieve a better responseRemain scientifically valid*Source: RS Owen in Quirks magazine, Feb 2002, p.24-26


The multi-mode bandwagonModes supportedProduct choice (42 packages)Source: Research Guide to Software 2003


Multi-mode: the challengeSurvey organizations, whether they are in universities like mine, in private-sector organizations or in government organizations, are going to have to change dramatically in some ways in order to do effective surveys as we bring these new technologies online and still use our other technologies where they work.Don Dillman, Washington State University


What do we mean by multi-mode?Multi-modeSurveys utilizing more than one research channelto reach different sub-samples, but confiningeach sub-sample to one channelMixed modeSerialSurveys that involve successive interviewing stages, each utilizing a different modeParallelSurveys that allows participants to choose the mode and even to switch modes


The issues in mixing modesCalibrationThe risk of differential measurement error due to modal effect on the respondentCoverageSampling issuesrisk of differential non-response from sub-samples for each modeComplexityDuplication of operational and programming effort in addressing more than one modeIncreased cost, delays and errors from this duplication


Does it work? Allison & OKonisCATI and Web88% of CATI respondents, many of whom would have terminated, agreed to a continue their interview on the web54% of them went on to completeOosterveld and Willems mixed CATI/Web surveys can be designed to have no influence on the answersTheir experimental design separated modal effect from population effect


Modal influencesPresentational influencesGanassali and Moscarola have measured increased responses when relevant visuals clues presented in web interviewsModerating effect of interviewerNoted by Poynter and Comely amongst others: under-reporting of socially unacceptable responsesDifferences in open-ended responsesOosterveld and Willems claimed longer and more detailed on the web than phoneAllison and OKonis claimed great similarity for same two modes


Modal influences (2)Differences with scale questionsMany experiments show that there is a difference between modesCATI respondents tend to use the extremesCAWI resps make fuller use of the entire scaleTop-2 and bottom-2 scores tend to be similarDifference in dont knowsMore answers recorded as DK/NA in Web surveys than same survey when interviewer-led in CATINon-response (non-participation)Don Dillman and others observed greater tendency for males not to participate in CATI and females in Web surveys


Operational complexity issuesDifferent recruitment and screeningCant always approach by same modeDuplication of the survey instrumentComplete duplication of effort may be requiredProblems managing multiple versionsData HandlingNeed data in one place in one formatProblems mixing online and offline modes Mode switchingMust be fast if response rate to be improvedMode-appropriate texts


Multi-mode case studyDutch study of 60,000 households60% response requiredConsortium of research organisations involvedStandardised on same software (Bellview)Tri-modal: CATI, CAPI and WebRespondents could choose methodNon-response followed up by another method


Multi-mode case study (2)The users verdictIn practice, things turned out to be much more difficult to streamline. The differences between face-to-face, phone and self-completion on the Web meant we had to create three different scripts for each interviewing method, and this made the database very large and unwieldy. It also required a lot more scripting hours than we imagined.Hester Rippen, Unified Fieldwork Organization, Netherlands


Suppliers contactedAskiaAskiaMercatorsnapMI ProMI Pro Research StudioNebuDub InterviewerOpinion OneCAVIPulse TrainBellview FusionSphinxSphinxSPSS MRDimensions


Who supports what?


Innovation: Calibration issuesReduction of modal influenceOpinion One CAVITotally consistent appearance for Web, CASI & CAPINovel method for unaided questions in self-completion modesSphinxExperimental approachMeasurement of modal differencesPulse Train collect paradata on mode for each question


Innovation: Complexity issuesModal independent design SPSS MRModal playersAskia, MI Pro, Pulse Train, Nebu, SPSS MRModal templates applied to same survey instrumentCentral databaseAll apart from snapWizards for importing offline data in Askia


Innovation: Complexity issuesMode switchingHandled well in Askia, Pulse Train, Nebu and Opinion OneEmail despatched automatically in Opinion OneNebu recognises static and dynamic swapsCall me button in Pulse Train linked to diallerRecall of interviews into CATI mode in Askia, Nebu, Pulse TrainSwitching in and out of paper in MI Pro


Ideal multi-mode features (1/2)A common survey authoring tool that generates a single survey instrument for all modes.Independence between design and execution, with mode-specific templates and rules.The ability to define mode-specific texts in addition to foreign language alternatives.A single, consolidated database for all survey data, updated in real time.Ability to determine the mode of initial contact from the sample subjects stated preference.Efficient switching between modes, initiated by the script or by the respondent.


Ideal multi-mode features (2/2)Ability to conceal all interviewer-recorded data when switching to self-completion modes.Support for reminders and fall-back strategies to revert to a prior mode if still incomplete.Single view management and reporting, which identifies response by mode.Quota controls implemented across all modes in real time.Question constructs for mixed mode, e.g. unprompted questions for CAWI, and the ability to have mode specific answer categories (e.g. dont know and not stated). Recording of mode applied, at a datum level not just a case level.


Further readingAllison J & OKonis C (2002) If Given the Choice, Quirks Marketing Research Review, July/August issue, p 20.Bckstrm, C & Nilsson, C (2002) Mixed mode: Handling method differences between paper and web questionnaires, http://gathering.itm.mh.se/modsurvey/pdf/MixedMode-MethodDiff.pdfDillman D A (1978) Mail and Telephone Surveys: The Total Design Method, Wiley Dillman D A, Phelps G, Tortora R, Swift K, Kohrell J & Berck J (2001) Response Rate Measurement Differences in Mixed Mode Surveys Using Mail, Telephone, Interactive Voice Response and the Internet, AAPOR Annual Conference, MontrealGanassali S & Moscarola J (2002) Protocoles denqute et efficacit des sondages par Internet, Journes E-Marketing AFM/AIM Conference, Nantes, FranceMacer, T (2003) Research Software Review, The Market Research Society, London.Oosterveld, P & Williams P (2003) Two Modalities, One Answer. ESOMAR Technovate Conference, Cannes. Owen R S (2002) A Matter of Trade-offs: Examining the advantages and disadvantages of online surveys, Quirk.s Marketing Research Review, February, pp 24-26. Poynter R and Comely P (2003) Beyond Online Panels. ESOMAR Technovate Conference, CannesRippen H (2003) One project, three methods, in Research Guide to Software (2003).Taylor H (2000) Does Internet Research Work? Comparing online survey results with telephone survey, International Journal of the Market Research Society, 42.1


www.meaning.uk.com


innovation in research software

Documents