This article was downloaded by: [68.181.178.98] On: 15 September 2017, At: 13:55Publisher: Institute for Operations Research and the Management Sciences (INFORMS)INFORMS is located in Maryland, USA

Marketing Science

Publication details, including instructions for authors and subscription information:http://pubsonline.informs.org

Consumer Preference Elicitation of Complex ProductsUsing Fuzzy Support Vector Machine Active LearningDongling Huang, Lan Luo

To cite this article:Dongling Huang, Lan Luo (2016) Consumer Preference Elicitation of Complex Products Using Fuzzy Support Vector MachineActive Learning. Marketing Science 35(3):445-464. https://doi.org/10.1287/mksc.2015.0946

Full terms and conditions of use: http://pubsonline.informs.org/page/terms-and-conditions

This article may be used only for the purposes of research, teaching, and/or private study. Commercial useor systematic downloading (by robots or other automatic processes) is prohibited without explicit Publisherapproval, unless otherwise noted. For more information, contact permissions@informs.org.

The Publisher does not warrant or guarantee the article’s accuracy, completeness, merchantability, fitnessfor a particular purpose, or non-infringement. Descriptions of, or references to, products or publications, orinclusion of an advertisement in this article, neither constitutes nor implies a guarantee, endorsement, orsupport of claims made of that product, publication, or service.

Copyright © 2016, INFORMS

Please scroll down for article—it is on subsequent pages

INFORMS is the largest professional society in the world for professionals in the fields of operations research, managementscience, and analytics.For more information on INFORMS, its publications, membership, or meetings visit http://www.informs.org

Vol. 35, No. 3, May–June 2016, pp. 445–464ISSN 0732-2399 (print) � ISSN 1526-548X (online) http://dx.doi.org/10.1287/mksc.2015.0946

© 2016 INFORMS

Consumer Preference Elicitation of Complex ProductsUsing Fuzzy Support Vector Machine Active Learning

Dongling HuangDavid Nazarian College of Business and Economics, California State University, Northridge, Northridge, California 91330,

dongling.huang@csun.edu

Lan LuoMarshall School of Business, University of Southern California, Los Angeles, California 90089, lluo@marshall.usc.edu

As technology advances, new products (e.g., digital cameras, computer tablets, etc.) have become increasinglymore complex. Researchers often face considerable challenges in understanding consumers’ preferences for

such products. This paper proposes an adaptive decompositional framework to elicit consumers’ preferencesfor complex products. The proposed method starts with collaborative-filtered initial part-worths, followed byan adaptive question selection process that uses a fuzzy support vector machine active learning algorithm toadaptively refine the individual-specific preference estimate after each question. Our empirical and syntheticstudies suggest that the proposed method performs well for product categories equipped with as many as 70 to100 attribute levels, which is typically considered prohibitive for decompositional preference elicitation methods.In addition, we demonstrate that the proposed method provides a natural remedy for a long-standing challenge inadaptive question design by gauging the possibility of response errors on the fly and incorporating the results intothe survey design. This research also explores in a live setting how responses from previous respondents may beused to facilitate active learning of the focal respondent’s product preferences. Overall, the proposed approachoffers new capabilities that complement existing preference elicitation methods, particularly in the context ofcomplex products.

Data, as supplemental material, are available at http://dx.doi.org/10.1287/mksc.2015.0946.

Keywords : new product development; support vector machines; machine learning; active learning; adaptivequestions; conjoint analysis

History : Received: November 25, 2013; accepted: March 16, 2015; Pradeep Chintagunta, Dominique Hanssens, andJohn Hauser served as the special issue editors and Theodoros Evgeniou served as associate editor for thisarticle. Published online in Articles in Advance March 2, 2016.

1. IntroductionAs technology advances, new products (e.g., digitalcameras, computer tablets, etc.) have become increas-ingly more complex. Researchers often face considerablechallenges in understanding consumers’ preferences forsuch products (e.g., Green and Srinivasan 1990, Hauserand Rao 2004). Conventional preference elicitationmethods such as conjoint analysis often become infeasi-ble in this context because the number of questionsrequired to obtain accurate estimates increases rapidlywith the number of attributes and/or attribute levels.Historically, researchers have relied primarily on com-positional approaches to handle preference elicitation ofsuch products (e.g., Srinivasan 1988, Scholz et al. 2010,Netzer and Srinivasan 2011). Adaptive question selec-tion algorithms have also been proposed for complexproduct preference elicitation due to their ability torapidly reveal consumer’s product preferences with rel-atively few questions (e.g., Netzer and Srinivasan 2011).While significantly enhancing our ability to under-stand consumers’ preferences for complex products,

the extant research has yet to address the followingchallenges.

First, although widely used in the literature, compo-sitional approaches may encounter obstacles such asunrealistic settings, inaccurate attribute weighting, etc.(e.g., Green and Srinivasan 1990, Sattler and Hensel-Börner 2000) Second, despite their high efficiency inuncovering consumers’ product preferences, adaptivequestion selection methods are often subject to responseerrors that could misguide the selection of each subse-quent question. Last, with the exception of Dzyaburaand Hauser (2011) in the context of considerationheuristics elicitation, this line of research has yet toexplore the possibility of using other respondents’ datato facilitate active learning of the focal respondent’sproduct preferences.

Our research proposes an adaptive decompositionalframework in response to these challenges. The pro-posed method starts with a collaborative-filtered initialpart-worths, followed by an adaptive question selec-tion process using a fuzzy support vector machine

445

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex Products446 Marketing Science 35(3), pp. 445–464, © 2016 INFORMS

(SVM) active learning algorithm to adaptively refine theindividual-specific preference estimate after each ques-tion. Compared to extant preference elicitation methods,our research offers the following new capabilities:

• Our adaptive decompositional approach is compu-tationally efficient for preference elicitation of complexproducts on the fly, as the algorithm primarily scaleswith the sample size of the training data, rather thanthe dimensionality of the data vector.

• While extant research either neglects responseerrors in adaptive question selection or sets possibilityof error instance as a priori, our algorithm gauges thepossibility of response errors on the fly and incorporatesit into adaptive survey design.

• Although most adaptive question selection meth-ods only use information from the focal respondent,we use responses from previous respondents in a livesetting via collaborative filtering to facilitate activelearning of the focal respondent’s product preferences.

We illustrate the proposed method in two computer-based studies involving digital cameras (with 30+ at-tribute levels) and computer tablets (with 70+ attributelevels). Our empirical investigation demonstrates thatthe proposed method outperforms the self-explicatedmethod, the adaptive Choice-Based Conjoint method,the traditional Choice-Based Conjoint method, andan upgrading method similar to Park et al. (2008)in its ability to correctly predict validation choices.Our synthetic data experiments further demonstratethat the proposed method can rapidly and effectivelyelicit individual-level preference estimates even whenthe product category is equipped with more than 100attribute levels. We also use synthetic data experimentsto compare the scalability, parameter recovery, andpredictive validity of the proposed algorithm withthat of Dzyabura and Hauser (2011) for considerationquestions and of Abernethy et al. (2008) for choicequestions. We show that the proposed question selec-tion algorithm may be used in conjunction with or asa substitute for algorithms by Dzyabura and Hauser(2011) and Abernethy et al. (2008) to uncover con-sumers’ preferences for complex products. Overall,our empirical and synthetic studies suggest that theproposed approach offers a promising new method tocomplement existing preference elicitation methods.

The remainder of the paper is organized as follows.In §2, we discuss the relationship of this research tothe extant literature. In §3, we present the proposedadaptive question design algorithm. In §4, we describeour two empirical applications. Details of our syntheticstudies are presented in §5. The final section summa-rizes key results, discusses limitations of this research,and offers directions for future research.

2. Relationship to Extant LiteratureThe algorithm used in our proposed framework isclosely related to the machine learning literature with

origins in computer science. An important applica-tion of machine learning is classification, in whichmachines “learn” to recognize complex patterns, todistinguish between exemplars based on their differentpatterns, and to make intelligent predictions on theirclasses. Many marketing problems require accuratelyclassifying consumers and/or products (or both) (e.g.,consumer segmentation; identification of desirableversus undesirable products). Therefore, marketingresearchers have recently begun to embrace machinelearning methods in the estimation of classic marketingproblems (e.g., Cui and Curry 2005; Evgeniou et al.2005, 2007; Hauser et al. 2010).

Built on this stream of literature, the current paperintroduces SVM-based active learning into adaptivequestion design. Arguably the most popular statisticalmachine learning method in the past decade (Toubiaet al. 2007a), SVM methods are well known for high-dimensional classification problems (e.g., Vapnik 1998,Tong and Koller 2001). In particular, we use a fuzzySVM method to adaptively select each subsequentquestion for each respondent on the fly. As a weightedvariant of the soft margin SVM formulation (the softmargin SVM was initially introduced by Cortes andVapnik 1995), the fuzzy SVM method assigns differentweights to different data points to enable greater flexi-bility of error control. Since the early 2000s, the classof fuzzy SVM methods has gained notable popularityin the SVM literature, mainly due to its effectiveness inreducing the effect of noises/errors in the data (e.g.,Lin and Wang 2002, 2004; Wang et al. 2005; Shiltonand Lai 2007; Heo and Gader 2009).

When used for preference elicitation of complexproducts, this algorithm exhibits a number of advan-tages over extant methods. One desirable propertyof the SVM-based active learning algorithm is thatthe optimization used to facilitate adaptive selectionof each subsequent question can be transformed to adual convex optimization problem (Tong and Koller2001). In our context, the primal problem (Equations (2)and (4)) is also constructed to be convex. Therefore, theproposed algorithm not only offers an explicitly definedunique optimum but also is easily solvable by mostsoftware for problems with dimensions that are likelyto be of interest to marketers. Indeed, the SVM-basedclassification is primarily scaled by the size of the train-ing data (i.e., the number of questions presented to eachconsumer in our context), rather than the dimensional-ity of the data vector (Dong et al. 2005). Consequently,the SVM-based active learning method is particularlysuitable for the problem at hand. In contrast, severalalternative adaptive methods (such as the adaptive fastpolyhedral methods by Toubia et al. 2003, 2004, 2007b)are scaled by the dimensionality of the product vector.This may become more computationally cumbersomeas the dimension of product attributes/attribute levelsincreases. Moreover, while the Hessian-based adaptive

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex ProductsMarketing Science 35(3), pp. 445–464, © 2016 INFORMS 447

methods (e.g., Abernethy et al. 2008, Toubia et al. 2013)require discrete transformations when used for discreteattributes, the SVM-based active learning method isflexible enough to directly accommodate discrete andcontinuous product attributes.

Another unique advantage particularly related to thefuzzy SVM active learning method is that it enablesresearchers to gauge the possibility of response errorson the fly and to incorporate it into adaptive questionselection. In the context of adaptive question design,response errors may be conceptualized as the ran-dom error component in consumer’s utility function(e.g., Toubia et al. 2003). Empirical data suggest thatresponse errors are approximately 21% of total util-ity (Hauser and Toubia 2005). Because each responseerror might set the adaptive question selection to thewrong path and negatively impact selection of allsubsequent questions, the presence of such errors posesa long-standing challenge to the adaptive questiondesign literature (e.g., Hauser and Toubia 2005, Toubiaet al. 2007a). To date, response errors have eitherbeen neglected (e.g., Toubia et al. 2004, Netzer andSrinivasan 2011) or set as a priori possibility for all indi-viduals and all questions (e.g., Toubia et al. 2003, 2007b,2013; Abernethy et al. 2008; Dzyabura and Hauser2011). We demonstrate that the proposed method canbe used not only to gauge possible response errors onthe fly but also to reduce the effects of such noise inadaptive question selection.

Last, inspired by Dzyabura and Hauser (2011) whosuggest that previous-respondent data may be usedto improve elicitation of consideration heuristics, theproposed method uses responses from previous respon-dents via collaborative filtering to facilitate active

Figure 1 Overall Flow of the Proposed Adaptive Question Design Method

Previous respondents’part-worths and self-configured profiles

Configurator(collaborative filtering)

Identify must-haves and/or unacceptablefeatures

Adaptive choice questions(choice-based fuzzy SVM active learning)

Initialpart-worths

Candidate pool ofprofiles

Adaptive consideration questions(consideration-based fuzzy SVM active learning)

learning of the focal respondent’s product preferences.The concept of collaborative filtering has been appliedin various contexts such as prediction of TV show pref-erences and movie recommendation systems (Breeseet al. 1998). We illustrate that such a technique can beincorporated in adaptive question design using actualrespondents in a live setting.

3. Adaptive Question Selection forComplex Product PreferenceElicitation

3.1. Overall FlowFigure 1 depicts the overall flow of our adaptive ques-tion design. We begin by prompting the consumer to con-figure a product that he is most likely to purchase, takinginto account any corresponding feature-dependentprices. Based on collaborative filtering between thefocal and previous respondents’ self-configured profiles,we obtain an individual-specific initial part-worthsvector (§3.2). Next, we provide each consumer withan option of selecting “must-have” and “unaccept-able” product features. We use information obtainedfrom such features to construct the pool of candidateprofiles to be evaluated in the consideration stage,as it is infeasible to evaluate all profiles using activelearning without excessive delays in between surveyquestions within our context (§3.3). Conditional on theinitial part-worths and the candidate pool of profilesobtained for each respondent, we use a two-stageconsider-then-choose process where the considerationstage asks the respondent whether he would con-sider a product profile (§3.4) and the choice stage asks

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex Products448 Marketing Science 35(3), pp. 445–464, © 2016 INFORMS

the respondent to choose among competing productprofiles (§3.5). In both stages, we use fuzzy SVM activelearning for adaptive question design, so that eachsubsequent question is individually customized torefine the consumer-specific preference estimate whileaccounting for possible response errors on the fly.

We explain the underlying rationale of our overallframework below. The primary goal of the proposedmethod is to estimate a part-worths vector for eachrespondent j . Given that the scale of the part-worthsvector is arbitrary (Orme 2010), before the respondentanswers any question, we may visualize the feasibleregion of the part-worths being all of the points onthe surface of a hypersphere with unit norm (i.e.,wj ∈ W � �wj� = 1). Such a feasible region is referredto as version space in Tong and Koller (2001) andHerbrich et al. (2001) and a polyhedron in Toubia et al.(2003, 2004, 2007a). Conceptually, each answer givenby the respondent provides constraint(s) that makesthis feasible region smaller.

Given the large number of attributes/attribute levelsassociated with complex products and the limitednumber of questions we can ask each respondent,an informative first question would enable us to effi-ciently construct the initial region of the feasible part-worths (§3.2). Similarly, by constructing a candidatepool with the majority of profiles satisfying the “must-have” and “unacceptable” criteria, we can maximizeour learning about the focal respondent’s productpreferences by asking whether he would consider aprofile based on his favorability towards other prod-uct features (§3.3). Essentially, the first two steps ofour overall framework aim to construct a suitablefoundation for the adaptive question selection later on.

We then use a consider-then-choice framework to elic-it each respondent’s product preference (§§3.3 and 3.4).Specifically, our algorithm aims to uncover a set ofpart-worths estimates that are consistent with theconsumer’s answers to these questions. Historically,researchers have often used conjunctive rules to cap-ture consumer’s decision rules in the considerationprocess (e.g., Hauser et al. 2010, Dzyabura and Hauser2011). It has been less common to use part-worths tocharacterize consumer’s responses to considerationquestions. Our synthetic data experiments reveal that,even when the true consideration model is driven byconjunctive decision rules, the part-worths estimatefrom our algorithm exhibits good ability to predictwhether the respondent would consider a profile (§5.1).Indeed, even if heuristic decision rules are adoptedby some respondents (particularly in the considera-tion stage), such preferences will be partially capturedin our individual-specific part-worths estimates thataim to optimally predict responses from these con-sumers. Therefore, while not explicitly portraying theserespondents’ consideration heuristics, our part-worths

estimates serve as an approximation of the decisionheuristics used by such individuals.1

Within this setup, we present an algorithm whereinwe use a set of part-worths estimates to characterizea respondent’s answers to consideration and choicedecisions. In this context, we aim to select each subse-quent consideration/choice question such that we canreduce the feasible region of the part-worths as rapidlyas possible. Intuitively, one good way to achieve thisgoal is to choose a question that halves such a region(Tong and Koller 2001; Toubia et al. 2003, 2004, 2007b).To accomplish this goal, we adapt the active learningapproach proposed by Tong and Koller (2001) to selecteach subsequent consideration/choice question on thefly. Similar to Toubia et al. (2003, 2004, 2007a), thisalgorithm relies on intermediate individual-level part-worths estimates to adaptively select each subsequentquestion. If the consumer makes no response errors,such an approach would rapidly shrink the feasibleregion of the part-worths. Nevertheless, response errorsare often inevitable in practice. To reduce the effects ofresponse errors, we adapt the fuzzy SVM estimationalgorithm (Lin and Wang 2002) in consideration andchoice stages to obtain the intermediate part-worths.Under this algorithm, each part-worths estimate isobtained as an interior point within the current feasibleregion of part-worths, via a simultaneous optimizationthat balances between data-imposed constraints andweighted classification violation. Consequently, whenselecting each subsequent question based on such inter-mediate part-worths estimates, the negative impact ofresponse errors in the process of adaptive questionselection would be alleviated.

We also conjecture that our current multistage frame-work may help reduce the effect of response errors.In adaptive question design, early response errors areconsiderably more detrimental than errors that occurtoward the end of the adaptive survey (e.g., Hauserand Toubia 2005, Toubia et al. 2007a). Therefore, whenpresented with the less demanding self-configurationand consideration questions before the more challeng-ing choice questions, respondents may be less likelyto incur response errors early on. Pseudo code of ouralgorithm is provided in Web Appendix A (availableas supplemental material at http://dx.doi.org/10.1287/mksc.2015.0946). Screenshots of the survey interface

1 Note that, in practice, some consumers may not use the sameutility function for consideration and choice decisions. For example,an individual might emphasize different sets of attributes in theconsideration phase versus in choice phase. In §5.1, we discuss howour algorithm can be used in conjunction with the algorithm byDzyabura and Hauser (2011) to accommodate a consider-then-choiceframework wherein conjunctive rules are used to examine answers toconsideration questions and conjoint part-worths are used to captureproduct preferences reflected in choice questions. In such cases,different utility functions may also be used to model considerationand choice decisions separately.

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex ProductsMarketing Science 35(3), pp. 445–464, © 2016 INFORMS 449

from our computer tablet applications are presented inWeb Appendix B.

3.2. Collaborative-Filtered Initial Part-Worths VectorOur framework begins by asking each respondentto configure a product profile that he is most likelyto purchase, taking into account any correspondingfeature-dependent prices.2 Such a self-configured prod-uct profile provides substantial information about aconsumer’s product preferences (Dzyabura and Hauser2011, Johnson and Orme 2007). In our context, theinformation from the self-configured profile is used asfollows.

First, we generate an initial individual-specific part-worths vector based on collaborative filtering betweenthe focal and previous respondents’ self-configuredproduct profiles. The basic intuition is that we maylearn about the focal respondent’s product preferencesby examining the preferences of previous respondentswho have configured similar product profiles. For exam-ple, if two respondents self-configured two identicalcomputer tablets, they are likely to share some com-monality in their overall preferences towards computertablets. Analogous to the role of informative prior inthe Bayesian literature, consumer-specific initial part-worths obtained through collaborative filtering mayenhance active learning of the focal respondent’s prod-uct preference at the outset of our adaptive questionselection. Specifically, after the focal respondent j con-figures a profile that he is most likely to purchase, thefollowing equation is used to obtain the respondent’sinitial part-worths vector w̃0

j on the fly:

w̃0j =

1j − 1

j−1∑

j ′=1

s4j1 j ′5 · w̃j ′ with

s4j1 j ′5=c′j · cj ′

�cj��cj ′�1 (1)

where w̃j ′ is the estimated part-worths of previousrespondent j ′ with j ′ = 11 0 0 0 1 j − 1. In Equation (1), cjdenotes the vector that represents product features ofthe self-configured profile for the focal respondent j ,cj ′ represents the corresponding vector from previousrespondent j ′, and s4j1 j ′5 measures the degree of cosinesimilarity between vectors cj and cj ′ .

The cosine similarity measure is widely used to cap-ture the similarity between two vectors in informationalretrieval and collaborative filtering literature (e.g.,

2 Following Johnson and Orme (2007), we include feature-dependentprices in the self-configuration task to increase the realism of thistask (otherwise respondents may self-configure the most advancedproduct profile with the lowest price). In the subsequent considerationand choice questions, we follow Orme (2007) by adopting a summedprice approach with a plus/minus 30% random price variation inboth empirical studies.

Salton and McGill 1986, Breese et al. 1998). Given thatour method uses aspect type coding with attribute-leveldummies, this measure is bounded between 0 and 1. Inparticular, s4j1 j ′5= 0 if there is no overlap between cjand cj ′ ; 0 < s4j1 j ′5 < 1 if there is partial overlap betweencj and cj ′ ; and s4j1 j ′5= 1 if cj = cj ′ . The resulting initialpart-worths is then used to identify the next set ofprofiles shown to the consumer.

Second, we use information from the configurator toset the two initial anchoring points in our fuzzy SVMactive learning algorithm. As a classification method, awell posed SVM problem entails training data from bothclasses. In our context, we first give the self-configuredproduct profile (i.e., the respondent’s favorite) a labelof “1” (meaning that the consumer would consider it).Next, we select a profile among those that are the mostdifferent from the self-configured profile and give ita label of “−1.” As such a profile is not unique, the“opposite” profile is randomly chosen among thosethat do not share any common feature with the self-configured profile (our synthetic data experimentssuggest that, in over 99% of cases, consumers wouldnot consider such an opposite profile). After the next setof profiles is queried based on the collaborative-filteredinitial part-worths, we combine the two anchoringpoints with the newly labeled profiles in the trainingdata to ensure that the SVM problem is well posed.When previous respondents are absent, only the twoanchoring points are used to obtain the initial part-worths vector for the focal respondent.

3.3. Identify Must-Have and/or UnacceptableProduct Features

After the configuration task, we provide each respon-dent with an option of selecting some “must-have”and “unacceptable” product features. In the contextof complex products, it is often infeasible to evaluateall profiles using active learning without excessivedelays between survey questions (Dzyabura and Hauser2011). One major advantage of identifying the “must-have” and “unacceptable” product features is that suchinformation can be leveraged to construct the pool ofcandidate profiles to be evaluated in the considerationstage.3

In particular, among all possible product profilesto be queried, we develop an individual-specific poolcontaining (e.g., 90%) profiles that satisfy the “unaccept-able” and “must-have” criteria (denoted as N 1

j 5, withremaining profiles randomly chosen from those that do

3 Note that one potential caveat of this approach is that the inclu-sions of “must-have” and “unacceptable” features might prime therespondent into conjunctive-style decision making. An alternativewould be to use the uncertainty sampling method used by Dzyaburaand Hauser (2011) to construct the candidate pool of product profiles,with the trade-off that it might be challenging to accurately identifythe most uncertain profiles during the first few queries.

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex Products450 Marketing Science 35(3), pp. 445–464, © 2016 INFORMS

not satisfy these criteria (denoted as N 2j 5.

4 The rationalefor having the majority of profiles in the candidate poolsatisfying the “must-have” and “unacceptable” criteriais that we can maximize our learning about the focalrespondent’s product preferences by asking whetherhe would consider a profile based on his favorabilitytowards other product attributes. The remaining pro-files are chosen to account for the possibility that someindividuals may identify “desirable”/“undesirable”features as “must-haves”/“unacceptables” (Johnsonand Orme 2007). As long as the size of N 2

j is sufficientlylarge, our adaptive algorithm will update the estimatedpart-worths vector so that profiles not satisfying theinitial criteria may also be queried in subsequent surveyquestions.

3.4. Consideration-Based Fuzzy SVM ActiveLearning

We next present the consideration-based fuzzy SVM ac-tive learning algorithm. We first describe the algorithmused to estimate the individual-specific part-worthsvector on the fly. Then we elaborate the algorithm usedto adaptively select profiles queried in each subsequentquestion.

3.4.1. Algorithm to Estimate Individual-SpecificPart-Worths on the Fly. Let xi

j 4i = 1121 0 0 0 1 I3 j =

1121 0 0 0 1 J 5 denote the aspect type coded product pro-file i labeled by respondent j as yi

j = 1 to indicatethat he would consider the profile and yi

j = −1 if hewould not consider it. Following the tradition in theconjoint literature, we use a main-effects only modelwhere wI

j · xij denotes the utility estimate of product

profile i 4i = 1121 0 0 0 1 I5 for respondent j and �Ij being

the utility estimate of his “no-choice” option afterI profiles are labeled. The utility of the “no-choice”option represents the decision boundary where theconsumer would only consider a profile if its utilityis no less than the baseline utility associated withthe “no-choice” option (Haaijer et al. 2001). That is,consumer i will only consider profile j , i.e., yi

j = 1, ifwI

j · xij −�I

j ≥ 0; yij = −1 otherwise.

In this context, the primary purpose of the SVMestimation algorithm is to find an individual-specificpart-worths (i.e., w̃I

j = 4wIj 1�

Ij 55 that can correctly clas-

sify labeled profiles into the two classes of “wouldconsider” and “would not consider.” Finding such apart-worths vector can be challenging in practice as

4 Synthetic data experiments reveal that, as long as the total numberof candidate profiles (i.e., Nj =N 1

j +N 2j 5 is sufficiently large, we can

recover a part-worths estimate that is close to the true part-worthsunder our active learning method. In the empirical applications,we set Nj to be 20,000. Our synthetic studies suggest that this issufficiently large to recover the true part-worths while keeping thequestion selection at less than 0.25 second between questions atthe consideration stage. Similar approaches are used to determinethe number of profiles to be considered in the choice stage.

(1) there may be response errors in the data, and (2) thetrue decision process may not be representable by alinear utility function as specified above. Regardingthe first issue, we use a fuzzy SVM algorithm thatassigns a different weight to each labeled profile alongwith a regularization parameter to enable classificationviolation when we present the algorithm later in thissection. The second issue can be alleviated by usingaspect type coded product utility or by introducing anonlinear kernel to the SVM algorithm. Compared withthe alternative continuous/attribute level order coding,the aspect type coded product utility functions enablegreater flexibility to accommodate nonlinear preferencewithin each attribute (e.g., the utility function doesnot require monotone preferences for screen size, suchas smaller/bigger size is strictly better). Additionally,nonlinear kernels could be used if there are nonlinearpreferences across attributes. For example, if priorknowledge suggests that interaction effects exist amongtwo or more product attribute levels, the SVM estima-tion algorithm can be readily adapted to accommodatesuch a nonlinear utility function. Vapnik (1998) andEvgeniou et al. (2005) provide detailed discussions onthe generalization of the SVM estimation algorithmto such nonlinear models, which also maintains itscomputational efficiency even with highly nonlinearutility functions.

For simplicity, we demonstrate our estimation algo-rithm below using the example of the main-effectsonly model. Formally, upon labeling of I profiles (i =1121 0 0 0 1 I ), the following algorithm is used to estimaterespondent j’s part-worths vector (Vapnik 1998, Tongand Koller 2001, Lin and Wang 2002):

minwI

j 1�Ij 1 �

ij

[

12�wI

j� +CI∑

i=1

uij�

ij

]

s.t. yij 4w

Ij · xi

j −�Ij 5≥ 1 − �i

j

�ij ≥ 01

(2)

where C is an aggregate-level regularization parameterthat allows a certain degree of prior misclassification atthe aggregate level, �i

j is a slack variable that can bethought of as a measure of the amount of misclassifica-tion associated with profile i, and ui

j assigns a differentweight to each labeled profile. When ui

j = 1, Equation (2)corresponds to the soft margin SVM algorithm.

The regularization parameter C in Equation (2) mustbe determined outside the SVM optimization.5 Whilethis parameter may partially absorb the negative impactof noises in the data by allowing a certain degree of

5 Following the approaches described in Evgeniou et al. (2005) andToubia et al. (2007b), we use a cross-validation method based onpretest data from the same population as the main study to determinethe values of C in our two empirical applications.

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex ProductsMarketing Science 35(3), pp. 445–464, © 2016 INFORMS 451

prior misclassification at the aggregate level, we discussbelow how the fuzzy membership method providesadditional flexibility to gauge potential response errorson the fly.

Instead of assuming that all labeled data belongto one of the two classes with 100% accuracy, thefuzzy SVM method assigns a fuzzy membership toeach labeled profile so that data points with a highprobability of being corrupted by noises will be givenlower values of fuzzy memberships (Lin and Wang2002). Therefore, rather than giving each labeled datapoint equal weight in the optimization, profiles withhigher probabilities of being meaningless will be givenless weight in the estimation under the fuzzy SVMmethod.

In practice, researchers often do not have completeknowledge about the causes or nature of noises inthe data. Therefore, in the machine learning literature,researchers have explored various approaches to dis-cern noises/outliers in the data (e.g., Lin and Wang2002, 2004; Wang et al. 2005; Shilton and Lai 2007; Heoand Gader 2009). We adopt a method similar to Linand Wang (2002) to assign each labeled profile with afuzzy membership ui

j (0 <uij ≤ 1) as follows:

uij =

1 −�xi

j − x̄Ij+�

r Ij+ +�if yi

j = 11

1 −�xi

j − x̄Ij−�

r Ij− +�if yi

j = −11

(3)

where r Ij± = maxi∈Nj±4�xi

j − x̄Ij±�5 represents the radius

of each class (with the two classes being considerversus not consider), x̄I

j± = 41/N Ij±5∑

i∈Nj±xij denotes

each class’s group center, N Ij+ = 8i � yi

j = 19 and N Ij− =

8i � yij = −19 indicates the number of profiles in each

class upon labeling of I profiles; � is a small positivenumber to ensure that all ui

j > 0.Because the respondent’s true part-worths are un-

known to researchers, given the two classes of labeledprofiles, we use the center of each class to approximateour current best guess about a profile that is representa-tive of its class. We then define the fuzzy membershipas a function of the Euclidean distance of each labeledprofile to its current class center. That is, given ourcurrent knowledge about the respondent’s productpreferences, we assess the possibility of response errorsby examining to what extent the labeled profile differsfrom other labeled profiles in its class.

Therefore, depending on its distance to the currentclass center, the labeled profile may be assigned a 90%probability belonging to one class and a 10% probabilityof being meaningless, or a 20% probability belonging toone class and an 80% probability of being meaning-less. In Equation (2), profiles with higher probabilitiesof being meaningless (i.e., profiles with smaller ui

j

estimates) are given less weight in the fuzzy SVMestimation algorithm.

We repeat the process outlined in Equations (2) and (3)iteratively upon the labeling of each additional profile.Specifically, each time an additional profile is labeled,we assign it a fuzzy membership given the class centerof prior labeled profiles in its class, and based on whichupdated part-worths is estimated. Next, we updateour class center estimates and the ui

j 4i = 1121 0 0 0 1 I5estimate for each labeled profile to date. As a result, aswe gain additional information about respondent j’sproduct preferences, the algorithm can be used toiteratively refine the part-worths estimate and thefuzzy membership estimates. Because we use theintermediate part-worths estimate to adaptively selecteach subsequent question, the proposed fuzzy SVMmethod can be used not only to gauge possible responseerrors in the data but also to reduce the effects of suchnoises in the adaptive question design.

We have also used synthetic data experiments toexplore several alternative approaches to defining aprofile’s class membership probability, such as defininga profile’s class membership as a function of its dis-tances to both its own class center and the center ofthe opposite class, or imposing an underlying errordistribution assumption similar to the Logit, Probit orthe Gaussian Mixture models. To the extent that theestimation is feasible, we do not observe improvementin model performance by implementing such alterna-tive weighing schemes (details are provided in WebAppendix C).

It is worth noting that the class of fuzzy SVM meth-ods discussed above faces potential gains and losses inadaptive question selection. On the positive side, thismethod alleviates the negative effect of response errorsif such errors exist (see more investigation on thismatter in §5.3). On the negative side, if the respondentdoes not incur an error, our fuzzy membership esti-mates may render the estimation less efficient. Giventhat the slack variable �i

j in Equation (2) equals 0 for allnonsupport vectors in the solution, the efficiency lossonly occurs when the solutions to the optimization inEquation (2) are affected by the ui

j estimates associatedwith correctly classified support vectors. Synthetic dataexperiments reveal that, when used for data with noresponse errors, the fuzzy SVM active learning incursa minor efficiency loss compared to the soft marginSVM active learning (§5.3). Therefore, if researchersare uncertain about the degree of response errors inadaptive question design, the fuzzy SVM method de-scribed above may be used to alleviate the negativeeffect of possible response errors at the expense of apotentially minor efficiency loss. On the other hand,the soft margin SVM can be used to maximize theefficiency in active learning if prior experiences indicatethat consumer’s responses are highly deterministic(i.e., response errors play a negligible role).

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex Products452 Marketing Science 35(3), pp. 445–464, © 2016 INFORMS

3.4.2. Algorithm to Adaptively Select Profiles toBe Queried in Subsequent Question. In this sectionwe discuss how we adaptively select the next set ofprofiles shown to the respondent based on the latestestimate of his individual-specific part-worths. We usean approach adapted from Tong and Koller (2001). Ourprimary goal is to query each subsequent profile insuch a way that we can reduce the feasible region ofthe part-worths as rapidly as possible. Intuitively, onegood way of achieving this goal is to choose a querythat halves such a region.

Let w̃Ij = 4wI

j 1�Ij 5 denote the part-worths vector

obtained from the optimization in Equation (2) afterI profiles are labeled (w̃0

j in Equation (1) is used ifno profiles have been labeled). In its simplest form,Tong and Koller (2001) suggest that the next profile tobe queried can be the one with the smallest distance(margin) to the current hyperplane estimate representedby w̃I

j . In our context, the margin of an unlabeledprofile is computed as m

gj = �wI

j · xgj −�I

j �, with g beingthe index of unlabeled profiles. Tong and Koller (2001)show that, when the training data are symmetricallydistributed and the feasible region of part-worths isnearly sphere shaped, active learning via this simplemargin approach reduces the current version spaceby half.

In the context of adaptive question design, the train-ing data are likely to be asymmetrically distributedand/or the feasible region of the current part-worths canbe elongated. To overcome such restrictions in the sim-ple margin approach, Tong and Koller (2001) proposethe ratio margin approach as an augmentation. Whileconceptually appealing, the ratio margin approach isconsiderably more computationally burdensome thanthe simple margin approach. Here, we use the followinghybrid method to combine the two approaches.

Assume that, with the simple margin approach, wehave narrowed down to S trial profiles that are closestto the current hyperplane estimate represented by w̃I

j .We then take each trial profile s (s = 1121 0 0 0 1 S) fromthis set, give it a hypothetical label of 1, calculate anew part-worths by combining this new trial profilewith the labeled profiles, and obtain a hypotheticalmargin m′s+

j . Next, we perform a similar calculationby relabeling this profile as −1, calculating the result-ing part-worths vector, and obtaining a hypotheticalmargin m′s−

j . The ratio margin of this profile is definedas max4m′s+

j /m′s−j 1m′s−

j /m′s+j 5. After repeating these for

all of the S trial profiles, we pick the profile with thesmallest ratio margin as the next profile shown to theconsumer (i.e., mins=110001S6max4m′s+

j /m′s−j 1m′s−

j /m′s+j 575.

By asking the consumer to reveal his preference forsuch a profile iteratively, we can rapidly reduce thecurrent feasible region of the part-worths (Tong andKoller 2001).

In our empirical application, because more thanone profile is shown to the respondent at one time(Figure B3 in Web Appendix B), the set of profiles (e.g.,five) with the smallest ratio margins under the mostrecent part-worths estimate is selected to query therespondent. Additionally, upon satisfying the smallestratio margin criterion, if two or more profiles havethe same ratio margins, the profiles with the shortestoverall distances to both class centers will be chosen.We impose this modification to the original approachby Tong and Koller (2001) so that, in the context of ourfuzzy SVM estimation, if such profiles turn out to becorrectly labeled support vectors, the efficiency losswill be minimized.

This process is repeated iteratively until Q1 questionsare asked. As discussed in Dzyabura and Hauser (2011),the number of questions to be included may be chosenbased on prior experience or managerial judgment. Inour context, we conducted synthetic data experimentswith identical product dimensions as the two studies inour empirical investigation to determine the number ofquestions to be asked. We discovered that the proposedmethod could correctly classify the majority of profilesin various contexts after eight screens of profiles arequeried, with each screen composed of five profiles.Consequently, we adopted this stopping rule for thedata collection in our empirical application. Similarapproaches were used to determine the number ofquestions to be asked in the choice stage.

3.5. Choice-Based Fuzzy SVM Active LearningUpon completion of the consideration stage, we usethe latest part-worths estimate to compute the utilitiesof all candidate profiles for respondent j , from whicha set of Mj profiles is selected to be considered inthe choice stage. Similar in spirit to the “uncertaintysampling” rule adopted in Dzyabura and Hauser (2011),the profiles are selected such that their utility estimatesare the closest to the decision boundary determined bythe most recent part-worths estimate.

In choice tasks, when an individual frequently optsfor the no-choice option, we cannot efficiently learnabout his favorability towards various product features.In contrast, we obtain substantially more informationabout how an individual makes trade-offs amongdifferent product features when he chooses one profileover the competing profile(s) in a choice set. Therefore,in addition to selecting profiles whose utility estimatesare closest to the baseline utility estimate (i.e., theno-choice option), we use a selection rule in whichthe majority (e.g., 90%) of profiles in Mj have utilityestimates above the threshold defined by the “no-choice” option. The remaining profiles in Mj haveutility estimates less than the threshold to allow forpotential estimation error from our consideration stage.

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex ProductsMarketing Science 35(3), pp. 445–464, © 2016 INFORMS 453

3.5.1. Algorithm to Estimate Individual-SpecificPart-Worths on the Fly. For simplicity, we illustrate ourapproach using the example of a choice question withtwo product profiles and a “no-choice” option. Thegeneral principles are applicable to choice questionsconsisting of more than two profiles. Let us denote thetwo profiles in the kth choice question as xkA

j and xkBj .

After a total of T responses from the respondent (includ-ing both at the consideration and the choice stages),depending on respondent j’s choice among profile A,profile B, none of the two, we obtain the followinginformation correspondingly:

Chose A: wTj ·4xkA

j −xkBj 5≥0 and wT

j ·xkAj ≥�T

j 1

Chose B: wTj ·4xkA

j −xkBj 5<0 and wT

j ·xkBj ≥�T

j 1

Chose None: wTj ·xkA

j <�Tj and wT

j ·xkBj <�T

j 0

(4)

As shown in Equation (4), we obtain two data pointseach time the respondent makes a choice. For all in-equalities containing �T

j , the fuzzy membership ofthe labeled response can be obtained directly usingEquation (3), with class center and radius estimatescalculated from pooled responses from both the consid-eration and the choice stages. When the inequalities inEquation (4) entail utility comparison between the twoprofiles, we denote xkAB

j = xkAj − xkB

j and rewrite suchinequalities as wT

j · xkABj ≥ or < 0. Next, we assign a

fuzzy class membership to each data point obtained,with xkAB

j replacing xij in Equation (3). Under such

scenarios, the class center of each class captures themean differences between the two profiles when oneprofile is favored over the other. Conceptually, if theposition of xkAB

j considerably deviates from its classcenter, it implies that the labeled response does notalign with our current knowledge about the respon-dent’s product preferences. Therefore, we assign a lowclass membership to such a response. Formally, theoptimization we solve at this stage of the adaptivequestion design can be expressed as

minwT

j 1�Tj 1 �

ij 1 �

kABj

[

12�wT

j � +C

( I ′∑

i=1

uij�

ij +

K′

∑

k=1

ukj �

kABj

)]

s.t. yij 4w

Tj · xi

j −�Tj 5≥ 1 − �i

j 1

ykABj 4wT

j · xkABj 5≥ 1 − �kAB

j 1

�ij 1 �

kABj ≥ 01

(5)

with the first constraint denoting all labeled responsesrelated to �T

j (hence including responses from both con-sideration and choice stages) and the second constraintcontaining all responses related to utility compari-son between the two profiles. It is evident that thisoptimization remains convex.

Similar to §3.4.1, we update our fuzzy membershipestimates for all prior labeled responses (includingthose obtained in the consideration stage) each timeafter the respondent makes a choice among the twoproduct profiles and the no-choice option. Consequently,our fuzzy membership estimates are refined over timeas we accumulate additional knowledge about the focalrespondent’s product preferences.

3.5.2. Algorithm to Adaptively Select Profiles inthe Next Choice Question. Below we discuss howwe identify the next set of profile pair (i.e., profile g1and profile g25 to be shown to the respondent. In thechoice stage,mg11g2

j =�wTj ·4xg1 −xg25�,m

g1j =�wT

j ·xg1 −�Tj �,

and mg2j =�wT

j ·xg2 −�Tj � capture our most recent esti-

mates of respondent j’s relative preferences acrossthe two profiles and the no-choice option. Basedon these utility estimates, we first use the crite-rion min8max4mg11g2

j 1mg1j 1m

g2j 59 for all 84g11g25 ∈Mj9

to select a trial set of choice sets that we would con-sider for the next choice question. The utilities of alloptions in this choice set are a priori near equal. Thiscorresponds to the utility balance criterion commonlyadopted in the conjoint literature. As suggested byvarious prior studies (e.g., Huber and Hansen 1986,Haaijer et al. 2001, Toubia et al. 2004), utility balancedquestions are the most informative in further refiningthe part-worths estimates. This criterion is also consis-tent with the simple margin approach proposed byTong and Koller (2001), which aims to cut the feasibleregion of part-worths approximately in half.

Given that labeled responses are likely to be asym-metrically distributed and/or the feasible region ofpart-worths may be elongated, we use a hybrid ofsimple margin and ratio margin approaches similar tothat described in §3.4.2. After using a simple marginapproach to obtain a trial set of choice sets, we firstgive this choice set a hypothetical response of choosingprofile g1, calculating a new part-worths by combiningthis response with the prior obtained responses, andobtaining hypothetical margins m′g11g2+

j and m′g1+

j . Wethen give this choice set of a hypothetical response ofchoosing profile g2 and obtain hypothetical marginsof m′g11g2−

j and m′g2+

j . A similar approach is used toobtain m′g1−

j and m′g2−

j when the choice set is given ahypothetical response of no-choice. Then the pair ofprofiles with the smallest ratio margin

(

i.e., mins=110001S

[

max(

max(

m′g11g2+

j

m′g11g2−

j

1m′g11g2−

j

m′g11g2+

j

)

1

max(

m′g1+

j

m′g1−

j

1m′g1−

j

m′g1+

j

)

1max(

m′g2+

j

m′g2−

j

1m′g2−

j

m′g2+

j

))])

(6)

is selected as the next choice set shown to the consumer.Similar to §3.4.2, conditional on satisfying the smallest

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex Products454 Marketing Science 35(3), pp. 445–464, © 2016 INFORMS

ratio margin criterion, if two or more choice sets havethe same ratio margins, the choice set with the shortestoverall distance to the corresponding class centerswill be chosen to minimize efficiency loss. We repeatthis process iteratively to shrink the feasible region ofpart-worths as rapidly as possible, until Q2 questionsare asked.

4. Empirical InvestigationIn this section we describe two empirical studies involv-ing digital cameras and computer tablets. Our pretestindicates that both product categories are of interest tothe respondents’ population (undergraduate students).The overall complexity of the digital camera category(with 30+ attribute levels) is parallel to product cate-gories studied in prior research that elicits consumers’preferences for complex products (e.g., Park et al. 2008,Netzer and Srinivasan 2011, Scholz et al. 2010). Thecomputer tablet category (with 70+ attribute levels) isconsiderably more complex than those used in extantmethods, particularly in the context of decompositionalpreference elicitation methods. To keep our empiricalapplications meaningful and realistic, we conductedpretests to choose a set of attributes that the respon-dents typically consider. We then used retail websitessuch as BestBuy.com and Amazon.com to identify theranges and values of attribute levels used in bothempirical applications.

4.1. Digital Camera Study

4.1.1. Research Design. A total of 425 participantsare randomly assigned to one of the six preferencemeasurement conditions. We included two conditionsof the fuzzy SVM method: Condition 1: fuzzy SVMwith collaborative filtering; Condition 2: fuzzy SVMwithout collaborative filtering. The overall flow of thetwo fuzzy SVM conditions follows Figure 1, with theexception that in Condition 2 the initial part-worths isattained solely based on the focal respondent’s self-configured product profile. We further compare thepredictive validity of the proposed method with thefollowing four benchmark methods: Condition 3: theself-explicated method; Condition 4: an upgradingmethod similar to Park et al. (2008) with no incentivealignment; Condition 5: the adaptive Choice-BasedConjoint (ACBC); and Condition 6: the traditionalChoice-Based Conjoint (CBC). The list of attributes andattribute levels included in our digital camera study isprovided in Table A1 in Web Appendix D.

Similar to prior studies in the literature (e.g., Scholzet al. 2010, Netzer and Srinivasan 2011), participants inall conditions first complete the preference measure-ment task, followed by an external validation task anda post-survey feedback task. Identical across the sixexperimental conditions, the external validation task

comprises two choice questions, each including twocamera profiles. Generated using fractional factorialdesign, the profiles are carefully chosen so that oneprofile does not clearly dominate the other in eachchoice set.

4.1.2. Results. We obtain the individual-specificpart-worths estimates from the two fuzzy SVM condi-tions using the fuzzy SVM estimation algorithm. Theself-explicated estimates are obtained by multiplyingthe attribute importance weights with the correspond-ing desirability ratings (Srinivasan 1988). We use thehierarchical Bayesian estimation to obtain individual-level part-worths estimates from the upgrading method,the ACBC method, and the CBC method.

Following the tradition in this literature (e.g.,Evgeniou et al. 2005, Netzer and Srinivasan 2011, Parket al. 2008), we use the hit rate of the external validitytasks to gauge the predictive validity of the six prefer-ence measurement methods (Table 1).6 We find that thetwo fuzzy SVM conditions perform significantly betterthan all benchmark methods in correctly predictingrespondents’ choices in hold-out tasks (p < 0005).

Comparing hit rates across the two fuzzy SVMconditions, we find that collaborative filtering improvedpredictions but not significantly (0.801 versus 0.779,p > 0005). This finding is consistent with empiricalresults from Dzyabura and Hauser (2011). It is possiblethat, after all of the consideration and choice questionsare queried in our adaptive survey, incremental benefitsfrom the collaborative filtered initial part-worths havediminished. This matter is explored further in syntheticdata experiments in §5.4.

We also compared participants’ responses to the postsurvey feedback questions across the six experimentalconditions. Overall, participants provided more favor-able feedback to the format and questions generatedunder the fuzzy SVM conditions than those from thebenchmark methods (Table A2 in Web Appendix D).7

6 We also tried to incorporate the Kullback-Leibler (KL) measureused by Dzyabura and Hauser (2011), Ding et al. (2011), and Hauseret al. (2014) in our digital camera application. We discovered thatthis measure is only applicable with three or more validation tasks.Our digital camera application includes two choice validation tasks.In such cases, when the consumer chooses the profile on the leftin one task and the profile on the right in the other task, the KLmeasure equals zero regardless of whether both predictions arecorrect, wrong or one is correct and one is wrong. Essentially, the KLmeasure does not discriminate whether observed and predictedchoices are aligned when the total number of validation tasks is two.In our computer tablet study, we included six choice validation tasksand used the KL measure to gauge predictive validity.7 In both empirical applications, we also recorded the amount of timeit takes for each participant to complete the survey. This informationis provided in Tables A3 and A4 in Web Appendix D.

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex ProductsMarketing Science 35(3), pp. 445–464, © 2016 INFORMS 455

Table 1 Comparison of Predictive Validity: Digital Camera Study

Hit rate

Method (sample size) Avg. SE

Condition 1: Proposed method (N = 73) 00801a 0.030Condition 2: Fuzzy SVM without collaborative 00779a 0.032

filtering (N = 69)Condition 3: Self-explicated (N = 66) 00689 0.041Condition 4: Upgrading (N = 80) 00563 0.038Condition 5: ACBC (N = 75) 00660 0.038Condition 6: CBC (N = 62) 00605 0.045

aBest in column or not significantly different from best in column at the0.05 level.

4.2. Computer Tablet Study

4.2.1. Research Design. In a second empirical study,we examine the performance of the proposed methodfor the more complex product category of computertablets (with 70+ attribute levels). The complete list ofattributes and attribute levels included in this studyis provided in Figure A1 in Web Appendix B. Inaddition to the different product category and theincreased number of attribute levels, we made thefollowing modifications in this empirical application.First, given the high complexity of the product category,we included a warm-up task so that the participantscould get familiar with the different attribute levelsbefore the preference measurement task. This task hasbeen shown to improve the accuracy of preference elici-tation (Huber et al. 1993). Specifically, we provided thelist of 14 attributes used to describe computer tablets,followed by a brief verbal and graphic descriptionfor each attribute level, displayed for one attribute ata time.

Second, because a number of prior studies (e.g., Ding2007; Ding et al. 2005, 2009, 2011; Dong et al. 2010;Hauser et al. 2014) suggest that incentive alignmentoffers benefits such as greater respondent involvement,less boredom, and higher data quality, we incorporatedincentive alignment in this application. At the begin-ning of the experiment, we told the participants that wewould award a computer tablet device to one randomlyselected participant from this study, plus cash repre-senting the difference between the price of the tabletdevice and $900. We set $900 as the maximum prizevalue because the majority of computer tablets cost lessthan $900 at the time of our study. The participantswere told that the total number of participants for thisstudy would be approximately 150 (i.e., the chance ofwinning is about 1 in 150). Because we wanted thepreference elicitation tasks and the validation tasks tobe incentive aligned, participants were told that wewould randomly decide which of the two tasks to usewhen determining the final prize. We also told eachparticipant that, if chosen as a winner, he would receivea computer tablet based on: (1) his choice from one of

the validation questions or (2) his most preferred tabletamong a list of 25 tablets, inferred from his answers tothe preference elicitation questions. Following Dinget al. (2011) and Hauser et al. (2014), participants weretold that this list was pre-determined by the researchersand that it would be made public after the study.Therefore, the respondents have incentives to answerthe questions carefully and truthfully.

Last, we modified our validation procedure relativeto the digital camera study. We included initial anddelayed validation tasks as in Ding et al. (2011) andHauser et al. (2014). Following Hauser et al. (2014),the delayed validation questions were sent to the re-spondents by email one week after the preferencemeasurement task. Additionally, we included both pre-and post-preference measurement validation questionsas suggested by Netzer and Srinivasan (2011). Eachrespondent answered six validation choice questions,with three before the preference measurement task andthree in the delayed validation task. As pointed out byNetzer and Srinivasan (2011), the standard validationtask procedure used in our digital camera study mightbe susceptible to idiosyncrasies of the chosen validationquestions. In the computer tablet study, we followedNetzer and Srinivasan (2011) by including a broaderset of validation choice questions in the validationtask. First, we used a fractional factorial design togenerate a set of orthogonal balanced choice questions.Next, we scanned through the generated questions andretained only those comprising profiles available in themarketplace (as any one of the computer tablets in thevalidation questions could be awarded to a participant).To allow for appropriate comparison across conditions,we also followed Netzer and Srinivasan (2011) by usingthe same sets of randomly drawn validation questionsin all conditions.

In this study, 151 participants are randomly assignedto one of the four preference measurement conditions.To better understand the incremental benefit fromincluding consideration questions in our adaptivequestion design, we compared the proposed method(Condition 1) to an alternative fuzzy SVM method inwhich the choice questions are presented immediatelyafter the self-configuration task and the unacceptableand must-have questions (Condition 2). Furthermore,we included the self-explicated method (Condition 3)and the ACBC method (Condition 4) to replicate resultsfrom the first empirical application.

The flow of each experimental condition is describedbelow. First, the participant was introduced to thestudy along with a basic description of the incentivealignment mechanism. Second, the participant waspresented with the warm-up task. Third, three valida-tion questions, each consisting of two tablet profiles,are shown to the participant. Fourth, the participantwas presented with the preference measurement task

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex Products456 Marketing Science 35(3), pp. 445–464, © 2016 INFORMS

Table 2 Comparison of Predictive Validity: Computer Tablet Study

KL divergence

Initial validation Delayed validation Pooled Hit rate (pooled)

Method N Avg. SE N Ave. SE Ave. SE Ave. SE

Condition 1: Proposed method 35 00288a1b 0.062 26 00260a1b 0.073 00471a1b 0.054 00671a 0.041Condition 2: Fuzzy SVM without 36 00426 0.066 26 00407 0.077 00577 0.052 00565 0.049

consideration questionsCondition 3: Self-explicated 39 00443 0.058 27 00479 0.080 00662 0.043 00585 0.031Condition 4: ACBC 41 00472 0.059 30 00531 0.062 00652 0.048 00520 0.033

aBest in column or not significantly different from best in column at the 0.05 level.bMarginally better than the fuzzy SVM without consideration questions condition (p < 001).

(which varies by experimental condition). Last, a weeklater, each participant received a follow-up email with asurvey link to a second set of three validation questions.

4.2.2. Results. Table 2 reports the predictive valid-ity of the four experimental conditions. In additionto the traditional hit rate measure, we used the KLdivergence measure to gauge the degree of divergencefrom predicted choices to those that are observed inthe validation data. The KL divergence measure isan information-theory-based measure of divergence(Kullback and Leibler 1951, Chaloner and Verdinelli1995). Dzyabura and Hauser (2011), Ding et al. (2011),and Hauser et al. (2014) demonstrate that, for consider-ation data, the KL divergence measure provides anevaluation of predictive ability that is rigorous andwhich discriminates well. We followed the formulaeprovided in Dzyabura and Hauser (2011) to calculatethe KL divergence measures. In our context, we concep-tualize a false-positive prediction as the case whereinthe respondent is predicted to choose a profile butdid not actually choose it in a validation question;and a false-negative prediction as the case whereinthe respondent is predicted not to choose a profile butactually chose it in a validation question. Because thismeasure evaluates divergence from perfect prediction,a smaller KL divergence measure indicates a bettermodel prediction.

Consistent with findings from our digital cameraapplication, the proposed method exhibits superiorpredictive ability when compared to the self-explicatedand ACBC methods in both the initial and delayedvalidation tasks. We also find that the proposed methodhas smaller KL divergence than the fuzzy SVM con-dition without consideration questions (marginallysignificant at p < 001). It is evident that the inclusionof consideration questions is helpful in improvingpreference elicitation in our context. It is possible that,when consideration questions are absent, respondentsmay encounter greater cognitive difficulty in makingaccurate trade-offs of choice questions. Respondentsmay also be less likely to incur early response errors

(which are known to be detrimental in adaptive ques-tion design), when they are presented with the lessdemanding consideration questions before the morecognitively challenging choice tasks.

We also compared the KL divergence measure fromthe initial validation task with that from the delayedvalidation task within each experimental condition. Nosignificant differences are observed. Therefore, whenpooling results across the initial and delayed validationtasks for each respondent, the KL divergence measuresfollow the same pattern as the one discussed above.8 Thehit rate comparisons are also provided in Table 2. Theproposed method also outperforms the three benchmarkmethods in terms of hit rate.

One of the main advantages of adaptive questiondesign is the opportunity to reduce respondents’ cog-nitive burden by asking fewer questions. We furtherexamine the out-of-sample performance of the pro-posed method when only the first q questions are usedfor each respondent (Table 3). We find that both theKL divergence and the hit rate measures graduallyimprove with the inclusion of self-configurator (q = 1),consideration questions (q = 2 to 9 with 8 screens ofconsideration questions with 5 profiles on each screen),and choice questions (q = 10 to 34), indicating that all ofthese questions positively contribute to our preferenceelicitation task. Table 3 also reveals that the proposedmethod performs well with much fewer choice ques-tions. In particular, the predictive validity after only 6–8choice questions (i.e., q = 15 or 17) is already similar tothe predictive validity with all 25 choice questions (i.e.,q = 34). Indeed, after only the first 8 choice questions(i.e., q = 17), the proposed method already exhibitssignificantly better predictive validity than the self-explicated and ACBC methods. This finding suggeststhat respondent burden in this application may be sub-stantially reduced with a much shorter survey. This isconsistent with Netzer and Srinivasan (2011) who also

8 Note that the values of the KL measure depend on the number ofthe validation tasks (Hauser et al. 2014). Therefore, the pooled KLmeasures differ in magnitude from those based on initial or delayedvalidation tasks only.

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex ProductsMarketing Science 35(3), pp. 445–464, © 2016 INFORMS 457

Table 3 Predictive Validity by Number of Questions Asked: Computer Tablet Study

No. of questions (q) KL-divergence (smaller is better) Hit rate (larger is better)

1 0.557 0.538 ← Self-configurator

3 0.600 0.5865 0.602 0.6197 0.611 0.6489 0.522 0.614

11 0.529 0.63813 0.504 0.64815 0.484 0.63817 0.494 0.67119 0.552 0.65721 0.498 0.66723 0.496 0.67625 0.500 0.66727 0.502 0.66229 0.502 0.67631 0.480 0.68133 0.480 0.65234 0.471 0.671

Consideration questions

Choice questions

report negligible improvement in predictive validityafter 5–7 adaptive paired comparison questions.

Overall, in comparing the predictive ability of theproposed method with that of the ACBC and self-explicated methods, our computer tablet applicationreplicated results from the digital camera application.The comparison between the two fuzzy SVM conditionsalso shows that inclusion of consideration questionsis useful in facilitating preference elicitation in ourcontext. In addition, this application illustrates that theproposed method scales well when the focal productcategory is considerably more complex than those usedin prior studies.

5. Synthetic Data ExperimentsIn this section we describe a series of synthetic dataexperiments conducted to complement our empiricalinvestigation. In §5.1, we compare the performanceof the proposed question selection algorithm withthat of Dzyabura and Hauser (2011) for considerationquestions when the true consideration decisions areconjunctive. In §5.2, we examine the performance ofthe proposed method with that of benchmark methodswhen the true consideration and choice decisions arebased on a part-worths model. Under this comparison,we use the question selection algorithm in Dzyaburaand Hauser (2011) as the benchmark question selec-tion method for consideration questions and that inAbernethy et al. (2008) as the benchmark method forchoice questions. To investigate the applicability ofthese question selection methods to high-dimensionalproblems, in §§5.1 and 5.2, we examine the scalabilityof each algorithm when the focal product category isequipped with up to 100+ attribute levels. To ensure

fair comparisons across methods, only the focal respon-dent’s responses are used in the adaptive questionselection in all comparisons described above. We alsotest the upper bound of parameter recovery and pre-dictive validity by assuming no response errors in suchcomparisons. In §5.3, we compare the performances ofthe fuzzy SVM versus soft margin SVM active learningwith and without response errors. In §5.4, we exam-ine the improvements of collaborative-filtered initialpart-worths over noninformative initial part-worths onparameter recovery and predictive validity. We reportour main findings below. Additional implementationdetails can be found in Web Appendix E.

5.1. Using Proposed vs. Conjunctive QuestionSelection When the True ConsiderationDecisions Are Conjunctive

In §3 we describe a framework wherein a set of part-worths is used to characterize consumer’s responses toconsideration and choice questions. In the literature,conjunctive-like criteria are often used to examineanswers to consideration questions (e.g., Bettman 1970,Hauser et al. 2010). Therefore, we conduct syntheticdata experiments to examine the performance of theproposed question selection algorithm for considera-tion questions when the true consideration model isconjunctive. Specifically, the adaptive question selectionalgorithm in Dzyabura and Hauser (2011) is used asthe benchmark method in this comparison.

Given our emphasis on complex products, we exam-ine three scenarios in which the focal product categorycomprises 15, 25, and 35 attributes with three levelseach. Under each scenario, we simulate 300 syntheticrespondents (i.e., 300 conjunctive decision rules) asdescribed in Dzyabura and Hauser (2011). We then

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex Products458 Marketing Science 35(3), pp. 445–464, © 2016 INFORMS

Table 4 Using Proposed vs. Conjunctive Question Selection When the True Consideration Decisions Are Conjunctive: Synthetic Data Experiments

(a) Parameter recovery and predictive validity comparisons

Question selection method Question selection method

Proposed method Dzyabura and Hauser (2011) Proposed method Dzyabura and Hauser (2011)

Product dimension Performance measure Conjunctive estimation Fuzzy SVM estimation

3 × 15 Hit rate 0.986 10000a 00966a 0.919KL divergence 0.072 00000a 00091a 0.264

U2 0.928 10000a 00728a 0.5153 × 25 Hit rate 0.948 00992a 00948a 0.910

KL divergence 0.252 00013a 00117a 0.246U2 0.818 00850 00693a 0.588

3 × 35 Hit rate 0.945 00994a 00936a 0.892KL divergence 0.232 00018a 00131a 0.267

U2 0.685 00743a 00614a 0.577

(b) Average time to generate next question comparisons (in seconds)

Question selection method

Product dimension Proposed method Dzyabura and Hauser (2011)

3 × 15 00218a 208563 × 25 00214a 609203 × 35 00215a 110861

aSignificantly better than the alternative method at the 0.05 level.

perform active learning for each participant where40 consideration questions are adaptively selected,with each synthetic respondent labeling the profileas “would consider” or “would not consider” basedon the underlying true conjunctive decision rule. Tocompare the relative performance of the two questionselection methods, we generate 3,000 validation profilesfor each synthetic respondent, with the respondentconsidering 1,500 profiles and not considering theremaining profiles under the respondent’s true under-lying conjunctive decision rule. Because each syntheticrespondent considers 50% of the validation profiles, thenull model that predicts “randomly considers profiles”would achieve a hit rate of 50%. Therefore, while the hitrate measure can be misleading for consideration dataempirically, it provides a valid measure of predictivevalidity in our synthetic setting.

Table 4 provides comparison results. To comparethe question selection methods, we keep the estima-tion method constant. Specifically, we present thecomparison results using both the conjunctive estima-tion method as in Dzyabura and Hauser (2011) andthe fuzzy SVM estimation method proposed in thispaper. In terms of performance metrics, we use hitrate, KL divergence, and U 2 to measure predictivevalidity and parameter recovery (Table 4(a)). The metricU 2 is an information-theoretic measure of parameterrecovery (Hauser 1978). It measures the percentage ofuncertainties explained by the model; with U 2 = 100%

indicating perfect parameter recovery.9 To examinescalability of these two question selection methods,we also report the average time it takes to generatethe next question in seconds (Table 4(b)). The reportedcomputing times are all based on Matlab code run onan Intel 3.2 GHz personal computer with a Windows 7Operating System.

When the estimation method in Dzyabura andHauser (2011) is used (i.e., for each attribute level, themodel estimates a probability for which the respondentfinds it acceptable), the question selection algorithmby Dzyabura and Hauser (2011) exhibits a superiorhit rate, KL divergence, and U 2 to the focal questionselection algorithm in almost all problem instances.

9 While the original U 2 measure in Hauser (1978) is based on choiceprobabilities, the fuzzy SVM estimation gives rise to dichotomous(e.g., consider versus not consider; choose product A versus prod-uct B) rather than probabilistic predictions. Therefore, when thefuzzy SVM algorithm is used for model estimation, we calculated theU 2 measure based on a logit transformation, with the deterministiccomponent of the product utility calculated from the estimatedpart-worths given by the fuzzy SVM algorithm. Because the scale ofutility estimates matters in the magnitude of U 2 (if we multiply thepart-worths estimates by a constant, larger part-worths result inmore extreme choice probabilities, hence more extreme U 2 estimate),we normalize the estimated part-worths to the scale of the truepart-worths and use the relative U 2 (the U 2 calculated from thefuzzy SVM part-worths estimates divided by the U 2 calculated fromthe true part-worths) to remove the effect of scaling. This measurewas not used in our empirical studies because the true part-worthsare unknown empirically and it is ambiguous as to how to determinethe baseline scale in our empirical comparisons.

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex ProductsMarketing Science 35(3), pp. 445–464, © 2016 INFORMS 459

Indeed, when the product dimension is relatively mod-erate (15 attributes and 3 levels), Dzyabura and Hauser(2011)’s method yields perfect parameter recovery andholdout prediction after 40 questions. Such resultsreveal that, when the true consideration model is con-junctive, the algorithm proposed by Dzyabura andHauser (2011) works exceptionally well in estimatingthe probability that the respondent finds each attributelevel acceptable.

Because our approach aims to obtain a set of part-worths estimates for each respondent, we also comparethe performance of the two question selection methodswhen the fuzzy SVM method is used for estimation. Insuch cases, an individual-specific part-worths vectoris estimated after all 40 questions are queried undereach question selection algorithm. The estimated part-worths are then used to predict the validation profiles.Interestingly, we discover that, when the fuzzy SVMestimation is used, the proposed method outperformsthe method used by Dzyabura and Hauser (2011) interms of hit rate, KL divergence, and U 2 measures.Such results are likely to be driven by the fact that theDzyabura and Hauser (2011) algorithm is specificallydeveloped to uncover the probability for which therespondent considers each attribute level, rather thana utility estimate associated with the attribute level.Therefore, when the primary focus is to estimate part-worths, this method does not perform as well, evenwhen the true consideration model is conjunctive.

We further study scalability of the two methods byexamining the average time it takes to generate the nextquestion under each question selection algorithm. Whenthe proposed question selection algorithm is used, onaverage it takes less than 0.25 seconds to generate thenext question in all scenarios. By contrast, under thequestion selection algorithm by Dzyabura and Hauser(2011), the average time it takes to generate the nextquestion is considerably longer (ranging from 2.856seconds to 11.861 seconds in the three scenarios). Notethat such discrepancies may diminish considerably ifwe were to optimize or code both algorithms in a morecomputationally efficient language such as C or C++.

Overall, our synthetic data experiments reveal thefollowing. First, even when the true considerationmodel is conjunctive, the part-worths estimate fromthe proposed method exhibit a reasonably good abilityto predict whether the respondent would consider aprofile. Given that the part-worths model of consider-ation and the conjunctive model of consideration asin Dzyabura and Hauser (2011) aim to capture a setof linear decision rules in the consideration process,we believe that such findings are quite reasonable.Second, the proposed framework is not restricted to thequestion selection algorithm discussed in §3.4. Indeed,the algorithm by Dzyabura and Hauser (2011) could beused in conjunction with the proposed algorithm in

a consider-then-choice framework to uncover consid-eration heuristics and conjoint part-worths. In suchcases, different utility functions may also be used toseparately model consumer’s consideration and choicedecisions.

5.2. Performance Comparisons When TrueConsideration and Choice Decisions AreBased on a Part-Worths Model

We now compare the performance of the proposedmethod with that of benchmark methods when thetrue consideration and choice decisions are based ona part-worths model. For simplicity, we examine thecase wherein the same utility function is used forconsideration and choice decisions. If different part-worths models were specified for consideration andchoice questions, we expect that our key findings wouldnot change qualitatively.

In each scenario under study, we perform activelearning wherein 40 consideration questions are adap-tively designed for each respondent, followed by 25adaptive choice questions with 2 alternatives each.In the first benchmark condition, the question selectionalgorithm by Dzyabura and Hauser (2011) is used forthe adaptive design of consideration questions. In thesecond benchmark condition, the question selectionalgorithm by Abernethy et al. (2008) is used for theadaptive design of choice questions. Because the algo-rithm by Dzyabura and Hauser (2011) is designedfor consideration questions only and that by Aber-nethy et al. (2008) is for choice questions only, fuzzySVM active learning is used as the question selectionmethod for choice questions in the first benchmarkcondition and for consideration questions in the secondbenchmark condition to ensure fair comparison. Forsimilar reasons, the fuzzy SVM method is used as theestimation method after all questions are queried in allthree conditions.

As before, we examine three scenarios in whichthe focal product category comprises 15, 25, and 35attributes with three levels each. Three-hundred syn-thetic respondents are simulated in each scenario. Thepart-worths for each respondent are randomly gener-ated with (−�, 0, �) for each attribute, with �∼N41135.To compare performances across conditions, 3,000validation questions consisting of two profiles are gener-ated for each respondent, with the respondent choosingthe profile on the left 50% of the time and choosing theprofile on the right in the remaining questions. Underthis setup, the null model for the hit rate and U 2 is theone that predicts “randomly choose among the twoprofiles.” In addition to the hit rate, KL divergence,and U 2 measures, we use mean absolute error (MAE)and root mean square error (RMSE) to measure theability of each question selection method to recover thetrue part-worths. For comparability across methods,

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex Products460 Marketing Science 35(3), pp. 445–464, © 2016 INFORMS

Table 5 Performance Comparisons When True Consideration and Choice Decisions Are Both Based on a Part-Worths Model: Synthetic Data Experiments

(a) Parameter recovery and predictive validity comparisons

Product dimension Performance measure Proposed method Dzyabura and Hauser (2011) Abernethy et al. (2008)

3 × 15 Hit rate 00948a 00929 00937KL divergence 00279a 00365 00327

U2 00736a 00701 00701MAE 00745a 00858 00834

RMSE 00920a 10057 10037

3 × 25 Hit rate 00933a 00925 00870KL divergence 00343a 00373 00553

U2 00679a 00623 00596MAE 00839a 00877 10109

RMSE 10040a 10086 10374

3 × 35 Hit rate 00901a 00893 00812KL divergence 00460a 00480 00694

U2 00625a 00583 00487MAE 10015a 10042 10347

RMSE 10254a 10291 10668

(b) Average time to generate next question comparisons (in seconds)

Product dimension Question type Proposed method Dzyabura and Hauser (2011) Abernethy et al. (2008)

3 × 15 Consideration 00269a 5.779 —Choice 00612 — 00002a

3 × 25 Consideration 00206a 3.075 —Choice 00609 — 00002a

3 × 35 Consideration 00217a 4.702 —Choice 00690 — 00004a

aSignificantly better than the alternative method at the 0.05 level.

we normalize the estimated part-worths to the scale ofthe true part-worths in each problem instance.

Our comparison results are shown in Table 5. Whenthe true consideration and choice decisions are based ona part-worths model, the proposed question selectionmethod outperforms the two benchmark methods inparameter recovery and predictive validity (Table 5(a)).Not surprisingly, the question selection algorithm byDzyabura and Hauser (2011) does not work as well inthis setting, as the algorithm is specifically developedto uncover consideration heuristics when the trueconsideration decisions follow a set of conjunctiverules, rather than a part-worths model. Meanwhile,the lack of performance from the question selectionalgorithm by Abernethy et al. (2008), particularly inhigh dimensional problems, is likely related to thediscrete transformation required by its gradient-basedalgorithm when used for a large number of discreteattributes in our setting.10

10 While we obtain significantly better performance from the proposedmethod, the hit rate differences between the proposed method andDzyabura and Hauser (2011) are rather small in the case of 35attributes with three levels each (0.901 versus 0.893 as in Table 5(a)).We further examine the percentages of times that the estimatedpart-worths from the two methods could correctly predict themost preferred attribute level in each attribute. We do not find

We also report the average time it takes to generatethe next question under each question selection methodin Table 5(b). Because the algorithm by Dzyaburaand Hauser (2011) is used for consideration questionsonly and that by Abernethy et al. (2008) is used forchoice questions only, the corresponding average timeis reported in this table. For consideration questions,the algorithm by Dzyabura and Hauser (2011) takesapproximately three or more seconds on average togenerate the next question (again, the computing speedmay improve considerably if the code were optimizedor written in a more computationally efficient lan-guage). Regarding choice questions, the algorithm byAbernethy et al. (2008) is very fast computationallybecause the solution used to generate the next question

significant differences between the two methods in this case (both cancorrectly predict the most preferred attribute levels about 86% of thetime). We also compare the mean absolute percentage error (MAPE)between the predicted and actual part-worths from the two methods.Consistent with findings from the MAE and RMSE measures inTable 5, the proposed method provides more accurate part-worthsestimates than Dzyabura and Hauser (2011) in all cases. Managerially,if the primary focus of the firm is to identify the most preferredattribute level in each attribute, we think that such differences in hitrates do not produce additional insights. Nevertheless, if the firm’scentral goal is to obtain a precise forecast of market share or productprofit, we believe that the improved accuracy in our part-worthsestimates would be beneficial.

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex ProductsMarketing Science 35(3), pp. 445–464, © 2016 INFORMS 461

Table 6 Performance Comparisons Between Fuzzy SVM and Soft MarginSVM Active Learning: Synthetic Data Experiments

(a) Parameter recovery and predictive validity comparisonswith response errors

Error Performance Fuzzy Softpositions measure SVM margin SVM

Early Hit rate 00760a 0.720KL divergence 00769a 0.827

U2 00280a 0.201MAE 10694a 1.834

RMSE 20103a 2.275Middle Hit rate 00834a 0.819

KL divergence 00634a 0.669U2 00417a 0.394

MAE 10519a 1.596RMSE 10889a 1.978

Late Hit rate 00893 0.890KL divergence 00483 0.492

U2 00511 0.507MAE 10319 1.338

RMSE 10636 1.649

(b) Parameter recovery and predictive validity comparisonswith no response error

Performance measure Fuzzy SVM Soft margin SVM

Hit rate 0.901 00904b

KL divergence 0.460 00448U2 0.625 00648a

MAE 1.015 00992a

RMSE 1.254 10226a

aSignificantly better than the alternative method at the 0.05 level.bMarginally better than the alternative method at the 0.1 level.

can be derived in closed form. Therefore, when thefocal product consists of a large number of continu-ous attributes, this method is a good alternative foradaptive design of choice questions.

5.3. Performance Comparisons Between Fuzzy SVMand Soft Margin SVM Active Learning Withand Without Response Errors

A key advantage of our proposed method is its abilityto gauge response errors on the fly. In this section,we investigate use of the fuzzy SVM active learningversus that of the soft margin SVM active learningwithout fuzzy membership probabilities. Because bothmethods scale well in high dimensional problems, weexamine the case wherein the focal product categoryconsists of 35 attributes with three levels each. We usemethods similar to those used in §5.2 to simulate thetrue part-worths and the holdout profiles for the 300synthetic respondents used in each problem instance.

We first investigate performance comparisons of thetwo methods when there are response errors. Giventhat the positions of response errors play an integralrole in the performance of adaptive question design, weexamine the two question selection methods’ abilities to

recover true part-worths and to predict holdout profilesunder the scenarios of (1) early versus (2) middle versus(3) late response errors. To ensure fair comparisons,we set the instances of response errors at 15% acrossthe three scenarios. In the early/middle/late responseerrors scenario, all response errors occur during thefirst/middle/late 1/3 of adaptive questions. For eachsynthetic respondent, we use random draws from auniform distribution to determine the positions of theerrors. Within each scenario, we hold error positionsconstant across the two question selection methods sothat our results are comparable.

Our performance comparisons are reported inTable 6(a).11 When response errors take place duringearly or middle portions of the adaptive study, thefuzzy SVM active learning method exhibits a superiorability to recover the true part-worths and to predictholdout questions than the soft-margin SVM activelearning. Nevertheless, if response errors occur towardsthe end of the adaptive survey, both question selectionmethods perform similarly. This pattern is quite reason-able because early response errors are in general moredetrimental than errors taking place later in adaptivequestion design. Given its primary goal of alleviatingthe negative effect from response errors on the fly,the fuzzy SVM active learning method provides themost improvement over the soft-margin SVM methodwhen the impact of response errors is salient. In con-trast, because the negative effect from response errorsis limited when errors take place towards the endof the adaptive question selection, the advantage ofusing fuzzy SVM over soft margin SVM diminishescorrespondingly.

We also examine performances of the two methodswhen respondents do not make any response error. Asexpected, use of the fuzzy SVM active learning incurredan efficiency loss, which originated from the less thanperfect fuzzy membership probabilities assigned to thecorrectly labeled support vectors (Table 6(b)). Neverthe-less, such an efficiency loss is relatively minor, possiblydue to the fact that active learning is inherently lesschallenging in the absence of response errors.

11 For simplicity, we report the performance metrics with 40 adaptivequestions where each synthetic respondent labels the profile as“would consider” or “would not consider.” Similar patterns are foundwhen choice questions are added after the consideration questions.In line with Table 6(a), the advantage of fuzzy SVM active learningis the most salient when errors take place towards the beginningand middle portions of the consideration/choice questions. Wealso conducted similar comparisons under varying levels of errorinstances and product dimensions. We find that the general resultshold qualitatively as long as the error instance is not excessive (if therespondents incur too many errors, neither method can effectivelyrecover the true part-worths).

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex Products462 Marketing Science 35(3), pp. 445–464, © 2016 INFORMS

5.4. Tests of Improvements Over NoninformativeInitial Part-Worths

In the synthetic experiments above, we use only thefocal respondent’s information in the design of adap-tive questions. In this section, we examine the use ofcollaborative-filtered initial part-worths versus thatof noninformative initial part-worths in parameterrecovery and predictive validity. Because the initialindividual-specific part-worths in our proposed frame-work is obtained via collaborative filtering between thefocal and previous respondents’ self-configured productprofiles, we also investigate whether the degree ofheterogeneity in synthetic respondents’ configuratorsplays a role in the usefulness of incorporating datafrom other respondents. Intuitively, if all respondentshave the same product preferences and self-configurethe same profile, past respondents’ data should bequite informative in determining the focal respondent’sinitial part-worths. By contrast, when respondentsdiffer greatly in their most favorite product profiles,collaborative filtering may not be very helpful.

We consider the following two scenarios of homoge-nous versus heterogeneous configurators accordingly.In the homogenous case, we simulate 300 syntheticrespondents with identical part-worths (hence identicalself-configured profile). In the heterogeneous case, weconsider 300 synthetic respondents with different self-configurators. By excluding perfect overlap in the focaland previous respondents’ self-configured profiles, theaverage cosine similarity (s4j1 j ′5 in Equation (1)) in theheterogeneous case is 0.331.

In each case described above, we consider a baselinemodel where noninformative initial part-worths areused at the outset of the adaptive question design.Instead of using information from the focal and previousrespondents’ configurators as in the two collaborative fil-tering conditions, the noninformative initial part-worths

Table 7 Performance Improvements (in Percentage) Over Noninformative Initial Part-Worths

Heterogeneous configurators (%) Homogenous configurators (%)

No. of questions Hit rate KL U2 MAE RMSE Hit rate KL U2 MAE RMSE

1 40089 27061 443033 21023 22028 54033 45078 652014 30064 320205 21012 14003 186049 12026 12024 23093 16047 209010 15007 15021

10 13081 10021 104038 9000 8070 13079 9097 99054 9057 905915 9053 8022 60092 6095 6043 8037 6093 51038 6095 607020 6057 6085 36056 5051 5003 4097 4087 26021 4085 405325 5003 6003 25050 4075 4022 3047 4001 16047 4019 305530 4025 5085 20036 3045 3054 2042 3040 10032 3013 206335 3026 5025 13087 2093 2091 2017 3064 8026 2043 203940 2061 4079 10028 2073 2060 1045 2087 5040 2003 200545 0090 2004 3024 1020 1025 −0042 −1005 −1058 −0055 −005150 1018 3019 4046 1053 1064 −0021 −0065 −00086 −0085 −005155 0088 2080 2097 1032 1049 −0016 −0066 −0057 −0041 −003960 0065 2032 2022 1021 1028 0004 0027 0004 0023 −001765 0044 1076 1054 1006 1009 −0010 −0081 −0036 −0035 −0042

are obtained by randomly querying one profile that thesynthetic respondent would consider and one profilethe respondent would not consider from the trainingdata. We then compare improvements obtained fromcollaborative-filtered initial part-worths over nonin-formative initial part-worths in the respective cases ofhomogenous versus heterogeneous configurators. Inboth comparisons, we examine the scenario whereinthe focal product category consists of 35 attributes withthree levels each. The adaptive question design is basedon 40 consideration questions and 25 choice questionsas in §5.2. We follow the same procedure describedin §5.2 to construct the 3,000 validation questions foreach synthetic respondent.

Table 7 provides comparison results from these twocases as a function of the number of questions queried.Consistent with our conjecture, at the outset of theadaptive question design, incremental benefits fromthe collaborative-filtered versus the noninformativeinitial part-worths are considerably more salient in thehomogenous configurator case. This finding is ratherintuitive because previous respondents’ estimated part-worths are information rich if all respondents havethe same product preferences. Interestingly, Table 7also reveals that, in both cases, the advantages fromcollaborative-filtered initial part-worths diminish dur-ing the course of the adaptive question survey. Indeed,after the respondents answer about 40 considerationquestions, the benefits from the collaborative-filteredinitial part-worths become more or less negligible. Thisfinding implies that, analogous to the role of priorsin the Bayesian literature, benefits from informativeinitial part-worths can lessen considerably as more databecomes available. Similar results hold when we usean alternative baseline model wherein the initial part-worths vector is calculated based on the focal respon-dent’s configurator alone. These synthetic experimentssuggest that collaborative-filtered initial part-worths

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex ProductsMarketing Science 35(3), pp. 445–464, © 2016 INFORMS 463

in the proposed method may only be beneficial whenconsumers exhibit similar product preferences andpractical concerns preclude longer surveys.

6. ConclusionsIn this paper, we propose an adaptive decompositionalframework to elicit consumers’ preferences for complexproducts. Our research suggests that the proposedmethod could provide the following new capabilitiesto complement existing preference elicitation meth-ods. First, compared to extant methods, the proposedalgorithm is particularly suitable for high dimensionalproblems. Our empirical and synthetic studies demon-strate that the proposed framework can rapidly andeffectively elicit individual-level preference estimatesfor product categories equipped with 70–100 attributelevels. This is typically considered prohibitive fordecompositional preference elicitation methods. Second,we demonstrate that the fuzzy SVM active learningmethod provides a natural remedy for a long-standingchallenge in adaptive question design by gauging thepossibility of response errors on the fly and incorporat-ing it into the survey design. Through synthetic dataexperiments we show that the proposed algorithmis particularly effective when response errors takeplace towards the beginning or middle portions ofadaptive questions. Last, while most adaptive questionselection methods only use information from the focalrespondent, our research explores in a live settinghow previous respondent data may be used to assistactive learning of the focal respondent’s product prefer-ences. Overall, our research suggests that the proposedapproach is a promising new method that can be usedto complement extant preference elicitation methods,particularly in the context of complex products.

Our research is also subject to limitations and sug-gests promising avenues for future research. First, whilewe use part-worths models to characterize consumers’responses to consideration and choice questions, ouralgorithm does not explicitly uncover decision heuris-tics used by consumers. Future research may adapt theproposed algorithm to directly capture such heuristics.In such cases, the SVM classification would be per-formed at the product feature level, rather than at theproduct level as in the proposed method. Separateutility functions may also be used in the considerationand choice stages if the underlying decision rules forthe consideration phase versus the choice phase areknown to be different.

Second, while the current research is among thefirst efforts to explore the use of previous respondents’data in complex product preference elicitation in alive setting, we only observe incremental benefits ofcollaborative filtering at the outset of the adaptive ques-tion survey. Future research may consider alternative

approaches to take better advantage of this technique.For example, richer covariates such as demographic,socioeconomic, and product use information may beincorporated into collaborative filtering. Furthermore,advantages from using other respondents’ data mightbecome more salient if researchers were to incorporatecollaborative filtering into the entire course of adaptivequestion design rather than only in the selection ofinitial questions.

Last, while the proposed algorithm is flexible enoughto accommodate nonlinear utility functions, the specifi-cations of such nonlinear utility functions are ad hocby nature. Managerial insights or pretests (or both)are needed to determine the exact form of the kernelfunction. If an inappropriate kernel is used, responseerrors may be indistinguishable from the incorrectlyspecified utility form. Therefore, before the adaptivequestion design, managerial consultation or pretests(or both) should be used to carefully specify the exactutility model to be estimated.

With regard to extensions, future research may fur-ther explore the use of semi-supervised active learningin marketing context, particularly in the area of adap-tive question design. For example, a common challengefaced by adaptive question design is the lack of labeleddata points, particularly at the beginning of the survey.The basic idea of semi-supervised active learning isto iteratively identify unlabeled data points that aresimilar to labeled data, and to assign pseudo labelsto such points so that the training data set can beenlarged. Recent research has shown that such effortscan effectively alleviate the problem of small-sizedtraining data (e.g., Wu and Yap 2006, Hoi et al. 2009,Leng et al. 2013). Future research may further explorehow to use such methods to improve extant adaptivequestion design, or in any marketing context whereindividual-level consumer data are relatively sparse.Additionally, if consumer responses are classified inmultiple categories (e.g., not preferred, neutral, pre-ferred, etc.), researchers can leverage recent methodsthat use support vector machine classifiers for activelearning of multiclass classification (e.g., Patra andBruzzone 2012). Such endeavors are fruitful areas forfuture research.

Supplemental MaterialSupplemental material to this paper is available at http://dx.doi.org/10.1287/mksc.2015.0946.

AcknowledgmentsThe authors thank seminar participants at MIT, ColumbiaUniversity, the University of Texas at Austin, the Universityof British Columbia, Georgetown University, the INFORMSMarketing Science Conference, and the AMA AdvancedResearch Techniques (ART) Forum for their constructivecomments. This study also benefited from a grant of DonMurray to the USC Marshall Center for Global Innovation.

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.

Huang and Luo: Consumer Preference Elicitation of Complex Products464 Marketing Science 35(3), pp. 445–464, © 2016 INFORMS

ReferencesAbernethy J, Evgeniou T, Toubia O, Vert JP (2008) Eliciting consumer

preferences using robust adaptive choice questionnaires. IEEETrans. Knowledge Data Engrg. 20(2):145–155.

Bettman J (1979) An Information Processing Theory of Consumer Choice(Addison-Wesley, Reading, MA).

Breese JS, Heckerman D, Kadie C (1998) Empirical analysis ofpredictive algorithms for collaborative filtering. Proc. 14th Conf.Uncertainty Artificial Intelligence, 43–52.

Chaloner K, Verdinelli I (1995) Bayesian experimental design:A review. Statist. Sci. 10(3):273–304.

Cortes C, Vapnik V (1995) Support-vector networks. Machine Learn.20(3):273–297.

Cui D, Curry D (2005) Prediction in marketing using the supportvector machine. Marketing Sci. 24(4):595–615.

Ding M (2007) An incentive-aligned mechanism for conjoint analysis.J. Marketing Res. 44(2):214–223.

Ding M, Grewal R, Liechty J (2005) Incentive-aligned conjoint analysis.J. Marketing Res. 42(1):67–82.

Ding M, Park YH, Bradlow ET (2009) Barter markets for conjointanalysis. Management Sci. 55(6):1003–1017.

Ding M, Hauser JR, Dong S, Dzyabura D, Yang Z, Su C, Gaskin SP(2011) Unstructured direct elicitation of decision rules. J. Market-ing Res. 48(1):116–127.

Dong JX, Krzyzak A, Suen CY (2005) Fast SVM training algorithmwith decomposition on very large data sets. IEEE Trans. PatternAnal. Machine Intelligence 27(4):603–618.

Dong S, Ding M, Huber J (2010) A simple mechanism to incentive-align conjoint experiments. Internat. J. Res. Marketing 27(1):25–32.

Dzyabura D, Hauser JR (2011) Active machine learning for consider-ation heuristics. Marketing Sci. 30(5):801–819.

Evgeniou T, Boussios C, Zacharia G (2005) Generalized robustconjoint estimation. Marketing Sci. 24(3):415–429.

Evgeniou T, Pontil M, Toubia O (2007) A convex optimizationapproach to modeling consumer heterogeneity in conjointestimation. Marketing Sci. 26(6):805–818.

Green PE, Srinivasan VS (1990) Conjoint analysis in marketing:New developments with implications for research and practice.J. Marketing 54(4):3–19.

Haaijer R, Kamakura WA, Wedel M (2001) The no-choice alternativein conjoint choice experiments. Internat. J. Market Res. 43(1):93–106.

Hauser JR (1978) Testing the accuracy, usefulness, and significance ofprobabilistic choice models: An information-theoretic approach.Oper. Res. 26(3):406–421.

Hauser JR, Rao VR (2004) Conjoint analysis, related modeling, andapplications. Wind Y, Green PE, eds. Marketing Research andModeling: Progress and Prospects (Springer, New York), 141–168.

Hauser JR, Toubia O (2005) The impact of utility balance andendogeneity in conjoint analysis. Marketing Sci. 24(3):498–507.

Hauser JR, Dong S, Ding M (2014) Self-reflection and articulatedconsumer preferences. J. Product Innovation Management 31(1):17–32.

Hauser JR, Toubia O, Evgeniou T, Befurt R, Dzyabura D (2010) Dis-junctions of conjunctions, cognitive simplicity, and considerationsets. J. Marketing Res. 47(3):485–496.

Heo G, Gader P (2009) Fuzzy SVM for noisy data: A robust mem-bership calculation method. 2009 Fuzzy Systems Conf. Proc.,431–436.

Herbrich R, Graepel T, Campbell C (2001) Bayes point machines.J. Machine Learn. Res. 1(September):245–279.

Hoi SC, Jin R, Zhu J, Lyu MR (2009) Semisupervised SVM batchmode active learning with applications to image retrieval. ACMTrans. Inform. Systems 27(3):Article 16.

Huber J, Hansen D (1986) Testing the impact of dimensional com-plexity and affective differences in adaptive conjoint analysis.Adv. Consumer Res. 14(1):159–163.

Huber J, Wittink DR, Fiedler JA, Miller R (1993) The effectiveness ofalternative preference elicitation procedures in predicting choice.J. Marketing Res. 30(1):105–114.

Johnson RM, Orme BK (2007) A new approach to adaptive CBC.Sawtooth Software Research Paper, Sequim, WA.

Kullback S, Leibler RA (1951) On information and sufficiency. Ann.Math. Statist. 22(1):79–86.

Leng Y, Xu X, Qi G (2013) Combining active learning and semi-supervised learning to construct SVM classifier. Knowledge-BasedSystems 44(May):121–131.

Lin CF, Wang SD (2002) Fuzzy support vector machines. IEEE Trans.Neural Networks 13(2):464–471.

Lin CF, Wang SD (2004) Training algorithms for fuzzy supportvector machines with noisy data. Pattern Recognition Lett. 25(14):1647–1656.

Netzer O, Srinivasan VS (2011) Adaptive self-explication of multiat-tribute preferences. J. Marketing Res. 48(1):140–156.

Orme B (2007) Three ways to treat overall price in conjoint analysis.Sawtooth Software Research Paper, Sequim, WA.

Orme B (2010) Getting Started with Conjoint Analysis2 Strategies forProduct Design and Pricing Research, 2nd ed. (Research PublishersLLC, Madison, WI).

Park YH, Ding M, Rao VR (2008) Eliciting preference for complexproducts: A web-based upgrading method. J. Marketing Res.45(5):562–574.

Patra S, Bruzzone L (2012) A batch-mode active learning techniquebased on multiple uncertainty for SVM classifier. GeoscienceRemote Sensing Lett. 9(3):497–501.

Salton G, McGill MJ (1986) Introduction to Modern Information Retrieval(McGraw-Hill, New York).

Sattler H, Hensel-Börner S (2000) A comparison of conjoint measure-ment with self-explicated approaches. Gustafsson A, Huber F,eds. Conjoint Measurement2 Methods and Applications (Springer-Verlag, Berlin Heidelberg), 147–159.

Scholz SW, Meissner M, Decker R (2010) Measuring consumerpreferences for complex products: A compositional approachbased on paired comparisons. J. Marketing Res. 47(4):685–698.

Shilton A, Lai DT (2007) Iterative fuzzy support vector machineclassification. 2007 Fuzzy Systems Conf. Proc., 1391–1396.

Srinivasan VS (1988) A conjunctive-compensatory approach tothe self-explication of multiattributed preferences. Decision Sci.19(2):295–305.

Tong S, Koller D (2001) Support vector machine active learningwith applications to text classification. J. Machine Learn. Res.2(March):45–66.

Toubia O, Evgeniou T, Hauser JR (2007a) Optimization-based andmachine-learning methods for conjoint analysis: Estimationand question design. Gustafsson A, Herrmann A, eds. ConjointMeasurement: Methods and Applications, 4th ed. (Springer-Verlag,Berlin Heidelberg), 231–258.

Toubia O, Hauser JR, Garcia R (2007b) Probabilistic polyhedralmethods for adaptive choice-based conjoint analysis: Theoryand application. Marketing Sci. 26(5):596–610.

Toubia O, Hauser JR, Simester DI (2004) Polyhedral methods foradaptive choice-based conjoint analysis. J. Marketing Res. 41(1):116–131.

Toubia O, Johnson E, Evgeniou T, Delquié P (2013) Dynamic experi-ments for estimating preferences: An adaptive method of elicitingtime and risk parameters. Management Sci. 59(3):613–640.

Toubia O, Simester DI, Hauser JR, Dahan E (2003) Fast polyhedraladaptive conjoint estimation. Marketing Sci. 22(3):273–303.

Vapnik VN (1998) Statistical Learning Theory (John Wiley & Sons,New York).

Wang Y, Wang S, Lai KK (2005) A fuzzy support vector machine toevaluate credit risk. IEEE Trans. Fuzzy Sys. 13(6):820–831.

Wu K, Yap KH (2006) Fuzzy SVM for content-based image retrieval.IEEE Comput. Intelligence Magazine 1(2):10–16.

Dow

nloa

ded

from

info

rms.

org

by [

68.1

81.1

78.9

8] o

n 15

Sep

tem

ber

2017

, at 1

3:55

. Fo

r pe

rson

al u

se o

nly,

all

righ

ts r

eser

ved.