a product feature inference model for...

Proceedings of The ASME 2015 International Design Engineering Technical Conferences &Computers and Information in Engineering Conference

IDETC/CIE 2015August 2-5, 2015, Boston, Massachusetts, USA

DETC2015-47225

A PRODUCT FEATURE INFERENCE MODEL FOR MINING IMPLICIT CUSTOMERPREFERENCES WITHIN LARGE SCALE SOCIAL MEDIA NETWORKS

Suppawong TuarobComputer Science and Engineering

Industrial and Manufacturing EngineeringThe Pennsylvania State University

University Park, PA 16802Email: [email protected]

Conrad S. TuckerEngineering Design and Industrial and Manufacturing Engineering

Computer Science and EngineeringThe Pennsylvania State University

University Park, PA 16802Email: [email protected]

ABSTRACTThe acquisition and mining of product feature data from on-

line sources such as customer review websites and large scalesocial media networks is an emerging area of research. In manyexisting design methodologies that acquire product feature pref-erences form online sources, the underlying assumption is thatproduct features expressed by customers are explicitly stated andreadily observable to be mined using product feature extractiontools. In many scenarios however, product feature preferencesexpressed by customers are implicit in nature and do not directlymap to engineering design targets. For example, a customer mayimplicitly state “wow I have to squint to read this on the screen”,when the explicit product feature may be a larger screen. The au-thors of this work propose an inference model that automaticallyassigns the most probable explicit product feature desired by acustomer, given an implicit preference expressed. The algorithmiteratively refines its inference model by presenting a hypothesisand using ground truth data, determining its statistical validity.A case study involving smartphone product features expressedthrough Twitter networks is presented to demonstrate the effec-tiveness of the proposed methodology.

1 IntroductionSocial media such as Twitter, Facebook, and Google Plus

allows users from over the world to connect and exchange infor-mation in a timely and seamless manner. Literature has shownsuccessful usages of knowledge mined from this timely and everincreasing social media data in diverse applications such as de-tecting earthquake warnings and emergence needs due to naturaldisasters [1,2], mining healthcare-related information for disease

prediction [3,4], predicting financial market movement [5,6], etc.In the product design domain, Tuarob and Tucker have pro-

posed a set of algorithms to identify notable features imple-mented in existing products in order to provide suggestions todesigners regarding incorporating/removing such features intothe next generation products [7, 8]. In a subsequent work, theyalso proposed an automated algorithm for identifying innovativeusers in social network communities [9, 10].

Implicit speech is a sophisticated form of language usage inwhich the speaker conveys the information in an implicit man-ner, and is becoming a standard of social media usage. Commonmanifestations of implicit speech include vagueness and sar-casm. While social media data contains useful information per-taining to many applications, the applicability of the existing nat-ural language processing techniques is primarily limited to socialmedia data whose content is explicit. However, being colloquialin nature, a majority of social media textual data is present in animplicit form. Maynard and Greenwood reported that roughly22.75% of all the tweets in general are implicit [11]. Therefore,the ability to correctly interpret this implicit social media datawould not only reduce the errors caused by methodologies thatare not specifically designed to handle implicit information, butalso allow the methodologies to make use of additional implicitdata that would have traditionally been disregarded due to beingtreated as noise.

Examples of explicit and implicit social media messages aregiven below:

Explicit “Dell venue pro size is perfect.not too small not to big.”

Implicit “I actually believed all that crazy

1 Copyright c⃝ 2015 by ASME

stuff about the IPhone 5 when reallyits just longer with a bigger screen.”

The first example is considered explicit because it can be di-rectly inferred from the grammatical structure that the user maybe satisfied by the perfect size of his Dell Venue Pro. However,the second example does not give any direct information aboutthe screen feature of his/her iPhone 5, and hence is implicit,though it may be inferred that this particular user may feel thatthe bigger screen of his/her iPhone 5 is not an impressive inno-vation. If these implicit social media messages remain untreated,two problems could occur:

1. Most text mining algorithms for large scale data areextraction-based (where each of the objects or documentsis classified whether it is useful or not), and would disre-gard such implicit data where explicit knowledge could notbe extracted, resulting in low utilization of useful data.

2. Some implicit social media messages are in the form ofsarcasm which may either exaggerate (i.e. “Apparentlythe new iphone 5 helps you lose weight,you buy it and you can’t afford foodfor a month.”) or oppose (i.e. “HOLY SH**!...The iPhone 5 can now have 5 rows oficons. Too amazing. #sarcasm”) the originalmeaning. If the traditional techniques are applied on theseuntreated messages, they would interpret false meaning.

Regardless of all the useful applications that emerge fromsocial media data, being able to automatically explicate the im-plicit social media data would not only increase the performanceof the existing natural language processing techniques, but alsoallow more data to be used.

Processing social media data has been one of the biggestchallenges for researchers. Traditional natural processing tech-niques that have been shown to work well on traditional docu-ments are reported to fail or under-perform when applied on so-cial media data, whose natures differ from traditional documentsin the following ways:

1. Social media data is sparse and high-dimensional. A unitof social media document (aka message) is short, contain-ing only one or two sentences. Some social media servicessuch as Twitter enforce the length of a message, urging theusers to be creative and use their own combination of wordforms to express their opinions within limited context. Tra-ditional techniques for interpreting semantics from docu-ments would fail on social media data due to insufficiencyin textual content. Furthermore, the high-dimensionalitycaused by using creative word forms would prevent such tra-ditional techniques from finding semantic similarity amongthe pool of social media messages.

2. Social media data is noisy. Noise in social media datacomes in multiple forms such as grammatical errors, in-tentional/unintentional typographical errors, and symbolicword forms. Since traditional text processing techniques as-sume documents to be well-formed and grammatically cor-

rected, they would fail to operate on social media data.

Existing attempts to interpret the semantic meaning behindimplicit social media and relevant kinds of data (i.e. product re-views) include machine learning based implicit sentence detec-tion algorithms proposed by Tsur et al. [12, 13]. However, theirmethods only identify whether a piece of textual information issarcastic or not. The work presented in this paper extends theprevious literature by further extracting true meaning from so-cial media messages whose context related to products/productfeatures are implicit.

Specifically, this paper presents a mathematical modelbased on the heterogeneous co-word network patterns in orderto translate implicit context towards a particular product orproduct feature into the explicit equivalence. A co-word network(or word co-occurrence network) is a graph where each noderepresents a unique word, and an undirected edge representsthe frequency of co-occurrence of the two words. In this work,the network is augmented to incorporate parts of speech intoeach word. The intuition behind using the co-word networkis that even though a message may be implicit, the similarcombination of the words may have been used by other userswho express their messages more explicitly. For example,given an implicit message “wow I have to squint toread this on the screen”, other users may have usedthe terms squint and screen in a more explicit context such as“Don’t make me squint @user - your mobilebanner needs work on my tiny screen iPhone5S.” If the combination of the words squint and screen occursin the messages that contain the word tiny frequently enough,then the system would be able to relate the original message toa more explicit set of terms. Particularly, the system would beable to interpret that the user thinks that the screen feature ofthis particular product is small.

Specifically, this paper has the following main contributions:

1. The authors adopt the usage of the co-word network in prod-uct design context. The co-word network has shown to beuseful in multiple semantic extraction applications in infor-mation retrieval literature. To the best of our knowledge, thistechnique has first been used in the design literature.

2. The authors propose a probabilistic mathematical model inorder to map implicit product-related information in socialmedia data into the equivalent explicit context.

3. The authors illustrate the efficacy of the proposed method-ology using a case study of real world smartphone data andTwitter data.

2 Related WorksWhile the use of implicit language such as indirect speech

and sarcasm has been well explored in multiple psycholinguis-tic studies [14–16], automatic semantic interpretation of implicitinformation in social networks is still in an infancy stage. Thissection first surveys the use of social media data pertaining to theproduct design applications, and then discusses existing natural


language processing techniques that have been used to extractsemantics from social media data.

2.1 Applications of Large Scale Social Media Data inProduct Design Domain

Social media has recently been established as a viablesource for product design and development. Previous studiesclaimed that knowledge extracted from social media data couldbe more beneficial than traditional product design knowledgesources such as product reviews (from popular online electroniccommerce website such as Amazon.com, BestBuy.com, Wal-mart.com) and user study campaigns [7–9]. Asur et al. was ableto use Twitter data collected during a 3 month period to predictthe demand of theatre movies [17]. They claimed that the predic-tion results are more accurate than those of the Hollywood StockExchange. Their study also found that sentiments in tweets canimprove the prediction after a movie has been released. Tuaroband Tucker found that social media data could be a potential datasource for extracting user preferences towards particular prod-ucts or product features [7, 8]. In a later work, they presenteda methodology for automatic discovery of innovative users (aka.lead users) in online communities, using a set of mathematicalmodels to extract latent features (product features not yet imple-mented in the market space), then identify lead users based onthe volume of innovative features that they express in social me-dia [9, 10]. Recently, Stone and Choi presented a visualizationtool which allows designers to extract useful insights from on-line product reviews [18].

Since all the above techniques rely on the assumption thatsocial media data is explicit, these techniques would fail to cor-rectly process implicit social media messages which could resultin error or inaccurate results. With these emerging product de-sign applications that rely on social media as a knowledge source,it is crucial that the algorithms behind these applications are ableto correctly interpret the true meaning of the data.

2.2 Natural Language Technology for Semantic Inter-pretation in Social Media

In this subsection, technologies used to process social mediadata that go beyond just keyword detection (which works only onexplicit data) are reviewed. Multiple studies in the InformationRetrieval field have agreed that it is necessary to develop specialtext processing techniques for social media messages, since theyare different from traditional documents due to smaller textualcontent, heterogeneous language standards, and higher level ofnoise [19–21].

Social media holds sentiments expressed by its users (pri-marily in the form of textual data). Sentiment analysis in socialmedia refers to the use of natural language processing, text anal-ysis and computational linguistics to identify and extract sub-jective information in social media. Thelwall et al. found thatimportant events lead to increases in average negative sentimentstrength in tweets during the same period [22]. The authors con-cluded that negative sentiment may be the key to popular trends

in Twitter. Kucuktunc et al. studied the influence of several fac-tors such as gender, age, education level, discussion topic, andtime of day on sentiment variation in Yahoo! Answers [23].Their findings shed light towards an application on attitude pre-diction in online question-answering forums. Weber et al. pro-posed a machine learning based algorithm to mine tips, short,self-contained, concise texts describing non-obvious advice [24].Sentiment of each short text is extracted and used as part of thefeatures. Even though sentiment analysis could prove to be use-ful when designers would like to know how customers feel abouta particular product or product feature, most sentiment extrac-tion techniques only output sentiment level in two dimension(i.e. Positive and Negative). Hence, more advanced techniquesare needed in order to narrow down what actually the customerswant to say.

Besides sentiment analysis, multiple studies have found thattopical analysis could be useful when dealing with noisy textualdata such as social media. Even though social media is highin noise due to the heterogeneity of the writing styles, formal-ity, and creativity, such noise bears undiscovered wisdom of thecrowd. Paul and Dredze utilized a modified Latent Dirichlet Al-location [25] model to identify 15 ailments along with descrip-tions and symptoms in Twitter data [26, 27]. Tuarob et al. pro-posed a methodology for discovering health related content insocial media data by quantifying topical similarity between doc-uments as a feature type [3,4]. Furthermore, a number of studieshave devoted to using topical models to detect emerging trendsin social networks [28–30].

The techniques mentioned above rely on explicit content ofsocial media data and would likely fail or not produce correctresults when applied on documents whose meanings are implicit.

Implicit document processing has posed challenges to com-putational linguists. Researchers have studied on the nature ofimplicit uses of language; however, none have successfully de-veloped a computational model to translate implicit content intothe equivalent explicit form. In dealing with implicit context insocial media data, multiple algorithms have been proposed to de-tect the presence of implicit content in social media [12, 31, 32];however, these algorithms do not further attempt to map the im-plicit content to the equivalent explicit forms. To the best of ourknowledge, we are the first to explore the problem of identifyingexplicit user preference towards a product/product feature fromlarge scale social media data.

3 MethodologyMultiple recent studies have noted large scale social media

data for its usefulness in the product design domain. Being col-loquial and ubiquitous allows social media users to express theirthoughts anywhere and anytime, resulting in a large amount ofdata which is instantly accessible.

The methodology proposed in this paper mines language us-ages in the form of word co-occurrence patterns, in order to mapimplicit context commonly found in social media data to equiva-lent explicit ones. Figure 1 outlines the overview of the proposed


Social Media Data

Preprocessing

Social Media Data

Indexing co-

word network

Query

Processing

Result

Processing

Co-word Network

User

wow I have to squint to read this on the screen

FIGURE 1. Overview of the proposed system.

methodology.First, social media data is collected and preprocessed (Sec-

tion 3.1). The textual content is then fed to the indexer in order togenerate the co-word network (Section 3.2). Once the network isgenerated and indexed, the user could give the system an implicitmessage as the query. The query is processed and the results arereturned to the users as a ranked list of relevant keywords classi-fied by parts of speech (Section 3.3).

A practical usage of the proposed implicit message infer-ence system would be to aid designers in synthesizing productfeatures, mined from users’ feedback in large scale social mediadata, into the next generation products. A framework was pre-sented in [8], where designers iteratively identify notably goodand bad features from existing products, and incorporate/removethem from the next generation products. The method proposedin this paper could be incorporated into such a framework to im-prove the notable product feature extraction process.

The following subsections will discuss each component inFigure 1 in detail.

3.1 Social Media Data PreprocessingSocial media provides a means for people to interact, share,

and exchange information and opinions in virtual communitiesand networks [33]. For generalization, the proposed method-ology minimizes the assumption about functionalities of socialmedia data, and only assumes that a unit of social media is a tu-ple of unstructured textual content, a user ID, and a timestamp.Such a unit is referred to as a message throughout the paper. Thisminimal assumption would allow the proposed methodology togeneralize across multiple heterogeneous pools of social mediasuch as Twitter, Facebook, Google+, etc., as each of these socialmedia platforms has this data structure. Social media messages,corresponding to each product domain, are retrieved by a queryof the product’s name (and its variants) within the large streamof social media data.

3.1.1 Data Cleaning. Most social media crawlingAPIs provide additional information with each social media mes-sage such as user identification, geographical information, andother statistics. Though this additional information could be use-ful, it is disregarded and removed not only to save storage spaceand improve computational speed, but also to preserve the mini-mal assumption about the social media data mentioned earlier.

Raw social media messages are full of noise that could pre-vent further steps from achieving the expected performance. Inorder to remove such noise, the data cleaning process does thefollowing:

1. Lowercasing the textual content2. Removing hashtags, usernames, and hyperlink3. Removing stop words1

Note that misspelled words (e.g. hahaha, lovin, etc.)and emoticons (e.g. :-), (")(- -)("), etc.) are intention-ally preserved. Even though they are not well-formed and do notexist in traditional dictionaries, they have been shown to carryuseful information that infers semantic meaning behind the mes-sages [4,34]. Furthermore, unlike traditional preprocessing tech-niques for reducing noise in documents, the social media datais not stemmed, since previous studies have shown that stem-ming could excessively reduce the dimensionality of the data(especially in short messages, each of which contains roughly 14words on average [35, 36]), and would likely to result in poorerperformance [3].

3.1.2 Sentiment Extraction The technique devel-oped by Thelwall et al. is employed to quantify the emotion in amessage. The algorithm takes a short text as an input, and out-puts two values, each of which ranges from 1 to 5 [34]. The firstvalue represents the positive sentiment level, and the other rep-resents the negative sentiment level. The reason for having thetwo sentiment scores instead of just one (with −/+ sign repre-senting negative/positive sentiment) is because research findingshave determined that positive and negative sentiments can coex-ist [37]. The positive and negative scores are then combined toproduce an emotion strength score using the following equation:

Emotion Strength(ES) = PositiveScore−NegativeScore (1)

A message is then classified into one of the 3 categoriesbased on the sign of the Emotion Strength score (i.e. posi-tive (+ve), neutral (0ve), negative (-ve)). The EmotionStrengthscores will later be used to identify whether a particular messageconveys a positive or negative attitude towards a particular prod-uct or product feature.

3.1.3 Feature Extraction. Product features are ex-tracted from each social media message. In this paper, the feature

1Stop words are words that are filtered out before processing of textual in-formation. Such words are typically too common to infer meaningful semantics.Examples of stop words include the, is, at, which, and on.


Algorithm 1: The feature extraction algorithm from a col-lection of documents

Input: D: Set of free-text documents to extract product features.Output: E: Set of extractions. Each e ∈ E is a tuple of

⟨ f eature, f requency⟩, for examplee = ⟨‘onscreen keyboard′,5⟩

1 preprocessing;2 for d ∈ D do3 Clean d ;4 POS tag d ;5 Extract multi-word features ;6 end7 initialization;8 E = ⊘ ;9 T = ⊘ ;

10 F = Seed Features ;11 while E can still grow do12 Learn templates from seed features;13 Add new template to T;14 foreach d ∈ D do15 foreach Sentence s ∈ d do16 e← Extract potential feature-opinion pair using T;17 Add e to E ;18 end19 end20 Update F;21 end22 E← Clustering and normalizing features ;23 return E;

extraction algorithm used in [9] is employed. The pseudo-codeof the algorithm is outlined in Algorithm 1. At a high level, thealgorithm takes a collection of social messages corresponding toa product as input, and outputs a tuple of ⟨ f eature, f requency⟩such as ⟨‘onscreen keyboard′,5⟩, which infers that the on-screenkeyboard feature of this specific product was mentioned 5 timeswithin the given corpus of social media messages. Interestedreaders are encouraged to consult [9] for additional details aboutthe feature extraction algorithm.

The features are extracted because the proposed methodol-ogy infers explicit opinions towards a particular product feature,hence it is imperative that product features can automatically beidentified.

3.1.4 Part of Speech Tagging. The final step of thesocial media data preprocess is to tag each word in a social mes-sage with a par of speech (POS). In this paper, Carnegie MellonARK Twitter POS tagger2 is used for this purpose. This particu-lar POS tagger has not only been developed specially for socialmedia data, but has also been successfully used in the productdesign domain [8]. Table 1 lists the POS tags used in this paper,along with their descriptions.

The part of speech information is needed in order to dis-ambiguate words with multiple meanings (i.e. homonyms) [38],

2http://www.ark.cs.cmu.edu/TweetNLP/

TABLE 1. Statistics of the co-word network generated using the Twit-ter data associate with the 27 smartphone products. The number ofnodes and average degrees are categorized by the types of nodes.

Node

TypeDescrip on

# of

Nodes

Avg

Degree

PRODUCT Smartphone model name 27 4589.17

N common noun 24931 56.79

A adjec!ve 6169 64.93

V verb including copula, auxiliaries 13508 62.11

^ proper noun 32562 30.62

! interjec!on 4566 39.46

Ppre- or postposi!on, or subordina!ng

conjunc!on840 57.13

Gother abbrevia!ons, foreign words,

possessive endings, symbols, garbage9354 38.30

R adverb 2325 48.09

Lnominal + verbal (e.g. i’m ), verbal +

nominal (let’s )432 66.48

D determiner 361 79.07

~discourse marker, indica!ons of

con!nua!on across mul!ple tweets92 22.63

E emo!con 88 35.73

Opronoun (personal/WH; not

possessive)452 50.16

$ numeral 25 79.56

, punctua!on 35 20.43

& coordina!ng conjunc!on 62 55.50

Z proper noun + possessive 90 15.47

T verb par!cle 42 11.00

S nominal + possessive 25 10.28

X existen!al there , predeterminers 13 16.38

which can be commonly found in social media. For exam-ple, the word “cold” in “Who waits for an iphone5in this cold weather?” and “I’ve got a coldthis morning. will skip class.” may have dif-ferent meanings.

Besides standard linguistic POS tags offered by the POS tag-ger tool, a special POS tag PRODUCT is also introduced to dis-tinguish a word that represents a product name (e.g. iPhone 4,Samsung Galaxy SII, Nokia N9, etc) from other words.

3.2 Generating and Indexing Co-word NetworkA co-word network is the collective interconnection of terms

based on their paired presence within a specified unit of text.Traditional co-word networks represent a node with only textualrepresentation of a word. Variants of co-occurrence networkshave been used extensively in the Information Retrieval field ina wide range of applications that involve semantic analysis suchas concept/trend emergence detection [39, 40], discovering newwords, finding/clustering relevant items [41, 42], semantic inter-pretation [3, 43], and document annotation [44, 45].

In this paper, a node also incorporates part of speech in-formation for word-sense disambiguation purposes. Concretely,a co-word network is an undirected, weighted graph whereeach node is a pair of ⟨Word,POS Tag⟩ (e.g., ⟨squint,V ⟩ and⟨iPhone 4,PRODUCT ⟩) that represents a POS tagged word, and


each edge weight is the frequency of co-occurrence. Let D bethe set of all social media messages, and T be the vocabulary ex-tracted from D. Formally, the co-word network G is defined asfollows:

G= ⟨V,E⟩V={⟨Word,POS Tag⟩ ∈ T }E={(a,b)| a,b ∈ T}Weight(a,b)=|{d | d ∈ D, (a,b)∈ E, d contains both aand b}|

Algorithm 2: The co-word generation algorithm from acollection of social media messages.

Input: D: Set of free-text social media messages.Output: G: Co-word network.

1 initialization;2 V = ⊘ ;3 E = ⊘ ;4 foreach Document d ∈ D do5 /*Extracting nodes from message*/ ;6 Compound c = ⊘ ;7 foreach Word w ∈ d do8 Node n = ⟨w.text,w.pos⟩ ;9 Add n to c ;

10 end11 /* Update V */ ;12 foreach Node n ∈ c do13 if n /∈ V then14 Add n to V ;15 end16 end17 /* Update E */ ;18 foreach Possible combination of word pair ⟨a,b⟩ in D do19 Edge e = (a,b) ;20 if e /∈ E then21 Add e to E ;22 end23 Increment e.weight by 1 ;24 end25 end26 return G = ⟨V,E⟩;

A compound is defined as a set of nodes. A social mediamessage is converted to a compound by converting each word inthe message into a node. The nodes are then combined. Repli-cated nodes are removed. Algorithm 2 explains how the co-wordnetwork is generated from a corpus of social media messages.First, the set of nodes, V , and the set of edges, E, are initializedto empty sets. For each social media message d in the corpus D,all the words are tagged with appropriated POS tags, and thenconverted into nodes which are then combined into a compoundc. For each node n in the compound c, update V by including n.Then for each possible combination pair of nodes in c, the weight

of the edge that links these two nodes is incremented by 1. Theco-word network generation is finished once all the messages areprocessed. In this paper, the open-source graph database Neo4J3

is used to store and index the network. Neo4J is used in this taskdue to its scalability that allows a network with millions of edgesto be efficiently stored and indexed.

3.3 Query and Result ProcessingA query is a free text message with implicit content. Ex-

ample queries includes “I can’t express how much Ilove the price of iPhone 5 on black Friday”and “I have to squint the screen to readthis on Nokia N9”. This section describes how a userquery is transformed into the network-compatible format, ora compound Q, for further processing. In particular, in orderto process a free text query QText , the following steps areperformed:

1. Preprocess the query QText using the mechanism describedin Section 3.1, in order to clean the raw message, extractfeatures, and assign POS tags.

2. Form the query compound Q, by converting each POStagged word into a node, and combining them into a set.

3. Remove the nodes in Q that are stop words or do not exist inthe co-word network.

The resulting query compound Q is then fed into the systemfor further processing.

The implicit message translation problem in transformedinto a node ranking problem so that traditional Information Re-trieval techniques can be applied. In this context, a node in theco-word network is equivalent to a combination of a word andits POS. Given the set of products in the same domain (productspace) S, the set of all features (feature space) F, the co-wordnetwork G = ⟨V,E⟩, and query compound Q. The node rankingalgorithm takes the following steps:

STEP1 For each node t ∈V , compute P(t|Q, f ,s), the likelihood(relevant to product features) of the node t given the querycompound Q, target product feature f ∈ F, and the products ∈ S.

STEP2 Rank the nodes by their likelihood.STEP3 Top nodes are returned.

P(t|Q, f ,s) represents the likelihood that the node t is rele-vant to the feature f of the product s, given the query compoundQ. The relevance of a node is quantified by its relatedness andexplicitness to the query compound Q. Hence, mathematicallyP(t|Q, f ,s) is defined as follows:

P(t|Q, f ,s) = ∑q∈Q

wq ·Relatedness(t,q) ·Explicitness(t|q) (2)

3http://neo4j.com/


Where,

Relatedness(t,q) = weight(t,q)∑n∈Ad j(q) weight(n,q) (3)

Explicitness(t|q) = degree(t)∑n∈Ad j(q) degree(n) (4)

wq is the weight for the node q ∈ Q, and ∑q∈Q wq = 1.Ad j(q) is the set of adjacent (neighbor) nodes to q. In the im-plementation, feature (i.e. f ) and product nodes (i.e. s) are giventwice the weight of other nodes in the compound. This is be-cause, by giving higher weight to the target feature and product,the likelihood given to each node will be more relevant towardsthe feature of the product of interest. weight(t,q) is the weightof the edge linking t and q, which is the co-occurrence frequencyof the two nodes. Note that if t and q have never been mentionedtogether, then the Relatedness(t,q) is evaluated to zero.

Relatedness(t,q) hence quantifies how frequently t and qare mentioned together. The score is normalized to range be-tween [0,1] for consistency when combined with other compo-nents.

Explicitness(t|q) quantifies explicitness of the term repre-sented by the node t when presented in the same context as theterm represented by the node q, and is measured by the normal-ized degree of the node t. A term is explicit if it makes the con-text clearer or easier to understand to the readers. An intuitiveassumption is made that terms that have explicit meanings tendto be commonly used and mentioned frequently in multiple con-texts. Such properties are captured by the degree of the noderepresenting the term, since the higher degree a node has, themore diverse it is co-mentioned with other words. Table 2 pro-vides examples of 10 highest degree nodes and 10 lowest degreenodes, classified by parts of speech. From the example, it canbe seen that words with high degrees have explicit meanings andwould make the context simpler and more clarified. On the otherhand, the words with low degrees tend to be spurious words thatdo not directly associate with the product domain. These wordstend to make the context implicit, especially when talking abouta product or product feature.

Finally, P(t|Q, f ,s) is then a weighted sum of the relevancebetween the node t ∈ V and each node in the query compoundQ. P(t|Q, f ,s) ranges between [0,1], using to approximate theprobability of the node t being relevant to the query compoundq. Once P(t|Q, f ,s) is computed for all the nodes in the co-wordnetwork, they can then be ranked using this score. The final out-put of the system would then be the top words classified by theirparts of speech.

4 Case Study, Results, and DiscussionThis section introduces a case study used to verify the pro-

posed methodology and discusses the results.A case study of 27 smartphone products is presented that

uses social media data (Twitter data) to mine relevant productdesign information. Data pertaining to product specifications

from the smartphone domain is then used to validate the pro-posed methodology. The selected smartphone models includeBlackBerry Bold 9900, Dell Venue Pro, HP Veer, HTC Thunder-Bolt, iPhone 3G, iPhone 3GS, iPhone 4, iPhone 4S, iPhone 5,iPhone 5C, iPhone 5S, Kyocera Echo, LG Cosmos Touch, LG En-lighten, Motorola Droid RAZR, Motorola DROID X2, Nokia E7,Nokia N9, Samsung Dart, Samsung Exhibit 4G, Samsung GalaxyNexus, Samsung Galaxy S 4G, Samsung Galaxy S II, SamsungGalaxy Tab, Samsung Infuse 4G, Sony Ericsson Xperia Play, andT-Mobile G2x.

Smartphones are used as a case study in this paper becauseof the large volume of discussion about this product domain insocial media. Previous work also illustrated that social mediadata (i.e. Twitter) contains crucial information about product fea-tures of other more mundane products such as automobiles [8].The proposed algorithms may not work well for products whichare not prevalently discussed in social media as the correspond-ing sets of social media messages may be too small to extractuseful knowledge from.

4.1 Social Media Data CollectionTwitter4 is a microblog service that allows its users to send

and read text messages of up to 140 characters, known as tweets.The Twitter dataset used in this research was collected randomlyusing the provided Twitter API, and comprises 2,117,415,962 (˜2.1 billion) tweets in the United States during the period of 31months, from March 2011 to September 2013.

Tweets related to a product are collected by detecting thepresence of the product name (and variants), and preprocessedby cleaning and mapping sentiment level as discussed in Section3.1. Table 3 lists the number of tweets, number of unique Twitterusers, and number of extracted features.

4.2 Co-word Network GenerationThe co-word network is generated using the procedure out-

lined in Algorithm 2, using all the social media data associatedwith the 27 smartphone models. The resulting network contains95,999 nodes and 2,288,723 edges. A node has a degree of 47.7and is used 160 times on average. Table 1 lists the numbers andaverage degrees of nodes categorized by parts of speech.

4.3 Query and Result ProcessingThis section reports notable results from the proposed

methodology.Given a textual query with implicit content, the system

first transforms it into a compound, by removing stop wordsand converting each remaining distinct word into a node.For example, a textual query “I have to squint thescreen to read this on Nokia N9” would be trans-lated into the compound { ⟨read,V ⟩, ⟨squint,V ⟩, ⟨screen,N⟩,⟨Nokia N9,PRODUCT ⟩ }. Note that not all the words in thequery are converted into nodes since they could be stop words

4https://twitter.com/


TABLE 2. (Left) Top 10 nodes (words) with highest degree, classified by parts of speech. (Right) Bottom 10 nodes (words) with lowest degree,classified by parts of speech.

N V A ! N V A !

phone got good lol synergy obstruct conten ous heeh

case need free haha granddaddy expel uncomplicated ofan

today buy great ya seeds configure sowable wordddd

me wait cool yeah average cook disconnected soz

day love bad lmao pervert violate democra c lololololol

screen gonna nice wow hugs deploy doub"ul eiiishhh

people make long hey orphan redirect inappropriate yayayayay

app upgrade happy damn swimsuit bleed prac camente wujuuu

charger sell big yo chauffeur impersonate unrecoverable oooou

camera feel fast omg paradigm reign heartbroken naaaaaw

Highest degree nodes (words) Lowest degree nodes (words)

FIGURE 2. Graphical example of the words co-occurring with the query compound.

(e.g., I, have, the, this, and on). Figure 2 shows part of thegenerated co-word network where all the nodes co-occur withthe queries nodes (red nodes). The thickness of the edges areproportionate to the actual edge weight. Similarly, the size ofeach node represents its relative degree.

Table 4 illustrates the actual results from the proposedmethodology on 10 sample social media messages whose prefer-ences associated to the target product features are implicit. Thetable lists actual Twitter messages, target features, manual inter-pretation (by the authors) and the resulting top 3 relevant key-words (out of 10 keywords returned by the system), classifiedby parts of speech. Only nouns (N), verbs (V), and adjectives(A) are showed since the combination of these words are mostlysufficient in order to interpret the explicit semantic behind eachmessage.

From the results, it is seen that the combination of the topwords returned by the system could potentially provide explicitmeaning of the implicit message. For example, the meaningbehind “I can’t express how much I love theprice of iPhone 5 on black Friday” may inferthat the user would like to buy and iPhone 5 today (which maybe a Black Friday) because the price is cheap. Similarly, theuser who posts “eh DroidRazr HD resolution? Idon’t think so.” may convey that the display of his/herDroid Razr is bad, and needs to be upgraded.

Most traditional semantic interpretation techniques includ-ing sentiment analysis assume that documents are explicit, andwould fail when dealing with these implicit social media mes-sages. The Column “Sentiment Level (From Implicit Context)”shows quantified sentiment level using the algorithm described


TABLE 3. Statistics of the Twitter data used in this paper, classifiedby smartphone products.

ModelNum

Tweets

Num

Users

Num

Features

Feature

U liza on

Feature

Intensity

Average

Feature

Diversity

BlackBerry Bold 9900 308 252 126 1.7460 0.7143 0.0219

Dell Venue Pro 96 64 50 1.8800 0.9792 0.0380

HP Veer 143 110 76 1.7632 0.9371 0.0265

HTC ThunderBolt 1157 851 335 2.5522 0.7390 0.0071

iPhone 3G 2154 1874 532 2.8459 0.7029 0.0050

iPhone 3GS 3803 3119 775 3.0361 0.6187 0.0041

iPhone 4 68860 43957 6057 5.2196 0.4591 0.0010

iPhone 4S 63500 39145 5922 6.0750 0.5666 0.0010

iPhone 5 211311 124461 13493 7.8739 0.5028 0.0006

iPhone 5C 5533 4475 833 5.1477 0.7750 0.0046

iPhone 5S 15808 12417 1962 5.9210 0.7349 0.0023

Kyocera Echo 52 42 22 1.3636 0.5769 0.0877

LG Cosmos Touch 23 20 11 1.4545 0.6957 0.1313

LG Enlighten 18 17 5 1.6000 0.4444 0.2000

Motorola Droid RAZR 2535 1981 593 3.4840 0.8150 0.0056

Motorola DROID X2 471 378 162 2.1790 0.7495 0.0134

Nokia E7 26 18 14 1.2143 0.6538 0.0879

Nokia N9 208 153 83 1.7470 0.6971 0.0224

Samsung Dart 29 28 10 1.5000 0.5172 0.1071

Samsung Exhibit 4G 23 22 10 1.2000 0.5217 0.1333

Samsung Galaxy Nexus 5218 2988 1147 3.2476 0.7139 0.0031

Samsung Galaxy S 4G 188 152 62 2.0161 0.6649 0.0293

Samsung Galaxy S II 4599 3517 801 3.1436 0.5475 0.0042

Samsung Galaxy Tab 3989 2578 884 3.1912 0.7072 0.0033

Samsung Infuse 4G 284 215 85 2.2000 0.6585 0.0192

Sony Ericsson Xperia Play 481 325 132 1.9394 0.5322 0.0148

T-Mobile G2x 83 69 39 1.4359 0.6747 0.0351

in Section 3.1.2 on the original tweets. The actual EmotionalStrength scores are in parentheses. The Column “Manual Senti-ment Evaluation” lists the manual evaluation by the authors onthe actual sentiment that each sample tweet infers towards thetarget product features (either Positive or Negative). The Column“Sentiment Level (From Translated Explicit Context)” shows thesentiment level using the same sentiment extraction algorithm,but on the translated explicit content generated by concatenatingthe top 20 keywords returned by the system into a single text(disregarding parts of speech). Surprisingly, the sentiment levelscomputed on the translated text agree with the manual evaluationin 7/10 samples (as shown in red bold font).

Not surprisingly, the sentiment level extracted from the orig-inal texts are all incorrect, since the sentiment extraction tech-nique is designed to detect explicit sentiment, and hence wouldnot give correct results when dealing with sarcasm or vague con-text. It is also interesting to note that the sentiment computedfor the implicit sample messages tend to be neutral (SentimentLevel ≈ 0), regardless of the fact that they are composed withemotion-inspired words (i.e., love, can’t, shit, beautifully, etc.).This agrees with prior findings that messages with implicit sen-timent (i.e. sarcasm) would be sentimentally neutralized sincesuch messages tend to have equally high volumes of both Posi-tive and Negative scores, causing the Emotion Strength score to

converge to 0 [46].Despite the promising preliminary qualitative results shown

above, the authors are aware that rigorous quantitative analysisinvolving human subjects identifying whether each keyword orcombination of keywords returned by the system is actually use-ful. However, with large scale data sets like the one used in thispaper, such quantitative evaluations would take quite an amountof human and time resource, and will be performed separately asa future work.

5 Conclusions and Future WorksThis paper proposes a knowledge based methodology for in-

ferring explicit sense from social media messages whose con-notations related to products/product features are implicit. Themethodology first generates a co-word network from the corpusof social media messages, which is used as the knowledge sourcethat captures the relationship among all the words expressed inthe stream of large scale social media data. A set of mathe-matical formulations are proposed in order to identify a com-bination of keywords that would best infer explicit connotationto a given implicit message query. A case study of real-world27 smartphone models with 31 months’ worth of Twitter data ispresented. The results of selected examples show great promisesthat the proposed methodology is effective in translating implicitproduct preferences to their explicit equivalent connotation thatcould be readily used in further knowledge extraction applica-tions such as synthesizing product features [8], predicting futureproduct demands and long-term product longevity [7], and iden-tifying innovative users in online communities [9]. Future workscould strengthen the evaluation process by involving user stud-ies, and verify the generalizability of the proposed methodologyby examining diverse case studies of different product domainsand social media services. Machine learning approaches suchas [47] will also be explored.

References[1] Sakaki, T., Okazaki, M., and Matsuo, Y., 2010. “Earth-

quake shakes twitter users: real-time event detection by so-cial sensors”. In Proceedings of the 19th international con-ference on World wide web, WWW ’10, ACM, pp. 851–860.

[2] Caragea, C., McNeese, N., Jaiswal, A., Traylor, G., Kim,H., Mitra, P., Wu, D., Tapia, A., Giles, L., Jansen, B., et al.,2011. “Classifying text messages for the haiti earthquake”.In Proceedings of the 8th International Conference on In-formation Systems for Crisis Response and Management(ISCRAM2011).

[3] Tuarob, S., Tucker, C. S., Salathe, M., and Ram, N., 2014.“An ensemble heterogeneous classification methodologyfor discovering health-related knowledge in social mediamessages”. Journal of Biomedical Informatics.

[4] Tuarob, S., Tucker, C. S., Salathe, M., and Ram, N., 2013.“Discovering health-related knowledge in social media us-


TABLE 4. Sample results of 10 implicit product-related tweets.

Nouns Verbs Adjec ves

1I can't express how much I love the

price of iPhone 5 on black Fridayprice Price is cheap. Posi!ve Neutral (0) today sale price need wait buy good free cheap Posi ve (3)

2I have to squint the screen to read

this on Nokia N9screen

Screen should be

bigger.Nega!ve Neutral (0)

screen angle

iphone

need fix

upgradebigger larger extra Nega ve (-1)

3

@markgurman I passed on the iPhone

5 to get a FINGERPRINT SENSOR?!

Apple can't innovate shit.

fingerprint

sensor

Fingerprint sensor is not

ground-breaking.Nega!ve Neutral (0)

fingerprint sensor

scanner

need wait

upgrade

wrong bad

disappoin!ngNeutral (0)

4

At last my iPhone 5 is l ighter. It

was about time apple l istened to our

complaints of sore arms due to its

weight

weight iPhone 5 is light. Posi!ve Nega!ve (-1)apple screen

weightneed wait lose happy light thin Nega!ve (-1)

5

I LOVE when my iPhone 5 charger

stops working #deepsarcasm Have to

use my moms iPhone

charger Charger is broken. Nega!ve Posi!ve (1)charger ba#ery

lifehate wait dislike broken stupid bad Nega ve (-1)

6

I just ran my verizon xperia play

over with the ambulance and not a

scratch on the body…

body Body is strong. Posi!ve Neutral (0) case screen body love need work

impressed

indestructable

hard

Posi ve (1)

7eh DroidRazr HD resolution? I don't

think so.resolu!on Resolu!on is not good. Nega!ve Neutral (0)

display screen

camera

upgrade cut

support

bad low

panoramicNega ve (-2)

8

Maybe it's just me, but the new

Nokia N9 looks like a giant blue

iPod Nanolooks

Nokia N9 resembles a

big blue iPod Nano.

(with a hint of

disappointment)

Nega!ve Posi!ve (1)screen look

appearancefeel touch guess nice alike blue Posi!ve (1)

9

Just sitting on my iPhone 4 and the

screen already craked!?!? . . .this week

has been just beatifully spectacular.screen Screen is fragile. Nega!ve Posi!ve (1) iphone case screen crack suck hate damn bad extra Nega ve (-1)

10Here's A Phone I Wouldn't Mind

Getting--Nokia N9Nokia N9 I want Nokia N9. Posi!ve Neutral (0)

case screen

cameraneed want buy

good great

awesomePosi ve (1)

Sen ment Level

(From Translated

Explicit Context)

Manual

Sen ment

Evalua on

Sample

Number

Explicit Transla on

Tweet

Target

Product

Feature

Manual Context

Interpreta on

Sen ment Level

(From Implicit

Context)

ing ensembles of heterogeneous features”. In Proceedingsof the 22Nd ACM International Conference on Conferenceon Information & Knowledge Management, CIKM ’13,ACM, pp. 1685–1690.

[5] Bollen, J., Mao, H., and Zeng, X., 2011. “Twitter moodpredicts the stock market”. Journal of Computational Sci-ence, 2(1), pp. 1–8.

[6] Zhang, X., Fuehres, H., and Gloor, P., 2012. “Predictingasset value through twitter buzz”. Advances in CollectiveIntelligence 2011, pp. 23–34.

[7] Tuarob, S., and Tucker, C. S., 2013. “Fad or here to stay:Predicting product market adoption and longevity usinglarge scale, social media data”. In Proc. ASME 2013 Int.Design Engineering Technical Conf. Computers and Infor-mation in Engineering Conf.(IDETC/CIE2013).

[8] Tuarob, S., and Tucker, C. S., 2015. “Quantifying productfavorability and extracting notable product features usinglarge scale social media data”. Journal of Computing andInformation Science in Engineering.

[9] Tuarob, S., and Tucker, C. S., 2014. “Discovering next gen-eration product innovations by identifying lead user pref-erences expressed through large scale social media data”.In Proceedings of ASME International Design Engineer-ing Technical Conferences & Computers and Informationin Engineering Conference 2014, ASME.

[10] Tuarob, S., and Tucker, C. S., 2015. “Automated discoveryof lead users and latent product features by mining largescale social media networks”. Journal of Mechanical De-sign.

[11] Maynard, D., and Greenwood, M. A., 2014. “Who caresabout sarcastic tweets? investigating the impact of sarcasmon sentiment analysis”. In Proceedings of LREC.

[12] Tsur, O., Davidov, D., and Rappoport, A., 2010. “Icwsm-agreat catchy name: Semi-supervised recognition of sarcas-tic sentences in online product reviews.”. In ICWSM.

[13] Davidov, D., Tsur, O., and Rappoport, A., 2010. “Semi-supervised recognition of sarcastic sentences in twitter andamazon”. In Proceedings of the Fourteenth Conferenceon Computational Natural Language Learning, Associationfor Computational Linguistics, pp. 107–116.

[14] Muecke, D. C., 1982. Irony and the Ironic. Methuen.[15] Gibbs, R. W., 1986. “On the psycholinguistics of sarcasm.”.

Journal of Experimental Psychology: General, 115(1),p. 3.

[16] Gibbs, R. W., and Colston, H. L., 2007. Irony in languageand thought: A cognitive science reader. Psychology Press.

[17] Asur, S., and Huberman, B. A., 2010. “Predicting the fu-ture with social media”. In Web Intelligence and IntelligentAgent Technology (WI-IAT), 2010 IEEE/WIC/ACM Inter-national Conference on, Vol. 1, IEEE, pp. 492–499.

[18] Stone, T., and Choi, S.-K., 2014. “Visualization tool for in-terpreting user needs from user-generated content via textmining and classification”. In ASME 2014 InternationalDesign Engineering Technical Conferences and Comput-ers and Information in Engineering Conference, Ameri-can Society of Mechanical Engineers, pp. V02AT03A009–V02AT03A009.

[19] Zhao, W. X., Jiang, J., Weng, J., He, J., Lim, E.-P., Yan, H.,


and Li, X., 2011. “Comparing twitter and traditional mediausing topic models”. In Advances in Information Retrieval.Springer, pp. 338–349.

[20] YaJuan, D., WEIF uRu, C. Z., Heung, Z. M., and SHUM,Y., 2012. “Twitter topic summarization by ranking tweetsusing social influence and content quality”. In Proceedingsof the 24th International Conference on Computational Lin-guistics, pp. 763–780.

[21] Wang, Y., Wu, H., and Fang, H., 2014. “An explorationof tie-breaking for microblog retrieval”. In Advances inInformation Retrieval. Springer, pp. 713–719.

[22] Thelwall, M., Buckley, K., and Paltoglou, G., 2011. “Sen-timent in twitter events”. J. Am. Soc. Inf. Sci. Technol.,pp. 406–418.

[23] Kucuktunc, O., Cambazoglu, B. B., Weber, I., and Fer-hatosmanoglu, H., 2012. “A large-scale sentiment analy-sis for yahoo! answers”. In Proceedings of the fifth ACMinternational conference on Web search and data mining,WSDM ’12, ACM, pp. 633–642.

[24] Weber, I., Ukkonen, A., and Gionis, A., 2012. “Answers,not links: extracting tips from yahoo! answers to addresshow-to web queries”. In Proceedings of the fifth ACMinternational conference on Web search and data mining,WSDM ’12, ACM, pp. 613–622.

[25] Blei, D. M., Ng, A. Y., and Jordan, M. I., 2003. “La-tent dirichlet allocation”. J. Mach. Learn. Res., 3, Mar.,pp. 993–1022.

[26] Paul, M. J., and Dredze, M., 2011. A model for miningpublic health topics from twitter. Tech. rep.

[27] Paul, M. J., and Dredze, M., 2011. “You are what youtweet: Analyzing twitter for public health.”. In ICWSM,pp. 265–272.

[28] Ramage, D., Dumais, S. T., and Liebling, D. J., 2010.“Characterizing microblogs with topic models.”. ICWSM,10, pp. 1–1.

[29] Prier, K. W., Smith, M. S., Giraud-Carrier, C., and Han-son, C. L., 2011. “Identifying health-related topics on twit-ter”. In Social computing, behavioral-cultural modelingand prediction. Springer, pp. 18–25.

[30] Jin, O., Liu, N. N., Zhao, K., Yu, Y., and Yang, Q.,2011. “Transferring topical knowledge from auxiliary longtexts for short text clustering”. In Proceedings of the 20thACM international conference on Information and knowl-edge management, ACM, pp. 775–784.

[31] Gonzalez-Ibanez, R., Muresan, S., and Wacholder, N.,2011. “Identifying sarcasm in twitter: a closer look”. InProceedings of the 49th Annual Meeting of the Associationfor Computational Linguistics: Human Language Tech-nologies: short papers-Volume 2, Association for Compu-tational Linguistics, pp. 581–586.

[32] Reyes, A., Rosso, P., and Veale, T., 2013. “A multidimen-sional approach for detecting irony in twitter”. LanguageResources and Evaluation, 47(1), pp. 239–268.

[33] Ahlqvist, T., 2008. Social media roadmaps: exploring thefutures triggered by social media. VTT.

[34] Thelwall, M., Buckley, K., Paltoglou, G., Cai, D., and Kap-pas, A., 2010. “Sentiment in short strength detection in-formal text”. J. Am. Soc. Inf. Sci. Technol., 61(12), Dec.,pp. 2544–2558.

[35] Guo, W., Li, H., Ji, H., and Diab, M. T., 2013. “Linkingtweets to news: A framework to enrich short text data insocial media.”. In ACL (1), Citeseer, pp. 239–249.

[36] Ramaswamy, S. “Comparing the efficiency of two cluster-ing techniques”.

[37] Fox, E., 2008. Emotion science: cognitive and neurosci-entific approaches to understanding human emotions. Pal-grave Macmillan.

[38] Cutting, D., Kupiec, J., Pedersen, J., and Sibun, P., 1992.“A practical part-of-speech tagger”. In Proceedings of theThird Conference on Applied Natural Language Process-ing, ANLC ’92, Association for Computational Linguistics,pp. 133–140.

[39] Ozgur, A., Cetin, B., and Bingol, H., 2008. “Co-occurrencenetwork of reuters news”. International Journal of ModernPhysics C, 19(05), pp. 689–702.

[40] Jia, S., Yang, C., Liu, J., and Zhang, Z., 2012. “Animproved information filtering technology”. In FutureComputing, Communication, Control and Management.Springer, pp. 507–512.

[41] Tuarob, S., Mitra, P., and Giles, C. L., 2012. “Improving al-gorithm search using the algorithm co-citation network”. InProceedings of the 12th ACM/IEEE-CS Joint Conferenceon Digital Libraries, JCDL ’12, ACM, pp. 277–280.

[42] Tuarob, S., Bhatia, S., Mitra, P., and Giles, C., 2013. “Auto-matic detection of pseudocodes in scholarly documents us-ing machine learning”. In Document Analysis and Recog-nition (ICDAR), 2013 12th International Conference on,pp. 738–742.

[43] Evans, D. A., Handerson, S. K., Monarch, I. A., Pereiro, J.,Delon, L., and Hersh, W. R., 1998. Mapping vocabulariesusing latent semantics. Springer.

[44] Tuarob, S., Pouchard, L. C., and Giles, C. L., 2013. “Au-tomatic tag recommendation for metadata annotation us-ing probabilistic topic modeling”. In Proceedings of the13th ACM/IEEE-CS Joint Conference on Digital Libraries,JCDL ’13, ACM, pp. 239–248.

[45] Tuarob, S., Pouchard, L., Mitra, P., and Giles, C., 2015.“A generalized topic modeling approach for automatic doc-ument annotation”. International Journal on Digital Li-braries, pp. 1–18.

[46] Thelwall, M., 2013. “Heart and soul: Sentiment strengthdetection in the social web with sentistrength”. Cyberemo-tions, pp. 1–14.

[47] Tuarob, S., Strong, R., Blomberg, J., Chandra, A., Chowd-hary, P., Oh, S., and Tucker, C., 2015. “Automatic discov-ery of service name replacements using ledger data”. InServices Computing (SCC), 2015 IEEE International Con-ference on, IEEE.


a product feature inference model for...

Documents