The Elements of Fashion Style

Download The Elements of Fashion Style

Post on 02-Jan-2017




2 download

Embed Size (px)


<ul><li><p>The Elements of Fashion Style</p><p>Kristen Vaccaro, Sunaya Shivakumar, Ziqiao Ding, Karrie Karahalios, and Ranjitha KumarDepartment of Computer Science</p><p>University of Illinois at Urbana-Champaign{kvaccaro,sshivak2,zding5,kkarahal,ranjitha}</p><p>USER INPUT STYLE DOCUMENT TOP ITEMS</p><p>I need an outfit for a beach wedding that I'm going to early this summer. I'm so excited -- it's going to be warm and exotic and tropical... I want my outfit to look effortless, breezy, flowy, like Im floating over the sand! Oh, and obviously no white! For a tropical spot, I think my outfit should be bright and colorful.</p><p>beachweddingsummertropicalexoticeffortlessbreezyglowingradiantfloatingflowywarmbrightcolorful</p><p>Figure 1: This paper presents a data-driven fashion model that learns correspondences between high-level styles (like beach, flowy, and wedding)and low-level design elements such as color, material, and silhouette. The model powers a number of fashion applications, such as an automated personalstylist that recommends fashion outfits (right) based on natural language specifications (left).</p><p>ABSTRACTThe outfits people wear contain latent fashion concepts cap-turing styles, seasons, events, and environments. Fashion the-orists have proposed that these concepts are shaped by de-sign elements such as color, material, and silhouette. Whilea dress may be bohemian because of its pattern, material,trim, or some combination thereof, it is not always clear howlow-level elements translate to high-level styles. In this pa-per, we use polylingual topic modeling to learn latent fashionconcepts jointly in two languages capturing these elementsand styles. This latent topic formation enables translationbetween languages through topic space, exposing the ele-ments of fashion style. The model is trained on a set of morethan half a million outfits collected from Polyvore, a popularfashion-based social network. We present novel, data-drivenfashion applications that allow users to express their desires innatural language just as they would to a real stylist, and pro-duce tailored item recommendations for their fashion needs.</p><p>Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from</p><p>UIST 2016, October 16 - 19, 2016, Tokyo, Japan 2016 Copyright held by the owner/author(s). Publication rights licensed to ACM.ISBN 978-1-4503-4189-9/16/10. . . $15.00</p><p>DOI:</p><p>Author KeywordsFashion, elements, styles, polylingual topic modeling</p><p>ACM Classification KeywordsH.5.2 User Interfaces; H.2.8 Database Applications</p><p>INTRODUCTIONOutfits contain latent fashion concepts, capturing styles, sea-sons, events, and environments. Fashion theorists have pro-posed that important design elements color, material, sil-houette, and trim shape these concepts [24]. A long-standing question in fashion theory is how low-level fashionelements map to high-level styles. One of the first theorists tostudy fashion as a language, Roland Barthes, highlighted thedifficulty of translating between elements and styles [2]:</p><p>If I read that a square-necked, white silk sweater is verysmart, it is impossible for me to say without again hav-ing to revert to intuition which of these four features(sweater, silk, white, square neck) act as signifiers forthe concept smart: is it only one feature which carriesthe meaning, or conversely do non-signifying elementscome together and suddenly create meaning as soon asthey are combined?</p><p>To address this fundamental question, we present a model thatlearns the relation between fashion design elements and fash-ion styles. This model adapts a natural language processingtechnique polylingual topic modeling to learn latent</p><p>{kvaccaro, sshivak2, zding5, kkarahal, ranjitha}@illinois.edu</p></li><li><p>fashion concepts jointly in two languages: a style languageused to describe outfits, and an element language used to la-bel clothing items. This model answers Barthes question:identifying the elements that determine styles. It also powersautomated personal stylist systems that can identify peoplesstyles from an outfit they assemble, or recommend items fora desired style (Figure 1).</p><p>We train a polylingual topic model (PLTM) on a dataset ofover half a million outfits collected from a popular fashion-based social network, Polyvore1. Polyvore outfits have bothfree-text outfit descriptions written by their creator and itemlabels (e.g., color, material, pattern, designer) extracted byPolyvore. These two streams of data form a pair of paralleldocuments (style and element, respectively) for each outfit,which comprise the training inputs for the model.</p><p>Each topic in the trained PLTM corresponds to a pair of dis-tributions over the two vocabularies, capturing the correspon-dence between style and element words (Figure 2). For exam-ple, the model learns that fashion elements such as black,leather, and jacket are often signifiers for styles such asbiker, and motorcycle.</p><p>We validate the model using a set of crowdsourced, percep-tual tasks: for example, asking users to select the set of wordsin the element language that is the best match for a set in thestyle language. These tasks demonstrate that the learned top-ics mirror human perception: the topics are semantically co-herent and translation between elements and styles is mean-ingful to users.</p><p>This paper motivates the choice of model, describes both thePolyvore outfit dataset as well as the training and evaluationof the PLTM, and illustrates several resultant fashion applica-tions. Using the PLTM, we can explain why a clothing itemfits a certain style: we know whether it is the collar, color,material, or item itself that makes a sweater smart. We canbuild data-driven style quizzes, predicting style preferencesfrom a users outfit. We even describe an automated personalstylist which can provide outfit recommendations for a de-sired style expressed in natural language. Polylingual topicmodeling can help us better our understanding of fashion the-ory, and support a rich new set of interactive fashion tools.</p><p>MOTIVATIONAs more fashion data has become available online, re-searchers have built data-driven fashion systems that processlarge-scale data to characterize styles [6, 11, 12, 20, 23],recommend outfits [21, 22, 27, 32], and capture changingtrends [8, 10, 29]. Several projects have used deep learn-ing to automatically infer structure from low-level (typically)vision-based features [15, 28]. While these models can pre-dict whether items match or whether an outfit is hipster,they cannot explain why. For many applications, modelspredicated on human-interpretable features are more usefulthan models that merely predict outcomes [9]. For example,when a user looks for a party outfit, an explanation likethis item was recommended because it is a black miniskirthelps her understand the suggestion and provide feedback tothe system.</p><p>topic distribution</p><p>1 21 2</p><p>STYLEsummer, vintage, beach, american, relaxed, retro, unisex </p><p>ELEMENTshort, denim, highwaisted, shirt, top, cutoff, form, distressed</p><p>STYLEbiker, motorcycle, vintage, summer, college, varsity, military </p><p>ELEMENTjacket, black, leather, shirt, zip, denim, sleeve, faux</p><p>STYLEprom, occasion, special, party, holiday, bridesmaid</p><p>ELEMENTdress, shoe, cocktail, evening, mini, heel, costume</p><p>STYLEparty, summer, night, sexy, vintage, fitting, botanical</p><p>ELEMENTdress, mini, sleeveless, cocktail, skater, flare, out, lace, floral</p><p>1</p><p>2</p><p>1</p><p>2</p><p>topic distribution</p><p>Figure 2: Via polylingual topic modeling, we infer distributions over latentfashion topics in outfits that capture the elements of fashion style. Fashionelements like jacket, black, leather signify the biker, motorcycle style.Conversely, fashion styles like prom, special occasion label groups ofelements such as cocktail, mini, dress.</p><p>This paper presents a fashion model that maps low-level el-ements to high-level styles, adapting polylingual topic mod-eling to learn correspondences between them [18]. Both setsof features (elements and styles) are human interpretable, andthe translational capability of PLTMs can power applicationsthat indicate how design features are tied to user outcomes,identifying peoples styles from the elements in their outfitsand recommending clothing items from high-level style de-scriptions.</p><p>In addition to their translational capabilities, PLTMs offer anumber of other advantages. Unlike systems built on discrim-inative models [21, 27, 32], PLTMs support a dynamic set ofstyles that grows with the dataset and need not be specifieda priori. Moreover, topic modeling represents documents asdistributions over concepts, allowing styles to coexist withinoutfits rather than labeling them with individual styles [6, 11,12, 20]. Finally, the model smooths distributions so that sys-tems can support low frequency queries. Even though thereare no wedding outfits explicitly labeled punk rock inour dataset, we can still suggest appropriate attire for such anevent by identifying high probability fashion elements associ-ated with wedding (e.g., white, lace) and punk rock(e.g., leather, studded), and searching for clothing itemswhich contain them.</p><p>To build a PLTM for fashion, we require data that containsboth style and element information. Researchers have stud-ied many sources of fashion data, from independent fash-</p></li><li><p>Happy Valentines DayHappy Valentines Day! Have a nice time with your boyfriends, and dont forget about people who are alone (like me). The next few days will be in tones of romance, couples, blush colors. Have a nice weekend! Send warm hugs and love. #valentinesday #personalstyle #sweaterweather </p><p>Red cardigan,Long sleeve tops,Mango tops</p><p>Short sleeve shirts,White t shirt,Lightweight shirt,Mango shirt</p><p>Stack heel shoes,Oxford shoes</p><p>Retro sunglasses,Heart sunglasses,Hippie glasses</p><p>happy, love, hugs blush, valentines</p><p>boyfriends, warm couples, romance </p><p>alone, weekend retro, hippie </p><p>valentinesday personalstyle </p><p>sweaterweather</p><p>STYLE</p><p>red, short, sleeve shirts, white, tshirt </p><p>mango, oxford lightweight, stack tops, heel, shoes </p><p>sunglasses, heart cardigan, long</p><p>ELEMENT</p><p>POLYVORE OUTFIT PLTM DOCUMENTS</p><p>Figure 3: Polyvore outfits (left) are described at two levels: high-levelstyle descriptions (e.g., #valentinesday) and specific descriptions of theitems design elements (e.g., red cardigan, lightweight shirt). For eachoutfit, we process these two streams of data into a pair of parallel docu-ments (right).</p><p>ion items [3] to objects with rough co-occurrence informa-tion [15] to entire outfits captured in photographs [10, 11,31]. Each source has its own strengths, but most require pars-ing, annotation, or the use of proxy labels. We take advantageof Polyvores coordinated outfit data, where each outfit is de-scribed in both a low-level element language and a high-levelstyle one.</p><p>FASHION DATAPolyvore is a fashion-based social network with over 20 mil-lion users [25]. On Polyvore, users create collections of fash-ion items which they collage together. Such collages are com-mon in fashion: mood boards are frequently used to com-municate the themes, concepts, colors and fabrics that willbe used in a collection [24]. True mood boards are rarelywearable in a real sense, but on Polyvore collages typicallyform a cohesive outfit.</p><p>Polyvore outfits are described at two levels: specific descrip-tions of the items design elements (e.g., black, leather,crop top) and high-level style descriptions, often of the out-fit as a whole (e.g., punk). We leverage these two streams ofdata to construct a pair of parallel documents for each outfit,which become the training inputs for the PLTM.</p><p>Polyvore Outfit DataPolyvore outfit datasets contain an image of the outfit, a ti-tle, text description, and a list of items the outfit comprises(Figure 3). Titles and text descriptions are provided by usersand often capture abstract, high-level fashion concepts: theuse of the outfit; its appropriate environment, season, or evenmood. In addition, each outfit item has its own image andelement labels provided by Polyvore (Figure 3, bottom left).These labels are typically low-level descriptions of the itemsdesign elements, such as silhouette, color, pattern, material,trim, and designer.</p><p>We collected text and image data for 590,234 outfits using asnowball sampling approach [7] from Polyvores front page,sampling sporadically over several months between 2013 and2015. Our collection includes more than three million uniquefashion items, with an average of 10 items per outfit. Wecollected label data for 675,699 of those items, resulting in arepository of just over 4 million item labels.</p><p>Representing Outfits in Two LanguagesWith the outfit and item data collected, we create two vocab-ularies to process outfits into parallel style and element docu-ments. The style vocabulary is created by extracting termsfrom the repositorys text data relating to style, event, oc-casion, environment, weather, etc. Most of these words aredrawn from the text produced by Polyvore users since theyannotate outfits using high-level descriptors; however, wealso include Polyvore item labels that describe styles (e.g.,retro sunglasses). We manually process the 10,000 mostfrequent words from the title and description text to identifywords that should be added to the style vocabulary, keepinghashtags such as summerstyle and discarding common En-glish words that are irrelevant to fashion.</p><p>The element vocabulary is drawn from the repositorys set ofPolyvore item labels. We learn frequent bigrams, trigramsand quadgrams such as Oscar de la Renta or high heelsvia pointwise mutual information [14]. The element vocab-ulary comprises these terms and any remaining unigram la-bels not added to the style vocabulary. After processing therepositorys text, the style vocabulary has 3106 terms and theelement vocabulary 7231.</p><p>Using these vocabularies, we process each outfits text datainto a pair of parallel documents: one containing words fromthe style vocabulary, and a second containing words from theelement vocabulary (Figure 3, right). Both documents de-scribe the same set of items in two different languages: anoutfit might be goth in the style language, but the wordsused to describe it in the element language might be black,velvet, and Kambriel. These parallel documents becomethe training input for the PLTM.</p><p>FASHION TOPICSTo capture the correspondence between the fashion styles andelements exposed by the Polyvore dataset, we adapt polylin-gual topic modeling. Polylingual to...</p></li></ul>