using information composition to represent...
TRANSCRIPT
Using Information Composition to
Represent Connections Among
Events Across Time and Place
Rhema LinderInterface Ecology Lab
Texas A&M University
College Station, Texas
Andruid KerneInterface Ecology Lab
Texas A&M University
College Station, Texas
Permission to make digital or hard copies of all or part of this
work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or
commercial advantage and that copies bear this notice and the
full citation on the first page. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee.
CHI’12, May 5–10, 2012, Austin, Texas, USA.
Copyright 2012 ACM 978-1-4503-1016-1/12/05...$10.00.
AbstractWe present information composition as a medium forreflecting on events across space and time, combiningsocial media and personal interpretive annotation. Socialmedia sites contain a wealth of information rich in quality,but they are often information islands. Aggregating theseexperiences tempered with metadata and personalannotation enables readers to reflect on events frompersonal and cultural perspectives. We present Gulf Coasthurricanes Katrina, Rita, and Ike as an aggregate ofpersonal impressions from Flickr, using an informationcomposition to curate heritage memory.
Author Keywordsinformation composition; reflection-in-action; distributedexperience; social media
ACM Classification KeywordsH.5.m [Information interfaces and presentation (e.g.,HCI)]: Miscellaneous.
IntroductionTechnologies play a fundamental role in the sociotechnicalconstruction of individual and collective memory [12].Collecting individual impressions is easy, but connectingimpressions across time and place to understand and buildcomplex mental models of important, personal, and
cultural perspectives is less directly supported. Naturaldisasters are historically and personally important [8]events that, in modern times, are recorded in social mediaas disconnected individual experiences.
Finding relationships and common occurrences betweenevents is a scientific and human endeavor. Bergsonexplains humans as starting in an infant state, neitherself-aware nor aware of external surroundings, except for astream of images constituting sensory input that remainsdisassociated and useless until he or she can learn toremember and perceive [4].
We, as people living in the most recorded world, can peerinto the experiences of many people who leaveimpressions about themselves in social media. This, initself, gives us an ability to possess more sensory datathan we can use, but does not increase our ability toperceive more about cultural memories. Informationcomposition is a medium that a↵ords aggregating theseimpressions from social media.
Figure 1: Extracted metadata of
a Flickr photo containing useful
links and structured information
revealed in context over the
clipping, showing metadata
details on demand.
Information CompositionInformation composition is a lightly constrained mediuminfluenced by design and art for representing things andideas using an expressive spatial arrangement andcompositing of image and text clippings with richmetadata. Each clipping is a conjunction of visualrepresentation and metadata linkback. Manipulating eachclipping’s scale, compositing level, color, and position intoan aggregate enables an author to represent complexrelationships about things and ideas implicitly andexplicitly [6, 13]. As a whole, it a↵ords big picture ideationand reflection; constituent clippings function indexically,visualizing metadata when brushed over with the mouse(see Figure 1) and functioning as hyperlinks to their
source documents. The whole is greater than the sum ofits parts, yet, at the same time, includes all of its parts.
We created a tool for authoring information compositionscalled InfoComposer. InfoComposer is an open sourcecreativity support tool that helps users ideate, collect, andreflect on things and ideas [2]. Users can useInfoComposer to collect text and images from a browserthough drag and drop, which automatically extractssource specific (e.g. Flickr, Wikipedia, ACM) metadata[7] from the source of clippings, presenting a visualizationof the structured metadata as a user on brushes over itwith the cursor.
Suppose Kara creates an information composition. Shestarts with an empty composition space where she dragsin clippings from a web browser and authors personalannotations. She starts with some preconceived notionsabout the themes, information sources, and end productshe wants to create. While collecting information, shereflects on the metadata, nature, and quality of theclippings in her composition. She gathers similar clippingsinto groups and juxtaposes clippings to show contrast.She manipulates clippings, adds annotations, and thinksabout how her actions interact with what her compositionmeans as a whole. Moving her mouse over a clippinginstantly reveals its metadata, showing the source of theclipping and related documents (see Figure 1). Shiftingher actions and attention cyclicly between ideation,design, exploration, and reflection helps Kara create andformulate a narrative of her own thoughts. As shemanipulates her clippings and annotations into anunderstandable cohesive whole, she produces twoproducts: the formulation and framing of her thought andalso the product of the information composition itself.
This iterative and reflective design and thought process
exemplifies Schon calls Reflection-in-Action [10]. Eachlocalized action that the person performs has an e↵ect onthe composition as a whole, potentially changing themesor transferring emphasis and narrative. Repeating thisprocess constitutes a conversation with the situation andmaterials. Schon argues that making progress in a designbegins with first naming problems (in our case themes),and then framing the context to solve them (in our casesupporting documents and the clippings that representthem) [10]. Nakakoli et al. designed and tested ART, alightly constrained two dimensional externalization fordesign, finding that it reduced cognitive load andfacilitated reflecting on design and themes in writing tasks[9]. In contrast to ART, InfoComposer can be used toboth ideate with a fluid thinking structure and produce avisually appealing, rich in metadata, final product. Tuandescribes seeing as a creative process that requires aperson to organize stimuli into “flowing structures thatprovide signs meaningful to the purposeful organism” [11].The mind’s eye naturally visualizes relationships betweenthings in flowing mental structures. During authoring,InfoComposer a↵ords creating flowing structures whilereflecting-in-action. The structures that the author usesfor continuous feedback are the same that a reader canuse when viewing an information composition.
Figure 2: As a whole, the
information composition medium
a↵ords visceral big picture
ideation; each constituent
clipping contains indexical
information available though in
context metadata details on
demand.
The form of a gestalt and visceral composition serves as akind of map of thoughts. Turnbull finds all maps to be atleast partially indexical: needing human contextualinformation to be considered true [3]. The metadatabound to clippings in a composition provides indexicalfunction. In maps, humans select and interpretinformation to present meaningful ideas, focusingattention, and memory. By both explicitly and implicitlyconnecting elements, the information composition authorprovides the reader with an environment of stimuli for
forming opinions and understandings, supporting creativeseeing and exploration of the parts and whole.
Events as Collective ExperiencesAs social networks grow in size and content, individualimpressions of experiences will continue to be created,indexed, archived, and, perhaps, revisited [12]. Theconnections modeled through social media are imperfectbinary representations of complex relationships neither asdynamic nor as qualitative as reality.
Bergson addresses the problem of understanding therelationship between the inner and the outer by describinghuman experience as sensory input understood bymemory. A person’s ability to perceive is tightly coupledwith his or her ability to remember. Infants lack theability to understand sensory input [4] until they haverepeated exposure to sensation which they use to formmental models of self and the meaning of sensory input.
The relationship between social media observation andevents is like the relationship between our minds and theworld as observed by our central nervous system. Ournerves send information to our brains about the world,and from infancy we gradually build an understandingabout our own body and the world. At first, only rough,non-localized understandings of sensation are all that weaccomplish as infants. In the same way, unstructuredcollections of social media can have too little signal buriedunder too much noise. However, even with a personalizedand focused social media signal, lack of organization andcontextualization of the media yields diminishedperceptible meaning. Events experienced through multiplepeople and view points are best represented whencontextualized for meaning and juxtaposed with multiplerelated, aligned, and oppositional impressions.
One can choose to view social media as a stream ofexperiences from individual viewpoints. Each of theseviewpoints and contexts provide a signal. Social mediaimpressions are particularly situated because eachimpression is typically tied to an individual’s general onlinepresence. Comments from friends, strangers, and groupsattached to an impression enhance its social context.
Figure 3: A selection of photos
taken the same day by the same
author of Hurricane RitaDestruction[5] found by clicking
”photos that day” on extracted
metadata.
Figure 4: The location of
Hurricane Rita Destruction found
by clicking ”place” on extracted
metadata.
Information composition is suitable for representingcollections of individual experiences from social media. Aninformation composition is less rigid and linear than atraditional narrative, but more of an aggregate and storythan lists of social media impressions. Informationcompositions contain rich metadata on clippings, whichcannot be represented by an image alone. Informationcomposition a↵ords the reflective exploration andrepresentation for curation and understanding of eventsacross space and time while allowing authors to buildvisualized mental models and personalized annotations.
Use Case: Gulf Coast HurricanesIn Figure 5, we depict Katrina, Rita, and Ike with aninformation composition of individual impressions fromFlickr[1]. While these hurricanes occurred in di↵erenttimes and places, they are similar events. Note that threetendrils, one for each hurricane, extend from the centralimage: a photograph of a Hurricane from space. Eachtendril is separated by whitespace, connecting to theimage of a hurricane in the middle. We implicitly signifythe connectedness of similar damage e↵ects and socialreactions while acknowledging that each hurricaneoccurred separately. The circular and dynamic structure ofthe whole hints at the destructive force of hurricanes andsoftens chronological ordering.
We authored multiple layers to a↵ord focused comparison
to readers of this information composition. The reader isinvited to compare and study tendril to tendril, a clippingwithin a tendril to that same tendril, and any othercombination of aggregate and individual clipping. Whiletimes and places are heterogeneous, the clippings hereshare a common theme of hurricane disaster in the GulfCoast area. We see the same story of waiting, a storm, aflood, left over disaster, and clean up e↵orts juxtaposed assimilar events. Shared experience, community, andcleanup are layered together across each hurricane toimplicitly state that disaster can bring people togetherthrough need. The individuals who took the photos andposted them on Flickr probably do not know each otherpersonally, but they have a shared experience of hurricanedisaster: a heritage memory.
Viewing this work lets one view the rich, structuredmetadata juxtaposed in context as the mouse brushesover a clipping, a↵ording the exploration of personalcontext for each impression and related information. Forexample, the image Hurricane Rita Destruction has ageographical “place” link, which allows one to see otherphotos taken in that location (see Figures 3 and 4).Metadata fields often contain links to relevant web pagesthat include related information: the authors page, photostaken by that author that day, month, or year, and groupsor sets that that picture belongs to.
Figure 5: We present Heritage of Gulf Coast Disaster as an information composition as an example of representing heritage of events
across space and time as experiential aggregates from multiple perspectives. To see metadata and sources for the individual clippings,
please go to http://students.cs.tamu.edu/rhema/compositions/gulf.html.
ConclusionSocial media is a new sense. The information it conveys istrue to the same degree that one’s central nervous systemconveys true information. Information composition a↵ordscomparison and contextualization needed to align,represent, and reflect on related events across time andplace. The information composition medium supportsreflection during authoring and viewing processes, helpingusers develop perspectives of heritage memories.
References[1] Flickr - Photo Sharing. http://www.flickr.com.[2] Info Composer Website.
http://www.ecologylab.net/infocomposer.[3] Turnbull, David, and Watson-Verran, Helen, and
Deakin University. Maps are territories, science is anatlas : a portfolio of exhibits / David Turnbull ; witha contribution by Helen Watson with the Yolngucommunity at Yirrkala. Deakin University :distributed by Deakin University Press, Geelong, Vic.:, 1989.
[4] H. Bergson. Matter and Memory. SwanSonnenschein, London, 1911.
[5] B. Eyes and Bluebonnets. Hurricane RitaDestruction. http://www.flickr.com/photos/
almostlindy/1292493060/.[6] A. Kerne, E. Koh, S. M. Smith, A. Webb, and
B. Dworaczyk. combinformation: Mixed-initiativecomposition of image and text surrogates promotesinformation discovery. ACM Trans. Inf. Syst.,27:5:1–5:45, December 2008.
[7] A. Kerne, Y. Qu, A. M. Webb, S. Damaraju,N. Lupfer, and A. Mathur. Meta-metadata: ametadata semantics language for collection
representation applications. In Proceedings of the19th ACM international conference on Informationand knowledge management, CIKM ’10, pages1129–1138, New York, NY, USA, 2010. ACM.
[8] S. B. Liu. Grassroots heritage in the crisis context: asocial media probes approach to studying heritage ina participatory age. In Proceedings of the 28th of theinternational conference extended abstracts onHuman factors in computing systems, CHI EA ’10,pages 2975–2978, New York, NY, USA, 2010. ACM.
[9] K. Nakakoji, Y. Yamamoto, S. Takada, and B. N.Reeves. Two-dimensional spatial positioning as ameans for reflection in design. In Proceedings of the3rd conference on Designing interactive systems:processes, practices, methods, and techniques, DIS’00, pages 145–154, New York, NY, USA, 2000.ACM.
[10] D. A. Schon. The Reflective Practitioner: HowProfessionals Think In Action. Basic Books, 1edition, Sept. 1984.
[11] Y.-f. Tuan. Space and place : the perspective ofexperience / Yi-Fu Tuan. University of MinnesotaPress, Minneapolis :, 1977.
[12] N. Van House and E. F. Churchill. Technologies ofmemory: Key issues and critical perspectives.Memory Studies, 1(3):295–310, Sept. 2008.
[13] A. M. Webb and A. Kerne. Integrating implicitstructure visualization with authoring promotesideation. In Proceedings of the 11th annualinternational ACM/IEEE joint conference on Digitallibraries, JCDL ’11, pages 203–212, New York, NY,USA, 2011. ACM.