rae

Supporting multimodal media recommendationand annotation using social network analysis

Adam Rae

[email protected]

Supervisors Stefan Rüger, Suzanne Little, Roelof van ZwolDepartment The Knowledge Media InstituteStatus Full TimeProbation Viva AfterStarting Date October 2007

Research Hypothesis

By analysing and extracting information from the social graphs de-scribed by both explicit and implicit user interactions, like thosefound in online media sharing systems like Flickr1, it is possibleto augment existing non-social aware recommender systems andthereby significantly improve their performance.

Large scale web based systems for sharing media continue to tackle the problemof helping their users find what they are looking for in a timely manner. To dothis, lots of good quality metadata is required to sift through the data collectionto pick out exactly those documents that match the information need of theuser. In the case of finding images from the online photo sharing website Flickr,this could be from over 4 billion examples. How can we help both the systemand the user in enriching the metadata of the media within the collection inorder to improve the experience for the user and to reduce the burden on theunderlying data handling system? Can modelling users, by themselves andwithin the context of the wider online community help? Can this modeling beused to improve recommender systems that improve the experience and reducecognitive burden on users?

Existing approaches tend to treat multimedia in the same way they havedealt with text documents in the past, specifically by treating the textual meta-data associated with an image as a text document, but this ignores the inherentlydifferent nature of the data the system is handling. Images are visual data, andwhile they can be described well by textual metadata, they cannot be describedcompletely by it. Also, the user cannot be ignored in the retrieval process, andlearning more about a user provides information to the system to tailor results totheir specific requirements. Users interact online, and these interactions form a

1http://www.flickr.com/

2010 CRC PhD Student Conference

Page 91 of 125

new type of data that has yet to be fully explored nor exploited when modellingusers.

The work presented here combines the mining of social graphs that occur inFlickr with visual content and metadata analysis to provide better personalisedphoto recommender mechanisms and the following experiment and its analysisare a major component in my overall thesis.

Interaction ScenarioIn order to address this research question, multiple experiments have been car-ried out, one of which I present here:

Envisage an incoming stream of photos made available to a user. Insystems of a scale similar to Flickr, this could be thousands of im-ages per second. Can a system that uses cues from the social, visualand semantic aspects of these images perform better than one thatuses the more traditional approach of using only semantic informa-tion, according to specifically defined objective metrics? How doesperformance vary between users?

An experiment was carried out that mines data from the social communities inFlickr, from the visual content of images and from the text based metadata anduses a machines learning mechanism to merge these signals together to form aclassifier that, given a candidate image and prospective viewing user, decideswhether the user would label that image as a ‘Favourite’2 - see Figure 1.

Related Work

The significant influence that our peers can have on our behaviour online hasbeen studied by researchers such as Lerman and Jones[3], and the particularinteraction that occurs between users and visual media in particular in thework of Nov et al.[4]and Kern et al[2]. Their insights into the importance ofunderstanding more about a user in order to best fulfil their information needsupports the hypothesis that this kind of information can be usefully exploitedto improve systems that try to match that need to a data set supported by socialinteraction. Here I extend their ideas by incorporating this valuable social datainto a complementary multimodal framework that takes advantage of multipletypes of data.

The use of social interaction features in the work of Sigurbjörnsson and vanZwol[7] and Garg and Weber[1] inspired my more comprehensive feature setand its analysis. Their findings that aggregating data generated from onlinecommunities is valuable when suggesting tags is important and I believe alsotransfers to recommendation in general as well as to the specific task of recom-mending images. In fact, I demonstrated this in previous work on social mediatag suggestion[6].

I use some of the human perception based visual features outlined in thework of San Pedro and Siersdorfer[5], as these have been shown to work wellin similar experimental scenarios and cover a range of visual classes. I extendthem further with a selection of other high performing visual features.

2A binary label Flickr users can use to annotate an image they like.


Page 92 of 125

User BMember of urban animals group

User AHas tagged beaches before

Incoming stream of previously unseen candidate images

Potential Favourite Imagesfor User A

Potential Favourite Imagesfor User B

TextualUser information User information

Social

Trained Classifier

Feature Extraction

Visual

Figure 1: Diagram of the image classification system used with Flickr data.

Experimental Work

400 users of varying levels of social activity were selected from Flickr and their‘Favourite’ labelled images collected. This resulted in a collection of hundredsof thousands of images. To train my classifier, these images were treated aspositive examples of relevant images. I generated a variety of negative examplesets to reflect realistic system scenarios. For all photo examples we extractedvisual and semantic features, and social features that described the user, theowner of the photo, any connection between them as well as other behaviourmetrics. We then tested our classifier using previously unseen examples andmeasured the performance of the system with a particular emphasis on theinformation retrieval metric of precision at 5 and 10 to reflect our envisaged usecase scenario.

Results

An extract of the results from the experiment are shown in Table 1. They canbe summarised thus:

• It is possible to achieve high levels of precision in selecting our positiveexamples, especially by using social features. This performance increaseis statistically significantly higher than the baseline Textual run. Thesesocial signals evidently play a significant rôle when a user labels an imagea ‘Favourite’ and can be usefully exploited to help them.

• The value of individual types of features is complex, but complementary.The combined systems tend to perform better than the individual ones.

• It is far easier to classifier photos that are not ‘Favourites’ than those thatare, as shown by the high negative values. This can be used to narrowdown the search space for relevant images by removing those that areobviously not going to interest the user, thus reduing load on both theuser and the system.


Page 93 of 125

System Accuracy + Prec. + Rec. - Prec. - Rec.Textual 0.87 0.48 0.18 0.88 0.97Visual 0.88 1.00 0.09 0.88 1.00Social 0.92 0.80 0.56 0.94 0.98

Textual+Visual 0.88 0.62 0.27 0.90 0.97Textual+Social 0.92 0.77 0.62 0.94 0.97Visual+Social 0.93 0.89 0.56 0.94 0.99

Text+Vis.+Soc. 0.93 0.84 0.62 0.94 0.98

Table 1: Accuracy, precison and recall for various combinations of features usingthe experiments most realistic scenario data set. Photos labelled as ‘Favourites’are positive examples, and those that are not are negative examples. Highernumbers are better.

• As is typical in this style of information retrieval experiment, we can trade-off between precision and recall depending on our requirements. As we areinterested in high precision in this particular experiment, we see that thecombination of the Visual+Social and Text+Visual+Social runsgive good precision without sacrificing too much recall.

References

[1] Nikhil Garg and Ingmar Weber. Personalized, interactive tag recommenda-tion for flickr. In Proceedings of the 2008 ACM Conference on Recommender

Systems, pages 67–74, Lausanne, Switzerland, October 2008. ACM.

[2] R. Kern, M. Granitzer, and V. Pammer. Extending folksonomies for imagetagging. In Workshop on Image Analysis for Multimedia Interactive Services,

2008, pages 126–129, May 2008.

[3] Kristina Lerman and Laurie Jones. Social browsing on flickr. In Proceedings

of ICWSM, December 2007.

[4] Oded Nov, Mor Naaman, and Chen Ye. What drives content tagging: thecase of photos on flickr. In Proceeding of the twenty-sixth annual SIGCHI

conference on Human factors in computing systems, pages 1097–1100, Flo-rence, Italy, 2008. ACM.

[5] Jose San Pedro and Stefan Siersdorfer. Ranking and classifying attractive-ness of photos in folksonomies. In WWW, Madrid, Spain, April 2009.

[6] Adam Rae, Roelof van Zwol, and Börkur Sigurbjörnsson. Improving tagrecommendation using social networks. In 9th International conference on

Adaptivity, Personalization and Fusion of Heterogeneous Information, April2010.

[7] Roelof van Zwol. Flickr: Who is looking? In IEEE/WIC/ACM Inter-

national Conference on Web Intelligence, pages 184–190, Washington, DC,USA, 2007. IEEE Computer Society.


Page 94 of 125

The effect of Feedback on the Motivation of Software

Engineers

Rien Sach

[email protected]

Supervisors Helen Sharp

Marian Petre

Department/Institute Computing

Status Fulltime

Probation viva After

Starting date October 2009

Motivation is reported as having an effect on crucial aspects of software engineering

such as productivity (Procaccino and Verner 2005), software quality (Boehm 1981),

and a project’s overall success (Frangos 1997). Feedback is a key factor in the most

commonly used theory in reports published on the motivation of software engineers

(Hall et al. 2009), and it is important that we gain a greater understanding of the effect

it has on the motivation of software engineers.

My research is grounded in the question “What are the effects of feedback on the

motivation of software engineers?”, and focuses on feedback conveyed in human

interactions. I believe that before I can focus my question further I will need to begin

some preliminary work to identify how feedback occurs, what types of feedback

occur, and the possible impact of this feedback.

Motivation can be understood in different ways. For example, as a manager you might

consider motivation as something you must maintain in your employees to ensure

they complete work for you as quickly as possible. As an employee you might

consider motivation as the drive that keeps you focused on a task, or it might simply

be what pushes you to get up in the morning and go to work.

Herzberg (1987) describes motivation as “a function of growth from getting intrinsic

rewards out of interesting and challenging work”. That’s quite a nice definition; and

according to Herzberg motivation is intrinsic to one’s self. Ryan and Deci (2000)

describe intrinsic motivation as “the doing of activity for its inherent satisfaction

rather than for some separable consequence” (Page 60).

Herzberg (1987) defines extrinsic factors as movement and distinguishes it from

motivation, stating that “Movement is a function of fear of punishment or failure to

get extrinsic rewards”. Ryan and Deci (2000) state that “Extrinsic motivation is a

construct that pertains whenever an activity is done in order to attain some separable

outcome”.

There are 8 core motivational theories (Hall et al. 2009) and some of the theories

focus on motivation as a “a sequence or process of related activities” (Hall et al. 2009)

called process theories, while others focus on motivation “at a single point in time”

(Couger and Zawacki 1980) called content theories.


Page 95 of 125

As reported in a systematic literature review conducted by Beecham et al (2007), and

their published review of the use of theory inside this review in 2009 (Hall et al 2009),

the three most popular theories used in studies of motivation in Software Engineering

were Hackman and Oldman’s Job Characteristics Theory (68%), Herzberg’s

Motivational Hygiene Theory (41%), and Maslow’s Theory of Needs (21%)1.

Hackman and Oldman’s Job Characteristics Theory focuses on the physical job, and

suggests five characteristics (skill variety, task identity, task significance, autonomy,

and feedback) that lead to three psychological states which in turn lead to higher

internal motivation and higher quality work. Herzberg’s Hygiene Theory suggests that

the only true motivation is intrinsic motivation, and this leads to job satisfaction,

where extrinsic factors are only useful in avoiding job dissatisfaction.

One of the five key job characteristics in Hackman and Oldman’s theory is feedback.

Feedback is not explicitly mentioned in Herzberg’s Motivational Hygiene Theory, but

he notes that it is a part of job enrichment, which he states is “key to designing work

that motivates employees” (Herzberg 1987). However this is a managerial view point.

Software Engineers are considered to be current practitioners working on active

software projects within the industry. This includes programmers, analysts, testers,

and designers who actively work and produce software for real projects in the real

world.

From a management perspective, gaining a greater understanding of what motives

employees could prove invaluable in increasing productivity and software quality, and

from an individual perspective the prospect of being given feedback that motivates

you and makes your job more enjoyable and improves the quality of your work

experience could lead to a more successful and enjoyable work life.

My proposed research is divided into stages. In the first stage I plan to conduct

interviews and diary studies to identify the types of feedback in software engineering

and how feedback is experienced by software engineers. I then plan to conduct

additional studies to identify what impact this feedback has on software engineers and

how that impact is evident. Finally, I plan to observe software engineers at work to

see feedback in context, and to compare those observations to the information

gathered during the first two stages.

At the end of my PhD I hope to accomplish research that leads to a greater

understanding of what feedback is inside software engineering and how it is given or

received. Subsequently I wish to gain an understanding of how this feedback alters the

motivation of software engineers and how this manifests as something such as

behaviour, productivity or attitude.

1 The percentages are a representative of how many of 92 papers the theories were found to be

explicitly used in. There can be multiple theories used in any one paper, and the 92 papers were part of

a systematic literature review conducted by Hall et al (2007) sampling over 500 players.


Page 96 of 125

References

B.W. Boehm, Software Engineering Economics, Prentice-Hall, 1981.

COUGER, J. D. AND ZAWACKI, R. A. 1980. Motivating and Managing Computer Personnel.

John Wiley & Sons.

S.A. Frangos, “Motivated Humans for Reliable Software Products,” Microprocessors and

Microsystems, vol. 21, no. 10, 1997, pp. 605–610.

Frederick Herzberg, One More Time: How Do You Motivate Employees? (Harvard Business

School Press, 1987).

J. Procaccino and J.M. Verner, “What Do Software Practitioners Really Think about Project

Success: An Exploratory Study,” J. Systems and Software, vol. 78, no. 2, 2005, pp. 194–203.

Richard M. Ryan and Edward L. Deci, “Intrinsic and Extrinsic Motivations: Classic Definitions

and New Directions,” Contemporary Educational Psychology 25, no. 1 (January 2000): 54-67.

Tracy Hall et al., “A systematic review of theory use in studies investigating the motivations of

software engineers,” ACM Trans. Softw. Eng. Methodol. 18, no. 3 (2009): 1-29.

Sarah Beecham et al., “Motivation in Software Engineering: A systematic literature review,”

Information and Software Technology 50, no. 9-10 (August 2008): 860-878.


Page 97 of 125