writing & reading workshop

Writing & Reading Workshop

Today´s programme

Workshop on: writing experiment reports how to read research papers

– expect to be challenged ...

Will not go through pensum in detail – focus on utilizing your acquired knowledge Ask questions as we get to each section of

the reports

Today´s programme

Exercise: Writing a review of Law et al.: ”Understanding,

Scoping and Defining User Experience: A Survey Approach” Following the guidelines in Philip Fong: ”How to Read a CS

Research Paper?”

We will do the first parts in steps throughout the day, the latter parts you will complete at home and hand in next week

Some points may be to answer because they require knowledge of the field of user experience – take your best shot or skip them.

Practical information

We have been approved (horray!) with breaking from 11.45 to 12.45 – same exercise room (4A58)

Re. the 28/9 – heard from 2 students – weren´t there 3???

Today´s test if you read the curriculum: Handout by Graham Hole is exact copy of chapter 9 in the book – how many discoverd this?

Handouts for exercises on the course website

The game lecture series

Center for Computer Games Research Game Lecture Series

Runs on Thursdays at 15.00 around every 2 weeks in the semester

Talks of interest to you: 24th of September: Janus Sørensen from

IO Interactive on user testing 19th November: Lennart Nacke on bio-

physiological measures of user experience

Paper review - assignment

1-2 page review following the recipe of Philip Fong

1) Summary of the paper in your own words

2) Evaluation of the work Is the research problem significant? Are the contributions significant? Are the claims valid?

3) Synthesis: summarise your thoughts on the paper

Writing scientific reports


Reports are based on the idealised scientific journal article

Reports are divided into sections which provide a specific piece of information each

This standard format is used in virtually all the sciences (not humanities) and with minor modification, the industry

Being able to write reports is a key skill


Purpose:

To communicate research findings to others in a clear, systematic and standardised way.

Clear: So we get it ...Systematic and standardized: So

we can quickly read the paper, know where things are

Sections

Sections of a report:

Title Abstract (summary) Introduction Method: Participants Design Procedure Results Discussion/Conclusion References Appendices

10-20%

50-70%

15-25%

Aims of a report

1. "Why?". Why did I do this particular experiment? What did I expect to find out by doing it? This question is dealt with in the Introduction.

2. "How?". How did I actually carry it out - what procedures and apparatus did I use?

This question is covered in the Method section.

3."What?". What did I find? What were my results? This information is provided in the results section.

4."So What?" What does it all mean? How do my results relate to previous research on the same topic, and what are their theoretical implications? What are my conclusions?These issues are all dealt with in the Discussion and Conclusion sections.

Other components

Title Provides quick idea what the report is

about

Abstract Summary of the contents and main

conclusions Should keep the attention of the reader

References Documents arguments in report and

provides means for locating further knowledge

Title

Ask yourself: Do you want anyone to actually read the report?

If yes, use a title that catches people´s attention If no, use a really long and boring title

Informative but succinct (10-12 words maximum) - e.g.:

“Gender differences in attitudes to "Bambi" amongst Navaho Indians”

“The effects of nationality and age on sun-bed claiming behaviour”

Report titles ...

Abstract

An abstract is a brief summary of the report

Typically 120-150 words

Give reader: (a) what you did: Quick overview of

contents (b) what you found: The main results (c) what it means: Theoretical/practical

implications

Typically easiest to write the last

Keywords

Typically 3-5 keywords of the report Nominally provides the reader another quick

idea about the contents + is used as search words in databases/browsers

Consider One thing is what the paper is about, another

is what the current buzzwords in your field is Ensure you know which keywords to use to

make people find/not find your report in library/company databases

Executive Review

Used business reports and white papers, or experiment reports in the industry

The same thing as an abstract, but tuned to the specific audience/stakeholder Design department Management Shareholders Etc.

Introduction

Introduces the reader to the overall problem area first, then focuses on the specific problem under investigation

General problem introduction and statement (why?), e.g.: - gaps in previous knowledge; - methodological flaws in previous work.

Review of current work in the field Discussion of any relevant literature and theories Overview of the report contents

Brief outline of our experiment, and its possible outcomes and their implications.

Why are we bothering to do our study?

The METHOD section (dun-dun!)

Method

Describes how the experiments were run

Describes what data that were collected and how they were analyzed

Argues for why the approach is correct

Introduction + hypotheses Design Participants Apparatus Procedure

Method

Provide enough detail that experiment can be

replicated, but not any irrelevant information

Hypotheses

In HCI research papers, there will normally be specific hypotheses and research questions that are sought answered

These are normally – not always - described in the method section

In commercial HCI reports, often there are no hypotheses, but specific purposes with the test/experiment (sort of the same thing...)

Participants

Who took part in the experiment?What were their characteristics?

(demographics – age, gender, nationality, experience)

Volunteers or paid?Randomly selected or not?How were they allocated to the

different conditions of the experiment?

Experiment conditions

Typically we have a control group and an experiment group

The experiment group we do something with – ”Will this pill turn people purple?”

The control group we do NOT submit to an experiment – they permit us to measure the effect of the experiment reliably

Materials (apparatus, equipment...)

Also called ”setup” in many HCI situations

Basically describes the equipment of the experiment + measurement instruments

Equipment: computers, watches, cameras, software etc.

Instruments: surveys, pulse watches etc.

Diagrams and photographs are often used

Procedure

Explain how the experiment was carried out

Usually pretty detailed with exact description of what was done to the participants, and what they had to do

Order of the tests being administered

Temporal frame: How long? When? Etc.

Results

An objective walkthrough of what results that have come out of the experiment

Do not interpret numbers here – just report them

Only critical raw data and summary statistics are included in the report ”76% of people turned purple...” The rest is in appendices

Results

Two overall parts in the results

1) Descriptive statistics These describe datasets ”35 samples, mean weight 2 kgs, SD=0.2

kgs”

2) Inferential statistics These allow us to answer research

questions ”An ANOVA shoved that the iron meteors

had statistically different weights than ice meteors”

Results

Descriptive statistics

Start by providing details for each group or experiment condition, e.g. Means

Follow up with inferential statistics (the results of statistical tests), to decide whether there was any difference between the groups or conditions

Results

inferential statistics ...

State which test you used The value of the ”test statistic”Number of degrees of freedom

(where appropriate)Significance level

We will get back to these – just remember that you

need this stuff in the results!

Results

Example:

”An independent-means t-test was performed. This showed that participants who had received 40kg of the drug ‘Coffee' recalled significantly fewer words than those who had consumed 20 mg (t (29) = 3.65, p < . 001)"

Statistical test

Significance level

Test statistic value

Results

If you have many results, summarise them in tables and graphs

Do not dublicate information (in both tables and graphs)

All graphs and tables should be clearly labeled with self-explanatory titles and legends (i.e. no need to read the main text to understand what the figure shows)

Results

Figure X: An initial analysis of the causes of death in Fragile Alliance as function of time. The figure charts the percentage distribution of kills for AI-controlled autonomous agents and players.

Axis labels + measurementunits Graph label

Title

Figure text

Clear data presentation

Discussion

Interpretation of the results What do the results mean? Explain what the statistics mean

Relation to other works What have other people found?

Impact What is the importance?

Critical reflection Criticize your own work, indicate possible

weaknesses, design flaws, limits of application of results?

Research agenda (what now?)

Discussion: Research agenda

Worthwhile suggestions for future research? Extension to other situations, to see

if the results are generalisable [applicable to entire population we are interested in]

Replication with a better-matched set of participants.

Explain briefly why these studies would be worth doing.

Conclusion

Can be sub-sumed under discussion

Quick summary of the results, their meaning and importance, and setting of future research agenda.

If these are in conclusion, remove from discussion

Acknowledgements

This section should give thanks to the major people (supervisors, associates) and organizations (sponsoring agencies, funders) that helped you.

Mostly in research articles

For example: “The authors thank Spiderman, who provided constructive feedback on earlier drafts of this report. Also a heartfelt thanks to Captain Kirk, who financed the project”.

Task 1: Research problem You get 1 hour to give the paper a quick

read-through (”skimming”)

Use this time to orient yourself in the paper (hint: start with the abstract), and discuss in the group if there are components you do not understand

After 45 minutes, discuss in the group: What is the research problem/-s the paper attempts to address? (use Fong, section 2 point 1 as a guide)

Sections

Sections of a report:

Title Abstract (summary) Introduction Method: Participants Design Procedure Results Discussion/Conclusion References Appendices

10-20%

50-70%

15-25%

References

ALL papers, reports, artwork or similar was cited in the report, are listed here in full detail

In alphabetical order, very standardized format

E.g.: Buonaparte, N. and Nelson, H. (1805,

March 21). An analysis of sun-bed claiming behavior in Western Europe. Seafaring Weekly, 75, 46-49.

References

References must provide enough informtion for the reader to be able to track it down

(unless you don´t want to of course)

This allows the reader to: Seek further knowledge on the subject Evaluate if you have used the reference

correctly (science is all about mistrust, as Graham and

Andy puts it)

References

There are many referencing formats

Journals, conferences and companies have different standards

Some common standards, like APA, CHI, Science, Chicago, ACM, Harvard etc.

Here we use the APA format – described in detail in our textbook

You must use this in all reports for this course

References

In-text references usually something along the lines of (single-author): (Spock, 2005) Studies by Spock (2005) indicate that ... Studies by Spock (2005, 2007) indicate

that ... It has been claimed that aliens are

illogical (Spock, 2005) The results of Spock (2005) are at odds

with those of Kirk (2004) on the matter of aliens

References

If multiple authors

”Spock and Kirk (2006) agreed that aliens are indeed weird”

If more than 2 authors

”Spock et al. (2007) argue that aliens are green”

Figure 1: Are aliens weird?

References

Multiple references ...

”Aliens have been claimed to be alien (Kirk, 2004; Spock, 2005; Spock et al., 2007)”

References

Primary and secondary references

Primary: Those you have read (ideally, ALL your references

Secondary: Those you quote because it makes you look

smart ... Because someone else referenced them and you

should probably do it too ... (looks important) Because the primary is in Swahili and you don´t

read Swahili To avoid getting bogged down with detail – if

someone else has done the work, quote that -

References

People say you should be honest about it if you use a secondary

reference:

"It has been claimed that aliens can be cured by killing them (Kirk, 1995, as cited in Spock, 2000)".

“Kirk (1995, as cited in Spock, 2000) claimed that aliens can be cured by killing them".

I say: Adopt and adapt ...

References

LEARN TO DO PROPER REFERENCING!

Appendices (research+industry)

Experimental materials Location of programs, instructions, questionnaires, test materials, transcripts of sessions, photos of screen presentations if required, etc.

Pilot study results Preliminary/pilot experiment record, what you did and saw, how and why you modified the experiment.

Appendices (research+industry)

Raw Data Raw data elicited from subjects, verbal comments from subjects, videotapes, etc. Large amounts of data are best kept on discs. Public data should be available on ftp for replication purposes.

Statistical runs/output from softwarePrintouts of your statistical analyses.

Experimental consent forms/participant instructionProof that your subjects consented to do the experiment, and any other relevant documentation

Stimulus material

Summary

A report is build over a standardized model

A report always contains specific sections, each with their specific purpose

Always use a consistent referencing style

Task 2: Contributions – 20 mins.

Re-browse the abstract, introduction and conclusion sections

What contributions do the authors claim to make with the paper? Answering a question? Providing new

knowledge? Evidence for something?

(use Fong, section 2 point 2 as a guide)