1info4990 research methods, s2 2008 info4990 research methods bing bing zhou info4990/ lecture based...

1INFO4990 Research Methods, s2 2008

INFO4990 Research MethodsINFO4990 Research Methods

Bing Bing ZhouBing Bing Zhouhttp://www.cs.usyd.edu.au/~info4990/http://www.cs.usyd.edu.au/~info4990/

Lecture based in part on materials by Alan Lecture based in part on materials by Alan Fekete, Mary Lou Maher, Joseph Davis, Irena Fekete, Mary Lou Maher, Joseph Davis, Irena

Koprinska and othersKoprinska and others

Literature Review and Literature Review and Literature SearchLiterature Search


OutlineOutline Literature reviewLiterature review

why do you need to review the literaturewhy do you need to review the literature goals of literature reviewsgoals of literature reviews

How to searchHow to search Finding relevant papersFinding relevant papers Search practicalitiesSearch practicalities

How to read the literatureHow to read the literature

INFO4990 Research Methods, s2 2008 3

““If I have seen further, it is only If I have seen further, it is only because I have stood on the because I have stood on the

shoulders of giants”.shoulders of giants”.

Attributed to Sir Isaac NewtonAttributed to Sir Isaac Newton


Why do you Need to Review the Why do you Need to Review the Literature?Literature?

• To help you find a research questionTo help you find a research question• What is the current state of the art?What is the current state of the art?• Where are the Where are the gaps or the next steps?gaps or the next steps?

• To help you establish the importance of the research To help you establish the importance of the research questionquestion• Why is this problem important and worth researching?Why is this problem important and worth researching?

• To establish your work as part of the knowledge To establish your work as part of the knowledge base of your research community, e.g.base of your research community, e.g.

• ““[Cite] defined the following concept, about which we [Cite] defined the following concept, about which we prove …”prove …”

• ““[Cite1, cite2, cite3] have all worked on systems like [Cite1, cite2, cite3] have all worked on systems like this”this”

• ““our performance is better than that of [Cite]”our performance is better than that of [Cite]”• To solve your problemTo solve your problem

• E.g. combining ideas from several papersE.g. combining ideas from several papers


Goals of the Literature Goals of the Literature ReviewReview

• Demonstrate that you know the fieldDemonstrate that you know the field• Your research question follows from the Your research question follows from the

literature reviewliterature review

• Justifies your research, provides the Justifies your research, provides the rationale for the researchrationale for the research• how does your work connect to previous workhow does your work connect to previous work• how does your work differ from previous workhow does your work differ from previous work

• Allows you to establish the conceptual Allows you to establish the conceptual framework and methodological focusframework and methodological focus


Literature ReviewLiterature Review• Not just a summary of related papers but a Not just a summary of related papers but a

critical examination of these papers critical examination of these papers • What are their strengths and limitations? What are their strengths and limitations? • How exactly do they relate to your question? How exactly do they relate to your question?

• Should be coherent and structuredShould be coherent and structured• Learning from examplesLearning from examples

• Ask your supervisor to suggest good previous thesis Ask your supervisor to suggest good previous thesis (Honours/Masters/ PhD) and research papers (Honours/Masters/ PhD) and research papers (journal/conference)(journal/conference)

• Many available on-lineMany available on-line• Discuss the literature review section and the overall Discuss the literature review section and the overall

thesis/paper structure together!thesis/paper structure together!


How to Find Relevant How to Find Relevant PapersPapers

• Papers given by your supervisorPapers given by your supervisor• In many cases your research question will be In many cases your research question will be

defined as modification/extension of existing workdefined as modification/extension of existing work

• Papers you found by searching and following Papers you found by searching and following upup• Based on terms of interestBased on terms of interest• Based on links from other work you findBased on links from other work you find

• Reference chainsReference chains• Papers listed as references in an important paper are Papers listed as references in an important paper are

also worth examining also worth examining • Community links – important people, journals, Community links – important people, journals,

conferencesconferences


Communities of InterestCommunities of Interest

• Your work should be targeted at a Your work should be targeted at a communitycommunity• It must be It must be explainedexplained in relation to work in that community in relation to work in that community

• Find who are the important people in the Find who are the important people in the community and where their work is community and where their work is publishedpublished• Find their websites using GoogleFind their websites using Google• What else they have written?What else they have written?• Who are their students?Who are their students?• Do they continue to work on the same topic?Do they continue to work on the same topic?• What are their current projectsWhat are their current projects


Searching for Papers - Some Searching for Papers - Some PracticalitiesPracticalities

• Given a reference to a paper, you need to get the Given a reference to a paper, you need to get the actual contentactual content

• Look onlineLook online • most of the literature is available as pdf even very old most of the literature is available as pdf even very old

papers have been scannedpapers have been scanned• ACM Digital Library – full access via the library’s home ACM Digital Library – full access via the library’s home

pagepage• IEEE Digital Library – full access via the library’s home IEEE Digital Library – full access via the library’s home

pagepage• Publisher’s site, e.g. Springer, Elsevier, etc. – hundreds Publisher’s site, e.g. Springer, Elsevier, etc. – hundreds

of journals accessible via the library’s home pageof journals accessible via the library’s home page• Google ScholarGoogle Scholar• CiteseerCiteseer• Just using Google Just using Google

• with the author’s name and a few rare words from the titlewith the author’s name and a few rare words from the title


Effective Search Effective Search

• We are all used to Google-style searchWe are all used to Google-style search• List words of interestList words of interest

• Make sensible choices for your searchMake sensible choices for your search • prefer words that are not very common in general prefer words that are not very common in general

EnglishEnglish• not much point searching for “design” or “information” or not much point searching for “design” or “information” or

“relation” but “congestion” and “heterogeneity” will work“relation” but “congestion” and “heterogeneity” will work

• often add a term for the field as a whole, e.g. often add a term for the field as a whole, e.g. “database” or “network”“database” or “network”

• Start with a few targeted words and add more if Start with a few targeted words and add more if too much junk is returnedtoo much junk is returned


Effective Search (2) Effective Search (2) • Most search tools offer a variety of features, Most search tools offer a variety of features,

especially important when words of interest are especially important when words of interest are commoncommon

• http://www.google.com.au/intl/en/help/basics.htmlhttp://www.google.com.au/intl/en/help/basics.html• http://www.google.com.au/intl/en/help/refinesearch.htmlhttp://www.google.com.au/intl/en/help/refinesearch.html

• AND (implicit in Google)AND (implicit in Google)• OROR• Exact phrase match (Goggle: “words in exact order”)Exact phrase match (Goggle: “words in exact order”)• Allow synonyms (Google: ~tern)Allow synonyms (Google: ~tern)• Exclude terms (Google: -term) - useful for terms with more than Exclude terms (Google: -term) - useful for terms with more than

one meaning. one meaning. • Include common terms (Google: +term) – common terms are Include common terms (Google: +term) – common terms are

excluded by default, e.g. “how’, where’excluded by default, e.g. “how’, where’


Effective Search (3) Effective Search (3)

• Topic-Specific Searches in GoogleTopic-Specific Searches in Google• Google Book SearchGoogle Book Search - Search the full text - Search the full text

of booksof books• Google Code SearchGoogle Code Search - Search public - Search public

source codesource code• Google ScholarGoogle Scholar - Search scholarly papers - Search scholarly papers

Google News archive searchGoogle News archive search - Search - Search historical newshistorical news


How to Reference Web How to Reference Web DocumentsDocuments

• Some publications exist only on the webSome publications exist only on the web• No formal journal or conference publicationNo formal journal or conference publication• May be TRMay be TR

• Reference them by giving the URL used Reference them by giving the URL used and the date on which it was accessedand the date on which it was accessed• web information changesweb information changes

• Also, be careful when getting a version Also, be careful when getting a version from the author’s web page as it may be from the author’s web page as it may be posted in preliminary not final versionposted in preliminary not final version


Guide to Research Guide to Research LiteratureLiterature

Types of publicationsTypes of publications conference and workshop papersconference and workshop papers journal papersjournal papers technical reports technical reports monographsmonographs


Conference PapersConference Papers Call for papers - ~1 year before meetingCall for papers - ~1 year before meeting Paper submission - ~4-8 months before meetingPaper submission - ~4-8 months before meeting

Page limit e.g. 8 pagesPage limit e.g. 8 pages Details often omitted (proofs, design technicalities)Details often omitted (proofs, design technicalities)

Program Committee reviews the papersProgram Committee reviews the papers Criteria: relevance, significance, originality, soundness, Criteria: relevance, significance, originality, soundness,

readabilityreadability Final version for proceedings due ~3 months Final version for proceedings due ~3 months

before meetingbefore meeting revise by author in light of reviewsrevise by author in light of reviews but not checked againbut not checked again

Annual or bi-annual conferencesAnnual or bi-annual conferences


Selection ProcessSelection Process Typically 3 reviewersTypically 3 reviewers Acceptance rate – variesAcceptance rate – varies

Some 10-15%, others 50%Some 10-15%, others 50% Some review “blind” (author details not Some review “blind” (author details not

shown to reviewers), others do notshown to reviewers), others do not

Ask your supervisor for guidance about Ask your supervisor for guidance about which are the reliable and important which are the reliable and important conferences in your field!conferences in your field!


Workshop PapersWorkshop Papers A workshop is typically a smaller meeting A workshop is typically a smaller meeting

than a conference than a conference Sometimes workshop papers are just like Sometimes workshop papers are just like

conference papersconference papers Other workshops are more preliminaryOther workshops are more preliminary

can publish a position paper (draft of an idea can publish a position paper (draft of an idea without evidence, or proposal for future work)without evidence, or proposal for future work)

less rigorously reviewed, the goal is mainly to less rigorously reviewed, the goal is mainly to allow the community to meetallow the community to meet


Journal ArticleJournal Article Typically longer than a conference paperTypically longer than a conference paper Often based on a conference paper with additions, Often based on a conference paper with additions,

corrections and improvementscorrections and improvements Refereed by Refereed by

at least 3 reviewers, experts in the fieldat least 3 reviewers, experts in the field they spend months on the paper checking details, etc. they spend months on the paper checking details, etc.

Decisions: accepted, accepted with minor revisions, Decisions: accepted, accepted with minor revisions, major revisions and resubmission, rejectedmajor revisions and resubmission, rejected

Revisions, refereed againRevisions, refereed again Accepted, published after several months (journal issues Accepted, published after several months (journal issues

have limited capacity)have limited capacity) Time from submission to publication varies, typically Time from submission to publication varies, typically

1-1.5 years but may be 3-4 years1-1.5 years but may be 3-4 years


Standard of JournalsStandard of Journals Many journals in each area with different Many journals in each area with different

standardsstandards Typically IEEE Transactions and ACM Typically IEEE Transactions and ACM

Communications are some of the top-ranked Communications are some of the top-ranked journalsjournals

Not all IEEE Trans. and ACM Comm. are top Not all IEEE Trans. and ACM Comm. are top journalsjournals

Not all papers published in top journals are good Not all papers published in top journals are good paperspapers

Ask you supervisor which journals are the top-Ask you supervisor which journals are the top-ranked and most important in your area!ranked and most important in your area!


Technical ReportTechnical Report Issued by the author’s department, Issued by the author’s department,

with a number and datewith a number and date May be based on a conference paperMay be based on a conference paper

Longer, includes all the boring details that Longer, includes all the boring details that are omitted from the conference paper are omitted from the conference paper due to space limitationsdue to space limitations

Used to establish priority Used to establish priority E.g. produce TR before submitting to E.g. produce TR before submitting to

conference or journal + conference conference or journal + conference and journal papers may get rejectedand journal papers may get rejected


PhD or MSc ThesisPhD or MSc Thesis Very extensive accountVery extensive account

Show much of the research processShow much of the research process Extensive survey of the literatureExtensive survey of the literature Very complete evaluation of the workVery complete evaluation of the work

The goal is to establish that the author is The goal is to establish that the author is ready to become independent researcherready to become independent researcher

i.e. PhD and MSc provide research trainingi.e. PhD and MSc provide research training Typically checked by 2 or 3 reviewersTypically checked by 2 or 3 reviewers


MonographMonograph A collection of selected papers from a A collection of selected papers from a

conference or workshopconference or workshop A bit more checking than for the A bit more checking than for the

conference/workshopconference/workshop An author can offer a coherent and unified An author can offer a coherent and unified

account of a whole research topicaccount of a whole research topic often combines their own results with other often combines their own results with other

people’speople’s Revisits several papers using unified notation, Revisits several papers using unified notation,

better exposition, better literature review, etc.better exposition, better literature review, etc. Publisher may get reviewers but their focus is Publisher may get reviewers but their focus is

“will it sell” not “is it correct”“will it sell” not “is it correct”


WarningsWarnings There's a whole bunch of There's a whole bunch of

conferences/journals in your community.conferences/journals in your community. You could spend all your time reading them. You could spend all your time reading them.

Fortunately, only a few are worth looking at. Fortunately, only a few are worth looking at. Quality of conferences and journals varies, Quality of conferences and journals varies,

and this is reflected in the checking of the and this is reflected in the checking of the paperspapers

Read papers with a critical eye!Read papers with a critical eye! Some communities are very clique-Some communities are very clique-

dominateddominated Unpopular opinions are not welcomeUnpopular opinions are not welcome Clique leaders can publish anything, even half-Clique leaders can publish anything, even half-

baked ideas without evidencebaked ideas without evidence


Fake Conferences and Fake Conferences and Random PapersRandom Papers http://pdos.csail.mit.edu/scigen/http://pdos.csail.mit.edu/scigen/

A random paper accepted to a journal?A random paper accepted to a journal?


The Research CommunityThe Research Community A community has conferences and journals A community has conferences and journals

of high prestige which they read and of high prestige which they read and publish inpublish in

They meet often, and each knows (more or less) They meet often, and each knows (more or less) what others are doingwhat others are doing

You must place your work in the context of a You must place your work in the context of a communitycommunity


Quality MetricsQuality Metrics How important is an article? How influential is an How important is an article? How influential is an

author?author? Based on citation analysis - number of times a paper or Based on citation analysis - number of times a paper or

author is citedauthor is cited How to calculate citations – Google Scholar + other softwareHow to calculate citations – Google Scholar + other software

Assumption: important authors and articles are cited Assumption: important authors and articles are cited more often than the others more often than the others

Increasingly used by governments, funding bodies, promotion Increasingly used by governments, funding bodies, promotion committees to evaluate the quality of author’s workcommittees to evaluate the quality of author’s work

Some drawbacksSome drawbacks Citing errors – authors with the same names are not separatedCiting errors – authors with the same names are not separated Cliques (friends, colleagues) cite each other in turn to build Cliques (friends, colleagues) cite each other in turn to build

their citation indextheir citation index Negative citations are included (citations to incorrect results)Negative citations are included (citations to incorrect results)


ISI Citation DatabaseISI Citation Database Very popular, established in 1960, contains Very popular, established in 1960, contains

>40million records, contains>40million records, contains Arts and Humanities Citation Index (A&HCI)Arts and Humanities Citation Index (A&HCI) Science Citation Index (SCI)Science Citation Index (SCI) Social Sciences Citation Index (SSCI)Social Sciences Citation Index (SSCI)

HoweverHowever it doesn’t index a large number of journalsit doesn’t index a large number of journals ignores open-access journalsignores open-access journals doesn’t index conferencesdoesn’t index conferences

““Read the Rise and Rise of Citation Analysis” by L. Read the Rise and Rise of Citation Analysis” by L. Meho!Meho!


Journal’s Impact FactorJournal’s Impact Factor Journal impact factorsJournal impact factors

Used to determine the importance of a journalUsed to determine the importance of a journal E.g. journal impact factor for 2007 =E.g. journal impact factor for 2007 =

# citations in 2007 to articles published in the # citations in 2007 to articles published in the journal in 2005-6journal in 2005-6

= -------------------------------------------------------------------------------= -------------------------------------------------------------------------------

# articles published in the journal in 2005-6# articles published in the journal in 2005-6


CORE’s ratingsCORE’s ratings Computing Research and Education Computing Research and Education

Association of Australasia (CORE)Association of Australasia (CORE) Australia and New ZealandAustralia and New Zealand

Ranking of journals and conferences in CS Ranking of journals and conferences in CS – not finalised– not finalised

http://www.core.edu.au/http://www.core.edu.au/


How to Read the How to Read the LiteratureLiterature

• Reading papers is a skill that takes practice. Reading papers is a skill that takes practice. • You can't afford to read in full all the papers that You can't afford to read in full all the papers that

come to you. come to you. • Normally there are three phases to reading one. Normally there are three phases to reading one.

• The first is to see if there's anything of interest in it at The first is to see if there's anything of interest in it at all. Abstracts, the table of contents, conclusion section, all. Abstracts, the table of contents, conclusion section, and introduction are good places to look. and introduction are good places to look.

• Once you've figured out what in general the paper is Once you've figured out what in general the paper is about and what the claimed contribution is, you can about and what the claimed contribution is, you can decide whether or not to go on to the second phase, decide whether or not to go on to the second phase, which is to find the part of the paper that has the good which is to find the part of the paper that has the good stuff. stuff.

• Finally, you may go back and read the whole paper Finally, you may go back and read the whole paper through if it seems worthwhile.through if it seems worthwhile.


How to Read the How to Read the Literature(2)Literature(2)

• With a critical eyeWith a critical eye• Take notes Take notes

• What is the contribution? What is the contribution? • What is the evidence? What is the research method What is the evidence? What is the research method

used?used?• How does this contribution relate to the previous How does this contribution relate to the previous

work?work?• What are the important references cited in the What are the important references cited in the

paper? (follow them up) paper? (follow them up) • Do you have any doubts, questions, ideas? Do you have any doubts, questions, ideas?

• Keep reading the important journals and Keep reading the important journals and conference papers in your fieldconference papers in your field• IT and Eng research ages quicklyIT and Eng research ages quickly

• Maintain annotated bibliography from the Maintain annotated bibliography from the startstart


How to Read the How to Read the Literature(3)Literature(3)

• Don’t rely on second hand summaries! Go to Don’t rely on second hand summaries! Go to the original source if you can!the original source if you can!• Get attributions right in your own writing: don’t Get attributions right in your own writing: don’t

just accept citations from other work, even with just accept citations from other work, even with full reference!full reference!

• When reading a well written paper When reading a well written paper • What makes this paper easy to read?What makes this paper easy to read?• What level of detail is provided?What level of detail is provided?• What examples are used to demonstrate important What examples are used to demonstrate important

concepts?concepts?• What questions are left unanswered (future work)?What questions are left unanswered (future work)?


Organising the Organising the LiteratureLiterature

• Isolate issues and highlight the findings and Isolate issues and highlight the findings and contributions that are central to your researchcontributions that are central to your research

• Group together papers that deal with a common Group together papers that deal with a common or related theme or issueor related theme or issue

• Organise the materials with diagrams, tables, Organise the materials with diagrams, tables, concept mapsconcept maps• Try out different structures for organising; they should Try out different structures for organising; they should

be most relevant to the goals of your researchbe most relevant to the goals of your research• Chronological order is not particularly useful Chronological order is not particularly useful but but

citation chains are usefulcitation chains are useful• WarningWarning

• papers often don’t use common terminology, or focus on papers often don’t use common terminology, or focus on common issues, or explain relationships fairly, clarifying common issues, or explain relationships fairly, clarifying these aspects is a key contribution you can makethese aspects is a key contribution you can make

1info4990 research methods, s2 2008 info4990 research methods bing bing zhou info4990/ lecture based...

Documents

research question

literature slide

research community

6info4990 research methods

2info4990 research methods

4info4990 research methods

5info4990 research methods

1info4990 research methods