Dr Stephen Dann
The Australian National University
Best Journals versus Best Fit Journals
A Strategic Orientation to
Research Quality
Knight to RQF’s Bishop 3
• "What we're looking to do is distribute funding according to research that's of high quality and high impact."
So what’s quality?
• Quality is a nebulous entity– Eye of Beholder Model
• I know what quality looks like when I see it
– Groucho Marx Model• I wouldn’t want to be published in a journal that
would publish me
– Quality is Greener on the Other Side• Whatever they did is better than what we did.
RQF on Metrics
• Recommendation:– Metrics should be used to inform the peer review
process rather than replace it.– This recommendation was elaborated to make the
process for the interaction of metrics and the Discipline Panel assessment process more explicit:
• Metrics should not be used in isolation, such as in the ‘shadow exercise’ proposed for the 2008 UK RAE.
• The data should be made available to Assessment Panels at the beginning of the assessment process.
• No attempt should be made to aggregate indicators to produce a single ‘quality score’.
RQF on Ranked Outputs• Ranked outputs: Distribution of selected category(ies) of research
output listed in the Research Group’s ‘body of work’ into ranked tiers, based on predetermined discipline specific rankings for the selected forms of output.
• The basis of the rankings will vary by discipline (they could be derived for – refereed journals, – professional journals, – book publishers, – conferences, or – performance venues
• No weightings should be applied to the different tiers
RQF on research output
• 2.3 Ranked outputs– the main thrust of this indicator is to
encourage researchers to publish in the most prestigious outlets for their discipline
• We set the rules on what’s prestigious.– So it’s time to set the rules in our favour
RQF on impact• The RQFDAG recommends that verifiable impact indicators
• The Impact Statement, of no more than ten pages, must be verifiable and auditable, and include a statement of claims against the impact criteria. The broad criteria will require a demonstration of:– the engagement with end-users recognising the importance of the
research to address a defined social, economic, environmental and/or cultural issue;
– the adoption of the research by the relevant end-user community to generate new policies, products, , processes, attitudes, behaviours and/ or outlooks;
– how the research has produced social, economic, environmental and/or cultural benefits for end-users regionally, nationally and/or internationally; and
– the extent of the benefit arising from the research.
Background to the Model
• Over 20 years, over 20 different ranking scales
• Once the reader looks outside of the top three or five journals, the inconsistency of these lists stand as a tribute to "one size fits nobody" production orientations– Polonsky and Whitelaw (2004, 2005, 2006)
• If only the only consistency was change– Top 3 unshakeable
Ageing of the (sample) Population
Rank Name D.O.B.
(1) Journal of Marketing (JM) 1937
(2) Journal of Consumer Research (JCR) 1974
(3) Journal of Marketing Research (JMR) 1964
(4) Marketing Science (MS) 1982
(5) Journal of the Academy of Marketing Science (JAMS) 1973
(6) Journal of Retailing (JR) 1925
(7) International Journal of Research in Marketing (IJRM) 1959
(8) Journal of Business Research (JBR) 1978
(9) Journal of Advertising (JA) 1972
(10) Journal of Advertising Research (JAR) 1961
Quadrant Model
Break the old rules to make the new rules
Shifting from the League Table
• the use of a one-dimensional scale of "best" is inconsistent with the fundamental principles of the discipline and the practice of the industry – (Corfman, 1991)
Assumption of the Model
• Research output is assumed to have two core components– the goal of developing knowledge within a
specific discipline (academic freedom)
– the business strategy component which represents the organisational research goals of the University (academic pragmatism)
Discipline Research Goals
• Development of the marketing discipline– Pure marketing– Applied marketing (impact)– Industry focused research. (impact)
• Two forms– Academy driven goals
• Eg Vargo and Lusch, services dominant logic
– Academic research freedom• Personal pursuit of research interests
University Research Goals
• Organisational research goals / objectives– University / Funding body– Faculty / School / Marketing Unit
• AACSB (2006)– intellectual contributions should be consistent
with the school’s mission and strategic management processes
Four categories
1. Developing the discipline of marketing
2. Addressing Institutional Research Goals
3. Research Agenda Based Objectives
4. Independent Research Output
• First draft mistakenly presented these as tiers of objectives, and naturally, it was seen as a best to worst priority list
Discipline and University Research Goals
AcademyDriven
Research(Tier 1)
Marketing UnitResearchAgenda(Tier 3)
AcademyDriven
Research(Tier 1)
Marketing UnitResearchAgenda(Tier 3)
University ResearchObjectives
(Tier 2)
Non aligned Research
(Tier 4
University ResearchObjectives
(Tier 2)
Non aligned Research
(Tier 4
Discipline Research Goals
University Research Goals
Marketing Academy Journals
(Quadrant 1)
Marketing Group
Agenda (Quadrant 3)
Research Objectives
(Quadrant 2)
Non aligned Research
(Quadrant 4)
Discipline Research Goals
University Research Goals
Discipline and University Research Goals
AcademyDriven
Research(Tier 1)
Marketing UnitResearchAgenda(Tier 3)
AcademyDriven
Research(Tier 1)
Marketing UnitResearchAgenda(Tier 3)
University ResearchObjectives
(Tier 2)
Non aligned Research
(Tier 4
University ResearchObjectives
(Tier 2)
Non aligned Research
(Tier 4
Discipline Research Goals
University Research Goals
Marketing Academy Journals
(Quadrant 1)
Marketing Group
Agenda (Quadrant 3)
Research Objectives
(Quadrant 2)
Non aligned Research
(Quadrant 4)
Discipline Research Goals
University Research Goals
engagement with end-users
adoption of the research by the
relevant end-user community
adoption and benefits for relevant end-users
adoption of the research by the relevant end-user
community
Quadrant 1Developing the discipline of marketing
• Research Agenda: – pursuit of the enhancement and development of the discipline
• Theoretical developments• Methodological developments• Pure science approaches• RQF End user is the marketing academy.
• Which journals are listed?– Constitutes a list of the journals published by the respective
academies of marketing across the major geographic areas• Australasia, Europe and America in current list
– Same nature, same cluster
Only list that should remain constant between universities
Quadrant 1 List
• Academy of Marketing Science Review• Australasian Marketing Journal• International Journal of Research in Marketing• Journal of International Marketing• Journal of Marketing• Journal of Marketing Management• Journal of Marketing Research• Journal of Public Policy & Marketing• Journal of the Academy of Marketing Science
Quadrant 2Addressing Institutional Research Goals
• Research Agenda: – framework to encourage academics to pursue the research
interest of the university / funding agency– End user: University stakeholders, university
• Which journals are listed?– Journal inclusion is based on providing a matching the research
strength of the organisation against the market demand for the research output product
• Requires research and hack work
• Should be relatively similar for organisations with shared/common goals– Opportunity to pursue diversity as sought by Bishop & DEST
Sample Quadrant 2
• Sample Institutional Goal– Advance knowledge of Australian research
• Journals selected based on the extent to which they publish Australian research– Method 1: Journal keyword search
• search by university title• Search by “australian/australia”
– Method 2: Michael Polonsky– Polonsky, Garma and Mittelstaedt (2006) “An examination of
the globalisation of authorship in publishing in 20 leading marketing journals” European Business Review 18 (6)
– Method 3: Cross institutional dataset
Sample Quadrant 2
• Key journals– Asia-Pacific Journal of Marketing and Logistics– European Journal of Marketing*– Industrial Marketing Management– Journal of Brand Management (The)– Journal of Business Research– Journal of Consumer Behaviour– Journal of Consumer Marketing (The)– Journal of Product and Brand Management– Marketing Intelligence and Planning
• *Side note: I was once reprimanded by the editorial board of EJM for describing them on ELMAR as “The European Journal of Australians Publishing in Marketing”
Quadrant 3: Research Agenda
• Research Agenda– Diversification based on collective group research agenda
• Collective agenda of is based on the aggregated areas of research specialisations of the staff
• Self reported identification of research interest
– RQF End user: Business, students, academics
• Which journals are listed?– Discipline specific journals– Sub-disciplinary specific journals – Journals with a specific research concentration in an sub
discipline / sub set of marketing
• Should have a wide range of variation between universities
Create the Research Agenda
• Method: Aggregated Research Agenda– Step 1: Have the research staff of the unit self identify their
research areas against the ANZMAC track stream lists• (eg primary research, secondary research, PhD topic, grants
research)
– Step 2: Research streams are tallied to form an aggregate list of research areas, which is used to form a rank order list.
– Step 3: Cluster track streams against either Polonsky and Whitelaw’s (2006) “areas of research interest” or Lehmann (2006) journal categories
Convert the Research Agenda
• Method: Convert Research Agenda into Target Journals– Step 1: Identify Research Agenda Output Journals
• Option 1: Using Lehmann (2006) “Journal Evolution and the Development of Marketing” Journal of Public Policy & Marketing 24(1) pp138-140
• Option 2: Independent research to select journals based on the number of articles containing the relevant research keyword, and the stated research goal of the journal.
• Option 3: Cross institutional dataset of categorised journals
– Step 2: Prioritise research agenda journals according to the match between the research project and the needs of the target market journal
• Is the paper the best match for the objectives of the journal?
Sample Research Agenda
Sum P&W(2006) ANZMAC Track Areas
12 • Marketing & Society
• Corporate Responsibility, Social Issues and Social Responsibility Marketing and Society Non profit marketing Political Marketing Social Marketing
11 • Marketing Strategy
• Entrepreneurship, Firms in Networks, Marketing Strategy, New Product Development, Supply Chain Management, Financial & Value-Based Marketing
10 • Marketing Communications
• Advertising / Marketing Communication / IMC, Branding
10 •Marketing Communications
•Advertising / Marketing Communication / IMC, Branding
• Sample Quadrant based on Lehmann 2006– Journal of Advertising Research – Journal of Advertising – Journal of Current Issues and Research in Advertising – International Journal of Advertising– Journal of Marketing Communications – Journal of Promotion Management
Quadrant 4Independent Research Output
• Research Agenda– Academic freedom.– Personal research agenda that is not constrained by
the discipline, organisation or unit goals– RQF End user: Other academics/practitioners within
the research area
• Which journals are listed?– The rest of the journals
• Should vary between individuals within a group
Quadrant 4• recognition that it is possible for a quality research outlet to not be
relevant to the research agenda of an institution. – Inclusion in Quadrant 4 is not a criticism of the quality of the journal or
the research.
• used to defend the ability of the individual researcher to pursue research beyond the goal of the university, academy or marketing unit agenda.
• Can easily be destroyed if it is used for sniping between marketing academics
• This is not the Fourth Tier (Group of Death) of the UQBS /AoU lists
Supporting materials
Cross Institution Journal Verification
• Cross verification of journals– Potential ANZMAC Collaboration
• Database shared between marketing groups to identify the match between
– common objectives and journals– Research interest clusters and journals
• Creative Commons Licensed (Attribution Share-Alike) model
– allows for institutions to contribute to the dataset without seeking prior permission, but prevents any single entity from capturing the dataset.
Supporting Behaviours• Publicise the research clusters
– Know what is being published where
• Lay down covering fire– Cite each other, Cite early, cite often– Cite ANZMAC papers
• Create an ANZMAC marketing research output index– shows who’s publishing what and where to accelerate the process of cross-citing
• Specify RQF target end user community.– Target academics. Target teaching. Cross cite, and use in the classrooms.
• Cheat to Win: RQF is one of the largest prisoner’s dilemma games invented – so let’s cheat and communicate with each other.
Questions?