Scopus’ Re-evaluation Program

Susanne SteigingaSr. Product Manager Scopus ContentASEP, Moscow, April 2019

Content curation and research integrity at its best

Agenda

• Predatory journals – what are they?

• Ongoing content curation via Scopus’ Re-evaluation program

• The pink elephant in the room

• Q&A

PREDATORY JOURNALS

Low quality or ‘predatory journals’

• ‘Predatory journals’ are ill defined and it can be difficult to determine whether a journal is predatory or not.

• There is controversy as to which journals are considered predatory. In many cases there is a lack of factual evidence and no guidance for that view.

• It is not always straightforward to identify predatory journals, because journals change their editorial policies over time and a journal that did not start off as predatory may become so over the years and vice versa.

• Low quality journals can be easier to identify, but some which may, from a bibliometric perspective, seem of low impact play a role in niche areas or serve a local community.

Therefore it is key that all journals in Scopus arethoroughly analyzed manually by the CSAB. This to

ensure the right decisions are made.

How does this work?

First of all: content enters Scopus after rigorous evaluation by the CSAB based on strict selection criteria

Content Selection and Advisory Board (CSAB)

Once a title is indexed in Scopus, the journal performance will be monitored on an ongoing basis

Structural approach: Ongoing content curation to ensure continuous high quality content

Identification of under performing journals

using metrics and benchmarks

Machine learning tool to identify journals with outlier performance

Direct feedback from users and stakeholders

on publication concerns

Re-evaluation by the Content Selection & Advisory Board (CSAB)

Content Curation

Ongoing curation of the full journal base is expected and essential to keep journals at high quality levels and keep the database trustable

Review:

Curate:

Set metrics and benchmarks all journals have to meet every year. If they do not meet any of these benchmarks for two consecutive years, the journal will be re-evaluated by the CSAB

For more info: https://blog.scopus.com/posts/scopus-checks-balances-maintaining-quality-content-on-scopus

Metric Benchmark Explanation

Self-citation rate ≥ 200%The journal has a self-citation rate two times higher, or more, when compared to peer journals in its subject field.

Total citation rate ≤ 50%

The journal received half the number of citations, when compared to peer journals in its subject field.

CiteScore ≤ 50%The journal has a CiteScore half or less than the average CiteScore, when compared to peer journals in its subject field.

Number of articles ≤ 50%

The journal produced half, or less, the number of articles, when compared to peer journals in its subject field.

Number of full-text clicks on Scopus.com

≤ 50%The journal's abstract are used half as much, or less, when compared to peer journals in its subject field.

Abstract usage on Scopus.com ≤ 50%

The journal's full text are used half as much, or less, when compared to peer journals in its subject field.

Outlier performance: data science is used to identify journal outlier behavior

Radar tool launched & added to Re-evaluation in 2017: data analytics tool which identifies outlier journals based on 40+ predictors.Examples of prediction behavior:

• Sudden and unexplainable change in article output and growth• Sudden and unexplainable shift in geographical diversity among authors and editors• Shift in received citations and percentage of self-citations

• Flagged journals are manually re-evaluated by the CSAB for (dis)continuation of Scopus coverage.

What is outlier behavior?

0

200

400

600

800

1000

2015 2016 2017 2018

Country 2OtherCountry 1

Big increase in article output after debut in Scopus Shift in geographical diversity

Re-evaluation 2018 results broken down: overall discontinuation rate of 51%

263 Titles re-evaluated

100

38

125

53

47

21

17

61

64

Underperformance

Outlier performance

Publication concerns

53% Discontinuation rate

55% Discontinuation rate

49% Discontinuation rate

Since the launch of the program 772 journals have been re-evaluated with a discontinuation rate of 58%.

Should the CSAB decide a title is not meeting the Scopus Title Selection criteria anymore:• The publisher is informed of the situation by Scopus Re-evaluation team.

• For those journals that are discontinued, no new content is accepted nor backfilled.

• Content already indexed will remain as a matter of scientific record and to ensure the stability and consistency of research trend analytics, a core value element of Scopus.

• In exceptional cases of proven severe unethical publication practice, content already indexed in Scopus may be removed.

Title selection criteria: https://www.elsevier.com/solutions/scopus/how-scopus-works/content/content-policy-and-selection

All journals discontinued in Scopus as part of the re-evaluation program are publicly available via the Scopus discontinued sources list.

The pink elephant in the room and question perhaps on your mind too:

Does Scopus index Predatory journals?

Predatory journals in Scopus paper

• In March 2017 the article Predatory Journals in Scopus, identifying

predatory tiles indexed in Scopus, was published by IDEA, the

Institute for Democracy and Economic Analysis, at the Czech

Academy of Science.

• The paper is based on 2015 Scopus data which is used to analyse

publication behaviour.

• All of the 137 suspicious titles mentioned in the paper have gone

through the re-evaluation process. 97 titles (71%) discontinued.

• All identified titles listed by Beall that are mentioned in the paper

(note that Beall’s List is no longer being maintained) have also gone

through the re-evaluation process. 65% discontinued.

Does Scopus index predatory journals?

• As a leading abstract & citation index, it is not surprising that Scopus has been the subject of an increasing number of questions about predatory content.

• The decision to re-evaluate and potentially exclude a journal takes time and is complex, given the ongoing risk of discontinuing legitimate sources.

• Re-evaluation is a state-of-the-art defence against predatory publishing. Scopus takes its

responsibility in terms of curating its content on an ongoing basis and ensures only the

highest quality content remains indexed in the database.

• Scopus, together with the CSAB, are committed to continue to develop new

approaches to ensure the combined quality and breadth of its research data continue to

be unrivalled and can be trusted for your decisions in research.

Thank youQuestions?

Overall re-evaluation (‘16, ‘17, ‘18) results broken down: overall discontinuation rate of 58%

772 Titles re-evaluated

332

68

367

168

164

42

26

237

130

Metrics

Radar

Publication concerns

51% Discontinuation rate

62% Discontinuation rate

65% Discontinuation rate

Low quality or predatory publishing & Scopus

• When making decisions about science, it is essential that these decisions are based on curated, high-quality data that you can trust. Therefore, low quality or predatory journals are a threat to the integrity of science.

• Because predatory publishing is ill-defined and subject to personal interpretation, independent review of individual journals by academic subject experts in each field is essential.

• Scopus has been rigorously addressing this issue for some years now and have developed a process of continuous monitoring and re-evaluation by the independent Content Selection and Advisory Board (CSAB), to determine whether coverage of the journals in Scopus should be continued or not.

Structural approach: Ongoing content curation to ensure continuous high quality content

Identification of under performing journals

using metrics and benchmarks

Machine learning tool to identify journals with outlier performance

Direct feedback from users and stakeholders

on publication concerns

Re-evaluation by the Content Selection & Advisory Board (CSAB)

Content Curation

Ongoing curation of the full journal base is expected and essential to keep journals at high quality levels and keep the database trustable

Review:

Curate:

Re-evaluation results

772 Titles re-evaluated

332

68

367

168

164

42

26

237

130

Under-performance

Outlier-performance

Publication concerns

51% Discontinuation rate

62% Discontinuation rate

65% Discontinuation rate

Source: Science Magazine. April 3, 2019