introducing the open citation experiment - jisc digifest 2016

/15

Introducing the open citation experiment

Petr Knoth (@petrknoth) & Drahomira Herrmannova (@damirah)Knowledge Media institute, The Open University

Verena Weigert, Jisc

1

Towards full-text based research metrics: exploring Semantometrics

/15

What are Semantometrics?• A new class of metrics for evaluating research• Build on the premise that full-text is needed to

asses the value of a publication• Make use of the full-text features of a given

resource rather than relying on outside evidence

2

/15

Semantometric contribution measure• Based on the idea of measuring the progress

of scholarly discussion• The hypothesis states that the added value of

publication p can be estimated based on the semantic distance from the publications cited by p to the publications citing p

• Demonstrator at http://semantometrics.org

3

/15

Comparative study• Analysis carried out to investigate the

properties of the contribution measure• Experiments carried out on a dataset obtained

by merging data from the Connecting Repositories (CORE) system, the Microsoft Academic Graph (MAG) and Mendeley

• After merging the datasets over 1.6 million publications

4

/15

ApproachPremise: Full-text needed to assess publication’s research contribution.Hypothesis: Added value of publication p can be estimated based on the semantic distance from the publications cited by p to publications citing p.

5

/15

ApproachPremise: Full-text needed to assess publication’s research contribution.Hypothesis: Added value of publication p can be estimated based on the semantic distance from the publications cited by p to publications citing p.

5

/15

ApproachPremise: Full-text needed to assess publication’s research contribution.Hypothesis: Added value of publication p can be estimated based on the semantic distance from the publications cited by p to publications citing p.

5

/15

Contribution measure

6

/15

Contribution measure

p

6

/15

Contribution measure

p

6

/15

Contribution measure

p

6

/15

Contribution measure

p

A

6

/15

Contribution measure

p

A B

6

/15

Contribution measure

p

A B

6

/15

Contribution measure

p

A B

6

/15

Contribution measure

p

A Bdist(a,b)

6

/15

Contribution measure

p

A Bdist(a,b)

6

/15

Contribution measure

p

A Bdist(a,b)

6

/15

Contribution measure

p

A Bdist(a,b)

6

/15

Contribution measure

p

A Bdist(a,b)

6

/15

Contribution measure

p

A Bdist(a,b)

6

/15

Contribution measure

p

A Bdist(a,b)

6

/15

Contribution measure

p

A Bdist(a,b)

Average distance of the set members

6

/15

Contribution measure

p

A Bdist(a,b)

Average distance of the set members

6

/15

Contribution measure

p

A Bdist(a,b)

dist(b1,b2)

Average distance of the set members

6

/15

Contribution measure

p

A Bdist(a,b)

dist(b1,b2)

Average distance of the set members

6

/15

Contribution measure

p

A Bdist(a,b)

dist(b1,b2)

Average distance of the set members

6

/15

Contribution measure

p

A Bdist(a,b)

dist(b1,b2)

Average distance of the set members

6

/15

Contribution measure

p

A Bdist(a,b)

dist(b1,b2)

Average distance of the set members

6

/15

Real examples• Contribution 0.8452

/15

Real examples• Contribution 0.9220

/15

Real examples• Contribution 0.8296

/15

Real examples• Contribution 0.9348

/15

Real examples• Contribution distribution

/15

Conclusions• First large scale analysis of the semantometric

contribution measure• As semantometrics should exhibit a number of

advantages over existing research metrics, we should continue studying this field to better understand which facets of research quality they can capture and how they can be applied

13