![Page 1: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/1.jpg)
Are we all measuring in miles or kilometers?
Building trust in metrics though standards
Todd CarpenterExecutive Director, NISO
September 26, 2014
![Page 2: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/2.jpg)
2
Non-profit industry trade association accredited by ANSI
Mission of developing and maintaining technical standards related to information, documentation, discovery and distribution of published materials and media
Volunteer driven organization: 400+ contributors spread out across the world
Responsible (directly and indirectly) for standards like ISSN, DOI, Dublin Core metadata, DAISY digital talking books, OpenURL, MARC records, and ISBN
About
October 1, 2014
![Page 3: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/3.jpg)
How fast are we going?
![Page 4: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/4.jpg)
Pound-foot/seconds or kilogram-meter/seconds
![Page 5: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/5.jpg)
No researcher wants this to be the end of their career?
![Page 6: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/6.jpg)
Are we measuring scholarship using “English” or “Metrics”
Image: Flickr user karindalziel
![Page 7: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/7.jpg)
![Page 8: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/8.jpg)
What are the infrastructure elements of alternative
assessments?
![Page 9: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/9.jpg)
Basic Definitions
(So we are all talking about the same thing)
Altmetrics, impact, article-level metrics, social media metrics, usage
![Page 10: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/10.jpg)
Element Identification
![Page 11: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/11.jpg)
At what granularity?
![Page 12: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/12.jpg)
How long do we measure?
![Page 13: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/13.jpg)
![Page 14: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/14.jpg)
Consistency across providers
Source: Scott Chamberlain, Consuming Article-Level Metrics:Observations And Lessons From Comparing Aggregator Provider Data, Information Standards Quarterly, Summer 2013, Vol 25, Issue 2.
![Page 15: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/15.jpg)
15
I often sound like a broken record
• Defining what is to count = standards• How to describe what to count = standards• Identification of what to count = standards• Aggregating counts from network = standards• Exchange of what was counted = standards• Procedures for counting or not = standards
October 1, 2014
![Page 16: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/16.jpg)
TRUST
=
![Page 17: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/17.jpg)
![Page 18: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/18.jpg)
![Page 19: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/19.jpg)
19
Steering Committee • Euan Adie, Altmetric• Amy Brand, Harvard University• Mike Buschman, Plum Analytics• Todd Carpenter, NISO• Martin Fenner, Public Library of Science (PLoS) (Chair)• Michael Habib, Reed Elsevier• Gregg Gordon, Social Science Research Network (SSRN)• William Gunn, Mendeley• Nettie Lagace, NISO• Jamie Liu, American Chemical Society (ACS)• Heather Piwowar, ImpactStory• John Sack, HighWire Press• Peter Shepherd, Project Counter• Christine Stohn, Ex Libris• Greg Tananbaum, SPARC (Scholarly Publishing & Academic Resources Coalition)October 1, 2014
![Page 20: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/20.jpg)
20October 1, 2014
Isn’t it too soon? How soon is now?
![Page 21: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/21.jpg)
Alternative Assessment Initiative
Phase 1 MeetingsOctober 9, 2013 - San Francisco, CA
December 11, 2013 - Washington, DCJanuary 23-24 - Philadelphia, PA
Round of 1-on-1 interviews – March/Apr
Phase 1 report published in June 2014
![Page 22: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/22.jpg)
22
Meeting Lightning Talks• Expectations of researchers• Exploring disciplinary differences in the use of social media in
scholarly communication• Altmetrics as part of the services of a large university library
system• Deriving altmetrics from annotation activity• Altmetrics for Institutional Repositories: Are the metadata
ready?• Snowball Metrics: Global Standards for Institutional
Benchmarking• International Standard Name Identifier• Altmetric.com, Plum Analytics, Mendeley reader survey• Twitter InconsistencyOctober 1, 2014
“Lightning" by snowpeak is licensed under CC BY 2.0
![Page 23: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/23.jpg)
23October 1, 2014
![Page 24: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/24.jpg)
24October 1, 2014
![Page 25: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/25.jpg)
25October 1, 2014
![Page 26: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/26.jpg)
26
30 One-on-One Interviews
October 1, 2014
![Page 27: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/27.jpg)
27
White Paper Released
October 1, 2014
![Page 28: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/28.jpg)
28
Potential work themes
DefinitionsApplication to types of research outputsDiscovery implicationsResearch evaluationData quality and gamingGrouping, aggregating, and granularityContext Adoption
October 1, 2014
![Page 29: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/29.jpg)
29
Potential work themes
DefinitionsApplication to types of research outputsDiscovery implicationsResearch evaluationData quality and gamingGrouping, aggregating, and granularityContext Adoption
October 1, 2014
![Page 30: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/30.jpg)
30
Potential work themes
Definitions
Application to types of research outputsDiscovery implicationsResearch evaluationData quality and gamingGrouping, aggregating, and granularityContext Adoption
October 1, 2014
![Page 31: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/31.jpg)
31
Potential work themes
DefinitionsApplication to types of research outputs
Discovery implicationsResearch evaluationData quality and gamingGrouping, aggregating, and granularityContext Adoption
October 1, 2014
![Page 32: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/32.jpg)
32
Potential work themes
DefinitionsApplication to types of research outputsDiscovery implications
Research evaluationData quality and gamingGrouping, aggregating, and granularityContext Adoption
October 1, 2014
![Page 33: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/33.jpg)
33
Potential work themes
DefinitionsApplication to types of research outputsDiscovery implicationsResearch evaluation
Data quality and gamingGrouping, aggregating, and granularityContext Adoption
October 1, 2014
![Page 34: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/34.jpg)
34
Potential work themes
DefinitionsApplication to types of research outputsDiscovery implicationsResearch evaluationData quality and gaming
Grouping, aggregating, and granularityContext Adoption
October 1, 2014
![Page 35: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/35.jpg)
35
Potential work themes
DefinitionsApplication to types of research outputsDiscovery implicationsResearch evaluationData quality and gamingGrouping, aggregating, and granularity
Context Adoption
October 1, 2014
![Page 36: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/36.jpg)
36
Potential work themes
DefinitionsApplication to types of research outputsDiscovery implicationsResearch evaluationData quality and gamingGrouping, aggregating, and granularityContext
Adoption & PromotionOctober 1, 2014
![Page 37: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/37.jpg)
Alternative Assessment Initiative Phase 2
Presentations of Phase 1 report (June 2014)Prioritization Effort (June - Aug, 2014)
Project approval (Sept 2014)Working group formation (Oct 2014)
Consensus Development (Nov 2014 - Dec 2015)Trial Use Period (Dec 15 - Mar 16)
Publication of final recommendations (Jun 16)
![Page 38: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/38.jpg)
38October 1, 2014
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Unimportant
Of little importance
Moderately important
Important
Very important
Community Feedback on Project Idea Themes
n=118
![Page 39: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/39.jpg)
39October 1, 2014
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Unimportant
Of little importance
Moderately important
Important
Very important
n=118
Community Feedback on Project Idea Themes
![Page 40: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/40.jpg)
40
Top-ranked ideas (very important & important >70%)
• 87.9% - 1. Develop specific definitions for alternative assessment metrics. • 82.8% - 10. Promote and facilitate use of persistent identifiers in scholarly
communications. • 80.8% - 12. Develop strategies to improve data quality through
normalization of source data across providers. • 79.8% - 4. Identify research output types that are applicable to the use of
metrics.• 78.1% - 6. Define appropriate metrics and calculation methodologies for
specific output types, such as software, datasets, or performances. • 72.5% - 13. Explore creation of standardized APIs or download or
exchange formats to facilitate data gathering. • 70.7% - 11. Research issues surrounding the reproducibility of metrics
across providers.
October 1, 2014
![Page 41: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/41.jpg)
Alternative Assessments of our Assessment Initiative
White paper downloaded 4,910 in 110 days21 substantive comments received
120 in-person and virtual participants at the meetings These 3 meetings attracted >400 RSVPs for live stream
Goal: generate about 40 ideas, in total, generated more than 250Project materials downloaded more than 18,000 times
More than 450 direct tweets using the #NISOALMI hashtagSurvey ranking of output by 118 people
Five articles in traditional news publications 15 blog posts about the initiative
![Page 42: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/42.jpg)
For more
Project Site:www.niso.org/topics/tl/altmetrics_initiat
ive/
White Paper:http://www.niso.org/apps/group_public/download.php/13295/niso_altmetrics_white_paper_draft_v4.pdf
October 1, 2014 42
![Page 43: Are we all measuring in miles or kilometers? Building trust in metrics though standards](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5681385a550346895da0065f/html5/thumbnails/43.jpg)
Questions?
Todd CarpenterExecutive Director
National Information Standards Organization (NISO)3600 Clipper Mill Road, Suite 302Baltimore, MD 21211 USA+1 (301) 654-2512www.niso.org
October 1, 2014 43