bobby allen, pam lach, stephanie barnwell 2012 october 2nd valuing & evaluating digital...
TRANSCRIPT
BOBBY ALLEN, PAM LACH, STEPHANIE BARNWELL 2012 October 2nd
Valuing & Evaluating Digital Humanities Practice
www.digitalhumanities.unc.edu
Digital scholarship is the inevitable future of the humanities and social sciences. [D]igital literacy is a matter of national competitiveness and a mission that needs to be
embraced by universities, libraries, museums, and archives. . . . A robust cyberinfrastructure should include centers that support collaborative work with
specialized methods. . . . Public funds should be at the forefront of support to such national centers of excellence in digital humanities and social science, as crucial
seedbeds of further innovation.Our Cultural Commonwealth | 2006
Report of the Commission on Cyberinfrastructure in the Humanities and Social Sciences American Council of Learned Societies
The practice and the profession of history are undergoing changes quite unlike any we have experienced before. I increasingly believe that the digital revolution is yielding transformations so profound that their
nearest parallel is to Gutenberg’s invention of moveable type more than half a millennium ago.
William Cronon | President, American Historical Association In “The Public Practice of History in and for a Digital Age” | January 2012
contexts of DIGITAL HUMANITIES practice
Gordon Gee | President, Ohio State University In “Right Here, Right Now: A Crystallization of Purpose” | 7 October 2009
I believe we must finally speak aloud the truth: that some arbitrary volume of published papers, on some narrowly defined points of debate, is not necessarily more worthy than other activities. . . .
The central point is that the time is right, at this moment, to focus intently on the quality of your work and its impact on our students,
our disciplines, and our communities. Quality and impact. Those must be the two central considerations in our reward system. And
I am urging the provost and the deans to attend particularly to those two criteria in promotion and tenure decisions.
contexts of DIGITAL HUMANITIES practice
Digital tools are transforming the practice of history, yet junior scholars and graduate students are facing obstacles and risks to their professional advancement in using methods unrecognized as rigorous scholarly work.
Their peers and evaluators are often unable or unwilling to address the scholarship on its merits. Opportunities to publish digital work, or to even have it reviewed are limited. Finally, promotion and tenure processes are largely built around 19th-century notions of historical scholarship that do
not recognize or appropriately value much of this work. The disconnect between traditional evaluation and training and new digital methods
means young scholars take on greater risks when dividing their limited time and attention on new methods that ultimately may not ever face
scholarly evaluation on par with traditional scholarly production.A Call to Redefine Historical Scholarship in the Digital Turn
Open Letter to AHA Research Division | 26 January 2012
contexts of DIGITAL HUMANITIES practice
DIGITAL Humanities
explore more resources: http://digitalhumanities.unc.edu/resources/
a few DH PROJECTS from UNC
the Carolina Digital Humanities Initiative is…a five-year coordinated intervention in humanities research, graduate training, undergraduate teaching, and public engagement that will produce an adaptable and sustainable model of transformative academic practice in the humanities,change the role and reach of humanities at UNC,and serve as a model for innovation in academic practice in the humanities for other research universities
contexts of DIGITAL HUMANITIES practice at UNC
how will the CDHI work?• cluster faculty hires• Mellon Graduate Fellowships in Digital Humanities• Mellon Postdoctoral Fellowships in Digital Humanities• DIL/IAH Faculty Fellowships• graduate certificate program in Digital Humanities• cyberinfrastructure taskforce• data studies curriculum
five-year deliverables of the CDHI• 3 tenure-track digital humanists• 12 Mellon Graduate Fellowships• 4 Mellon Postdoctoral Fellowships• 8 DIL/IAH Faculty Fellowships and projects• graduate certificate program • coordinated DH curriculum with Duke and NCSU• DH skills/tools training program • cyberinfrastructure best practices policies implemented
and evaluated• data studies curriculum• 19 new DH courses developed, offered, and assessed
what could DH PRACTICE transform?
• how we ask research questions • how and what we teach• how students learn• how we engage with the public• how we train graduate students• how humanists and scientists work together
what is distinctive about DH PRACTICE?• social, collaborative• interdisciplinary, trans-domain• dynamic, iterative, open-ended• use of sources, i.e. big data• cuts across traditional distinctions: research,
teaching & service• can leverage engaged scholarship, effective
use of digital technologies, and interdisciplinary collaborations
what is distinctive about DH PRACTICE?
• cost: from fellowship to grant funding- technology & technical know-how- external funding model not scalable
• requires access to technologies and the skills to use them
• experimental (risk of failure)
Critical need to reduce barriers to entry and create sustainable business models!
The Commission believes that digital scholarship is the inevitable future of the humanities and social sciences, and that digital literacy is a matter of national competitiveness and a mission that needs to be embraced by universities, libraries, museums, and archives. In order to foster digital research, teaching, and publishing, we recommend specifically that there be:
• policies for tenure and promotion that recognize and reward digital scholarship and scholarly communication; recognition should be given not only to scholarship that uses the humanities and social science cyberinfrastructure but also to scholarship that contributes to its design, construction, and growth.
Our Cultural CommonwealthReport of the Commission on Cyberinfrastructure in the Humanities and Social Sciences American Council of Learned Societies (2006)
VALUING digital humanities practice
VALUING digital humanities practice
Our Cultural CommonwealthReport of the Commission on Cyberinfrastructure in
the Humanities and Social Sciences American Council of Learned Societies (2006)
We might expect younger colleagues to use new technologies with greater fluency and ease, but with tenure at stake, they will also be more risk-averse. There is a widely shared perception that academic departments in the humanities and social sciences do not adequately reward innovative work in digital form. A handful of recent examples provide exceptions to the norm, but in the most elite universities, traditional scholarly work, in the form of a single-authored, printed book or article published by a university press or scholarly society, is the currency of tenure and promotion; work online or in new media—especially work involving collaboration—is not encouraged. Senior scholars now have both the opportunity and the responsibility to take certain risks, first among which is to condone risk taking in their junior colleagues and their graduate students, making sure that such endeavors are appropriately rewarded.
SUPPORTING digital humanities practice• articulate clear expectations upon hire• reformulate and tweak these expectations, each year, to set benchmarks for
achievement and for assessment• at mid-term review, collaboratively identify specific benchmarks, and the mode of
evaluation that will be used to determine successful completion -- this is a portion of a contract for tenure, in effect
• identify potential referees early on and revise as necessary• identify outside resources and potential collaborators at the same time (granting
agencies, donors, legitimating agencies like NINES, potential collaborators)• identify audiences and expected impact of the research: what is it, who is it for and why
is it significant?• document effort• use informal avenues to communicate with colleagues in a collegial manner so all
faculty feel an interest in and responsibility for success
Digital Humanities Scholarship: Recommendations for Chairs in Language and Literature
Departments | Alison Booth, Pamela K. Gilbert, Steve Olsen, Brad Pasanek and the NINES/NEH Summer Institute Group, 2011
FORMS of digital humanities practice• websites• online peer-reviewed publication• scholarly electronic editions• research tools• engaged digital scholarship
(partnerships with cultural heritage/community orgs)• blogs• standards and specifications• instructional technology• digital art/new media production• AND. . . .
2001
EVALUATING digital humanities practice
1. Fundamentals for Initial Review: The work must be evaluated in the medium in which it was produced and published
2. Crediting: Digital projects are often collaborative in nature, involving teams of scholars who work together in different venues over various periods of time. Authors of digital works should provide a clear articulation of the role or roles that they have played in the genesis, development, and execution of the digital project.
3. Intellectual Rigor: Digital projects vary tremendously and may not "look" like traditional academic scholarship; at the same time, scholarly rigor must be assessed by examining how the work contributes to and advances the state of knowledge of a given field or fields
4. Crossing Research, Teaching, and Service: Digital projects almost always have multiple applications and uses that enhance—at the same time—research, teaching, and service.
5. Peer Review: Digital projects should be peer reviewed by scholars in fields who are able to assess the project's contribution to knowledge and situate it within the relevant intellectual landscape
Presner, et al, “How to Evaluate Digital Scholarship”
EVALUATING digital humanities practice
6. Impact: Digital projects can have an impact on numerous fields in the academy as well as across institutions and even the general public. They often cross the divide between research, teaching, and service in innovative ways that should be remarked
7. Approximating Equivalencies: Is a digital research project "equivalent" to a book published by a university press, an edited volume, a research article, or something else? These sorts of questions are often misguided since they are predicated on comparing fundamentally different knowledge artifacts and, perhaps more problematically, consider print publications as the norm and benchmark from which to measure all other work. Reviewers should be able to assess the significance of the digital work based on a number of factors: the quality and quantity of the research that contributed to the project; the length of time spent and the kind of intellectual investment of the creators and contributors; the range, depth, and forms of the content types and the ways in which this content is presented; and the nature of the authorship and publication process.
8. Development Cycles, Sustainability, and Ethics: It is important that review committees recognize the iterative nature of digital projects, which may entail multiple reviews over several review cycles, as projects grow, change, and mature.
9. Experimentation and Risk-Taking: Digital projects in the Humanities, Social Sciences, and Arts share with experimental practices in the Sciences a willingness to be open about iteration and negative results. As such, experimentation and trial-and-error are inherent parts of digital research and must be recognized to carry risk.
EVALUATING digital humanities practice
Presner, et al, “How to Evaluate Digital Scholarship”
EVALUATING digital humanities practice
Lee, Valerie and Cynthia L. Selfe, Cynthia L. "Our Capacious Caper: Exposing Print-Culture Bias in Departmental Tenure Documents." ADE Bulletin 145 (Spring 2008): 51-58
EVALUATING digital humanities practice
Digital technologies do more than propose new ways of thinking, as did theory; they require new modes of being. To
put this in less dramatic terms, the digital revolution requires us as a profession to make conscious the motivations and
values inhering in material practices, from putting a manuscript in the mail to a publisher to requiring for tenure
“a book between covers.” We must transfer the values informing these activities and practices onto new modes of
activity, so that we understand, value, and evaluate theoretical decisions about database modeling, algorithms,
and information flows to best support new research and reading practices.
“Evaluating Digital Scholarship, Introduction”Susan Schreibman, Laura Mandell, and Stephen Olsen
Goals/Questions• Does the scholar state the basic purpose of the work and its value for
public good?• Is there an “academic fit” with the scholar’s role, departmental and
university mission?• Does the scholar define objectives that are realistic and achievable?• Does the scholar identify intellectual and significant questions in the
discipline and in the community? Context of theory, literature, “best practices”• Does the scholar show an understanding of relevant existing scholarship?• Does the scholar bring the necessary skills to the collaboration?• Does the scholar make significant contributions to the work?• Is the work intellectually compelling? Methods• Does the scholar use methods appropriate to the goals, questions and
context of the work?• Does the scholar describe rationale for election of methods in relation to
context and issue?• Does the scholar apply effectively the methods selected?• Does the scholar modify procedures in response to changing
circumstances?
Results• Does the scholar achieve the goals?• Does the scholar’s work add consequentially to the discipline and to the
community?• Does the scholar’s work open additional areas for further exploration and
collaboration?• Does the scholar’s work achieve impact or change? Are those outcomes
evaluated and by whom?
Evaluation Criteria for the Scholarship of Engagement
Purdue University (2001)
EVALUATING (engaged) digital humanities practice
Communication/Dissemination• Does the scholar use a suitable styles and effective organization to
present the work?• Does the scholar communicate/disseminate to appropriate academic and
public audiences consistent with the mission of the institution?• Does the scholar use appropriate forums for communicating work to the
intended audience?• Does the scholar present information with clarity and integrity? Reflective Critique• Does the scholar critically evaluate the work?• What are the sources of evidence informing the critique?• Does the scholar bring an appropriate breadth of evidence to the
critique?• In what way has the community perspective informed the critique?• Does the scholar use evaluation to learn from the work and to direct
future work?• Is the scholar involved in a local, state and national dialogue related to
the work?
EVALUATING (engaged)digital humanities practice
challenges to evaluating DH practice• collaboration/authorship• “curatorial practice” (scholarly
editions, digital archives, discovery tools, databases)
• tool-making (software, games, standards and specifications, search tools, visualization)
• research vs. service• interdisciplinarity• lack of evaluative expertise at
department level• assessing the intellectual value of
digital work
• time frame for evaluating open-ended work
• documenting, contextualizing, explaining DH practice & process
• evaluating digital work in the form in which it was meant to be experienced
• experimental nature of DH practice
• measuring impact
vehicles for evaluation• peer review
--18th Connect, NINES • reviews in journals (digital and analog)
--http://digitalhumanities.org/dhq/• prizes
--http://www.historians.org/prizes/Rosenzweig_Fellowship.cfm• financial support (fellowships, internal and external grants, gifts)
--http://www.neh.gov/divisions/odh• conference presentations
--http://www.dh2012.uni-hamburg.de/• external expert reviews• citations/scholarly references/blog posts
--http://www.dancohen.org/• qualitative user feedback
-- http://docsouth.unc.edu/support/about/readers.pdf• quantitative use/impact measures
--downloads, visitor counts (http://www.google.com/analytics/)
vehicles for evaluation
visualizations of 1-year Google Analytics data for Going to the Showhosted by CDLA at UNC-Chapel Hill
University of Victoria“Report on Academic Computing
Recognition” | 1998
EXAMPLES of promotion and tenure practice
EXAMPLES of promotion and tenure practice
