1

Product and process in digital scholarship Megan Poore

In the era of the Internet, new ways of distributing the various products of our educational academic labours are beginning to challenge established ways of publishing the scholarly. Online media are testing the traditional notion of public scholarly product as being tangibly and almost exclusively tied up in a single format – that of peer-reviewed print media, such as educational journal articles, monographs and conference papers. This paper explores issues of scholarship and scholarly product in the educational technology landscape, and argues that we have now entered an age where we must re-evaluate exactly what makes a work scholarly or not. I tackle the issue of what is scholarship, and I point to the key role that peer review plays in scholarship proper. Keywords: online scholarship, publication, peer review

Introduction Educational scholarship has gone online. In the era of the Internet, new ways of distributing the various products of our educational academic labours are beginning to challenge established ways of publishing the scholarly – new genres for scholarly educational writing and communication are emerging as many academics self-publish and pre-publish their work online in formats such as blogs, video, wikis and open-source software (ACRL, 2007; Arms and Larsen, 2007). These new media are testing the traditional notion of the public scholarly product as being tangibly and almost exclusively tied up in a single format – that of peer-reviewed print media, such as educational journal articles, monographs and conference papers. This means that now, more than ever, we need to be clear about what educational scholarship consists of, about what artefacts that scholarship produces, and how each of these things might be measured in the online world. This paper explores the issues of scholarship and scholarly products in the educational technology present, and argues that, although we have traditionally been able to measure educational scholarship relatively easily in terms of a ‘finished’ work (the aforementioned journal article and monograph and the like), we have now entered an age where the many processes of educational scholarship are available for scrutiny online as their own type of artefact, forcing us to re-evaluate exactly how we might come to decide what makes a work scholarly or not. I demonstrate that the current era of information and publication abundance has allowed us to increase our inventory of scholarly products in the teaching and learning arena, just as the processes of scholarship are becoming more accessible and visible thanks to easy self-publication. After a brief exploration of the new ways of publishing our educational scholarly endeavours, I tackle the issue of what is scholarship and how it relates to educational scholarly products, both traditional and online. I point to the key role that peer review plays in scholarship proper and I conclude by implying that the new metrics of educational scholarship may include not just the one form of scholarly product, but many. New ways of publishing Traditionally, academics have only ever really made public their ‘finished’ scholarship, that is, completed work (including works-in-progress) that was in fit enough a state to weather the steely critique of one’s peers. Such work was most often published in the form of print monographs, journal articles, and conference and seminar papers, and, in a tidy piece of circular reasoning, ‘being published’ was a pretty good measure of whether or not your work was, indeed, scholarly: you had managed to bring together and critique the literature of the discipline, pose a problem, distil the most important bits of your research, proffer a reasoned argument for a given solution, communicate it in written form, and it have it found to be good by your intellectual equals. In other words, a piece was scholarly because it was published, and it was published because it was scholarly – and all because your peers said so. This meant that we had a pretty tight handle on what constituted a scholarly work, but it also meant that only one type of scholarly product – the journal article or monograph or conference paper – was being evaluated for fitness for publication. But now, what is or is not ‘publishable’ no longer remains the purview of one’s peers, and ‘publication’ is certainly no longer restricted to a print medium characterised by editors and the

2

infrastructure of printing presses. If I have an internet connection I can ‘publish’ just about anything I jolly well please. These new ways of publishing – this new abundance of published products and formats (Anderson, 2006; Jensen, 2007) – poses challenges and opportunities for scholarship in teaching and learning. The challenge is to keep control of what constitutes educational scholarship proper; the opportunity is in having a wider variety of scholarly products to draw on and assess than ever before. This raises two main questions: firstly, What is educational scholarship, as traditionally understood and how does this relate to scholarly product and process in an online world? And, secondly, Does the emergence of new forms of publication and scholarly product change our definition and measurement of what educational scholarship is? Scholarly product and process What educational scholarship – or any scholarship for that matter – is is probably fairly uncontroversial; I’ll provide a makeshift list of things momentarily, but first we have to acknowledge that the focus on measuring scholarship through a particular type of ‘finished’ product, rather than its full processes, has tended to dominate the modern Academy (Green and Roy, 2007, p. 36). Of course, there is a reason for this attention to the completed journal article or monograph or whatever: in the modern era, tenure and promotion are tied to how much and how often we publish in these formats (ACRL, 2007, p. 8; Green and Roy, 2007, p. 36; Kingsley, 2007, p. 213). That’s one reason for the favouritism, but it’s a cynical one and there is a supplementary explanation, and that is that, quite simply, that in an age of publication scarcity, the journal article and related formats simply provided the best way of communicating and measuring scholarship, even if some of the more mundane processes of scholarship were skipped over or condensed in the reporting in the final product. The journal article and its related formats, came to be scholarship’s most obvious and public products. But, as we all know, the process of scholarship does not consist solely of the writing of journal articles. Or book chapters or seminar papers. These things might represent a final scholarly product by which we have traditionally measured scholarship, but they are not, as we know, all that counts for ‘scholarship.’ Few would disagree, for example, when I say that the process of scholarship – educational scholarship – includes engaging intellectually with a body of educational work and finding a legitimate problem to be solved. It means conducting original research, collating and analysing data, formulating an informed opinion, making an argument and providing evidence for it. And educational scholarship means advancing our understanding of a subject, and communicating and producing new knowledge in teaching and learning. In many areas, scholarship is also about co-operating, collaborating and sharing discoveries with a research team. But that’s not all: an essential process in determining what is or is not scholarship is having your work subjected to peer-review. You can do everything else that might be ‘scholarly,’ you can go through all the other processes of ‘scholarship,’ but until your work has undergone the scrutiny and approval of your equals, it isn’t scholarship complete: in fact, it’s probably best described, up until this point, as ‘research.’ The problem with all this is that the traditional scholarly product of peer review – the educational journal article or monograph – cannot even begin to capture the full depth of our scholarly activity, although, until now, perhaps, such a product has provided our best way of communicating the most ‘important’ bits of what we do – or at least the most important bits of what we’ve found in our educational academic projects. The format of the peer-reviewed (printed) artefact evolved to draw out the most ‘valuable’ and, perhaps, intellectually assessable elements of the scholarly process, namely, argument, critique and evaluation. This means that journal articles (and monographs and the like) were not, traditionally, good or proper places to publish your initial, primitive thoughts on a topic; they weren’t good or proper places to replicate the entire process of a school-based experiment and they weren’t good or proper places to include bulky magnetic tape-recordings of fieldwork interviews or focus group meetings (Arms and Larsen, 2007, p. 12). An economics of scarcity was operating, and that’s why the journal form evolved as it did. But in an age of abundance of (self-)publishing possibilities, these economics are being challenged (Anderson, 2006): the background processes of scholarship are moving more and more into view as education and other academics post their initial thoughts on a problem in their blogs, as they produce wikis to collaborate on a project with colleagues in other countries, as they take part in discussion forums and start describing their research findings, and as they make available their raw data to anyone who

3

wants to inspect it. Moreover, as an educational researcher, I now also can publish just about any artefact of my scholarship, not just my intellectual cogitations. The processes of scholarship have always produced concrete scholarly artefacts: field-based journals, experiments, graphs, tables, databases, ethics committee applications, recordings of interviews or focus group meetings, photographs, references lists, grant applications, letters and emails, and notes, notes, notes – notes on fieldwork, notes on the literature, notes on my thinking, notes on others’ thinking, notes on where all of this fits into education as a whole … . But in a system where, traditionally, most of these products never made it to print, there is now a multitude of digital possibilities for distributing raw data. Up until recently, there simply hasn’t been room for these artefacts in the world of scholarly publication. But all that is changing. We now have the opportunity to publish many forms of scholarly product – both old and new – not just one. So, where does this leave us? It leaves us with the fact that in the era of the Read-Write Web, we can easily communicate most, if not all, of the processes of scholarship in teaching and learning – not just that of the outcome of peer review. In other words, we can now publish more and different types of scholarly product in various formats, and those varied products and formats are challenging the metrics by which we assess and gauge scholarship (Jensen, 2007). It also means that we’re finding that things are getting messy because scholarship itself is messy, and there is now the chance that we might be asked to measure scholarship as incomplete and as made up of many processes and products, not just the process that produces a finished peer-reviewed article. Measuring educational scholarship in an online world We are now at the point where we can address our second question, that is, Does the emergence of new forms of publication and the ability to make public our many scholarly products and processes change our definition of what educational scholarship is? I will be up-front here, and say, ‘No.’ All of those things I outlined earlier as making up scholarship are still valid in the online world. The laying bare of the less-visible (meaning ‘less-reported’) processes of teaching and learning scholarship that the Read/Write Web affords us, by giving us the opportunity to make public previously unseen products and processes of our scholarship, is surely a boon to scholarship as a whole: it may not be the most stimulating thing in the world for you to view all the material I, myself, looked at but rejected on my academic expedition through a particular topic, but at least there’s the possibility of your now doing it. So, what educaitonal scholarship is won’t change, but the products of educational scholarly endeavour we are today able to publish so straightforwardly, and the processes we are able to expose so readily, are causing us to consider how we will evaluate scholarship online in the future (Green and Roy, 2008, p. 36). How will we conduct peer review of so many products and processes? Or shouldn’t we? Should we stick with the ways in which we currently do things in journal articles and the like, where we evaluate reasoning, argument and critique, but (usually) not the entire and very raw data reported on? If we do this latter – stick with what we’ve got going at the moment – perhaps we should be more flexible in how we think of what constitutes an ‘article’: if my product meets all the traditional criteria of ‘published’ scholarly work, it’s just that I’ve made in the form of a podcast, then what’s the problem? The problem is how to conduct the peer review. In the past, whether or not one could call oneself a scholar was largely up to one’s peers. And as I’ve already pointed out, the process of getting a peer-reviewed piece of work into the public realm was easily controlled. Because of this, the notion of ‘being published’ became intricately tied to what it meant to ‘be scholarly.’ But we are now at such a point of digital publication abundance that we need to extricate these two things – conceptually, at least. Today, ‘publication’ must be seen to include self-publishing through wikis, blogs, online data sets, and the like, but ‘publication’ can no longer be equated with scholarship. One cannot give oneself the designation of being an ‘educational scholar’ just because one’s work is ‘published’ online. Being a scholar must continue to include peer review (Kingsley 2007), but new ways of conducting peer review may need to be developed to account for the sudden abundance of scholarly products available for examination. This means that peer review itself needs to be laid bare. Peer review processes could be out there for all to see – and maybe even for all to post comments on. We now have the technology to do all of these things. How we might do them, however, must be left for another discussion. Conclusion

4

Educational scholarly activity has always constituted a multitude of processes and has always produced a variety of artefacts – from notes to references lists to experiments. But these products and processes had to be distilled into an essay or monograph format, for the sake of brevity and ease of evaluation by one’s peers, in a era of publication scarcity. We have now entered an era of publication abundance, which means that just about any product of scholarly activity you care to name (except, perhaps, pure thought) can be published … ‘published online,’ that is. This is presenting challenges to how we evaluate the quality of the work of a scholar of education in the modern era, but the biggest challenge, perhaps, is to how we can control the peer review process that I have argued is essential to that determination. To my mind, there is no doubt that peer review must continue to be the single-most important factor in assessing the worth of a scholarly product; just how peer review might be conducted to account for the many and varied scholarly products that are now available for inspection by one’s education colleagues is an issue that must be confronted by the academic community if we are to remain intellectually relevant and rigorous in the online world. One final point: it may take some time for these things to get sorted out. But no matter how or how long it takes for these issues get resolved, one thing is certain: those educators who are already establishing a sizeable digital footprint through blogging, setting up wikis, starting discussion forum conversations, and podcasting interviews with research participants, can only be advantaged when the new metrics of educational scholarship eventually kick in. References ACRL. (2007). Establishing a research agenda for scholarly communication: A call for community

engagement. Association of College & Research Libraries Scholarly Communications Committee. Anderson, C. The Long Tail. How endless choice is creating unlimited demand. London: Random House. Arms, W., and Larsen, R. (2007). The future of scholarly communication: Building the infrastructure for

cyberscholarship. Report of a workshop held in Phoenix, Arizona, April 17 to 19, 2007. National Science Foundation and the Joint Information Systems Committee.

Green, D., and Roy, M. (2008). Things to do while waiting for the future to happen: Building cyberinfrastructure for the liberal arts. Educause Review July/August 2008, 25-48.

Jensen, M. (2007). The new metrics of scholarly authority. The Chronicle Review, June 15, 2007. http://chronicle.com/free/v53/i41/41b00601.htm [viewed 28 Jul 2008]

Kingsley, D. (2007). The journal is dead, long live the journal. On the Horizon, 15(4), 211-221.