scientific software - what happens after the grant?
TRANSCRIPT
Sustaining scientific infrastructures: transitioning from grants to peer production
James HowisonSchool of Information
University of Texas at Austin2 September 2016
@jameshowison(slides on slideshare, see twitter for link)
This material is based upon work supported by the US National Science Foundation under Grant Nos. SMA- 1064209 (SciSIP), OCI-0943168 (VOSS) and ACI-145348 (CAREER).
@jameshowison
Supporting Scientific software after grants run out
• What happens when the grant ends?– It’s hard, hard work to keep the code from
inevitable “bit-rot”
@jameshowison
Extension needs up-to-date code
@jameshowison
Just open source it!
(How hard can it be???)
@jameshowison
Open projects are not like grants
1. Governance2. Collaboration infrastructures3. Contribution processes4. Service center vs. Base for community
“open sourcing” means full-on sociotechnical change
@jameshowison
A literature on transfer to open?
• Copious literature on commercialization, “Technology Transfer” but not communities
• Happily there are promising literatures – Studies of open source and online communities
(Resnick, Crowston, Wiggins, Kittur, Kraut, Lampe, Ellison, …)– Studies of scientific practice
(Palmer, Borgman, Vertesi, Edwards, Olsons, Finholt, Lee/Bietz, Østerlund, Sawyer, Tapia, Ludders, …)
– Studies of infrastructural work (Bowker, Jackson, Vertesi, Ribes, …)
@jameshowison
How can scientific software projects successfully transition from grant support to thriving peer
production communities?
Research Design:
1. Theoretically sampled case studies
2. Longitudinal panel study
@jameshowison
Questions for each case:How did they succeed or fail in building peer production? – What actions were taken to change the project?– How did routines in the project change as a
result?– What conditions are relevant to the success of
those actions in causing change?
@jameshowison
Sampling success and failure
• Very hard to have people talk about failures– Records are often unavailable– Constant problem in studies of open source
• Panel study offers help here– Enroll early, before outcome clear– Build trust, chart course, keep records– Selected the NSF SI2 funding program
(program officer support)
@jameshowison
Panel Study setup
• SI2 program contributed to over 350 grants• Three step qualitative content analysis:
1. Did the grant intend to create software2. What documents (URLs, Workshop reports, or
Publications) are available?3. Read these, apply coding scheme
Content analysis categoriesCode Description
Project Presents Separate From Grant
Does the grant support the project (e.g., pre-existing), Or is the project only there because of the grant
inviteToContributecontributionProcess
Is there an explicit invitation for outsiders to contribute? Is there a process for taking contributions?
highlightsPublication e.g., Does the project have a “publications tab”
creditsNonPIContributors
Are only the PIs credited “the PIs and their teams” or a wider group?
associatedRepositoryCodeAvailable license
Is code available? Is it openly hosted? Where? Under what license?
Collaborative setup (wiki, bugtracker)Online meetings?
What set of collaborative tools are they using?
Offline meetings Does the project organize offline meetings, what kinds (user workshop, hackathon).
@jameshowison
Build dataset over time
• Training new graduate student on scheme– May involve additional students over time
• Intend to code ~5 projects a weekday for two years– 300 projects, 250 weekdays in year, 5 projects a day,
2 coders, assume some missed days!– ~5-10 observations of each project a year
• Also analyze repositories, where available.• Adding content analysis codes over time
@jameshowison
Case Method: Sampling
@jameshowison
Case Method: analysis
• Identify work episodes– Ground interviews in specific production work.– Source-code repositories help immensely– “Digital trace ethnography” (Ribes and Geiger)
• Identify socio-technical changes that divide project into stages – Investigate actions that precipitated changes
• Project narratives with illustrative vignettes
@jameshowison
ENZO
@jameshowison
ENZO pilot study
Data:• 5 interviews, so far (thanks Eunyoung Moon!)• Publications, websites, workshop websites,
source code repositories• Analysis:– Creation of timeline– Identification of episodes and 4 project phases
(with their precipitating events)
@jameshowison
• No central base to which changes are coming and going• Copy and pasting features across personal branches• Single lab
@jameshowison
• ENZO lab reforms as “Service Center” (grant)• Mainline branch internally, releases externally• Little expectation of contributions coming back in• “Friendly user” labs internally functioning like “early days”
@jameshowison
The “Week of Code”
• Director of external lab (former post-doc) has new job at Stanford (with startup funds!)
• Learns of various versions through conversations at conferences and reviewing(!)
• Focus is on collaboration infrastructure, not governance.
• Begin with the code of those not present
@jameshowison
• Central branch to which both core and outsiders contribute• Development continues separately in external labs• Called “Wild West” by participants, autonomy concerns.
@jameshowison
• Introduction of “code revision” (pull requests)• External lab members on similar footing to Core members• Review helps members not “step on each other’s work”
@jameshowison
Change
• What hasn’t changed:– Motivations (code is side-effect of scientific
inquiry, papers first, code second), no commercial value
• Challenges to change– Leadership’s emotional connection, difficulty of
passing on leadership.– Giving up autonomy (being “blocked” in one’s
work)
@jameshowison
What worked
• Always: collaboration technology before governance (contra “Collaboration Readiness” (Olson et al.) TORSC?).
• Social proof: visible action in public• Inspiration from open source• Working alongside, rather than with.
Superposition rather than Teamwork.
@jameshowison
Additional CAREER elements
• Teaching course on online communities– Incorporating more on managing software
projects in science• Contributing modules to Software Carpentry– 2-3 day workshops with graduate students– Enough command line, python, SQL to get them
working– I’m going to contribute module on contributing to
and running software projects in science
@jameshowison
Conclusions
• Software engineering, but in a very specific context
• Organization of software work but different to design and testing of methodologies
• Can also link in resource and motivation situations
• Learning from open source, building alternative paths alongside commercialization.