scientific software - what happens after the grant?

Sustaining scientific infrastructures: transitioning from grants to peer production

James HowisonSchool of Information

University of Texas at Austin2 September 2016

@jameshowison(slides on slideshare, see twitter for link)

This material is based upon work supported by the US National Science Foundation under Grant Nos. SMA- 1064209 (SciSIP), OCI-0943168 (VOSS) and ACI-145348 (CAREER).

@jameshowison

Supporting Scientific software after grants run out

• What happens when the grant ends?– It’s hard, hard work to keep the code from

inevitable “bit-rot”

@jameshowison

Extension needs up-to-date code

@jameshowison

Just open source it!

(How hard can it be???)

@jameshowison

Open projects are not like grants

1. Governance2. Collaboration infrastructures3. Contribution processes4. Service center vs. Base for community

“open sourcing” means full-on sociotechnical change

@jameshowison

A literature on transfer to open?

• Copious literature on commercialization, “Technology Transfer” but not communities

• Happily there are promising literatures – Studies of open source and online communities

(Resnick, Crowston, Wiggins, Kittur, Kraut, Lampe, Ellison, …)– Studies of scientific practice

(Palmer, Borgman, Vertesi, Edwards, Olsons, Finholt, Lee/Bietz, Østerlund, Sawyer, Tapia, Ludders, …)

– Studies of infrastructural work (Bowker, Jackson, Vertesi, Ribes, …)

@jameshowison

How can scientific software projects successfully transition from grant support to thriving peer

production communities?

Research Design:

1. Theoretically sampled case studies

2. Longitudinal panel study

@jameshowison

Questions for each case:How did they succeed or fail in building peer production? – What actions were taken to change the project?– How did routines in the project change as a

result?– What conditions are relevant to the success of

those actions in causing change?

@jameshowison

Sampling success and failure

• Very hard to have people talk about failures– Records are often unavailable– Constant problem in studies of open source

• Panel study offers help here– Enroll early, before outcome clear– Build trust, chart course, keep records– Selected the NSF SI2 funding program

(program officer support)

@jameshowison

Panel Study setup

• SI2 program contributed to over 350 grants• Three step qualitative content analysis:

1. Did the grant intend to create software2. What documents (URLs, Workshop reports, or

Publications) are available?3. Read these, apply coding scheme

Content analysis categoriesCode Description

Project Presents Separate From Grant

Does the grant support the project (e.g., pre-existing), Or is the project only there because of the grant

inviteToContributecontributionProcess

Is there an explicit invitation for outsiders to contribute? Is there a process for taking contributions?

highlightsPublication e.g., Does the project have a “publications tab”

creditsNonPIContributors

Are only the PIs credited “the PIs and their teams” or a wider group?

associatedRepositoryCodeAvailable license

Is code available? Is it openly hosted? Where? Under what license?

Collaborative setup (wiki, bugtracker)Online meetings?

What set of collaborative tools are they using?

Offline meetings Does the project organize offline meetings, what kinds (user workshop, hackathon).

@jameshowison

Build dataset over time

• Training new graduate student on scheme– May involve additional students over time

• Intend to code ~5 projects a weekday for two years– 300 projects, 250 weekdays in year, 5 projects a day,

2 coders, assume some missed days!– ~5-10 observations of each project a year

• Also analyze repositories, where available.• Adding content analysis codes over time

@jameshowison

Case Method: Sampling

@jameshowison

Case Method: analysis

• Identify work episodes– Ground interviews in specific production work.– Source-code repositories help immensely– “Digital trace ethnography” (Ribes and Geiger)

• Identify socio-technical changes that divide project into stages – Investigate actions that precipitated changes

• Project narratives with illustrative vignettes

@jameshowison

ENZO

@jameshowison

ENZO pilot study

Data:• 5 interviews, so far (thanks Eunyoung Moon!)• Publications, websites, workshop websites,

source code repositories• Analysis:– Creation of timeline– Identification of episodes and 4 project phases

(with their precipitating events)

@jameshowison

• No central base to which changes are coming and going• Copy and pasting features across personal branches• Single lab

@jameshowison

• ENZO lab reforms as “Service Center” (grant)• Mainline branch internally, releases externally• Little expectation of contributions coming back in• “Friendly user” labs internally functioning like “early days”

@jameshowison

The “Week of Code”

• Director of external lab (former post-doc) has new job at Stanford (with startup funds!)

• Learns of various versions through conversations at conferences and reviewing(!)

• Focus is on collaboration infrastructure, not governance.

• Begin with the code of those not present

@jameshowison

• Central branch to which both core and outsiders contribute• Development continues separately in external labs• Called “Wild West” by participants, autonomy concerns.

@jameshowison

• Introduction of “code revision” (pull requests)• External lab members on similar footing to Core members• Review helps members not “step on each other’s work”

@jameshowison

Change

• What hasn’t changed:– Motivations (code is side-effect of scientific

inquiry, papers first, code second), no commercial value

• Challenges to change– Leadership’s emotional connection, difficulty of

passing on leadership.– Giving up autonomy (being “blocked” in one’s

work)

@jameshowison

What worked

• Always: collaboration technology before governance (contra “Collaboration Readiness” (Olson et al.) TORSC?).

• Social proof: visible action in public• Inspiration from open source• Working alongside, rather than with.

Superposition rather than Teamwork.

@jameshowison

Additional CAREER elements

• Teaching course on online communities– Incorporating more on managing software

projects in science• Contributing modules to Software Carpentry– 2-3 day workshops with graduate students– Enough command line, python, SQL to get them

working– I’m going to contribute module on contributing to

and running software projects in science

@jameshowison

Conclusions

• Software engineering, but in a very specific context

• Organization of software work but different to design and testing of methodologies

• Can also link in resource and motivation situations

• Learning from open source, building alternative paths alongside commercialization.