bill appelbe bill@vpac , vpac (vpac)

Thu, June 3rd 2010 1

eScience Infrastructure and the Changing Culture of Research

-Successes and Lessons Learned in

Australia

Bill Appelbe [email protected] , VPAC (vpac.org)

UK eScience Presentation

mailto:[email protected]


Outline

• A Short History of eResearch in Australia• The Current Australian eResearch

landscape• Future directions and lessons learned


eResearch in Australia

• HPC/Cyberinfrastructure in Australia is funded and organized very differently from the UK or USA

• In Australia– There are only ~40 Universities (and 20m. people)– One national government research agency (CSIRO)– National research telecom – Aarnet.edu– Since 2005, focus on National Collaboration rather than

competition for Research Infrastructure• So no “competitive bids” for Peak Computing facilities

Thu, June 3rd 2010 3UK eScience Presentation

Thu, June 3rd 2010UK eScience Presentation 4

Evolution of Australian eResearch

1995 2000 2005 2010

• Universities purchased and supported their

own HPC; little collaboration• Growing concern that Australia was falling behind

• National HPC collaboration & funding: National HPC Tier-1 Facility (ANU) & state-based Tier-2 HPC facilities (PACs)• Merit-allocation for Tier-1 HPC• Incipient grid portals & operations

• National research infrastructure collaboration & funding, including Advanced Computing (PfC)• Community-based• Maturing grid computing

• SuperScience Initiative• State investment –BRC, MASSIVE, VLSCI

eResearch in Australia – APAC Era

• APAC, the Australian Partnership for Advanced Computing, was formed in 2000

• Ambitious idea– A national Tier-1 system at the Australian National

University in Canberra– Commonwealth funding to States to set up and/or

support State-based Tier-2 HPC Centres• $6M to Victoria to set up VPAC• Matched by State funds and University subscriptions


VPAC – “Innovation powered by Advanced Computing”• VPAC is an Advanced Computing/Cyberinfrastructure

“research services” organization– Like SDSC, NCSA, EPCC…

• But organized and funded differently, and • Does not do independent research!

– Independent company; owned by the State’s Universities– Collaborative R&D with Universities & government

agencies• Such as Geosciences Australia (~USGS) and Geosciences Victoria

– Also provides services to companies such as GM, Boeing, …– Now operates most HPC facilities in the State

• But it is not “socialist” HPC! No mandate for anyone to use VPAC or State subsidies


APAC Successes and Lessons

APAC Successes– National collaboration, national merit allocation of

HPC– Acted as a catalyst for formation of regional HPC

Centres– Use of gateway machines (VMs) to support grid

computing


APAC Successes and Lessons

Lessons– Putting $ into Universities to develop HPC

courseware was not a success– Grid linking of sites did not generate a lot of use

• As there was not a lot of “resources” (HPC cycles) put in;• And each State HPC system was individually managed; • And users do not like moving their applications and data

– Grid portals were not very successful• Each individually funded, developed by grad. Students, not

sustainable or maintainable


APAC’s successors – 2007+

• NCRIS – research infrastructure funding– Including software development and funding– Programs in national priority areas such as

geosciences, biosecurity, ….

• Within NCRIS, an eResearch Infrastructure program– Plaforms for Collaboration www.pfc.org.au – Three subprograms:

• ANDS – Data standards www.ands.org.au • NCI – Compute www.nci.org.au • ARCS – Australian Research Collaboration Services

www.arcs.org.au


http://www.pfc.org.au/

http://www.ands.org.au/

http://www.nci.org.au/

http://www.arcs.org.au/

ARCS Organization and Programs

• ARCS was set up as an Unincorporated Joint Venture (UJV) of the regional HPC/eResearch Centres (the “MARCs”)– Staff distributed across the country/MARCs


ARCS/NCRIS Successes

Successes• Within NCRIS, ongoing funding for national software

platform development– E.g., StGermain open-source computational science

platform, used for geodynamics in AU, USA

• In ARCS, development of – Cloud computing and configurable web portals: grisu– Data services, the “national drop box”– Collaboration tools (national EVO support)– National authentication/id

• ARCS/NEAT Funding for community software projects Thu, June 3rd 2010 11UK eScience Presentation

ARCS/NCRIS Lessons

Lessons• Organizations that are UJVs have governance and

management problems– Not just ARCS, but also the VeRSI UJV, the Victorian

eResearch Strategic Initiative

• Engagement is critical to the success of an eResearch Service provider– With users– With University administration

• Managing highly distributed software development teams is problematicThu, June 3rd 2010 12UK eScience Presentation


Outline

• A Short History of eScience in Australia• The Current Australian eScience



Cyberinfrastructure in Australia(cont.)• VPAC has broad funding from Universities, state and

federal grants, industry, etc.– ~$7M p.a.– 70+ employees at 4 sites in the State

• Systems support, software engineers, engineers, scientists

– Operates HPC and data centers for researchers and industry• 600+ users across 8 Universities; 5 HPC clusters; GPGPU and viz. systems

– Professional software development teams for both academic and commercial projects, including computational scientists

– VPAC staff “embedded” in Universities; joint grant proposals

• Strong international links and collaboration– Joint software development for Geodynamics with USA since 2003

• The NSF Center for Computational Infrastructure for Geodynamics www.geodynamics.org


http://www.geodynamics.org/

Cyberinfrastructure in Australia(cont.)• As a state, Victoria has a mature but rapidly

increasing Cyberinfrastructure investments – VLSCI - $100M joint investment by the State and Melbourne

University in a Computational Life Sciences Center– MASSIVE - $10M investment in a Computational Imaging

and Visualization Center by a consortium including the Australian Synchrotron, CSIRO, Monash, and VPAC

– A $100M Biotechnology Research Centre

• This complements the new EIF national investments– EIF Data Fabric (see .pdf attached)

• Most Universities have “eResearch Directors”– Under PVCRs, facilitate eResearch access for researchers



Outline

• A Short History of eScience in Australia• The Current Australian eScience



eResearch Services Centres

• The theme of the workshop is “Advancing Computational Science in academia and HPC Centres”

• I’m going to answer a more general question“What should you do to create a successful eResearch Services Centre”– eResearch services include computational science– But not everyone needs computational science, it is

a “tool” just like “database design”

• By implication, solving the more general question solves the simpler one


eResearch Services Centres (cont.)

Q: What does “success” mean in the context of an eResearch Services Centre?

A: Common measures include– Sustainability and “critical mass”– Quality of services, or “value” to host/client

research institution(s)– Innovation and strategic impact– Engagement – with users, host/client research

institutions; HPC vendors; national and international collaborators


Lesson #1 – Engage the user community

• What are the research problems being addressed?• What resources are needed to tackle them?• Can the Centre help? How?

Key failures and pitfalls in engagement:– Focus on IT first - “build it and they will come”– Inflexible project management– Focusing either on just the researchers or academic

administration; or only high-end users– Communication: IT specialists and researchers do not

mixThu, June 3rd 2010 19UK eScience Presentation

Lesson #2 – Build and retain expertise

• An eResearch Centre’s core asset is its expertise, not its hardware

• You need both “breadth and depth”, and a culture of collaboration, not prima donnas

• Train your own staff, and get them to train others

Key failures and pitfalls in building expertise:– The NIH syndrome; or “solutions looking for problems”– Insufficient breadth/depth– Getting carried away with technology or stuck in a rut– Insufficient “outreach” or “eResearch Analyst” expertise


Lesson #3 – Get the organizational structure right!

• You need leaders who are both scientists and good managers

• You need to be agile and able to redeploy expertise (“matrix management”)

Key failures and pitfalls in organizational structure:– No governance Board, review, or oversight– No risk management/mitigation– Organizations too tied to, or hampered by, University

politics, rules, and regulations


Lesson #4 – Collaboration

• Build meaningful, mutually beneficial ties and alliances with– Other Centres: regional, national, international– Industry: HPC vendors– IT development communities– Standards organizations– Government

Key failures and pitfalls in collaboration:– Investing resources in “one sided” collaboration– Focus on “marketing” not “collaboration”


Lesson #5 – Grow to a sustainable size

• At least 40 to 50 technical staff is ideal, or – If smaller specialize and outsource skills through

collaboration

• Diversified funding

Key failures and pitfalls in organizational size:– Over a dozen staff requires experienced managers

• Researchers are notoriously bad managers

– Not planning for growth in HR, finance management, or project tracking

– Insufficient funds for organizational size or lack of discretionary funds Thu, June 3rd 2010 23UK eScience Presentation


The “old model” of HPC Centres

• Focus on HPC and “big iron”• Users are expert UNIX users and programmers

– Projects are small specialist research teams– Coding from scratch – “hero codes”– Data is secondary

• Staff are specialist systems administrators and software developers

• Limited in-house training



The “new model” of HPC Centres

• Focus on grid computing, diverse Advanced Computing infrastructure

• Emerging users are not “traditional scientists”– Projects are diverse, collaborative, involve industry

or government collaborators– Community codes; commercial software– Data may be primary (e.g., Biogrid, Biobank)

• Staff include outreach experts (“missionaries”) with scientific background

• Training and building skills is key



Thank You!

Questions?


bill appelbe bill@vpac , vpac (vpac)

Documents

escience infrastructure

national hpc tier

state hpc system

hpc courseware

socialist hpc

hpc little collaboration

australian national

independent research