creating a developer pipeline by teaching gateway technologies

Welcome to the SGCI Webinar!• We will be starting shortly.

• Your audio has been muted, and you are encouraged to turn off your video during the presentation.• Controls for these are near the bottom of

the right-side control panel for BlueJeans.

• You may submit questions at any time using Chat, and the moderator will share them with the presenter when appropriate.

• This presentation will be recorded and slides will be posted.

1

Creating a developer pipeline by teaching

gateway technologiesMarlon Pierce, Suresh Marru

Science Gateways Research Center, Indiana Universitymarpierc@iu.edu, smarru@iu.edu

http://sgrc.iu.edu

The Pipeline Problem

Finding and retaining people to work on science gateways is tough.

If only there was an untapped reservoir of talent close by....

The Pipeline Solution

Recruit students to work with us by teaching a graduate level class on the underpinnings of gateways.

Embrace change.

What Is a Science Gateway?A Science Gateway is a user-centric environment

for conducting online computational science

(Not actual IU students)

What if Mark Zuckerberg Read More Chemistry and Biology Books?

Goal 1: Apply basic distributed computing concepts to Science

Gateways.

Goal 2: Apply new architectures, methodologies, and technologies to

Science Gateways: Microservices, DevOps

Browser

Web Interface Server

Application Server

Server SDK

Client SDK

IU: Big Red 2

Resource Plugins

XSEDE: Stampede XSEDE: Comet Juelich: Jureca

HTTPS

HTTP or TCP/IP

Facebook doesn’t look like this, so why should science gateways?

Goal 3: Teach open source software practices

Challenges for Science Gateways• Providing a rich user experience • Defining an API for the application server• Defining the right microservices.• Implementing the components, wiring them together correctly.• Supporting multiple gateway tenants• Fault tolerance for components• State management• Continuous delivery• Security management• Supporting full scientific exploratory cycle

What Do We Teach?

A quick survey of course material

Micro-Services

DevOpsDistributed Systems

Putting It All Together: Gateways and Microservices

Virtualization, Containers, Docker

API and Metadata Model Design

Messaging Systems: RabbitMQ, Apache Kafka

Resource Management and Scheduling: Mesos, Aurora, and Torque

Distributed State Management: Zookeeper, Consul

Security: OAuth2 and OpenIDConnect

Putting It All In Place: Continuous Integration and Deployment

Mission Accomplished! Students learned on their own.

More Information and Next Steps

• http://courses.airavata.org• Additional material in Canvas

• Details on assignments• We are interested in taking this to a broader audience

Thank you for participating!

• Please offer your feedback with our 30-second evaluation:http://sciencegateways.org/webinareval

• Join us next month (March 8) for

Building a Modern Research Data Portal with Globus – Introduction to the Globus Platform

36

Science!

Spring 2016 Course Overview• Students organized into teams of 3-4• 8 biweekly project milestones, each depended on the previous

assignment.• Basic sequence, Part 1:

• Learn to submit jobs on IU’s Karst Cluster, MOAB scheduler• Develop core job management application• Design an API• Design a Web front end• Capture job execution metadata

• All code must be in GitHub, must build cleanly on the grader’s laptop

Spring 2016 Project Milestones, Part 2• Integrate with a continuous integration and deployment system

• Travis-CI and Amazon Code-Deploy• Apache Jenkins on Jetstream

• Break up your monolithic Web applications into microservices• Combine with CI/CD

Spring 2016: What Worked, What Didn’t?• Small class, we got to know the

students pretty well.• Students managed to build basic

functioning gateways that used CI/CD automatically linked to GitHub accounts.

• Students got bogged down in job submission issues.

• Students had trouble breaking up their monolithic Web applications into microservices.

Fall 2016: Begin at the End• For Fall 2016, we reverse the order.• Students learned CI/CD and microservices in the first assignments.• We moved aggressively to problems of load balancing, fault tolerance,

and containerization of services.• We de-emphasized job submission to traditional clusters

• Used Apache Mesos and Apache Aurora for this.

Fall 2016: What Worked, What Didn’t• All the students were able to

keep up with the assignments.• Many were using technologies for

the first time.

• Some surprises• Some teams took it as a challenge

to use different approaches.

• Didn’t have the chance to get to know students as well as before.

• Larger class, graders

• Monthly assignments were too large.

• Biweekly worked better.

Spring 2017: Advanced Topics• In the Spring 2016 and Fall 2016 courses, we knew the answers

(mostly).• Now in Spring 2017 course, we tackle problems we don’t know the

answers to.• Or, if you prefer, for a given problem,

• Introductory Class: Find an answer that works• Advanced Class: Find the best answer

• How do you do that?