deploying biobankcloud with karamel/chef and federated authentication in biobankcloud jim dowling,...
DESCRIPTION
*Configured stacks include apps, dependencies, and firewalls. BBC platforms have lots of different stacks* ResourceMgr NN ssh, agent, chef, collectd NodeMgr DN ssh, agent, chef, collectd MYSQLD MGMD ssh, agent, chef, collectd NDBD ssh, agent, chef, collectdTRANSCRIPT
![Page 1: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/1.jpg)
Deploying BiobankCloud with Karamel/Chefand Federated Authentication in BiobankCloud
Jim Dowling, KTH – Royal Institute of Technology
![Page 2: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/2.jpg)
2
Bringing together Biobankers & Bioinformaticians
• Biobankers- NGS data producers• Collections, samples
- Non-programmers
• Bioinformaticians- NGS data analysts- Programmers• Python, R, Matlab, scripts
ITAdmins
Hops-HDFS
Hops-YARN
Web Application
CharonFS
Cuneiform/HiWAY
Karamel
![Page 3: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/3.jpg)
*Configured stacks include apps, dependencies, and firewalls.
BBC platforms have lots of different stacks*
ResourceMgr
NN
ssh, agent, chef, collectd
NodeMgr
DN
ssh, agent, chef, collectd
MYSQLD
MGMD
ssh, agent, chef, collectd
NDBD
ssh, agent, chef, collectd
![Page 4: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/4.jpg)
Plus the Frontend Stack
REST APIs
Glassfish
collectd-server
AngularJS Frontend
Web Application
![Page 5: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/5.jpg)
How do we deploy our PaaS?
NDBD NDBD MGMD
Works NN NN
DN DN DN
Data Center
![Page 6: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/6.jpg)
Deployment on AWS from Karamel using Ssh/Chef
NDBD NDBD MGMD
NN
DN DN DN
Public Cloud (AWS)
DashB NN
![Page 7: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/7.jpg)
Deployment when few public IPs available
NDBD NDBD MGMD
NN
DN DN DN
Private Cloud (OpenStack)
Karamel NN
![Page 8: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/8.jpg)
How do we install the software on the VMs?
![Page 9: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/9.jpg)
Karamel Layers
Google Compute Engine
Virtual Machines
Software is defined in Chef
Chef Cookbooks are publicly available in Github
![Page 10: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/10.jpg)
Ssh and Chef Solo to Install software
ssh ssh ssh
Karamel
RecipesRecipesRecipes Chef Recipes are infrastructureas code: idempotent & composable
ResourceMgr
NN
ssh, agent, chef, collectd
MYSQLD
MGMD
ssh, agent, chef, collectd
NodeMgr
DNssh, agent,
chef, collectd
Dependency Injection of Attributes (No Server or agents)
![Page 11: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/11.jpg)
Karamel Stack
AWS OpenStack
Chef
Bare Metal
JClouds ssh
Karamel API (YAML)
VMIs BitTorrentReduce Install TimesCreate VMs
GCE EGI
![Page 12: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/12.jpg)
Cluster Definition in Karamelname: ApacheHadoopV2
ec2: type: m3.medium region: eu-west-1
cookbooks: hadoop: github: "hopshadoop/apache-hadoop-chef" version: "v0.1" attrs: hadoop: version: 2.7.1
groups: metadata: size: 1 recipes: - hadoop::nn - hadoop::rm - hadoop::jhs datanodes: size: 2 recipes: - hadoop::dn - hadoop::nm
egi: XX: ?? YY: ??
![Page 13: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/13.jpg)
13
Orchestration of Deployments
DSL Service
Cloud Clients
Karamel Engine
Physical Mapping
Orchestrator
![Page 14: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/14.jpg)
14
Karamel FrontEnd
![Page 15: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/15.jpg)
Jclouds/Chef in the Wild• Node provisioning can fail for a variety of reasons.
• Stragglers will appear as clusters grow in size.
• Rate Limiting on (with low defaults) the number of concurrent operations that can be issued to AWS, OpenStack, etc.
![Page 16: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/16.jpg)
Karamel Scheduler• Builds a Directed Acyclic Graph of Chef Recipes
• Scheduler executes Chef recipes as a series of phases- Chef recipes are decomposed into the following phases:
install, <service>
• Respawn & provision failed and slow VMs.
• Strategies for handling rate-limiting clouds.- Automatic retry for failed Chef Recipes
![Page 17: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/17.jpg)
Related Work• Virtualization and Hadoop
- Project Serengeti (VMWare)- Project Savanna (Hortonworks & OpenStack)- Elastic MapReduce (Amazon Web Services)
• Administration of Hadoop Clusters- Cloudera Manager with Puppet- Hortonworks Ambari with Puppet
• AWS OpsWorks• OpenStack Heat• Docker Swarm
![Page 18: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/18.jpg)
BiobankCloud Federation
18
![Page 19: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/19.jpg)
Connecting Biobanks
BiobankMeta-Data
Sample Availability Data
Omics Data (Big Data)
Biobank
Biobank
Biobank
![Page 20: Deploying BiobankCloud with Karamel/Chef and Federated Authentication in BiobankCloud Jim Dowling, KTH Royal Institute of Technology](https://reader033.vdocuments.mx/reader033/viewer/2022051105/5a4d1b757f8b9ab0599b6eec/html5/thumbnails/20.jpg)
20
Authentication• Authentication Provider
- JDBC Realm- 2-Factor Authentication- LDAP
• EGI as Federated Authentication Provider