asgard, the grails app that deploys netflix to the cloud

184
Joe Sondow, Netflix The Grails App that Deploys Netflix to the Cloud @joesondow

Upload: spring-io

Post on 31-Oct-2014

233 views

Category:

Technology


1 download

DESCRIPTION

Speaker: Joe Sondow Asgard is a free and open source Grails application built and used by Netflix to deploy code changes and to manage resources in the Amazon cloud at large scale. In this talk we'll delve into how Asgard works, covering topics such as: Open source motivation, presence, and support. Tour of the user interface. Customizing Asgard for your own company. How Netflix philosophies drive Asgard's design. The Netflix Cloud Model. Easy large deployments and fast rollback. Caching the cloud with Groovy. Visual language for the cloud. Publishing a REST API. Mocking AWS for offline testing. Comparison with the AWS Management Console.

TRANSCRIPT

Page 1: Asgard, the Grails App that Deploys Netflix to the Cloud

Joe Sondow, Netflix

The Grails App that Deploys Netflix to the Cloud

@joesondow

Page 2: Asgard, the Grails App that Deploys Netflix to the Cloud

Slides onlinehttp://slideshare.net/joesondow

@joesondow

Page 3: Asgard, the Grails App that Deploys Netflix to the Cloud

Who am I?

@joesondow

Page 4: Asgard, the Grails App that Deploys Netflix to the Cloud

Who am I?

@joesondow

Page 5: Asgard, the Grails App that Deploys Netflix to the Cloud

Who am I?Joe Sondow

@joesondow

Page 6: Asgard, the Grails App that Deploys Netflix to the Cloud

Who am I?Joe Sondow

Netflix since 2010

@joesondow

Page 7: Asgard, the Grails App that Deploys Netflix to the Cloud

Who am I?Joe Sondow

Netflix since 2010

Asgard lead

@joesondow

Page 8: Asgard, the Grails App that Deploys Netflix to the Cloud

Who am I?Joe Sondow

Netflix since 2010

Asgard lead

Grails

@joesondow

Page 9: Asgard, the Grails App that Deploys Netflix to the Cloud

Who am I?Joe Sondow

Netflix since 2010

Asgard lead

Grails

jQuery

@joesondow

Page 10: Asgard, the Grails App that Deploys Netflix to the Cloud

Why am I here?

Page 11: Asgard, the Grails App that Deploys Netflix to the Cloud

Why am I here?

Page 12: Asgard, the Grails App that Deploys Netflix to the Cloud

Sell you something

Why am I here?

Page 13: Asgard, the Grails App that Deploys Netflix to the Cloud

Sell you something

Discuss business plans

Why am I here?

Page 14: Asgard, the Grails App that Deploys Netflix to the Cloud

Sell you something

Discuss business plans

Answer technical questions

Why am I here?

Page 15: Asgard, the Grails App that Deploys Netflix to the Cloud

Sell you something

Discuss business plans

Answer technical questions

Be a smaller fish in AWS

Why am I here?

Page 16: Asgard, the Grails App that Deploys Netflix to the Cloud

Sell you something

Discuss business plans

Answer technical questions

Be a smaller fish in AWS

Give back to community

Why am I here?

Page 17: Asgard, the Grails App that Deploys Netflix to the Cloud

Sell you something

Discuss business plans

Answer technical questions

Be a smaller fish in AWS

Give back to community

Build cloud standards

Why am I here?

Page 18: Asgard, the Grails App that Deploys Netflix to the Cloud

Sell you something

Discuss business plans

Answer technical questions

Be a smaller fish in AWS

Give back to community

Build cloud standards

Steal your engineers

Why am I here?

Page 19: Asgard, the Grails App that Deploys Netflix to the Cloud

Asgard

Page 20: Asgard, the Grails App that Deploys Netflix to the Cloud

Asgard

Page 21: Asgard, the Grails App that Deploys Netflix to the Cloud

AsgardScreen shots

Page 22: Asgard, the Grails App that Deploys Netflix to the Cloud

AsgardApplication list

Page 23: Asgard, the Grails App that Deploys Netflix to the Cloud

AsgardAuto Scaling Group list

Page 24: Asgard, the Grails App that Deploys Netflix to the Cloud

AsgardCluster deployment, ready for fast rollback

Page 25: Asgard, the Grails App that Deploys Netflix to the Cloud

Asgard

Page 26: Asgard, the Grails App that Deploys Netflix to the Cloud

AsgardApplication deployment

Page 27: Asgard, the Grails App that Deploys Netflix to the Cloud

AsgardApplication deployment

Cloud management

Page 28: Asgard, the Grails App that Deploys Netflix to the Cloud

AsgardApplication deployment

Cloud management

Started 2010

Page 29: Asgard, the Grails App that Deploys Netflix to the Cloud

AsgardApplication deployment

Cloud management

Started 2010

Open source June 2012

Page 30: Asgard, the Grails App that Deploys Netflix to the Cloud

AsgardApplication deployment

Cloud management

Started 2010

Open source June 2012

http://netflix.github.io

Page 31: Asgard, the Grails App that Deploys Netflix to the Cloud

AsgardApplication deployment

Cloud management

Started 2010

Open source June 2012

http://netflix.github.io

100’s of Jira tickets

Page 32: Asgard, the Grails App that Deploys Netflix to the Cloud

AsgardApplication deployment

Cloud management

Started 2010

Open source June 2012

http://netflix.github.io

100’s of Jira tickets

Actively developed

Page 33: Asgard, the Grails App that Deploys Netflix to the Cloud

Source code and downloadhttps://github.com/Netflix/asgard

Page 34: Asgard, the Grails App that Deploys Netflix to the Cloud

User forumhttps://groups.google.com/group/AsgardUsers

Page 36: Asgard, the Grails App that Deploys Netflix to the Cloud

The Asgard Showhttps://youtube.com/TheAsgardShow

Page 37: Asgard, the Grails App that Deploys Netflix to the Cloud

Open Source, Closed Config

Page 38: Asgard, the Grails App that Deploys Netflix to the Cloud

Open Source, Closed ConfigPull company-specific details out of Asgard

Page 39: Asgard, the Grails App that Deploys Netflix to the Cloud

Open Source, Closed Config

Page 40: Asgard, the Grails App that Deploys Netflix to the Cloud

Open Source, Closed Config

Page 41: Asgard, the Grails App that Deploys Netflix to the Cloud

Open Source, Closed ConfigAsgard for Netflix is configured to use company-specific extension points such as standard utility links for instances

Page 42: Asgard, the Grails App that Deploys Netflix to the Cloud

Open Source, Closed ConfigOut-of-the-box Asgard installation has no instance utility links

Page 43: Asgard, the Grails App that Deploys Netflix to the Cloud

Open Source, Closed Config

link  {

       //  Avoid  GStrings  here  because  these  Strings  are  stored  dynamic  templates  for  arbitrary  server  names.        String  logUrlStart  =  'http://${server}:7777'        String  configUrlStart  =  'http://${server}:9999/AdminConfig'        instanceLinkGroupingsToLinkTemplateLists  =  [                        'Logs':  [                                        new  TextLinkTemplate(logUrlStart  +  '/Admin/list?view=tomcat/catalina.out',  'catalina.out'),                                        new  TextLinkTemplate(logUrlStart  +  '/Admin/list',  'Log  File  Archive'),                                        new  TextLinkTemplate(logUrlStart  +  '/Admin/threaddumps',  'Thread  Dumps'),                                        new  TextLinkTemplate(logUrlStart  +  '/AdminProxy',  'Admin  Proxy  Info'),                                        new  TextLinkTemplate(logUrlStart  +  '/AdminStatus',  'Admin  Proxy  Status'),                                        new  TextLinkTemplate(logUrlStart  +  '/GC/index',  'GC  Visualization')                        ],                        'Netflix  Configuration':  [                                        new  TextLinkTemplate(configUrlStart  +  '/prop.html',  'NetflixConfiguration  Properties  Console'),                                        new  TextLinkTemplate(configUrlStart  +  '/libs.html',  'Libraries  Console'),                                        new  TextLinkTemplate(configUrlStart  +  '/machineProps',  'Machine  Readable  Properties'),                                        new  TextLinkTemplate(configUrlStart  +  '/webapp/META-­‐INF/MANIFEST.MF',  'Manifest'),                        ]        ]}

Netflix specific $ASGARD_HOME/Config.groovy

Page 44: Asgard, the Grails App that Deploys Netflix to the Cloud

Open Source, Closed Config

asgardHome = System.getenv('ASGARD_HOME') ?: System.getProperty('ASGARD_HOME') ?: "${System.getProperty('user.home')}/.asgard"

// Locations to search for config files that get merged into the main config.// Config files can either be Java properties files or ConfigSlurper scripts.grails.config.locations = [ "file:${asgardHome}/Config.groovy", 'classpath:sourceVersion.properties']

grails-app/conf/Config.groovy references external configuration file~/.asgard/Config.groovyhttps://github.com/Netflix/asgard/blob/master/grails-app/conf/Config.groovy

Page 45: Asgard, the Grails App that Deploys Netflix to the Cloud

Open Source, Closed Config

grails  {        awsAccounts=["171234567890"]        awsAccountNames=["171234567890":  "prod"]}secret  {        accessId="AKOBIWANLANDOLUKEHAN"        secretKey="Od0INd/C3P0/R2D2atatTIEFIGHTERdeathstar"}cloud  {        accountName="prod"        publicResourceAccounts=["amazon"]}

External Config.groovy also holds the AWS account credentials, or references for finding them.

Page 46: Asgard, the Grails App that Deploys Netflix to the Cloud
Page 47: Asgard, the Grails App that Deploys Netflix to the Cloud

Netflix is the world’s leading Internet television network with nearly 38 million members in 40 countries enjoying more than one billion hours of TV shows and movies per month, including original series.

(from http://ir.netflix.com)

Page 48: Asgard, the Grails App that Deploys Netflix to the Cloud

Freedom and Responsibility

Page 49: Asgard, the Grails App that Deploys Netflix to the Cloud

Freedom and ResponsibilityCorporate culture and the Cloud

Page 50: Asgard, the Grails App that Deploys Netflix to the Cloud

Freedom and Responsibility

Page 51: Asgard, the Grails App that Deploys Netflix to the Cloud

Freedom and ResponsibilityCloud SOA

Page 52: Asgard, the Grails App that Deploys Netflix to the Cloud

Freedom and ResponsibilityCloud SOA

100’s of services

Page 53: Asgard, the Grails App that Deploys Netflix to the Cloud

Freedom and ResponsibilityCloud SOA

100’s of services

Small teams

Page 54: Asgard, the Grails App that Deploys Netflix to the Cloud

Freedom and ResponsibilityCloud SOA

100’s of services

Small teams

Independent releases

Page 55: Asgard, the Grails App that Deploys Netflix to the Cloud

Freedom and ResponsibilityCloud SOA

100’s of services

Small teams

Independent releases

Controlled chaos

Page 56: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment model

Page 57: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment modelApplications and Clusters

Page 58: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment model

Page 59: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment model

Auto ScalingGroup

Page 60: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment model

Auto ScalingGroup

LaunchConfiguration

Page 61: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment model

Auto ScalingGroup

LaunchConfiguration

Elastic LoadBalancer

Page 62: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment model

Auto ScalingGroup

LaunchConfiguration

Amazon MachineImage

Elastic LoadBalancer

Page 63: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment model

Auto ScalingGroup

LaunchConfiguration

SecurityGroup

Amazon MachineImage

Elastic LoadBalancer

Page 64: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment model

Auto ScalingGroup

LaunchConfiguration

SecurityGroup

Amazon MachineImage

Instances

Elastic LoadBalancer

Page 65: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment model

Auto ScalingGroup

LaunchConfiguration

SecurityGroup

Amazon MachineImage

Instances

Elastic LoadBalancer

Page 66: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment model

Auto ScalingGroup

LaunchConfiguration

SecurityGroup

Amazon MachineImage

Instances

Elastic LoadBalancer

Page 67: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment model

Auto ScalingGroup

LaunchConfiguration

SecurityGroup

Amazon MachineImage

Instances

Elastic LoadBalancer

Page 68: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment model

Page 69: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment model

Search

Page 70: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment model

API Search

Page 71: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment model

API

Ratings

Search

Page 72: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment modelStreaming Starts

API

Ratings

Search

Page 73: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment modelStreaming Starts

Autocomplete API

Ratings

Search

Page 74: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment modelStreaming Starts

Autocomplete API

Sign Up

Ratings

Search

Page 75: Asgard, the Grails App that Deploys Netflix to the Cloud

Cloud deployment modelStreaming Starts

Autocomplete API

Sign Up

Ratings

Search

ApplicationApplication

ApplicationApplication

Application

Application

Page 76: Asgard, the Grails App that Deploys Netflix to the Cloud

Inventing the Application

Page 77: Asgard, the Grails App that Deploys Netflix to the Cloud

Inventing the ApplicationProblem:

Application is not an Amazon concept

Page 78: Asgard, the Grails App that Deploys Netflix to the Cloud

Inventing the ApplicationProblem:

Application is not an Amazon concept

Solution:

Create an Application database in the cloud

Enforce naming conventions on Amazon objects

Page 79: Asgard, the Grails App that Deploys Netflix to the Cloud

Fast Rollback

Page 80: Asgard, the Grails App that Deploys Netflix to the Cloud

Fast RollbackOptimism causes outages

Page 81: Asgard, the Grails App that Deploys Netflix to the Cloud

Fast RollbackOptimism causes outages

Production traffic is unique

Page 82: Asgard, the Grails App that Deploys Netflix to the Cloud

Fast RollbackOptimism causes outages

Production traffic is unique

Keep old version running

Page 83: Asgard, the Grails App that Deploys Netflix to the Cloud

Fast RollbackOptimism causes outages

Production traffic is unique

Keep old version running

Switch traffic to new version

Page 84: Asgard, the Grails App that Deploys Netflix to the Cloud

Fast RollbackOptimism causes outages

Production traffic is unique

Keep old version running

Switch traffic to new version

Monitor results

Page 85: Asgard, the Grails App that Deploys Netflix to the Cloud

Fast RollbackOptimism causes outages

Production traffic is unique

Keep old version running

Switch traffic to new version

Monitor results

Revert traffic quickly

Page 86: Asgard, the Grails App that Deploys Netflix to the Cloud

Fast Rollback

Page 87: Asgard, the Grails App that Deploys Netflix to the Cloud

Fast Rollback

api-usprod-v007

api-frontend

Page 88: Asgard, the Grails App that Deploys Netflix to the Cloud

Fast Rollback

api-usprod-v007

api-frontend

api-usprod-v008

Page 89: Asgard, the Grails App that Deploys Netflix to the Cloud

Fast Rollback

api-usprod-v007

api-frontend

api-usprod-v008

Page 90: Asgard, the Grails App that Deploys Netflix to the Cloud

Fast Rollback

api-usprod-v007

api-frontend

api-usprod-v008

Page 91: Asgard, the Grails App that Deploys Netflix to the Cloud

Fast Rollback

api-usprod-v007

api-frontend

api-usprod-v008

Page 92: Asgard, the Grails App that Deploys Netflix to the Cloud

Fast Rollback

api-usprod-v007

api-frontend

Page 93: Asgard, the Grails App that Deploys Netflix to the Cloud

Inventing the Cluster

Page 94: Asgard, the Grails App that Deploys Netflix to the Cloud

Inventing the ClusterProblem:

Two ASGs with one function but different names

Page 95: Asgard, the Grails App that Deploys Netflix to the Cloud

Inventing the ClusterProblem:

Two ASGs with one function but different names

Solution:

Append version number in reserved format

Parse ASG name to determine long-term “cluster”

Page 96: Asgard, the Grails App that Deploys Netflix to the Cloud

Inventing the Cluster

api

api-usprod

api-usprod-v007

api-usprod-v008

Application

Cluster

Auto Scaling Group

Auto Scaling Group

Instead of keeping a database in sync, use naming conventions to store the source in truth in Amazon’s API

Page 97: Asgard, the Grails App that Deploys Netflix to the Cloud

Database Aversion

Page 98: Asgard, the Grails App that Deploys Netflix to the Cloud

Database AversionStoring metadata on cloud objects

Page 99: Asgard, the Grails App that Deploys Netflix to the Cloud

Database Aversion

Page 100: Asgard, the Grails App that Deploys Netflix to the Cloud

Database Aversion

Naming conventions

Page 101: Asgard, the Grails App that Deploys Netflix to the Cloud

Database Aversion

Naming conventions

Tagging conventions

Page 102: Asgard, the Grails App that Deploys Netflix to the Cloud

Database Aversion

Naming conventions

Tagging conventions

No GORM domain objects

Page 103: Asgard, the Grails App that Deploys Netflix to the Cloud

Database Aversion

Naming conventions

Tagging conventions

No GORM domain objects

AWS Java SDK

Page 104: Asgard, the Grails App that Deploys Netflix to the Cloud

Database Aversion

Naming conventions

Tagging conventions

No GORM domain objects

AWS Java SDK

Less to go out of sync

Page 105: Asgard, the Grails App that Deploys Netflix to the Cloud

Database Aversion

Naming conventions

Tagging conventions

No GORM domain objects

AWS Java SDK

Less to go out of sync

Shared source of truth

Page 106: Asgard, the Grails App that Deploys Netflix to the Cloud

Caching the Cloud

Page 107: Asgard, the Grails App that Deploys Netflix to the Cloud

Caching the CloudResponsive, massive, multi-region metadata

Page 108: Asgard, the Grails App that Deploys Netflix to the Cloud

Caching the Cloud

Page 109: Asgard, the Grails App that Deploys Netflix to the Cloud

Caching the Cloud

Large counts

Page 110: Asgard, the Grails App that Deploys Netflix to the Cloud

Caching the Cloud

Large counts

Many types

Page 111: Asgard, the Grails App that Deploys Netflix to the Cloud

Caching the Cloud

Large counts

Many types

Complex relationships

Page 112: Asgard, the Grails App that Deploys Netflix to the Cloud

Caching the Cloud

Large counts

Many types

Complex relationships

Multiple regions

Page 113: Asgard, the Grails App that Deploys Netflix to the Cloud

Caching the Cloud

Large counts

Many types

Complex relationships

Multiple regions

Consistent single objects

Page 114: Asgard, the Grails App that Deploys Netflix to the Cloud

Caching the Cloud

Large counts

Many types

Complex relationships

Multiple regions

Consistent single objects

Eventually consistent lists

Page 115: Asgard, the Grails App that Deploys Netflix to the Cloud

class  Caches  {        final  CachedMap<AppRegistration>  allApplications        final  CachedMap<ApplicationMetrics>  allApplicationMetrics        final  CachedMap<HardwareProfile>  allHardwareProfiles        final  MultiRegionCachedMap<MetricAlarm>  allAlarms        final  MultiRegionCachedMap<ApplicationInstance>  allApplicationInstances        final  MultiRegionCachedMap<AutoScalingGroup>  allAutoScalingGroups        final  MultiRegionCachedMap<AvailabilityZone>  allAvailabilityZones        final  MultiRegionCachedMap<Cluster>  allClusters        final  MultiRegionCachedMap<DBInstance>  allDBInstances        final  MultiRegionCachedMap<DBSecurityGroup>  allDBSecurityGroups        final  MultiRegionCachedMap<DBSnapshot>  allDBSnapshots        final  MultiRegionCachedMap<String>  allDomains        final  MultiRegionCachedMap<FastProperty>  allFastProperties        final  MultiRegionCachedMap<Image>  allImages        final  MultiRegionCachedMap<Instance>  allInstances        final  MultiRegionCachedMap<InstanceTypeData>  allInstanceTypes        final  MultiRegionCachedMap<KeyPairInfo>  allKeyPairs        final  MultiRegionCachedMap<LaunchConfiguration>  allLaunchConfigurations        //  etc.        //  etc.        //  etc.}

Caching the Cloud

Page 116: Asgard, the Grails App that Deploys Netflix to the Cloud

class  Caches  {        final  CachedMap<AppRegistration>  allApplications        final  CachedMap<ApplicationMetrics>  allApplicationMetrics        final  CachedMap<HardwareProfile>  allHardwareProfiles        final  MultiRegionCachedMap<MetricAlarm>  allAlarms        final  MultiRegionCachedMap<ApplicationInstance>  allApplicationInstances        final  MultiRegionCachedMap<AutoScalingGroup>  allAutoScalingGroups        final  MultiRegionCachedMap<AvailabilityZone>  allAvailabilityZones        final  MultiRegionCachedMap<Cluster>  allClusters        final  MultiRegionCachedMap<DBInstance>  allDBInstances        final  MultiRegionCachedMap<DBSecurityGroup>  allDBSecurityGroups        final  MultiRegionCachedMap<DBSnapshot>  allDBSnapshots        final  MultiRegionCachedMap<String>  allDomains        final  MultiRegionCachedMap<FastProperty>  allFastProperties        final  MultiRegionCachedMap<Image>  allImages        final  MultiRegionCachedMap<Instance>  allInstances        final  MultiRegionCachedMap<InstanceTypeData>  allInstanceTypes        final  MultiRegionCachedMap<KeyPairInfo>  allKeyPairs        final  MultiRegionCachedMap<LaunchConfiguration>  allLaunchConfigurations        //  etc.        //  etc.        //  etc.}

Caching the Cloudclass  Caches  {        final  CachedMap<AppRegistration>  allApplications        final  CachedMap<ApplicationMetrics>  allApplicationMetrics        final  CachedMap<HardwareProfile>  allHardwareProfiles        final  MultiRegionCachedMap<MetricAlarm>  allAlarms        final  MultiRegionCachedMap<ApplicationInstance>  allApplicationInstances        final  MultiRegionCachedMap<AutoScalingGroup>  allAutoScalingGroups        final  MultiRegionCachedMap<AvailabilityZone>  allAvailabilityZones        final  MultiRegionCachedMap<Cluster>  allClusters        final  MultiRegionCachedMap<DBInstance>  allDBInstances        final  MultiRegionCachedMap<DBSecurityGroup>  allDBSecurityGroups        final  MultiRegionCachedMap<DBSnapshot>  allDBSnapshots        final  MultiRegionCachedMap<String>  allDomains        final  MultiRegionCachedMap<FastProperty>  allFastProperties        final  MultiRegionCachedMap<Image>  allImages        final  MultiRegionCachedMap<Instance>  allInstances        final  MultiRegionCachedMap<InstanceTypeData>  allInstanceTypes        final  MultiRegionCachedMap<KeyPairInfo>  allKeyPairs        final  MultiRegionCachedMap<LaunchConfiguration>  allLaunchConfigurations        //  etc.        //  etc.        //  etc.}

Page 117: Asgard, the Grails App that Deploys Netflix to the Cloud

class  AwsRdsService  implements  CacheInitializer,  InitializingBean  {

       MultiRegionAwsClient<AmazonRDS>  awsClient        Caches  caches

       void  initializeCaches()  {                caches.allDBInstances.ensureSetUp({  Region  region  -­‐>  retrieveDBInstances(region)  })        }

       private  List<DBInstance>  retrieveDBInstances(Region  region)  {                awsClient.by(region).describeDBInstances(new  DescribeDBInstancesRequest()).getDBInstances()        }

       Collection<DBInstance>  getDBInstances(UserContext  userContext)  {                caches.allDBInstances.by(userContext.region).list()        }}

Caching the Cloudclass  AwsRdsService  implements  CacheInitializer,  InitializingBean  {

       MultiRegionAwsClient<AmazonRDS>  awsClient        Caches  caches

       void  initializeCaches()  {                caches.allDBInstances.ensureSetUp({  Region  region  -­‐>  retrieveDBInstances(region)  })        }

       private  List<DBInstance>  retrieveDBInstances(Region  region)  {                awsClient.by(region).describeDBInstances(new  DescribeDBInstancesRequest()).getDBInstances()        }

       Collection<DBInstance>  getDBInstances(UserContext  userContext)  {                caches.allDBInstances.by(userContext.region).list()        }}

Page 118: Asgard, the Grails App that Deploys Netflix to the Cloud

class  AwsRdsService  implements  CacheInitializer,  InitializingBean  {

       MultiRegionAwsClient<AmazonRDS>  awsClient        Caches  caches

       void  initializeCaches()  {                caches.allDBInstances.ensureSetUp({  Region  region  -­‐>  retrieveDBInstances(region)  })        }

       private  List<DBInstance>  retrieveDBInstances(Region  region)  {                awsClient.by(region).describeDBInstances(new  DescribeDBInstancesRequest()).getDBInstances()        }

       Collection<DBInstance>  getDBInstances(UserContext  userContext)  {                caches.allDBInstances.by(userContext.region).list()        }}

Caching the Cloudclass  AwsRdsService  implements  CacheInitializer,  InitializingBean  {

       MultiRegionAwsClient<AmazonRDS>  awsClient        Caches  caches

       void  initializeCaches()  {                caches.allDBInstances.ensureSetUp({  Region  region  -­‐>  retrieveDBInstances(region)  })        }

       private  List<DBInstance>  retrieveDBInstances(Region  region)  {                awsClient.by(region).describeDBInstances(new  DescribeDBInstancesRequest()).getDBInstances()        }

       Collection<DBInstance>  getDBInstances(UserContext  userContext)  {                caches.allDBInstances.by(userContext.region).list()        }}

class  AwsRdsService  implements  CacheInitializer,  InitializingBean  {

       MultiRegionAwsClient<AmazonRDS>  awsClient        Caches  caches

       void  initializeCaches()  {                caches.allDBInstances.ensureSetUp({  Region  region  -­‐>  retrieveDBInstances(region)  })        }

       private  List<DBInstance>  retrieveDBInstances(Region  region)  {                awsClient.by(region).describeDBInstances(new  DescribeDBInstancesRequest()).getDBInstances()        }

       Collection<DBInstance>  getDBInstances(UserContext  userContext)  {                caches.allDBInstances.by(userContext.region).list()        }}

Page 119: Asgard, the Grails App that Deploys Netflix to the Cloud

class  AwsRdsService  implements  CacheInitializer,  InitializingBean  {

       MultiRegionAwsClient<AmazonRDS>  awsClient        Caches  caches

       void  initializeCaches()  {                caches.allDBInstances.ensureSetUp({  Region  region  -­‐>  retrieveDBInstances(region)  })        }

       private  List<DBInstance>  retrieveDBInstances(Region  region)  {                awsClient.by(region).describeDBInstances(new  DescribeDBInstancesRequest()).getDBInstances()        }

       Collection<DBInstance>  getDBInstances(UserContext  userContext)  {                caches.allDBInstances.by(userContext.region).list()        }}

Caching the Cloudclass  AwsRdsService  implements  CacheInitializer,  InitializingBean  {

       MultiRegionAwsClient<AmazonRDS>  awsClient        Caches  caches

       void  initializeCaches()  {                caches.allDBInstances.ensureSetUp({  Region  region  -­‐>  retrieveDBInstances(region)  })        }

       private  List<DBInstance>  retrieveDBInstances(Region  region)  {                awsClient.by(region).describeDBInstances(new  DescribeDBInstancesRequest()).getDBInstances()        }

       Collection<DBInstance>  getDBInstances(UserContext  userContext)  {                caches.allDBInstances.by(userContext.region).list()        }}

class  AwsRdsService  implements  CacheInitializer,  InitializingBean  {

       MultiRegionAwsClient<AmazonRDS>  awsClient        Caches  caches

       void  initializeCaches()  {                caches.allDBInstances.ensureSetUp({  Region  region  -­‐>  retrieveDBInstances(region)  })        }

       private  List<DBInstance>  retrieveDBInstances(Region  region)  {                awsClient.by(region).describeDBInstances(new  DescribeDBInstancesRequest()).getDBInstances()        }

       Collection<DBInstance>  getDBInstances(UserContext  userContext)  {                caches.allDBInstances.by(userContext.region).list()        }}

class  AwsRdsService  implements  CacheInitializer,  InitializingBean  {

       MultiRegionAwsClient<AmazonRDS>  awsClient        Caches  caches

       void  initializeCaches()  {                caches.allDBInstances.ensureSetUp({  Region  region  -­‐>  retrieveDBInstances(region)  })        }

       private  List<DBInstance>  retrieveDBInstances(Region  region)  {                awsClient.by(region).describeDBInstances(new  DescribeDBInstancesRequest()).getDBInstances()        }

       Collection<DBInstance>  getDBInstances(UserContext  userContext)  {                caches.allDBInstances.by(userContext.region).list()        }}

Page 120: Asgard, the Grails App that Deploys Netflix to the Cloud

class  AwsRdsService  implements  CacheInitializer,  InitializingBean  {

       MultiRegionAwsClient<AmazonRDS>  awsClient        Caches  caches

       void  initializeCaches()  {                caches.allDBInstances.ensureSetUp({  Region  region  -­‐>  retrieveDBInstances(region)  })        }

       private  List<DBInstance>  retrieveDBInstances(Region  region)  {                awsClient.by(region).describeDBInstances(new  DescribeDBInstancesRequest()).getDBInstances()        }

       Collection<DBInstance>  getDBInstances(UserContext  userContext)  {                caches.allDBInstances.by(userContext.region).list()        }}

Caching the Cloudclass  AwsRdsService  implements  CacheInitializer,  InitializingBean  {

       MultiRegionAwsClient<AmazonRDS>  awsClient        Caches  caches

       void  initializeCaches()  {                caches.allDBInstances.ensureSetUp({  Region  region  -­‐>  retrieveDBInstances(region)  })        }

       private  List<DBInstance>  retrieveDBInstances(Region  region)  {                awsClient.by(region).describeDBInstances(new  DescribeDBInstancesRequest()).getDBInstances()        }

       Collection<DBInstance>  getDBInstances(UserContext  userContext)  {                caches.allDBInstances.by(userContext.region).list()        }}

class  AwsRdsService  implements  CacheInitializer,  InitializingBean  {

       MultiRegionAwsClient<AmazonRDS>  awsClient        Caches  caches

       void  initializeCaches()  {                caches.allDBInstances.ensureSetUp({  Region  region  -­‐>  retrieveDBInstances(region)  })        }

       private  List<DBInstance>  retrieveDBInstances(Region  region)  {                awsClient.by(region).describeDBInstances(new  DescribeDBInstancesRequest()).getDBInstances()        }

       Collection<DBInstance>  getDBInstances(UserContext  userContext)  {                caches.allDBInstances.by(userContext.region).list()        }}

class  AwsRdsService  implements  CacheInitializer,  InitializingBean  {

       MultiRegionAwsClient<AmazonRDS>  awsClient        Caches  caches

       void  initializeCaches()  {                caches.allDBInstances.ensureSetUp({  Region  region  -­‐>  retrieveDBInstances(region)  })        }

       private  List<DBInstance>  retrieveDBInstances(Region  region)  {                awsClient.by(region).describeDBInstances(new  DescribeDBInstancesRequest()).getDBInstances()        }

       Collection<DBInstance>  getDBInstances(UserContext  userContext)  {                caches.allDBInstances.by(userContext.region).list()        }}

class  AwsRdsService  implements  CacheInitializer,  InitializingBean  {

       MultiRegionAwsClient<AmazonRDS>  awsClient        Caches  caches

       void  initializeCaches()  {                caches.allDBInstances.ensureSetUp({  Region  region  -­‐>  retrieveDBInstances(region)  })        }

       private  List<DBInstance>  retrieveDBInstances(Region  region)  {                awsClient.by(region).describeDBInstances(new  DescribeDBInstancesRequest()).getDBInstances()        }

       Collection<DBInstance>  getDBInstances(UserContext  userContext)  {                caches.allDBInstances.by(userContext.region).list()        }}

Page 121: Asgard, the Grails App that Deploys Netflix to the Cloud

Caching the Cloud

Page 122: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

Page 123: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the CloudTango open source icons

Page 124: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

Page 125: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

AWS is intimidating

Page 126: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

AWS is intimidating

Many object types

Page 127: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

AWS is intimidating

Many object types

Help newbie users

Page 128: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

AWS is intimidating

Many object types

Help newbie users

Reduce cognitive load

Page 129: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

AWS is intimidating

Many object types

Help newbie users

Reduce cognitive load

Make it easy

Page 130: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

AWS is intimidating

Many object types

Help newbie users

Reduce cognitive load

Make it easy

Avoid surprises

Page 131: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

Page 132: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

At a glance, these nav bar items look alike.

Page 133: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

At a glance, these nav bar items look alike.

Page 134: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

Page 135: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

Some screens have multiple action buttons that look too similar.

Page 136: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

Some screens have multiple action buttons that look too similar.

Page 137: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

Page 138: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

Because of naming conventions, these links look alike.

Page 139: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

Because of naming conventions, these links look alike.

Page 140: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

Page 141: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the CloudThe indicators for the current AWS region are too easy to miss.

Page 142: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the CloudThe indicators for the current AWS region are too easy to miss.

Page 143: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

These availability zones are important to recognize at a glance but their names look similar, and they appear on many screens.

Page 144: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

These availability zones are important to recognize at a glance but their names look similar, and they appear on many screens.

Page 145: Asgard, the Grails App that Deploys Netflix to the Cloud

Visual Language for the Cloud

Page 146: Asgard, the Grails App that Deploys Netflix to the Cloud

Tango Icons

Page 147: Asgard, the Grails App that Deploys Netflix to the Cloud

Tango Iconshttp://tango.freedesktop.org/

Page 150: Asgard, the Grails App that Deploys Netflix to the Cloud

Tango Iconshttp://tango.freedesktop.org/http://tango.freedesktop.org/Tango_Icon_Theme_Guidelines

http://commons.wikimedia.org/wiki/Tango_icons

Used by Firefox, Jenkins, GIMP, OpenOffice, VMWare

Page 151: Asgard, the Grails App that Deploys Netflix to the Cloud

REST API in Grails

Page 152: Asgard, the Grails App that Deploys Netflix to the Cloud

REST API in GrailsEnable external mashups with cloud data

Page 153: Asgard, the Grails App that Deploys Netflix to the Cloud

REST API in Grails

Page 154: Asgard, the Grails App that Deploys Netflix to the Cloud

REST API in Grails

Page 155: Asgard, the Grails App that Deploys Netflix to the Cloud

REST API in Grails

Page 156: Asgard, the Grails App that Deploys Netflix to the Cloud

REST API in Grails

Page 157: Asgard, the Grails App that Deploys Netflix to the Cloud

REST API in Grails

Page 158: Asgard, the Grails App that Deploys Netflix to the Cloud

REST API in Grails

Page 159: Asgard, the Grails App that Deploys Netflix to the Cloud

REST API in Grails

Page 160: Asgard, the Grails App that Deploys Netflix to the Cloud

REST API in Grails

Page 161: Asgard, the Grails App that Deploys Netflix to the Cloud

REST API in Grails

Page 162: Asgard, the Grails App that Deploys Netflix to the Cloud

REST API in Grailsdef show = { String name = params.id UserContext userContext = UserContext.of(request) AppRegistration app = applicationService.getRegisteredApplication(userContext, name) def groups = awsAutoScalingService.getAutoScalingGroupsForApp(userContext, name) List<String> clusterNames = groups.collect { Relationships.clusterFromGroupName(it.autoScalingGroupName) }.unique() Map details = [ app: app, strictName: Relationships.checkStrictName(app.name), clusters: clusterNames, groups: groups, balancers: awsLoadBalancerService.getLoadBalancersForApp(userContext, name), securities: awsEc2Service.getSecurityGroupsForApp(userContext, name), appSecurityGroup: awsEc2Service.getSecurityGroup(userContext, name), launches: awsAutoScalingService.getLaunchConfigurationsForApp(userContext, name), ] withFormat { html { return details } xml { new XML(details).render(response) } json { new JSON(details).render(response) } }}

ApplicationController.groovy

Page 163: Asgard, the Grails App that Deploys Netflix to the Cloud

Offline Development

Page 164: Asgard, the Grails App that Deploys Netflix to the Cloud

Offline DevelopmentMakes on a plane

Page 165: Asgard, the Grails App that Deploys Netflix to the Cloud

Offline DevelopmentMock data

Mock behavior

System property switch offline=true

Page 166: Asgard, the Grails App that Deploys Netflix to the Cloud

Mock Datahttp://asgardprod/us-east-1/autoScaling/list.json

Page 167: Asgard, the Grails App that Deploys Netflix to the Cloud

Mock DataParse JSON

Page 168: Asgard, the Grails App that Deploys Netflix to the Cloud

Mock BehaviorOverride Amazon Java client methods

Page 169: Asgard, the Grails App that Deploys Netflix to the Cloud

System Propertygrails run-app -Doffline=true

Page 170: Asgard, the Grails App that Deploys Netflix to the Cloud

Why not the AWS console?

Page 171: Asgard, the Grails App that Deploys Netflix to the Cloud

Why not the AWS console?

Page 172: Asgard, the Grails App that Deploys Netflix to the Cloud

Why not the AWS console?

Page 173: Asgard, the Grails App that Deploys Netflix to the Cloud

Why not the AWS console?Hide keys

Page 174: Asgard, the Grails App that Deploys Netflix to the Cloud

Why not the AWS console?Hide keys

Customize model

Page 175: Asgard, the Grails App that Deploys Netflix to the Cloud

Why not the AWS console?Hide keys

Customize model

Enforce conventions

Page 176: Asgard, the Grails App that Deploys Netflix to the Cloud

Why not the AWS console?Hide keys

Customize model

Enforce conventions

Automate workflow

Page 177: Asgard, the Grails App that Deploys Netflix to the Cloud

Why not the AWS console?Hide keys

Customize model

Enforce conventions

Automate workflow

Log changes

Page 178: Asgard, the Grails App that Deploys Netflix to the Cloud

Why not the AWS console?Hide keys

Customize model

Enforce conventions

Automate workflow

Log changes

Integrate systems

Page 179: Asgard, the Grails App that Deploys Netflix to the Cloud

Why not the AWS console?Hide keys

Customize model

Enforce conventions

Automate workflow

Log changes

Integrate systems

Customize REST API

Page 180: Asgard, the Grails App that Deploys Netflix to the Cloud

@NetflixOSS

Page 181: Asgard, the Grails App that Deploys Netflix to the Cloud

@NetflixOSShttp://techblog.netflix.com

Page 182: Asgard, the Grails App that Deploys Netflix to the Cloud

@NetflixOSShttp://techblog.netflix.comhttp://netflix.github.io

Page 183: Asgard, the Grails App that Deploys Netflix to the Cloud

Thank you

http://github.com/Netflix/asgard

Page 184: Asgard, the Grails App that Deploys Netflix to the Cloud

Thank youQuestions?

@AsgardOSS @joesondow jobs.netflix.com

http://github.com/Netflix/asgard