keynote: paving the way and making a difference: aws in the public sector
DESCRIPTION
Teresa Carlson, Vice President, Worldwide Public Sector, Amazon Web Services with: Mark Davies, Technical Lead, ChEMBL Group, European Bioinformatics Institute (EMBL-EBI); Phil Young, Head of Online, Transport for London (TfL) Tim Marshall, Executive Director of Jisc Technologies and CEO of JanetTRANSCRIPT
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
AWS Government, Education, and Nonprofits SymposiumLondon I 21 October 2014
AWS in the Public Sector
Teresa CarlsonVice President
World Wide Public Sector
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Strong growth and adoption
800+
government
agencies
3000+
educational
institutions
10000+
nonprofit
organizations
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Used by Government Agencies & Educational Institutions
Worldwide
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
AWS Partners Focused on Public Sector
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
AWS Rapid Pace of Innovation
2009
Amazon RDS
Amazon VPC
Auto Scaling
Elastic Load
Balancing
+48
2010
Amazon SNS
AWS Identity
& Access
Management
Amazon Route 53
+61
2011
Amazon
ElastiCache
Amazon SES
AWS
CloudFormation
AWS Direct
Connect
AWS Elastic
Beanstalk
GovCloud
+82
Amazon
CloudTrail
Amazon
CloudHSM
Amazon
WorkSpaces
Amazon Kinesis
Amazon Elastic
Transcoder
Amazon
AppStream
AWS OpsWorks
+280
2013
Amazon SWF
Amazon Redshift
Amazon Glacier
Amazon
Dynamo DB
Amazon
CloudSearch
AWS Storage
Gateway
AWS Data
Pipeline
+159
2012
Since inception AWS has:
• Released 942 new services and features
• Introduced over 35 major new services
• Announced 45 price reductions
2008
+24Amazon EBS
Amazon
CloudFront
+285
2014
Amazon Cognito
Amazon Zocalo
Amazon Mobile
Analytics
*as of Aug 18, 2014
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Amazon
ElasticCache
reduces prices for
cache nodes by an
average of 34%
March 26, 2014
34%Amazon S3 reduces
prices for Standard and
Reduced Redundancy
Storage, by an average
of 51%
March 26, 2014
51%
We’ve announced price reductions 45* times since
our inception in 2006. Recent price drops included…
*as of July 31, 2014
Amazon Route 53
lowers prices for both
Standard Queries and
Latency Based
Routing Queries by
20%
July 31, 2014
20%
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Architected for Enterprise Security RequirementsCertifications and accreditations for
workloads that matter
AWS CloudTrail - AWS API call
logging for governance & compliance
Stores data in
S3, or archive
to Glacier
Log and review
user activity
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
AWS Helps Government Cut Costs
“We are in the process of putting most of our public-facing
data in an Amazon cloud service,” said Terry Halvorsen,
the Chief Information Officer of the Department of the
Navy, in a keynote at Meritalk’s Data Center Brainstorm
event Thursday. Halvorsen said the move could save
the Navy as much as 60 percent versus the cost of
managing that data in its own data centers.-Data Center Knowledge,
March 14, 2014
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
AWS Global Infrastructure
10 Regions
including GovCloud US
26 Availability
Zones
51 Edge Locations
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Why Does the Cloud Matter to Public Sector?
Pave the Way for
Disruptive Innovation
Make the World
a Better Place
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Paving the way
What do public sector customers need?
• Disruptive innovation
• Agility
• Capability
• Cost savings
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Evolution of cloud adoption in government
Policy
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Phase 0 – Stated Policy
“Cloud First”
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Evolution of cloud adoption in government
DefinitionPolicy
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Phase 1 – Standard Definition
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Evolution of cloud adoption in government
Security and Compliance
DefinitionPolicy
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Phase 2 – Security & Compliance
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Evolution of cloud adoption in government
Security and Compliance
Procurement
DefinitionPolicy
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Phase 3 – Cloud-Friendly Procurement
Department of Interior
Foundation Cloud
United Kingdom G-Cloud
Dept. of Treasury Public Cloud Web Hosting Services
US Communities
Texas Dept. of
Information Resources contract
Navy SPAWAR
US Intelligence Community
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Evolution of cloud adoption in government
DefinitionPolicy
Security and Compliance
Procurement
Culture
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Evolution of cloud adoption in government
Security and Compliance
Procurement
Culture
Broad Adoption
DefinitionPolicy
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Paving the way for…
Real change
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Make The World a Better Place
• Support world-changing projects
• Enable economic development
• Improve citizen services and engagement
• Improve research and education
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
When You’re Changing the World, You Can’t Afford to be Slow…
Add New Dev Environment
Add New Prod Environment
Go Global in Seconds
Add 1,000 Servers
Remove 1,000 Servers
Deploy 1 PB Data Warehouse
Shut down 1 PB Data Warehouse
AWS:
Resources in MinutesOld World:
Resources in Weeks
Everything changes with this
kind of agility and speed
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Increased agility has
become the #1 reason
organizations use the AWS
cloud
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Make The World a Better Place
1. Support world-changing projects
2. Enable economic development
3. Improve citizen services and engagement
4. Improve research and education
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
What better way to change the world…
Code.org ran a worldwide
“Hour of Code”
More than 20 million youth
coded on their website in a
single week, with a peak load of
330,000 concurrent users
“Running on the AWS Cloud
gave us the elasticity to keep
the website running when
traffic spiked from zero to 20
million coders during
campaign week, and then
scale back efficiently. AWS
was fantastic.”-Geoffrey Elliott, Code.org
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Ensuring democracy…
Migration of core business applications for secure global
access, reduced costs, focused resources and improved
availability
"The driver really was the ability to
be responsive. The way we did that
was to move to the cloud.“
-Chris Spence, CIO
National Democratic
Institute
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Helping Nonprofits Achieve Scale and Reach
"Working hand in glove with AWS,
LeanIn.Org's development team was
able to re-architect the site over a 64-
hour period to ensure it would stand
up to 200 click-throughs per second."
-Rachel Thomas
President, Lean In
“Ban Bossy” campaign
Featured on Google home page and
stayed up!
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Using Analytics to Understand the World
“Do the simplest
thing,” but do it at a
stupidly large scale.”
-David Milne, a postdoctoral
research fellow at CSIRO
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
AWS Government, Education, and Nonprofits SymposiumLondon I 21 October 2014
European Bioinformatics Institute
SureChEMBL: Patient Data in the Cloud
Mark Davies, Technical Lead
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
What is EMBL-EBI?• Part of the European
Molecular Biology Laboratory
• International, non-profit research institute
• Europe’s hub for biological data services and research
• 500 members of staff from 53 nations.
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
ChEMBL – Data for Drug Discovery
Bioactivity data
Compound
Assa
y/T
arg
et
>Thrombin
MAHVRGLQLPGCLALAALCSLVHSQHVFLAPQQARSLLQRVRRANT
FLEEVRKGNLERECVEETCSYEEAFEALESSTATDVFWAKYTACET
ARTPRDKLAACLEGNCAEGLGTNYRGHVNITRSGIECQLWRSRYPH
KPEINSTTHPGADLQENFCRNPDSSTTGPWCYTTDPTVRRQECSIP
VCGQDQVTVAMTPRSEGSSVNLSPPLEQCVPDRGQQYQGRLAVTTH
GLPCLAWASAQAKALSKHQDFNSAVQLVENFCRNPDGDEEGVWCYV
AGKPGDFGYCDLNYCEEAVE
3. Insight, tools and resources for translational drug discovery
2. Organization, integration, curation and standardization of pharmacology data
1. Scientific facts
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Who works with ChEMBL?
https://www.ebi.ac.uk/chembl/
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Patent Data and Drug Discovery
• Historically a closed and costly data source
– Out of reach to many academics and SMEs
• Patent literature 2-3 years ahead of published literature
• Prior art and freedom to operate
• Competitor intelligence
• Provides access to lots more data
– High cost to extract and lots of noise
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
SureChEMBL – Open Patent Data
• December 2013 EMBL-EBI acquired SureChem
• A leading chemistry patent mining product from
Digital Science, Macmillan Group
• Commercial to open transition ‘perfect fit’ for both
EMBL-EBI and Digital Science
• System rebranded SureChEMBL and migrated to
EMBL-EBI resources
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
SureChEMBL Migration Challenge
• SureChem was an AWS based system, should we stay or
should we go?
– Live system with many dependencies
– Supporting existing customers during migration
– Short timescale
– Limited resources
– Limited budget
– Existing AWS experience
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
We stayed with AWS
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
SureChEMBL System
Keyword
searches
Chemistry
searches
Access to
chemistry
https://www.surechembl.org
Complex
filters
Access to
patents
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
SureChEMBL AWS Resources
Elastic Load
Balancing
Amazon
VPC
Amazon
EC2
Elastic
IP
Amazon
S3
Amazon
EBS
Snapshot CloudWatch IAM
Amazon SQS
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
AWS Benefits
• AWS provides a comprehensive suite tools and services
– As a developer, everything required to build system is just there
• Fast design, build, deploy and teardown cycle
• Web based AWS console
• Use programmatic access to automate tasks and monitor system
– CLI and multiple SDKs available
• CloudWatch alarms to monitor system
• Cost calculator
• New services being added all the time
• Excellent support
– Account manager, solution architects, online forums
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
AWS Recommendations
• Make use of excellent online documentation
– Many whitepapers and best practice guidelines
• Use VPC to architect system
– Refresh network knowledge
– Tricky to migrate from ‘Classic’ EC2
• Review costs
– Use the cost calculator
– Use reserved instances
– Monitor storage: RAIDs, backups, snapshots, storage medium, logs
– Shutdown resources when not being used
• Prepare for unexpected events
– Check the ‘Events’ section EC2 dashboard
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
SureChEMBL Migration Success19th September 2014 SureChEMBL system released
https://www.surechembl.org
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
AcknowledgementsChEMBL team• John Overington
• Jon Chambers
• George Papadatos
• Mark Davies
• Nathan Dedman
• Anna Gaulton
Digital Science• Nicko Goncharoff
• James Siddle
• Richard Koks
Funding:• Wellcome Trust Strategic Award for
ChEMBL database (WT086151/Z/08/Z &
WT104104/Z/14/Z)
• Open PHACTS - Innovative Medicines
Initiative Joint Undertaking (grant no.
115191)
• European Molecular Biology Laboratory
• BioMedBridges - European Commission
FP7 Capacities Specific Programme (grant
no. 284209)
Technology Partners:
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Thank YouMark Davies, Technical Lead
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Make The World a Better Place
1. Support world-changing projects
2. Enable economic development
3. Improve citizen services and engagement
4. Improve research and education
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Public/private engagement for
economic development
• Geospatial collaborative
environment
• 80+ participating agencies
• 30+ new services
developed
• 100M+ users of the service
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Public/private engagement for
economic development
“AWS has helped my organization to
provide better service availability
and handle higher traffic load at a
lower cost”
-Chan Chin Wai, CIO, SLA
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
June 2014 Competition
Finalists and Winners
Partners in Innovation
Best Practices Award
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Best Practices Award Winners
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Make The World a Better Place
1. Support world-changing projects
2. Enable economic development
3. Improve citizen services and engagement
4. Improve research and education
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Turning Government Data into Real Insight
Scalable web application Big Data Analytics
and Collaboration
Rapid deployment of analytics
engine
Redesigned portions of
Healthcare.gov
Post “flash crash”
forensics on EC2
Collaboration
platform for SEC
Mining social media for
early warnings of food and
drug safety issues on
accelerated timeline
Healthcare.gov
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
AWS Government, Education, and Nonprofits SymposiumLondon I 21 October 2014
Taking Transport for
London into the cloud
Phil Young
Head of TfL Online
• Background TfL and our Online services
• Choosing the cloud what drove our selection
• Making it happen build and commissioning
• Outcomes our experiences so far
Overview
54
About TfL• London’s integrated transport authority
• Responsible for all forms of transport –Tube, buses, roads and
congestion charging, trams, river, DLR, Overground, cycling,
walking, coaches, freight, taxis and private hire and more
• Around 24 million journeys daily on our network
• Planning and developing for the future, new services to meet the
demands of an increasing population
• Iconic brand, embedded into the fabric of our city
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
What our customers want
56
Understand what we stand for
Value for MoneyProgress &
Innovation
Trust
Excellent
reliability and
customer
experience
TfL web visits 2006-2014
0
5
10
15
20
25
30
Mill
ion
s
Total Visits
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Legacy issues
• Previous site not refreshed since 2007
• Met core functional needs, but required user effort for more complex
tasks (eg finding maps, logins)
• No longer met customer expectations for
localisation, personalisation, mapping
and journey planning
• Poor experience on mobiles and tablets
Requirements
Customer
TechnicalBusiness
The new site needed to meet
customer needs first, while delivering
on business objectives and being
technically deliverable
1. Personalised information
2. Fast realtime info on-the-move
3. Mobile first, great on all devices
4. One stop shop for planning and info
5. Improved visual appearance
6. Humanised interaction
7. Single log-in
8. Integrated journey planning tools
9. Trusted information
10. Simplified experience
11. Consistent experience of all of TfL
12. Easy access to customer service
13. Reassurance around travel decisions
Key customer requirements
Our vision of a new TfL site
CHOOSING THE CLOUD
61
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Platform requirementsFunctional:
• Personalisation and localisation
• Separate presentation layer
• Disparate data sources
• Integrated and consistent data
needed
• CMS and web application
integration
• Responsive design
Performance:
• Scalable - 30x spikes
• 24 hour 100% availability
• Fast response times
• Accommodate limited back-end
systems
• Highly secure
• Deal in realtime services
• Cost effective
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Choosing the cloud
• Elastic capacity
• Speed and flexibility, automation
• Cost
• Resilience and referencability
AWS selected because:
• Best match at the time for our multi-OS stack
• Cost-effective, usage-based approach
• Rapidly developing platform with regular new features and falling costs
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Challenges of delivery
• Product evolving through an agile programme as platform being built
• In house team and multi-supplier environment
• Challenging timescales to move from experimental to production
• Lack of cloud experience
• Platform maturity – still developing
• Internal scepticism
OUTCOMES
65
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
What we delivered
• A new API with a canonical data model
• Complex web application through MVC
• Multiple environments, ‘push button’ creation and product deployment
• Autoscaling
• Blue/green release approach
• Varnish caching layer
• Web application firewall
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
New from the ground-up
67
Responsive design
Mobile first
New information architecture
New search
New Journey Planner
Google maps throughout
Google Streetview
New Roads service board and mapping
New River service board
Emirates Airline service board
Trams service board
New content throughout
Localisation
New cloud hosting – more resilient and auto-
scaled for capacityLower overall hosting costs
Varnish for intelligent caching
Normalised API for efficient development and open data
Faster live info – 5m to 30s
Integrated fares in Journey Planner
New ‘Nearby’ tool
Integration of all forms of travel information for the first time
Platform for further development, single sign-on,
TDM, Contactless etc
MVC approach
HTML5 schematic mapping
Personalisation
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Key outcomes - infrastructure
• Solid availability and performance
• Sustained massive load through April Tube strikes (1 million pages
an hour, plus all syndicated load)
• Ease of standing up environments proved
• Autoscaling and blue/green deployment working well
• Substantial savings on operating costs
• Now extending to other services – Journey Planner and wider as
appropriate
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014 69
• Journey Planner usage up 10% with highest ever number of visits (10
million) in April
• Mobile usage has overtaken desktop usage since launch
• Customer satisfaction from April survey stood at 87%
• 76% of Londoners are using the site, 20 million visits a month
• 12 industry awards so far (and counting)
Key outcomes - customers
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
What we learned
• Benefits of scripting and autoscaling are very significant, even though
it’s complex
• Specific cloud experience makes things much faster
• An integrated, co-located team of product development and
infrastructure really helps
• Need a mindset which allows for agility and can accommodate
change
• This type of transformation is not easy – but worthwhile, requires a
bold approach
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Thank YouPHIL YOUNG
Head of TfL Online
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Make The World a Better Place
1. Support world-changing projects
2. Enable economic development
3. Improve citizen services and engagement
4. Improve research and education
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Accelerating Research
University of Chicago needed a cost-effective way to provide
big-data analysis to labs around the world while providing
always-on service. The university now hosts its Globus
Transfer service on AWS, helping more than 12,000 users to
move data with 99% availability.
“AWS has helped us scale up and
lower the cost of doing analysis.”- Ravi Madduri,
Research Fellow and Project Manager
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Improve Education and Break the Mold
Migrated public facing web properties with large bursts of traffic from 38,000 visitors to
150,000 a day
Improved disaster recovery, handled major event spikes in usage and 40%
less expensive
Amazon Jisc Web Portal
21/10/14 Cloud Computing
Tim Marshall
CEO, Janet
Universities are migrating off campus
Benefits
»Sweating the network asset
»Options from colocation to cloud
»Larger storage facilities
»Agility
»Economies of scale
»National agreements
»Better use of campus estate
Jisc Amazon Cloud Computing 76
And have been for some time ...
But the cloud...
What about?
»Data Preservation
»Security
»Legal Issues
»Costs
»Logistics
»This is where Jisc adds even more…
Jisc Amazon Cloud Computing 77
Tier 1 Cloud Vendors
Janet Network
Microsoft
O365 &
Azure
Apps &…
Amazon
AWS &
Glacier
• Approach is to gain sector sponsorship and involvement• Largest global providers with large sector market share • Broad applicability of service including for research, teaching & learning and enterprise• Long term investment and long term partnerships (including on technology)• Looking for standardisation• Working with Géant and Global Services Group
Data
location &
processing
Pricing &
Costs
Data Egress
/ bandwidth
Federated
Identity
NREN
Aggregation
What sector issue did we seek to solve?Individual users with:
• No protection
• No aggregation benefits
• No institution awareness or control
Organisational Issues
1. Where is your data?
2. Are you compliant?
3. Are you legal?
Jisc Amazon Cloud Computing 79
Jisc & Amazon
Sector leadership:
»University of York
»University of Newcastle
»University of Cambridge
Commercial Partnering:
»Arcus
»Amazon
Jisc Amazon Cloud Computing 80
Building a national sector level portal
The Portal is now Live
The Portal offers:
»Federated login for quick easy access
»Monthly invoicing IN STERLING, no need to use a credit card
» Itemised billing to consolidate costs across users and departments
»The ability to set budget limits against accounts
»Volume discounts through aggregation across users
Jisc Amazon Cloud Computing 82
Academic institutions are beginning to benefit
Jisc Amazon Cloud Computing 83
“Frictionless access to Cloud Services that’s our job”Dan Perry Director of Product and Marketing - Jisc
Jisc
Google Apps
Data Archiving
IT Financial
X-Ray
Shared Data
CentresAmazon
Web Services
File Sync and
Share
Microsoft
Data Centre Frame-
work
Find out more ...
http://www.jisc.ac.uk/amazon-web-services
http://www.jisc.ac.uk/network/cloud
http://www.jisc.ac.uk/
http://community.ja.net/cloud
Title of presentation 00/00/2013 84
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Customer
obsession
1
Invention
2
Long-term
thinking
3
Some final thoughts about Amazon…
A Faster, Cheaper and Better
Path to a Better World
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
AWS Government, Education, and Nonprofits SymposiumLondon | 21 October 2014
Thank YouTeresa Carlson