Download - The Transformation of : Two years of innovation in how our government serves its citizens
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 1
Transformation of Healthcare.gov
1
Sean O’Neil, Optum Technology
The Transformation of Healthcare.govTwo years of innovation in how our government serves its citizens
Sean O’Neil
Optum Technology
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 3
Optum joins the .war room
• When the launch of Healthcare.gov ran into troubles, CMS looked for help from the tech sector and from Optum.
• QSSI, an Optum owned company, was named Master System Integrator.
• We held the MSI role for the next 24 months.
This story is how those
events have transformed our
Public Sector technology in
amazing ways.
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 4
October 2013 - What went wrong?
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 5
October 2013 - What went wrong?
• Nothing special. It was just like most IT projects today.
• There’s no need to cover all the details because you’ve all experienced them yourself.
• The following list will be déjà vu for anyone that has worked in complex organizations:
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 6
Problems we’ve seen over and over…
Processes missing: change, incident, problem, config, release, job control..
Automation and Monitoring missing.
Load Testing inadequate.
Unclear lines of responsibility.
A high profile customer with high expectations.
Technology change approval process complex and bureaucratic.
Functional requirements changed at the last minute.
Hundreds of serious defects known at launch.
“Reference Architecture” overly and unnecessarily complex”.
COTS software used inappropriately.
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 7
Partial list of key technology fixes:
• Turn off transparent huge pages.
• Turn down excessive logging (Layer 7, Log4j x 4, firewall).
• Fix memory leaks.
• Strategically replace small VMs with large VMs.
• Fix data cache priming procedure.
• Basic Apache worker tuning.
• Change helpdesk interface to prevent locking on the whole portal.
• Mark Logic xquery tuning (id generator).
• Load balancing problems everywhere (introduce F5s and tuned distribution algorithms).
• Physical tuning:
– Placement of the Mark Logic forests within the NAS.
– Spread hot spots around the compute pools.
– Breaking out Mark Logic data and query nodes.
– Moving VMs to consolidate subnets.
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 8
Partial list of process and culture changes:
1) Verbal communication is OK.
2) SOPs and pre-approved changes.
3) Perfection is an illusion.
4) Problem Management on a 12 hour cycle.
5) Diversity in thought is strength.
6) The passive voice (and passive thinking!) is too slow.
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 9
The Rules of the .war room
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 10
The Rules of the .war room
1) The .war room is for solving problems, not shifting blame.
2) The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank.
3) We need to stay focused on the most urgent issues - things that will hurt us in the next 24 – 48 hours.
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 11
• The problems that plagued healthcare.gov weren’t anything special.
• But the rescue of the site was the spark of a new revolution in IT that is transforming our public sector.
• It’s not just “stopping another healthcare.gov”, it’s about embracing cutting edge technology and methods!
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 12
Our government joins the 21st Century
The Past The Future is Now
Proprietary private clouds Public and hybrid clouds
Expensive proprietary hardware (Oracle Exadata) Amazon EC2
Expensive RDBMS (Oracle) MySQL – MariaDB
Expensive on premise monitoring (Tivoli) New Relic APM
Expensive decision support systems (CA Capacity
Manager)
New Relic Insights
Java Application Servers Node.js and Nginx
No automation Jenkins, Puppet
“Cold stand-by” datacenter for Disaster Recovery Multiple EC2 availability zones, geo-dispersed datacenters,
Akamai GTM
Off-the-shelf, shrink-wrapped “bloatware” Roll your own. Use Open Source.
Static content hosted in datacenters, backed by application
servers.
Dynamic, interactive JavaScript content hosted in Akamai.
Limited visibility into customer behavior End user analytics, A/B Testing
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 13
New Relic Custom Plugins drive technology advances
• Roll your own and community sponsored plugins allowed us freedom in our technology choices because we knew we could monitor it.
• Quickly roll out new monitoring to respond to surprises in production.
Community versions Developed by HC.gov team
• Nginx • Terracotta
• F5 LTM • CA Layer 7 XML Gateway
• Apache HTTP Server • NFS
• EC2 • Chartbeat
• Ehcache • Gluster
• MS SQL • RedHat Kernel
• Node.js • LDAP
• Oracle Identity Manager
• MarkLogic
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 14
The Public sector using Public Clouds
• The cloud is a wonderful place, right?
–Leverage the cloud for cheap capacity
• We’re all paying for it… you want the Government to use your money
wisely, right?
–Scale up for “Black Friday” scenarios
• The last day of Healthcare.gov enrollment is 10x volume burst from day
one.
–Scale down after the surge to save your money.
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 15
Are public clouds secure?
• Securing applications in of public clouds is top priority!
–Remediated with secure network access using dedicated hardware
–Hardened server images
–Multiple layers of threat detection / prevention, data loss prevention, strong encryption everywhere. (…and lots of other stuff I won’t talk about!!)
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 16
Hybrid Clouds – The Best of Both Worlds
• Because moving everything to the cloud is hard and expensive.
–Some technology doesn’t work well in the cloud (yet)
• Mark Logic still needs big iron.
• Move assets to the cloud as they are ready.
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 17
A Case Study: The Scalable Login System
• Oracle Identity Management Suite was the source of many problems.
An expensive and complex COTS product.
• Replaced with the Scalable Login System
Node.js and Maria DB
Amazon EC2, S3, ELBEnd of
Year One
End of
Year Two
SLS
currently
Logins per
sec
53 154 346
Accounts
per hour
60,000 120,000 132,000
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 18
A Case Study: App 2.0
• We found 70% of applications submitted the first year were “simple”.
• Simple apps could be completed in less than 16 steps.
• For year two, we created App 2.0Year One
“Classic”
Year Two
App 2.0
Application
submittal
time
40 minutes
average for
all apps.
10 minutes
Steps per
application
70+ 16 steps
NGNIX, Node.js and Maria DB
Calls into the restful layer of the
“classic” Healthcare.gov system.
No changes required to existing
system.
Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 19
The United States Digital Service
• A few of the original members of the Healthcare.gov rescue team have formed a new team within the government to transform it from the inside out.
• Change Agents: USDS members are embedded within select agencies.
• Change the culture.
• Change the technology.
• Oversight of agency technology budgets.
• You can help!
https://www.whitehouse.gov/digital/united-states-digital-service