the transformation of : two years of innovation in how our government serves its citizens

Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 1

Transformation of Healthcare.gov

1

Sean O’Neil, Optum Technology

The Transformation of Healthcare.govTwo years of innovation in how our government serves its citizens

Sean O’Neil

Optum Technology


Optum joins the .war room

• When the launch of Healthcare.gov ran into troubles, CMS looked for help from the tech sector and from Optum.

• QSSI, an Optum owned company, was named Master System Integrator.

• We held the MSI role for the next 24 months.

This story is how those

events have transformed our

Public Sector technology in

amazing ways.


October 2013 - What went wrong?


October 2013 - What went wrong?

• Nothing special. It was just like most IT projects today.

• There’s no need to cover all the details because you’ve all experienced them yourself.

• The following list will be déjà vu for anyone that has worked in complex organizations:


Problems we’ve seen over and over…

Processes missing: change, incident, problem, config, release, job control..

Automation and Monitoring missing.

Load Testing inadequate.

Unclear lines of responsibility.

A high profile customer with high expectations.

Technology change approval process complex and bureaucratic.

Functional requirements changed at the last minute.

Hundreds of serious defects known at launch.

“Reference Architecture” overly and unnecessarily complex”.

COTS software used inappropriately.


Partial list of key technology fixes:

• Turn off transparent huge pages.

• Turn down excessive logging (Layer 7, Log4j x 4, firewall).

• Fix memory leaks.

• Strategically replace small VMs with large VMs.

• Fix data cache priming procedure.

• Basic Apache worker tuning.

• Change helpdesk interface to prevent locking on the whole portal.

• Mark Logic xquery tuning (id generator).

• Load balancing problems everywhere (introduce F5s and tuned distribution algorithms).

• Physical tuning:

– Placement of the Mark Logic forests within the NAS.

– Spread hot spots around the compute pools.

– Breaking out Mark Logic data and query nodes.

– Moving VMs to consolidate subnets.


Partial list of process and culture changes:

1) Verbal communication is OK.

2) SOPs and pre-approved changes.

3) Perfection is an illusion.

4) Problem Management on a 12 hour cycle.

5) Diversity in thought is strength.

6) The passive voice (and passive thinking!) is too slow.


The Rules of the .war room


The Rules of the .war room

1) The .war room is for solving problems, not shifting blame.

2) The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank.

3) We need to stay focused on the most urgent issues - things that will hurt us in the next 24 – 48 hours.


• The problems that plagued healthcare.gov weren’t anything special.

• But the rescue of the site was the spark of a new revolution in IT that is transforming our public sector.

• It’s not just “stopping another healthcare.gov”, it’s about embracing cutting edge technology and methods!


Our government joins the 21st Century

The Past The Future is Now

Proprietary private clouds Public and hybrid clouds

Expensive proprietary hardware (Oracle Exadata) Amazon EC2

Expensive RDBMS (Oracle) MySQL – MariaDB

Expensive on premise monitoring (Tivoli) New Relic APM

Expensive decision support systems (CA Capacity

Manager)

New Relic Insights

Java Application Servers Node.js and Nginx

No automation Jenkins, Puppet

“Cold stand-by” datacenter for Disaster Recovery Multiple EC2 availability zones, geo-dispersed datacenters,

Akamai GTM

Off-the-shelf, shrink-wrapped “bloatware” Roll your own. Use Open Source.

Static content hosted in datacenters, backed by application

servers.

Dynamic, interactive JavaScript content hosted in Akamai.

Limited visibility into customer behavior End user analytics, A/B Testing


New Relic Custom Plugins drive technology advances

• Roll your own and community sponsored plugins allowed us freedom in our technology choices because we knew we could monitor it.

• Quickly roll out new monitoring to respond to surprises in production.

Community versions Developed by HC.gov team

• Nginx • Terracotta

• F5 LTM • CA Layer 7 XML Gateway

• Apache HTTP Server • NFS

• EC2 • Chartbeat

• Ehcache • Gluster

• MS SQL • RedHat Kernel

• Node.js • LDAP

• Oracle Identity Manager

• MarkLogic


The Public sector using Public Clouds

• The cloud is a wonderful place, right?

–Leverage the cloud for cheap capacity

• We’re all paying for it… you want the Government to use your money

wisely, right?

–Scale up for “Black Friday” scenarios

• The last day of Healthcare.gov enrollment is 10x volume burst from day

one.

–Scale down after the surge to save your money.


Are public clouds secure?

• Securing applications in of public clouds is top priority!

–Remediated with secure network access using dedicated hardware

–Hardened server images

–Multiple layers of threat detection / prevention, data loss prevention, strong encryption everywhere. (…and lots of other stuff I won’t talk about!!)


Hybrid Clouds – The Best of Both Worlds

• Because moving everything to the cloud is hard and expensive.

–Some technology doesn’t work well in the cloud (yet)

• Mark Logic still needs big iron.

• Move assets to the cloud as they are ready.


A Case Study: The Scalable Login System

• Oracle Identity Management Suite was the source of many problems.

An expensive and complex COTS product.

• Replaced with the Scalable Login System

Node.js and Maria DB

Amazon EC2, S3, ELBEnd of

Year One

End of

Year Two

SLS

currently

Logins per

sec

53 154 346

Accounts

per hour

60,000 120,000 132,000


A Case Study: App 2.0

• We found 70% of applications submitted the first year were “simple”.

• Simple apps could be completed in less than 16 steps.

• For year two, we created App 2.0Year One

“Classic”

Year Two

App 2.0

Application

submittal

time

40 minutes

average for

all apps.

10 minutes

Steps per

application

70+ 16 steps

NGNIX, Node.js and Maria DB

Calls into the restful layer of the

“classic” Healthcare.gov system.

No changes required to existing

system.


The United States Digital Service

• A few of the original members of the Healthcare.gov rescue team have formed a new team within the government to transform it from the inside out.

• Change Agents: USDS members are embedded within select agencies.

• Change the culture.

• Change the technology.

• Oversight of agency technology budgets.

• You can help!

https://www.whitehouse.gov/digital/united-states-digital-service

Thank you.

Sean O’Neil

Optum Technology

612-508-4855

[email protected]