stories from testing healthcare · 2015 67% private 9% uninsured •56% employment-based •16%...

Stories From Testing

HealthCare.govThe unexpected adventures of an

amphibious time-traveling context-driven

cyborg software tester.

Ben Simo

HealthCare.gov

HealthCare.gov

http://x.co/ObamaDemo

Context: Health insurance in the USA2

01

0 16% uninsured64% private• 55% employment-based• 9% direct-purchase

31% government• 15% Medicare• 16% Medicaid

20

15 9% uninsured

67% private• 56% employment-based• 16% direct purchase

37% government• 16% Medicare• 20% Medicaid• 5% military

All percentages are percent of total population in the United StatesSource: US Census Bureau, 2010: http://x.co/2010health, 2015: http://x.co/2015health* Research methods changed in 2014

PPACA

Context: Health insurance reform lawPatient Protection and Affordable Care Act of 2010

Public health insurance reform

Expand Medicare eligibility and

coverage

Incentivize Medicare providers to reduce costs and improve

quality

Private health insurance reform

Set minimum coverage standards

Ban the use of medical history as an insurability

and coverage factor

Provide tax credits to subsidize insurance

premiums

Penalties for being uninsured

Penalize individuals and companies for not being

insured

Health insurance marketplaces

Make buying health insurance easier and

more affordable

Context: Health Insurance in my householdSeeking insurance for Tiffany

My household

• Wife & I

• Teenage Son

• Adult daughter & her daughter

Our health insurance

• Employer-subsidized health insurance

– Adult daughter is eligible until age 26

– Granddaughter is not eligible because she is not my child

• AHCCCS (Arizona’s Medicaid)

– Granddaughter lost coverage in summer 2013

HealthCare.gov marketplace launch: 1 October 2013

Incredible messThe system is down

Incredible messThe system is down

An incredible mess

http://x.co/imess

An incredible messOn the first day

1,100

60,000

250,000

Tested

Expected

Actual

Concurrent website visitors

2,800,000website visitors

?accountscreated

?applicationssubmitted

6insurance

planenrollments

6,700

248end of 2nd day

end of 1st week

Washington Post: Obamacare’s Launch Looked Even Worse from the Inside, http://x.co/worseinsidUSA Today: Obama adviser: Demand overwhelmed HealthCare.gov, http://x.co/hcdemand

"These bugs were

functions of volume.

Take away the volume

and it works.”– Todd Park,

CTO of the United States

Step 1: Set up a Marketplace accountNo option to browse plans

Confusing restrictions

Step 1: Set up a Marketplace account

Confusing restrictions

Step 1: Set up a Marketplace account

Confusing restrictions

Step 1: Set up a Marketplace account

Please wait

Step 1: Set up a Marketplace account

System is unavailable

Step 1: Set up a Marketplace account

Your account couldn’t be created at this time

Step 1: Set up a Marketplace account

This username already exists

Step 1: Set up a Marketplace account

Sorry you can’t get what you need right now

Step 1: Set up a Marketplace account

“to ensure that your

personal data

can’t be hacked

…

personalized questions

that can

only be verified by you”– HHS Secretary

Kathleen Sebelius

Security questions

Security questions

Security questions

Security questionsGrant 3rd party helpers access

Step 1: Set up a Marketplace account

Your account couldn’t be created at this time.

Email address is not Unique.

Step 1: Set up a Marketplace account

We sent an email …

Internal Server Error

Step 1: Set up a Marketplace account

Step 1: Set up a Marketplace account

“We have a lot of visitors trying to use our website right now. This is

causing some glitches… The email can take up to 3 days.”

- HealthCare.gov customer service

Step 1: Set up a Marketplace account

Step 1: Set up a Marketplace account

LoginBad request

LoginUnexpected error

LoginIncognito

Login> 4600 bytes of cookie data in the request header

But wait, there’s more

Redirects to insecure HTTP

< my username

But wait, there’s more

Username and password reset code emailed together

But wait, there’s more

Personal info sent to 3rd parties

But wait, there’s more

Stack traces returned to the browser

But wait, there’s more

Password reset codes don’t change

But wait, there’s more

HTML injection

But wait, there’s more

Auto-suggested SQL injection

My tweeting and blogging attract attention

My tweeting and blogging attract attention

How to successfully register for health insurance on HealthCare.govWe got advice from a pro software testerPublished: October 16, 2013 06:00 PM

“…we talked with a Phoenix software tester named Ben Simo. When he got stuck trying to register a family member, Simo used his professional know-how to look beneath the hood and come up with some suggestions for creating a Healthcare.gov user account that actually works.”

“If all this is too much for you to absorb, follow our previous advice: Stay away from Healthcare.gov for at least another month if you can. Hopefully that will be long enough for its software vendors to clean up the mess they’ve made.”

http://x.co/crhcgov

My tweeting and blogging attract attention

Traffic Didn’t Crash the Obamacare Site Alone.Bad Coding Did Too.Oct. 24, 2013

http://x.co/badcoding

“Nearly 20 million Americans have now experienced the broken Obamacare website first hand. But Ben Simo … found something more than a cumbersome login or a blank screen—clear evidence of subpar coding on the site.”

“[Simo] discovered that one part of the website had created so much cookietracking data that it appeared to exceed the site’s capacity to accept his login information. That’s the mark of a fractured development team.”

Security vulnerability

No process for receiving bug reports

• I am told to contact:

– Federal Trade Commission

– Federal Bureau of Investigation

– My local police

Security vulnerability

I keep blogging… carefully

My reports attract more attention

Security vulnerability

Congressional hearings

Security vulnerability

http://x.co/breachblog

Congressional hearings

There was not a breach.

There was a blog by a sort of skilled hacker,

that if a certain of series of incidents occurred

you could possibly get in and

obtain somebody’s personally identifiable …

It was a theoretical problem that

was immediately fixed.- HHS Secretary Kathleen Sebelius

Security vulnerability

A theoretical problem?

Security vulnerability

Resource Input Output

updateForgottenUsername First & last name, Email address Username

fetchSecurityQuestions First & last name, Email address Security questions

confirmUserLogin Username Password Reset UUID

forgotPasswordQuestions Username, Password Reset UUID Security questions

updateForgottenPassword Username, Password Reset UUID Email address

updateForgottenPassword Username, Password Reset UUID, Security questions, Security question answers

< Password reset

A certain series of events?

Exploiting the vulnerability

1. Get lists of names and email addresses (public info, marketing lists, another breach)

2. Get usernames for those names and addresses in the system (updateForgottenUsername)

3. Get password reset UUIDs (confirmUserLogin)

4. Get security questions (fetchSecurityQuestions)

5. Get security question answers (social engineering, Facebook, phishing)

6. Change passwords

7. Access personal information in user accounts

Security vulnerability

15 minutes of fame

A distributed denial of service attack

from• Reporters and talking heads

• TV• Radio• Print• Online

• Educators• Congressional committees

via• Email• Phone• Txt messages• Twitter• LinkedIn• Facebook

15 minutes of fame

15 minutes of fame

Hackers can’t get much?

”we are storing the minimum amount of data,because we think that’s

very important.The hub is not a data collector.

It is actually using data centers

at the IRS,at Homeland Security,

at Social Securityto verify information,

but it stores none of that data.”- HHS Secretary Kathleen Sebelius

Hackers can’t get much?

Not a data collector?

Hackers can’t get much?

Stores none of that data?

Hackers can’t get much?

Stores none of that data?absentParentAgreementIndicatorabsentParentNameageLeftFosterCareCodeamountIRSAnnualIncomeamountSocialSecurityBenefitsIncomeamountStateQuarterlyIncomeamountStateUnemploymentIncome

avgHoursPerWeekbabyDueQuantityblindDisabledIndicatorcaretakerRelativeIndicatorchildLivesWithBothParentschildOfVeteranIndicator

completeImmigrationInformationdateGainedEligibleImmigrationStatusdateReleasedFromIncarcerationdiscrepantMonthlyIncomeIndicatorfutureDependentsincarcerationEndDate

medicaidEligibilityReasonTextmotherAvgHoursWeekpersonSSNpregnancyIndicatorsameSexSpousetobaccoLastUsedemploymentTerminationDate

Hackers can’t get much?

Stores none of that data?

A web portal into internal government

data systems?

Application

Start your application

Application

Confusing questions

Application

Multiple personalities

Application

No data available in table

Application

Uncaught type error

Application

Uncaught type error

Application

Uncaught type error

Application

Dead end

Application

Processed an application I did not submit

Application

Processed an application I did not submit

Application

Processed an application I did not submit

Application performance

>8 seconds to go to the next question

Application performance

Huge payload

Application performance

Wow!

Application results

After about a month of trying

Application results

You don’t qualify

Application results

Eligibility requirements

Application results

Eligibility requirements

What went wrong?Testing failure

What went wrong?Testing failure

What went wrong?Implementation failure

What went wrong?Implementation failure

Browse Plans

Create Account 1Login 2

Verify Identity 3Apply for Insurance 4Submit Application 5

Determine Eligibility 6

What went wrong?Management failure

What went wrong?Management failure

• 55 companies involved in building the mess– 0 were responsible for overseeing the others

– “eternal loop of damnation” getting companies to work

together

• 0 monitoring– 0 were responsible for making sure system was usable

– Watched CNN to learn about problems

• 0 sense of urgency– Government software projects fail all the time

– This was just like every other project

“Everything’s been done wrong, almost.

Almost no place we can point to a decision

where we made the right one.”

- Mikey Dickerson,

United States Digital Services

Mikey Dickerson: One Year After Healthcare.gov, http://x.co/1yearafter

Your turn

Put on your tester hat and x-ray specs

• Testing is investigation

• Requirements documents are not required

• Communicate carefully

• Ethical behavior is essential

Testing is investigation

Testing is the process of evaluating a product

by learning about it through experimentation

which includes to some degree:

– questioning,

– study,

– modeling,

– observation,

– and inference.

- James Bach & Michael Bolton, Testing and Checking Refined

Consistency heuristicsJames Bach & Michael Bolton

Requirements documents are not required

(F)

Familiar

E

Explainable

W

World

HHistory

IImage

CComparable Products

CClaims

UUser Expectations

PProduct

PPurpose

SStatutes

Requirements documents are not required

OWASP Top 10

Sensitive data exposure6

Function-level access controls7

Cross-site request forgery8

Components with known vulnerabilities9

Unvalidated redirects and forwards10

Injection1

Broken authentication & session management2

Cross-site scripting3

Insecure object reference4

Security misconfiguration5

Failure mnemonicBen Simo

Requirements documents are not required

Usability heuristics for user interface designJakob Nielsen

Requirements documents are not required

Visibility of system status

Match between system and the real world

User control and freedom

Consistency and standards Error prevention

Recognition rather than recall

Flexibility and ease of use

Aesthetic and minimalist design

Help users recognize, diagnose, and recover from errors

Help and documentation

Communicate carefully

• Be accurate and precise

• Distinguish between what you observe and what you conclude

• Avoid speculation and blame

• Explain that which “goes without saying”

• Demonstrate the problem

• Explain the potential consequences

• Admit and correct your mistakes

Understand and honor ethical and legal boundaries

• Do no harm

• Honor terms of use

• Use the interfaces provided

• Don’t attempt to gain access to others’ data

• Don’t enable others to do harm

Ethical behavior is essential

IsThereAProblemHere.com