stories from testing healthcare · 2015 67% private 9% uninsured •56% employment-based •16%...
TRANSCRIPT
![Page 1: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/1.jpg)
Stories From Testing
HealthCare.govThe unexpected adventures of an
amphibious time-traveling context-driven
cyborg software tester.
Ben Simo
![Page 2: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/2.jpg)
HealthCare.gov
![Page 3: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/3.jpg)
HealthCare.gov
http://x.co/ObamaDemo
![Page 4: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/4.jpg)
Context: Health insurance in the USA2
01
0 16% uninsured64% private• 55% employment-based• 9% direct-purchase
31% government• 15% Medicare• 16% Medicaid
20
15 9% uninsured
67% private• 56% employment-based• 16% direct purchase
37% government• 16% Medicare• 20% Medicaid• 5% military
All percentages are percent of total population in the United StatesSource: US Census Bureau, 2010: http://x.co/2010health, 2015: http://x.co/2015health* Research methods changed in 2014
PPACA
![Page 5: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/5.jpg)
Context: Health insurance reform lawPatient Protection and Affordable Care Act of 2010
Public health insurance reform
Expand Medicare eligibility and
coverage
Incentivize Medicare providers to reduce costs and improve
quality
Private health insurance reform
Set minimum coverage standards
Ban the use of medical history as an insurability
and coverage factor
Provide tax credits to subsidize insurance
premiums
Penalties for being uninsured
Penalize individuals and companies for not being
insured
Health insurance marketplaces
Make buying health insurance easier and
more affordable
![Page 6: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/6.jpg)
Context: Health Insurance in my householdSeeking insurance for Tiffany
My household
• Wife & I
• Teenage Son
• Adult daughter & her daughter
Our health insurance
• Employer-subsidized health insurance
– Adult daughter is eligible until age 26
– Granddaughter is not eligible because she is not my child
• AHCCCS (Arizona’s Medicaid)
– Granddaughter lost coverage in summer 2013
![Page 7: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/7.jpg)
HealthCare.gov marketplace launch: 1 October 2013
![Page 8: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/8.jpg)
Incredible messThe system is down
![Page 9: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/9.jpg)
Incredible messThe system is down
![Page 10: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/10.jpg)
An incredible mess
http://x.co/imess
![Page 11: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/11.jpg)
An incredible messOn the first day
1,100
60,000
250,000
Tested
Expected
Actual
Concurrent website visitors
2,800,000website visitors
?accountscreated
?applicationssubmitted
6insurance
planenrollments
6,700
248end of 2nd day
end of 1st week
Washington Post: Obamacare’s Launch Looked Even Worse from the Inside, http://x.co/worseinsidUSA Today: Obama adviser: Demand overwhelmed HealthCare.gov, http://x.co/hcdemand
"These bugs were
functions of volume.
Take away the volume
and it works.”– Todd Park,
CTO of the United States
![Page 12: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/12.jpg)
Step 1: Set up a Marketplace accountNo option to browse plans
![Page 13: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/13.jpg)
Confusing restrictions
Step 1: Set up a Marketplace account
![Page 14: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/14.jpg)
Confusing restrictions
Step 1: Set up a Marketplace account
![Page 15: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/15.jpg)
Confusing restrictions
Step 1: Set up a Marketplace account
![Page 16: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/16.jpg)
Please wait
Step 1: Set up a Marketplace account
![Page 17: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/17.jpg)
System is unavailable
Step 1: Set up a Marketplace account
![Page 18: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/18.jpg)
Your account couldn’t be created at this time
Step 1: Set up a Marketplace account
![Page 19: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/19.jpg)
This username already exists
Step 1: Set up a Marketplace account
![Page 20: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/20.jpg)
Sorry you can’t get what you need right now
Step 1: Set up a Marketplace account
![Page 21: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/21.jpg)
“to ensure that your
personal data
can’t be hacked
…
personalized questions
that can
only be verified by you”– HHS Secretary
Kathleen Sebelius
Security questions
![Page 22: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/22.jpg)
Security questions
![Page 23: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/23.jpg)
Security questions
![Page 24: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/24.jpg)
Security questionsGrant 3rd party helpers access
![Page 25: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/25.jpg)
Step 1: Set up a Marketplace account
Your account couldn’t be created at this time.
Email address is not Unique.
![Page 26: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/26.jpg)
Step 1: Set up a Marketplace account
We sent an email …
Internal Server Error
![Page 27: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/27.jpg)
Step 1: Set up a Marketplace account
![Page 28: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/28.jpg)
Step 1: Set up a Marketplace account
“We have a lot of visitors trying to use our website right now. This is
causing some glitches… The email can take up to 3 days.”
- HealthCare.gov customer service
![Page 29: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/29.jpg)
Step 1: Set up a Marketplace account
![Page 30: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/30.jpg)
Step 1: Set up a Marketplace account
![Page 31: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/31.jpg)
LoginBad request
![Page 32: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/32.jpg)
LoginUnexpected error
![Page 33: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/33.jpg)
LoginIncognito
![Page 34: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/34.jpg)
Login> 4600 bytes of cookie data in the request header
![Page 35: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/35.jpg)
But wait, there’s more
Redirects to insecure HTTP
< my username
![Page 36: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/36.jpg)
But wait, there’s more
Username and password reset code emailed together
![Page 37: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/37.jpg)
But wait, there’s more
Personal info sent to 3rd parties
![Page 38: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/38.jpg)
But wait, there’s more
Stack traces returned to the browser
![Page 39: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/39.jpg)
But wait, there’s more
Password reset codes don’t change
![Page 40: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/40.jpg)
But wait, there’s more
HTML injection
![Page 41: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/41.jpg)
But wait, there’s more
Auto-suggested SQL injection
![Page 42: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/42.jpg)
My tweeting and blogging attract attention
![Page 43: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/43.jpg)
My tweeting and blogging attract attention
How to successfully register for health insurance on HealthCare.govWe got advice from a pro software testerPublished: October 16, 2013 06:00 PM
“…we talked with a Phoenix software tester named Ben Simo. When he got stuck trying to register a family member, Simo used his professional know-how to look beneath the hood and come up with some suggestions for creating a Healthcare.gov user account that actually works.”
“If all this is too much for you to absorb, follow our previous advice: Stay away from Healthcare.gov for at least another month if you can. Hopefully that will be long enough for its software vendors to clean up the mess they’ve made.”
http://x.co/crhcgov
![Page 44: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/44.jpg)
My tweeting and blogging attract attention
Traffic Didn’t Crash the Obamacare Site Alone.Bad Coding Did Too.Oct. 24, 2013
http://x.co/badcoding
“Nearly 20 million Americans have now experienced the broken Obamacare website first hand. But Ben Simo … found something more than a cumbersome login or a blank screen—clear evidence of subpar coding on the site.”
“[Simo] discovered that one part of the website had created so much cookietracking data that it appeared to exceed the site’s capacity to accept his login information. That’s the mark of a fractured development team.”
![Page 45: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/45.jpg)
Security vulnerability
No process for receiving bug reports
• I am told to contact:
– Federal Trade Commission
– Federal Bureau of Investigation
– My local police
![Page 46: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/46.jpg)
Security vulnerability
I keep blogging… carefully
![Page 47: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/47.jpg)
My reports attract more attention
Security vulnerability
![Page 48: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/48.jpg)
Congressional hearings
Security vulnerability
http://x.co/breachblog
![Page 49: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/49.jpg)
Congressional hearings
There was not a breach.
There was a blog by a sort of skilled hacker,
that if a certain of series of incidents occurred
you could possibly get in and
obtain somebody’s personally identifiable …
It was a theoretical problem that
was immediately fixed.- HHS Secretary Kathleen Sebelius
Security vulnerability
![Page 50: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/50.jpg)
A theoretical problem?
Security vulnerability
Resource Input Output
updateForgottenUsername First & last name, Email address Username
fetchSecurityQuestions First & last name, Email address Security questions
confirmUserLogin Username Password Reset UUID
forgotPasswordQuestions Username, Password Reset UUID Security questions
updateForgottenPassword Username, Password Reset UUID Email address
updateForgottenPassword Username, Password Reset UUID, Security questions, Security question answers
< Password reset
![Page 51: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/51.jpg)
A certain series of events?
Exploiting the vulnerability
1. Get lists of names and email addresses (public info, marketing lists, another breach)
2. Get usernames for those names and addresses in the system (updateForgottenUsername)
3. Get password reset UUIDs (confirmUserLogin)
4. Get security questions (fetchSecurityQuestions)
5. Get security question answers (social engineering, Facebook, phishing)
6. Change passwords
7. Access personal information in user accounts
Security vulnerability
![Page 52: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/52.jpg)
15 minutes of fame
A distributed denial of service attack
from• Reporters and talking heads
• TV• Radio• Print• Online
• Educators• Congressional committees
via• Email• Phone• Txt messages• Twitter• LinkedIn• Facebook
![Page 53: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/53.jpg)
15 minutes of fame
![Page 54: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/54.jpg)
15 minutes of fame
![Page 55: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/55.jpg)
Hackers can’t get much?
”we are storing the minimum amount of data,because we think that’s
very important.The hub is not a data collector.
It is actually using data centers
at the IRS,at Homeland Security,
at Social Securityto verify information,
but it stores none of that data.”- HHS Secretary Kathleen Sebelius
![Page 56: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/56.jpg)
Hackers can’t get much?
Not a data collector?
![Page 57: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/57.jpg)
Hackers can’t get much?
Stores none of that data?
![Page 58: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/58.jpg)
Hackers can’t get much?
Stores none of that data?absentParentAgreementIndicatorabsentParentNameageLeftFosterCareCodeamountIRSAnnualIncomeamountSocialSecurityBenefitsIncomeamountStateQuarterlyIncomeamountStateUnemploymentIncome
avgHoursPerWeekbabyDueQuantityblindDisabledIndicatorcaretakerRelativeIndicatorchildLivesWithBothParentschildOfVeteranIndicator
completeImmigrationInformationdateGainedEligibleImmigrationStatusdateReleasedFromIncarcerationdiscrepantMonthlyIncomeIndicatorfutureDependentsincarcerationEndDate
medicaidEligibilityReasonTextmotherAvgHoursWeekpersonSSNpregnancyIndicatorsameSexSpousetobaccoLastUsedemploymentTerminationDate
![Page 59: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/59.jpg)
Hackers can’t get much?
Stores none of that data?
A web portal into internal government
data systems?
![Page 60: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/60.jpg)
Application
Start your application
![Page 61: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/61.jpg)
Application
Confusing questions
![Page 62: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/62.jpg)
Application
Multiple personalities
![Page 63: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/63.jpg)
Application
No data available in table
![Page 64: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/64.jpg)
Application
Uncaught type error
![Page 65: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/65.jpg)
Application
Uncaught type error
![Page 66: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/66.jpg)
Application
Uncaught type error
![Page 67: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/67.jpg)
Application
Dead end
![Page 68: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/68.jpg)
Application
Processed an application I did not submit
![Page 69: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/69.jpg)
Application
Processed an application I did not submit
![Page 70: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/70.jpg)
Application
Processed an application I did not submit
![Page 71: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/71.jpg)
Application performance
>8 seconds to go to the next question
![Page 72: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/72.jpg)
Application performance
Huge payload
![Page 73: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/73.jpg)
Application performance
Wow!
![Page 74: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/74.jpg)
Application results
After about a month of trying
![Page 75: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/75.jpg)
Application results
You don’t qualify
![Page 76: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/76.jpg)
Application results
Eligibility requirements
![Page 77: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/77.jpg)
Application results
Eligibility requirements
![Page 78: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/78.jpg)
What went wrong?Testing failure
![Page 79: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/79.jpg)
What went wrong?Testing failure
![Page 80: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/80.jpg)
What went wrong?Implementation failure
![Page 81: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/81.jpg)
What went wrong?Implementation failure
Browse Plans
Create Account 1Login 2
Verify Identity 3Apply for Insurance 4Submit Application 5
Determine Eligibility 6
![Page 82: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/82.jpg)
What went wrong?Management failure
![Page 83: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/83.jpg)
What went wrong?Management failure
• 55 companies involved in building the mess– 0 were responsible for overseeing the others
– “eternal loop of damnation” getting companies to work
together
• 0 monitoring– 0 were responsible for making sure system was usable
– Watched CNN to learn about problems
• 0 sense of urgency– Government software projects fail all the time
– This was just like every other project
“Everything’s been done wrong, almost.
Almost no place we can point to a decision
where we made the right one.”
- Mikey Dickerson,
United States Digital Services
Mikey Dickerson: One Year After Healthcare.gov, http://x.co/1yearafter
![Page 84: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/84.jpg)
Your turn
Put on your tester hat and x-ray specs
• Testing is investigation
• Requirements documents are not required
• Communicate carefully
• Ethical behavior is essential
![Page 85: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/85.jpg)
Testing is investigation
Testing is the process of evaluating a product
by learning about it through experimentation
which includes to some degree:
– questioning,
– study,
– modeling,
– observation,
– and inference.
- James Bach & Michael Bolton, Testing and Checking Refined
![Page 86: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/86.jpg)
Consistency heuristicsJames Bach & Michael Bolton
Requirements documents are not required
(F)
Familiar
E
Explainable
W
World
HHistory
IImage
CComparable Products
CClaims
UUser Expectations
PProduct
PPurpose
SStatutes
![Page 87: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/87.jpg)
Requirements documents are not required
OWASP Top 10
Sensitive data exposure6
Function-level access controls7
Cross-site request forgery8
Components with known vulnerabilities9
Unvalidated redirects and forwards10
Injection1
Broken authentication & session management2
Cross-site scripting3
Insecure object reference4
Security misconfiguration5
![Page 88: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/88.jpg)
Failure mnemonicBen Simo
Requirements documents are not required
![Page 89: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/89.jpg)
Usability heuristics for user interface designJakob Nielsen
Requirements documents are not required
Visibility of system status
Match between system and the real world
User control and freedom
Consistency and standards Error prevention
Recognition rather than recall
Flexibility and ease of use
Aesthetic and minimalist design
Help users recognize, diagnose, and recover from errors
Help and documentation
![Page 90: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/90.jpg)
Communicate carefully
• Be accurate and precise
• Distinguish between what you observe and what you conclude
• Avoid speculation and blame
• Explain that which “goes without saying”
• Demonstrate the problem
• Explain the potential consequences
• Admit and correct your mistakes
![Page 91: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/91.jpg)
Understand and honor ethical and legal boundaries
• Do no harm
• Honor terms of use
• Use the interfaces provided
• Don’t attempt to gain access to others’ data
• Don’t enable others to do harm
Ethical behavior is essential
![Page 92: Stories From Testing HealthCare · 2015 67% private 9% uninsured •56% employment-based •16% direct purchase 37% government ... Username and password reset code emailed together](https://reader036.vdocuments.mx/reader036/viewer/2022081612/5f767dda0afcb315365cbd6a/html5/thumbnails/92.jpg)
IsThereAProblemHere.com