software fails 2016
TRANSCRIPT
© 2017 Tricentis. All rights reserved.
Software Fail Watch:
2016in Review
© 2017 Tricentis. All rights reserved.
How It Works The Big
PictureSoftware Fails by Industry
One Year Rewind
Software TypeType of Software Fail
Software Fails in the Stock Market Government Software Fails
Retail Software Fails Entertainment Software Fails Transportation Software Fails
Service Software FailsFinance Software
Fails The Software Fail Awards Biggest Hacks of
2016Biggest Fails of 2016 That Went
Un-noticed (Almost) Most Shocking Fails of 2016
ConclusionResources
The Software Fail Watch: 2016 in Review is a sobering reminder of how even a single software bug can cripple an enterprise. With 4.4 billion people and 1.1 trillion in assets impacted by software failures in 2016, it’s hard to argue that “more of the same” is the best path forward for software development professionals.
As the demand for the latest and greatest in technology and convenience grows, so does the need for software testers to protect their users and their brand from the potential influx of software failures. Our goal at Tricentis is to help testers succeed in this role—enabling fast, efficient, comprehensive testing that’s designed to support Continuous Testing, Agile, and DevOps.
Ultimately, we want to ensure that the inevitable software bugs are found by your testers, not your customers.
Preface
The Software Fail Watch is a collection of software bugs found in a year’s worth of English language news articles. To find the stories, we set up a Google account with an alert for phrases such as “software glitch” and “software bug”.
Then we manually sorted through each of the alerts, picking out promising headlines, reading the articles for relevance, and noting down any specific details of interest. If the article reported a software bug, we logged it into an Excel sheet (which you can download here), and extracted as much information as possible.
What industry does this fall into? Does the article say how much the affected software cost to implement?
Does it mention how many cars were recalled? How long was the system down? Is the associated company public, private, or a government contractor? You get the idea.
No exact numbers were recorded unless they were explicitly stated in the article itself.
Stories that appeared in multiple news alerts and were covered by multiple reputable sources were marked as having a “high level” of brand damage.
In the end we categorized all the stories into 6 broad industry categories: Entertainment, Finance, Government, Retail, Services, and Transportation.
How It Works
1159363
Stories
548 Software Fails Companie
s
© 2017 Tricentis. All rights reserved.
Statistically, there is a very good chance that you have been personally impacted by a software failure this year – perhaps even in the last quarter, or week.
The Big Picture To find the total amount of people affected, we added all the numbers we had pulled: i.e., cars recalled, mobile phones with malware, paychecks undelivered, accounts hacked, etc. and assumed that each item directly affected one person.
That means that in reality, our end sum is on the conservative side. It is far more likely that for every car recalled, a larger total number of people were affected, with further implications that rippled outwards in a software “butterfly effect”.
7.4 bn
4.4 bnPeople
AffectedAmounts to over 50% of the world‘s population
0
© 2017 Tricentis. All rights reserved.
10
60
50
40
30
20
70
Jan. Feb. Mar.April
Software fails by month, 2016
May June July Aug. Sept. Oct. Nov. Dec.
315 years,
6 months,2 weeks,
6 days,
16 hours,
& 26 minutes.
Accumulated time lost
$1,062,106,142,949
Assets affected (USD)
17,701Gulf Stream G-550
jets(valued at $60,000,000 USD)
The Big Picture, Pt.2
© 2017 Tricentis. All rights reserved.
© 2017 Tricentis. All rights reserved.
Software Fails by Industry
stories picked up by multiple news outlets47
2
Software fails by industry, 2016
A high-level view reveals clear patterns in where and how these software fails occur. Government-related software fails dominate the charts, with an average of 15 fails per month. Retail and Transportation are tied in second place, both clocking in an average of 9 fails per month.
Many trends observed in last year’s Software Fail Watch continued this year. For example, Transportation’s software fails peaked in late spring, while Retail’s software fails rose steadily in the months leading up to the Christmas holidays. The Finance and Entertainment industries kept a fairly low profile over the course of the year, both averaging just 2 software fails per month.
The wild card in 2016 is the Servicesrepresenting internet,
and telecom, etc. Theof software fails jumped
industry, electricity, numbers erratically
from monthwith
tomonth,
11 recordedpeaking in Maysoftware fails.
0
5
10
15
20
25
Jan. June July Aug. Sept. Oct. Nov. Dec.Feb.
Mar.
Entertainment
AprilMay
GovernmentFinance Retail Services Transportation
© 2017 Tricentis. All rights reserved.
© 2017 Tricentis. All rights reserved.
There are far more software bugs in theworld than we will likely ever know about.
Software Failsby Industry, Pt. 2
12%yearly increasein software fails
Comparing 2016 data vs. 2015 throws the picture into even sharper relief. If anything, the need for better software testing is only growing.
Ever the buggiest industry - Government - clocked in an additional 42 fails in 2016 over 2015. Retail, Transportation, and Service exceeded 2015’s numbers, while the Entertainment and Finance industries managed a year-on-year decrease.
Finance’s numbers decreased by nearly 35%, indicating that either the testing or the public relations in this industry have improved over the past year.
0
20
40
60
80
100
120
140
160
180
200
Entertainment Finance Government
Software fails by industry, 2015 and 2016
Retail Services Transportation
Year 2015 Year 2016
© 2017 Tricentis. All rights reserved.
One Year Rewind
4.3 billionPeople affected
vs.
4.4 billionPeople affected
239Companies
vs.
363Companies
4.2 billion
Assets affected
vs.
1.1 trillion
Assets affected
2015
2016
0
10
20
30
40
50
60
70
80
90
100
Jan. Feb. Mar. April May JuneJuly
Software fails by month, 2015 and 2016
Oct.Aug. Sept.
Total 2015
Nov. Dec.
Total 2016
© 2017 Tricentis. All rights reserved.
Software Type190Mobile/Cloud
“Mobile/Cloud” encompasses all web or app-based software. If a website went down or an app failed, it went into this category. While each industry was well represented within this category (showing how even industries that traditionally run legacy software have jumped on the mobile-boat), Retail predictably took the lead with72 stories. Of those 72 stories, 44 featured hardware and smart phone manufacturers such as Apple, Google, or Samsung.
217
On Premise
“On Premise” encompasses all software that requires installation in a specific location. If an organization’s internal system, such as an ERP (Enterprise Resource Program) or accounting software crashed, it went into this category. On Premise software exists within every industry. However Government overwhelmingly dominates this list, making up 59% of the stories in this category.
141
Embedded“Embedded” includes all software that is pre-installed on a device or piece of hardware. If a casino’s slot machine experienced a glitch, or a car’s airbag sensor malfunctioned, it was placed into this category. Transportation
makes up the majority of this category,141 stories. This is
given that mostwith 67 of unsurprising Transportation related software failsstem from a problem with the vehicle itself, be it car, plane, or train. Fails by software type, 2016
© 2017 Tricentis. All rights reserved.
Type of Software Fail
4 3 2 Software Bugs
3 8 Usability Glitches
78 Security
Vulnerabilities
In recording the stories, we identified three main types of software failures. The first, and most common, is a software bug: an instance in which a software application does not work as designed. The second type is a usability glitch: a design flaw that decreases the usability of the product or application. The third is a security vulnerability: a flaw that attackers can exploit to alter a system’s behavior.
Type of fail by industry, 2016
Type of fail by month, 2016
0
10
20
30
40
50
60
70
Aug. Sept. Oct. Nov. Dec.Jan. Feb. Mar. April
Software Bug
May JuneJuly
Usability GlitchSecurity Vulnerability
0
20
40
60
80
100
120
140
160
180
Entertainment Finance Government Retail
Software Bug Usability Glitch
ServicesTransportation
Security Vulnerability
© 2017 Tricentis. All rights reserved.
395
390
400
405
410
415
420
425
430
435
18. Jul 19. Jul 20. Jul 21. Jul 22. Jul
Approximately 40% of the companies hit by software fails in 2016 were public companies. While it is not always possible to trace the effects of a software fail in the rise and fall of a company‘s stock prices, there are times when the correlation is unmistakable.
British Airways’ parent company, International Consolidated Airlines Group, S.A (IAG), is one such example. British Airways implemented a new global check-in system in 2016, which quickly became a source of frustration and embarrassment. British Airways faced 5 major computer outages between May and September, resulting in thousands of flights canceled or delayed, and a cumulative stock market loss of 10.54% or £92.9 billion. The graph above shows a snapshot of IAG’s stocks following the July 17t h outage, in which the stock prices dipped 2.28% in the first dayalone, for a loss of £20.8 billion.
Software Failsin the
Stock Market
148Public Companies
Affected
Upon news of the software fail, the stock prices dropped-2.28%, for a loss of-£20,776,000,000 inmarket capitalization.
By July 22nd, a combination of factors (including Brexit), had caused IAG’s stock price to fall 25 points since July 18th.Stock markets
open Monday, July 18, after a severe British Airways software fail over the weekend.
© 2017 Tricentis. All rights reserved.
© 2017 Tricentis. All rights reserved.
"Every company is a software company. You’re building software that is going to deliver business outcomes and software is the differentiator for your business.”
https://www.skytap.com/blog/theresa-lanowitz-on-solving-age-old-problems-in-the-enterprise/
Theresa Lanowitz, voke
© 2017 Tricentis. All rights reserved.
Government organizations undeniably top the charts when it comes to software fails. The cause is multifaceted. Government organizations typically rely on contracted developers to custom build software that can fulfill complex operations and handle massive amounts of data. Furthermore, many organizations do not have in-houseresources for implementation. revealed during decision makers
testing or software Even if defects are implementation, many are pressured to move
ahead since they have already invested massive amounts of public funds into the project. This also explains why so many government organizations are still using legacy software – the resources, time, and money required for a software upgrade is simply too high for many to justify.
Government Software Fails
$5,703,579,938
Averaged known cost
of failed government software (USD)
can buy
Government fail by sector, 2016
30,193Houses in the United States(median price of a house in 2016: $188,900 USD)
0
© 2017 Tricentis. All rights reserved.
5 10 15 20 25 30 35 40 45 50
Transport
Taxes
Space
Exploration
School
Justice
Healthcare
Emergencies
Elections
Admin
© 2017 Tricentis. All rights reserved.
Retail Software Fails
The “Internet of Things” (IoT) has expanded the reach of software within our homes and across even mundane activities. Connecting your tea kettle, thermostat, and interior lighting to the internet is opening up new realms of possibility that the retail technology sector - and hackers - are just beginning to explore. The vast majority of the “security vulnerability” software fails recorded in 2016 were linked to the IoT in some way. Even the most harmless IoT hacks highlight both the weaknesses in our current IoT devices, and the increasing need for security as our lives become more web- dependent. However, many of the software fails featured hacks with dire implications – including a “botnet” DDOS attack that took down large portions of America’s internet for most of October 21st.
2,666,600 , 00 0
Mobile PhonesInfected with Malware
201816141210
86420
Jan. Feb. Mar.April
Retail software fails by month, 2015 and 2016
May June July Aug. Sept. Oct. Nov. Dec.
Year 2015 Year 2016
© 2017 Tricentis. All rights reserved.
A significant portion of the entertainment-related software fails in 2016 were discovered as part of Bug Bounty programs. Bug Bounty Hunting has become increasingly popular in the last few years, with big- name enterprises like Apple, Google, and Facebook offering cash rewards for newly discovered bugs. Even government organizations have joined the fray, such as the United State’s “Hack the Pentagon” initiative announced this year. The increase in Bug Bounty Hunting reveals two interesting trends: the widespread acceptance of crowd-sourcing quality assurance, and the mainstreaming of the white or gray-hat hackers. One of the notable stories from 2016 was Facebook’s $10,000 USD award to a 10- year-old Finnish hacker-in-the-making. Though$10,000 is not the highest award recorded, a 10- year-old boy certainly is the youngest person to be awarded.
Entertainment Software Fails
0
2
4
6
8
10
12
14
Jan. Feb. Mar. April May JuneJuly
Entertainment software fails by month, 2015 and 2016
Aug. Sept. Oct.
Year 2015
Nov. Dec.
Year 2016
© 2017 Tricentis. All rights reserved.
© 2017 Tricentis. All rights reserved.
Transportation Software Fails
2 1 , 2 2 8 , 0 6 6 Cars recalled
8,831 Planes grounded
22,712,987 People affected
Self-driving cars made the headlines a few times in 2016, as key players such as Google and Tesla sought to mainstream their use. In early January, Google released a report recounting the numbers of times a human took control of the self-driving car during tests (354 times within 15 months). In July, it was revealed that Tesla would likely be facing serious consequences after an Autopilot failure resulted in one man’s death. In October, the Singapore-based nuTonomy reported an accident while conducting testing for their flagship self-driving car service. We expect the headlines related to self-driving cars will only increase in the coming years. This showcases yet again that comprehensive risk-based software testing is now a necessity in all industries.
12
10
8
6
4
2
0Jan. Feb. Mar. April
May
Transportation software fails by month and mode of travel
June July Aug. Sept. Oct. Nov. Dec.
Road Air Rail
© 2017 Tricentis. All rights reserved.
Service Software Fails
The service industry plays a significant role in underpinning our every day lives. Whether it be within healthcare, internet, or telecom, a service-oriented company‘s influence interweaves into other industries like government, retail, or transportation. Roughly 50% of the service software fails from 2016 fell within the internet or telecom sectors. Many dramatic service-related stories made the news in 2016, one of the most notable being Yahoo’s admission of a cyber-breach that stole data from a staggering 1.5 billionaccounts. Another big story detailed how a recently discovered bug could negate 15 years worth of MRI findings and research.
27%
Of the stories explained the cause of the software fail
18
16
14
12
10
8
6
4
2
0Jan. Feb. Mar.
April
Service software fails by month, 2015 and 2016
May June July Aug. Sept. Oct. Nov. Dec.
Year 2015 Year 2016
© 2017 Tricentis. All rights reserved.
© 2017 Tricentis. All rights reserved.
Finance Software Fails
1.3 yearsAverage time lost
$521m ill
ionBiggest single loss
In our experience, software fails in the finance industry rarely hit the headlines. It is not that the industry does not have software fails – rather, they simply seem to be reported less. As opposed to some other industries, much of a financial institution’s software lies behind the scenes. This makes it easier for bugs to be patched quickly before their repercussions can snowball into a major news story. The finance stories that do come to light either take place in public-facing venues (such as a stock market), or feature bugs so catastrophic that the story cannot be buried. It seems that financial institutions are extremely vigilant when it comes tomaintainingWhen we 2014 and fails, many
their publicimage.recently reviewed our 2015 finance software of the original reportswere removed from news sites and
scrubbed from search results.0
2
4
6
8
10
12
14
16
18
Jan. Feb. Mar. April MayJune
Finance software fails by month, 2015 and 2016
July Aug. Sept. Oct.
Year 2015
Nov. Dec.
Year 2016
© 2017 Tricentis. All rights reserved.
© 2017 Tricentis. All rights reserved.
We’ve said it before and we’ll say it again: software may come and go, but software testing is here to stay.
Scariest FailMultiple incidents have been reported of creeps hacking into internet-connected baby monitors. Parents have discovered hackers taking advantage of a securityvulnerability in the baby monitor to control the camera, watch the room, play creepy music, and speak to the babies in the night.
The Software Fail Awards
Biggest “WTF?“ FailA US-based IP mapping firm accidentally mapped over 600million “unknown” IP Addresses onto a single address in Kansas. The unfortunate family living at the address has dealt with theconsequences for the past 10 years: police and FBI showing up regularly, having “followed” an IP address to the house looking forkidnappers, child pornographers, drug cartels, thieves, and more.
Funniest FailAn English programmer purchased a Wi-Fi enabled, voice-activated tea kettle. He then spent the next 11 hours attempting to make a cup of tea. An entire day’s worth of reprogramming later, the poor man finally got his morning cup of tea.
© 2017 Tricentis. All rights reserved.
Biggest Hacks of 2016
politicians, celebrities,
“The Panama Papers” hack leaked11.5 million documents and 2.6TB of data from the Mossack Fonseca law firm based in Panama – a known tax haven. The leak revealed the illicit financial dealings of
anddignitaries worldwide.
The CIA formally accused Russia of attempting to influence the United State’s presidential elections in favor of the Republican Party by releasing private emails from the Democratic National Committee. While the hack itself certainly occurred, whether Russia was responsible is still being debated.
Multiple banks in India were affected by a massive financial data breach, compromising 3.2 million debit cards. The source of the breach was traced back to malware in Hitachi’s Payment Services, allowing hackers to collect sensitive banking data and steal funds.
Hackers managed to steal $81 million from a bank in Bangladesh by exploiting a vulnerability in SWIFT, a financial transfer system. The dramatic story made headlines worldwide, and brought to light accounts of smaller bank heists exploiting the same vulnerability in recent years.
Yahoo admitted to two damaging hacks that occurred in 2013 and 2014, resulting in data stolen from1.5 billion accounts. It is unclear why this incident only came to light in 2016, however it does not bode well for Yahoo’s future.
© 2017 Tricentis. All rights reserved.
© 2017 Tricentis. All rights reserved.
Biggest Fails of 2016 That Went Un-noticed (Almost)CGTechnology, a sports gambling company, paid a fine of $1.5 million USD and fired its CEO after willfully ignoring a software glitch that underpaid bettors $700,000 USD in winnings. The company was not only accused of purposefully avoiding fixing the bug, but also of interfering with the Nevada Gaming Control Board‘s investigations into the matter.
Worldpay, a UK-based payment processing firm that processes over 36 million payments per day, experienced a software fail that crippled their services for over three weeks. The bug was traced back to a server software update that resulted in an overload of error messages. The failure comes as a particular embarrassment after the enterprise had recently invested over $500 million USD in updating the system’s software.
A secretive database containing 2.2 million names of people suspected of terrorism and organized crime was leaked into the public domain in July 2016. The database, which is managed by Thomson Reuters and used by banks and intelligence agencies alike, was not hacked (as might be expected), but simply
the public sphere after an “databasesoftwareerror”
dropped into unexplained occurred.
A teenager in Arizona faced three felony charges after creating a bug that took out911 emergency phone services in
oldcreateda an attempt to
three “non
- win a
states. The 18 year harmful” iOS bug in bounty from Apple’s The bug, which he
bug-hunting program. shared via a link on
Twitter, caused iPhones to call 911 on repeat until the phone was shut off. The link was reportedly opened thousands of times, swamping local 911 emergency services with hundreds of hang-up calls per minute.
The DAO, an investment fund containing Ethereum, Bitcoin’s rival crypto-currency, was robbed of over $50 million USD worth of “Ether” in late June. Not only did the hackers manage to successfully exploit the fund’s security vulnerabilities, but the necessary patch was so complicated that developers could not fix the vulnerability for several days after the theft. During that delay, a half-dozen smaller copy cat heists occurred.
Most Shocking Fails of 2016UK’s National Health Service admitted to mis- prescribing medication to over 300,000 heart patients due to a software error. SystmOne, the software used to calculate the risk of a heart attack, has reportedly producedincorrect results since 2009. As a result, some patients suffered from otherwise preventable heart attacks or strokes. Others needlessly dealt with the serious side-effects of taking unnecessary medication.
Fiat Chrysler recalled over 1 million vehicles after a government investigation revealed that a gear-shift flaw resulted in 266 accidents, 68 injuries, and at least 1 death (of a celebrity, no less). A software update was issued to address the flaw, but the update reportedly failed to fix at least 29,000 of the recalled vehicles.
A security researcher disclosed a firmware bug found in CCTV point-of-sale security cameras sold by over 70 different vendors. The bug, nicknamed ”Backoff”, is suspected to be a contributing factor to the spate of credit card breaches that occurred in majorretailers in the past years. TheU.S. Department of HomelandSecurity haswarned that up to 1,000 US businesses may be infected with the bug.
Thanksto system,
afaultyautomatedsoftware the
UnemploymentstateofMichigan’s
InsuranceAgency (UIA)mistakenly accused over 20,000 innocent
people of fraud. Midas, the data analysis system used, was wrongly accusing claimants of fraud in a whopping 93% of cases, resulting in unemployed persons losing their benefits and facing fines of up to $100,000 USD. The system, implemented in 2013, had been operating unchecked until 2015, when Michigan’s auditor general issued a scathing report on the system’s shortcomings.An Australian hospital suffered a string of
software outages, the longest lasting for 10 hours in early November. The software failreportedly cutting off medication, admit and
threw the hospital in chaos, access to patient records and and forcing staff to manually track patients by hand. The
hospital administration had reportedly complained several times in the previous months that the new software (implemented in June) would eventually have “fatal consequences” for patients if not fixed soon.
At one time (long, long ago), software was just a way of getting things done. It was nothing more than the convenience of using a calculator instead of doing your sums by hand. Those days are gone, however. The average enterprise software landscape contains 52 interconnected systems.* The average person’s software landscape is far vaster – as the saying goes, “no man is an island”, and the bridges between ourselves and the world are increasingly built with software.
Logically, this means that we could only expect the numbers we collect for the annual Software Fail Watch to increase year-by-year. That is not, however, the future Tricentis sees for software. Software testing methods will not simply plateau while software development and innovation climbs. The mainstreaming of DevOps and Agile has already done much to fulfill today’s demands for modern software. The future of software testing, however, belongs to Continuous Testing.
Conclusion
Love details?Download the full list of software bugs collected in 2016.
CREDITS