big data small testing -...

Big Data, Small Testing ?

Jayant Haritsa

Database Systems Lab

Indian Institute of Science

Feb 2017 Parcomptech CDAC 1

Old Concept, New Hype

• VLDB: Premier international database

conference, started in 1975

– Very Large Data Bases

• Large ≈ Big, ∴ Very Large ≫ Big

• Only a few enterprises really have big data,

the others just say so for bragging rights!

– World Data Center for Climate ~ 1 petabyte


NYT Op-ed Article [April 2014]

• Eight (No, Nine!) Problems With Big Data • Gary Marcus, Ernest Davis (NYU faculty)

“big data is prone to giving scientific-sounding solutions to hopelessly imprecise questions”

Who’s Bigger? Where Historical Figures Really Rank

(Book by MIT/Google: Hitler ranks higher than Aristotle!)

We need to ensure that Big Data does not wind up

becoming Huge Nonsense …


CallingBullshit.org [2017]

Univ. of Washington, Seattle

Profs. Carl Bergstrom and Jevin West

1 credit course: Calling Bullshit in the Age of Big Data

”We will focus on bullshit that comes clad in the trappings of scholarly discourse. Traditionally, such highbrow nonsense has come couched in big words and fancy rhetoric, but more and more we see it presented instead in the guise of big data and fancy algorithms — and these quantitative, statistical, and computational forms of bullshit are those that we will be addressing in the present course.”


Research Landscape

• Current Focus: Architecting the “plumbing”

infrastructure for Big Data environments • programming models, stream processing and summarization,

sketching and approximation algorithms, storage architectures,

cloud hosting, analytics, security …

• These techniques are unlikely to work in practice!

• The elephant in the room is the lack of testing methodologies for such deployments

5 Feb 2017 Parcomptech CDAC

Quotes†

50% of our cost is on testing (QA)

(Bill Gates @ Opening of Gates Building)

Testing alone takes up six months of the

18 month product release cycle

(SAP Executive)

Estimated damage of 60 billion dollars

per year in USA caused by software bugs

(US Department of Commerce, 2004)


† From Donald Kossmann’s Stanford talk

Big Data Disasters


1. UK Immigration [2013]

A Home Office text message campaign accusing people of being illegal immigrants has received numerous complaints after several people were contacted in error. Officials have sent messages to almost 40,000 people they suspect of not having a right to be in the UK, instructing them to contact border officials to discuss their immigration status. Government commissioned Capita, the outsourcing company, to trace people believed to have outstayed their visas.


UK Immigration (contd)

In a few months, Capita was accused of mishandling cases and getting just as mixed up as the bureaucrats it was supposed to be replacing!

In November, Capita admitted a backlog of 150,000 notifications to foreign students it hadn't been able to process and therefore determine if they should or shouldn't still be in the country.

In IT terms, it's been at the center of a billion dollar botched "e-borders" system, which has been missing deadlines and delivery dates since the middle of the last decade and which may not even be legal under European Union legislation!


2. Obama HealthCare.gov [2013]

Severe problems were caused by unexpected high volume when the site drew 250,000 simultaneous users instead of the 50,000-60,000 expected. More than 8 million people visited the site from October 1 to 4. White House officials subsequently conceded that it was not just an issue of volume, but involved software and systems design issues. Also, stress tests done by the contractors one day before the launch date revealed that the site became too slow with only 1,100 simultaneous users !

HealthCare.gov problems persisted even weeks after the launch. For example, a networking error at the related data services hub killed the website's functionality. This occurred the exact day after Health & Human Services head Kathleen Sebelius had highlighted designing that data hub as a government success.


3. Flipkart → Flopkart [2014]

Deccan Herald: Big Apology Day follows Flipkart's Big Billion Day – After its Big Billion Day on Monday, which fetched Flipkart.com $100

million by way of sales and the ire of hordes of angry customers who complained of technical glitches and false promises on discounts, the Bangalore-based online giant was quick to apologise for its drawbacks on Tuesday.

– “Though we saw unprecedented interest in our products and traffic like never before, we also realised that we were not adequately prepared for the sheer scale of the event. We didn't source enough products and deals in advance to cater to your requirements. To add to this, the load on our server led to intermittent outages, further impacting your shopping experience on our site,” the Bansals said.

– Noting that it took enormous effort from everyone at Flipkart, many months of preparation and pushing its “capabilities and systems to the limit” for the big day, the Bansals said that they were looking at deals and offers painstakingly put together for months.


Flipkart → Flopkart [2014]

Price Changes

– Even as Flipkart prepares various deals and promotional pricing in the lead-up to the sale, the pricing of several products gets changed to non-discounted rates for a few hours.

Out of stock – The website ran out of stock for many products within a few minutes

(and in some cases, seconds) of the sale going live. Most special deals were sold out as soon as they went live.

Cancellations

– A large number of people bought specific products simultaneously. This led to some instances of orders getting overbooked for a product sold out just a few seconds ago.

Website Issues

– Nearly 5000 servers were deployed and had prepared for 20 times the traffic growth. But the volume of traffic at different times of the day was much higher than this.


Testing Times


Software Mindset†

Everybody loves writing code

Everybody hates testing it

– emphasis on developing new models

than on evaluating current setups

– solution: automate the testing

computers are cheap and do not complain

Feb 2017 Parcomptech CDAC 14 14 14

† From Donald Kossmann’s Stanford talk

Automated Testing

Test Automation is a DB Problem – several optimizations in different flavors

– all about logical data independence

– far cry from being solved!

Research on Testing Methodology (as opposed to doing the testing) is fun! – interesting intellectual problems with immediate practical impact

– theory, algorithms, data structures, experiments, prototypes, …

– Testing and Tuning of Database Systems Special Issue of IEEE Data Engineering Bulletin, 31(1), 2008

Feb 2017 Parcomptech CDAC 15 15

ftp://ftp.research.microsoft.com/pub/debull/A08mar/issue1.htm

ftp://ftp.research.microsoft.com/pub/debull/issues-list1.htm

Research Problems [1]

• Given a large workload of test queries, how to

summarize them into a small set that provides

equivalent coverage? [SIGMOD 2002]

• automatic index selection in databases, workload-based sampling

• NP-hard from Minimum k-median; constant-factor approximation

for metric spaces

• Retaining select data characteristics in

outsourced data processing [AMCIS 2012]

• retain data quality problems (missing data, incorrect format,

domain violations) in encrypted customer data!


Research Problems [2]

• Given a database D, a parametrized query Q,

and cardinality constraints C on subexpressions,

how to assign the parameter values to Q to

satisfy the constraints? [TKDE 2002]

• problem is NP-hard from Subset-sum; hill-climbing heuristic

• Reverse query processing [ICDE 2007]

• Given query Q on database schema D, and desired result R,

generate a database instance D s.t. Q(D) = R; ideally, minimal D !

• Undecidable whether such a D even exists!

• Model checking, symbolic processing, reverse relational algebra


Trichotomy†


Answers

Database

Queries

18

† From Mike Franklin, UCB

Trichotomy†


Answers

Database

Queries

Query Processing

19


Trichotomy†


Answers

Database

Queries

Reverse

Query Processing

20


Trichotomy†


Answers

Database

Queries

Programming

By Example

21


DB Testing Practices


Basic Question

How do you know the output delivered

for the user objective is correct?

Checking is hard because of the

magnitude of data involved and the

complexity of the queries


Types of Errors

English-to-SQL translation errors

“Public demands change” Public is demanding change in society

Public demands are changing over time

Public is demanding loose change (coins)

– Big problem (only about 40% are correct !)

– Further, more than 80% are written correctly

only after two to four attempts!


Types of Errors (contd)

Syntactic errors

– easy to check with automatic parser generators

Semantic errors

– Schema/type errors (easy to check from catalogs)

– Arithmetic errors (easy to check at runtime)

– Optimizer rewriting errors

e.g. infamous Count Bug [1986]

– Operator implementation errors

– Index maintenance errors

– Transaction management errors

e.g. ARIES checkpoint error


Hard to find

Library Approach

SQL test libraries designed by the

engine developers or application

specialists

Run regression tests on this workload

– Very limited coverage


Stochastic Approach

Stochastic (random) generation of SQL

queries [Microsoft]

– highly expensive on real databases

– unlikely to catch “boundary-value” errors

– will not easily catch self-join and other such

specific errors

– correlated random variables


Moving on to the

Big Data World

Feb 2017 28 Parcomptech CDAC

Test Environment

• Underlying infrastructure is a hybrid of

ETL/IR/KM/DB components

• e.g. IBM Infosphere (DataStage, QualityStage, MDM,

DB2, Big Insights, Metadata repository, …)

• Need to test

• “functionality” (programs/data)

• “compilation” (query/model planning)

• “execution” (query/model processing)


Sample Scenario

• Wish to test “yottabyte” (1024 byte) scale

Big Data environment for InfoSphere

• Metrics: Functionality, Correctness,

Performance, Scalability

• Impractical (time) or infeasible (space) to

explicitly create and process test data


Pie-in-the-sky

A complete testing environment for Big

Data management systems, wherein the

entire data and meta-data is virtual or

transient, supporting efficient evaluation

of arbitrary deployment scenarios.


Metadata Testing


Our Approach

• Build metadata construction tools that

“fool” the underlying information systems

into thinking that the data is actually

present even though it had never been

created or stored

• Developed tool called CODD

(Constructing Dataless Databases)

• Edgar Codd, IBM, father of RDBMS / Turing awardee

• In archaic English, “cod” means “empty shell”


CODD Metadata Processor

• Easy-to-use graphical tool for the automated

creation, verification, retention, scaling and

porting of database meta-data configurations

• Entirely written in Java (~50K LOC) and

operational on industrial-strength db engines (DB2, Oracle, SQL Server, SQL-MX, PostgreSQL)

• Released as free software after receiving

copyright from the Indian government

• In use at several industrial and academic

research labs


Feb 2017 Parcomptech CDAC

Metadata Construction

35

• Users can directly input statistics on:

• Relational Tables (row cardinality, row length, disk blocks)

• Attribute Columns (column width, number of distinct values,

value distribution histograms)

• Attribute Indexes (number of leaf blocks, clustering factor)

• System Parameters (cores, memory size, CPU utilization)

35 35


Graphical Histogram

36

36 36


Metadata Validation

37

Need to ensure that the input information is

– Legal (valid type and range)

– Consistent (compatible with other metadata values)

Validation Approach – Construct a directed acyclic constraint graph CG(V,E)

– V is the set of individual metadata entities while E is the set of

statistical value dependencies

– Super Nodes: used to represent collapsed chain of nodes for

compactness

– Run topological sort on CG to obtain CGlinear

– CODD uses this linear ordering to guide the user

37 37


Constraint Graph [DB2]

38

Legality

Constraint

Statistical Dependency:

Direction chosen as per

abstraction hierarchy

Super

Nodes

Dashed edges represent

missing constraints

Node Processing Order

38 38

Unique features of CODD

• Supports creation of arbitrary “what-if” scenarios

• Carries out automatic validation of user input

• Supports both space-based scaling and time-

based scaling

• Provides graphical histogram operations

• Supports inter-engine metadata transfer


CODD in action

• Successfully simulated yottabyte environment on

a vanilla laptop

• Demonstrated deep bug in a popular commercial

database system that only surfaces at Big Data

scale

• Query optimizer saturates at 1020 bytes

• Hidden constant in the code inserted by a programmer

to symbolize “infinity”


Take Away

Research on Automating Big Data Testing is great technical fun with immediate practical relevance ...

Stop Protesting, be Pro-Testing !


More Details

For publications, software and documentation, visit:

http://dsl.cds.iisc.ac.in/projects/CODD/index.html


http://dsl.cds.iisc.ernet.in/projects/CODD/index.html

Questions ?

Feb 2017 43 Parcomptech CDAC

Parcomptech CDAC

Example Application Scenario

52

TPC-H 1 GB (baseline) and TPC-H 100TB (scaled) (both are metadata shells created using CODD)

Commercial DBMS; Laptop with 64 GB disk

Query 9 of TPC-H

select n_name, o_year, sum(amount)

from (select n_name, o_orderdate, l_extendedprice

from part, supplier, lineitem, partsupp, orders, nation

where s_suppkey = l_suppkey and ps_suppkey = l_suppkey and

ps_partkey = l_partkey and p_partkey = l_partkey and o_orderkey

= l_orderkey and s_nationkey = n_nationkey and p_name like

%green% and s_acctbal :varies and ps_supplycost :varies

) as all_nations

group by n_name, o_year

order by n_name, o_year desc

Feb 2017 52

Scalability: 1GB to 100TB Database

53

Baseline Dataset (1 GB) Scaled Dataset (100TB)

Number of Plans

increased from

32 to 77!

Significant change in

geometries of plan

optimalityregions


big data small testing -...

Documents