white paper unifying data with the 1010data insights platform · unifying data with the 1010data...

UNIFYING DATA WITH THE 1010DATA INSIGHTS PLATFORMUnifying your data is more important than ever. Learn how you can overcome barriers to data unification and achieve powerful new business insights.

WHITE PAPER

Unifying Data with the 1010data Insights Platform

TABLE OF CONTENTS

THE RACE TO UNIFY DATA 3

Emergence of SaaS and Cloud Point Solutions 3

Proliferation of New Data that May Matter 4

Rising Business Expectations 5

BARRIERS TO DATA UNIFICATION 6

Point-to-Point Data Integration 6

Aggregated Data Prevents Ad-Hoc Analysis 6

Systems Built for Experts 7

THE 1010DATA APPROACH TO DATA UNIFICATION 8

Loading All the Data 8

Enabling Immediate Analysis 9

Building Hybrid Data Structures 10

Scaling Across the Business 11

Updating Data Structures 12

CONCLUSION 13

ABOUT 1010DATA 14

2

To enable analytics that deliver insights, businesses have long recognized the importance of unifying disparate sources of data. Recently, however, an intense and renewed focus has been placed on data unification.

A few factors lie at the heart of why data unification has once again become a hot topic:

In the 90’s and 00’s, enterprises invested heavily in implementing and integrating on-premise ERP systems, along with the data warehouse and business intelligence (BI) systems for reporting and analysis against ERP-generated data. Many businesses anticipated that ongoing technology innovation would come in the form of enhancements and extensions that lived within the ERP paradigm.

THE RACE TO UNIFY DATA

EMERGENCE OF SAAS AND CLOUD POINT SOLUTIONS

Many elements of traditional ERP systems have been displaced by next-generation solutions delivered in the cloud. With business teams, ranging from call center to marketing to sales, now reliant on different services and generating disparate data, the need to unify this data for analysis is more urgent than ever before.

Cloud and SaaS Solutions Drive the Need for Data Unification

Customer Insights Call Center Loyalty Analysts Sales Marketing

Cloud CRMSaaS Marketing

Automation Web Analytics

SaaS + CloudPoint Solutions

Proliferation of External Data

Expectations of Business

? $

3


But in the new millennium, the cloud arrived – bringing with it an explosion of innovation in cloud software and SaaS solutions. These solutions enable new capabilities and new business models. They are delivered by an entirely new crop of vendors, atop next-generation data structures that look quite different than those of the incumbent ERP systems. And SaaS solutions – along with their underlying data – have enhanced and extended faster than ever before.

Rather than an evolution, businesses are experiencing a revolution in the way enterprise data is generated, and how quickly new sources and types of data need to be unified for analysis. The traditional data warehouse and BI systems that evolved in lockstep with ERP simply weren’t designed to (cost effectively) keep pace with the data integration demands that cloud solutions are now placing on businesses.

PROLIFERATION OF NEW DATA THAT MAY MATTER

In addition to the new sources of data generated by internal systems, the last few years have also seen a proliferation of new, external sources of big data. Massive troves of data are being generated from mobile devices, social media, internet browsing, instrumentation, the burgeoning Internet of Things and many more sources. And a new economy of syndicated, public and other third-party data – that promises to deliver profit-driving insights – has emerged as well. Specific examples of these data sources include:

The businesses that can capitalize on this data will have a powerful competitive edge into the future.

But which data is relevant? Until a business can begin testing and utilizing the data, there is no way to know which new data sources will deliver actionable insights that help boost performance. And with traditional BI infrastructure, it’s simply too difficult to explore and prove out new sources of data quickly enough. There isn’t enough time or budget for multiple IT projects to test the efficacy of all the new – and rapidly growing – sources of data available in today’s digital economy.

Census surveyClickstreamConsumer creditDemographicsDSD receipts

E-receipts Gas pricesInventoryLoyaltyMobile GPS

PromotionsSales Store trafficWeatherWebsite traffic

4

RISING BUSINESS EXPECTATIONS

The “datafication” of the modern business – and the empowerment of the modern individual by the web, social media and mobile devices – has given rise to a culture where everything is “more and faster.”

Consumer expectations have increased. Competition has increased. And businesspeople increasingly expect the insights locked in big data to provide the answers for succeeding in this new and challenging environment.

But for most businesses, extracting significant value from big data analytics has proven elusive. In 2011, McKinsey Global Institute published a study estimating that businesses stood to gain trillions of dollars in annual value, worldwide, by leveraging big data analytics. In 2016, McKinsey went back to analyze how much value had been captured during the 5 years that had passed. The results were as follows:

Across nearly every industry, challenges with data unification presented as the main reason why companies around the world remain unable to realize the hundreds of billions of dollars in business value that’s locked in big data.

1 The Age of Analytics: Competing in a Data-Driven World,” McKinsey Global Institute, December 2016

% of Potential Value Realized from Big Data, 2011-20161

Industry % Value Captured Major Barriers

US Retail 30-40% • Lack of analytical talent• Siloed data within companies

Manufacturing 20-30% • Siloed data in legacy IT systems• Leadership skeptical of impact

EU Public Sector 10-20% • Lack of analytical talent• Siloed data within different agencies

US Healthcare 10-20% • Need to demonstrate clinical utility• Interoperability and data sharing

More Faster

Consumer touch pointsProduct choicesNimble competitors

Data that mattersMore, more and more

Expected response timesProduct developmentDelivery of goods and servicesSolutions to problemsFaster, faster and faster

5


Why do businesses face so many barriers when attempting to unify data?

The majority of incumbent analytic systems and processes are effective at delivering standardized reporting and interactive analytics that serve a well-defined specification. But when it comes to asking the questions that have never been asked before – which often requires the unification of new data sources with existing data – traditional BI and reporting processes can struggle.

These struggles are typically a result of one or more of these common barriers to data unification:

BARRIERS TO DATA UNIFICATION

POINT-TO-POINT DATA INTEGRATION

To meet immediate departmental reporting needs, many businesses deploy point-to-point data integrations. In this approach, a small number of source system data feeds are integrated at an aggregate level using cubes. Today’s reporting requirements are met – but when a new question arises, the data in the cubes isn’t detailed or flexible enough to provide answers. Additional point-to-point integrations are created. Soon, it becomes untenable to maintain consistency across the proliferation of cubes.

Is everyone calculating profit the same way? Are data transformations consistent across all integrations? Over time, the answer becomes a consistent “no.”

AGGREGATED DATA PREVENTS AD-HOC ANALYSIS

Ad-hoc analytics – the ones that seek insights about the latest market trends, the latest corporate initiatives, or seek to glean value from new data – can deliver the greatest returns of all. But with an infrastructure that struggles with data unification, a very complex process for answering ad-hoc questions emerges: one that takes too long and requires too many valuable resources. Unfortunately, businesses begin to avoid asking the very questions whose answers could yield the greatest competitive edge.

Point-to-PointIntegration

Aggregated Data Prevents Ad-Hoc

Analysis

Systems Built for Experts

EXPERTS

6

SYSTEMS BUILT FOR EXPERTS

Delivering analysis has become tremendously complex. As such, the businesspeople who need to gain insights from big data have become reliant on experts who are increasingly difficult to find and retain. For instance, the modern data scientist requires multidisciplinary skills that include math, statistics, programming, query processing, BI & visualization tools, vertical industry expertise, and whole range of soft skills. It’s no wonder the scarcity of experts has become one of the leading bottlenecks to enabling unification of data throughout the enterprise.

Complex Process to Answer Ad-Hoc Questions

1. The business executive asks a question that can’t be answered with an existing report2. A business analyst translates the question into a reporting specification3. An IT project manager identifies the systems, data and people resources needed4. The project is placed in the busy IT queue5. Experts in each of the source systems extract the data6. An IT analyst builds a one-off data integration and generates a report7. A business analyst adds business context and terminology to the report8. The executive gets their answer, then asks a follow-up question9. The painstaking process begins all over again

“What is the impact of this new competitor on our revenue? Which sales channels are most a�ected?”

Business Analyst

IT Project Manager

IT Queue IT Analyst Business

Analyst “Great! I’d also like this report

broken down by department.”

Executive

Executive

Brick & Mortar

Syndicated

Ecommerce

1

2 3 4 6 78

9

5

7


The 1010data data unification approach, enabled by the innovative technology of the 1010data Insights Platform, operates under a few key principles: 1. Make all the data available for analysis2. Enable immediate analysis 3. Build granular-level hybrid data structures4. Scale across the business5. Allow for straightforward updating of data structures

THE 1010DATA APPROACH TO DATA UNIFICATION

Data that looks like “noise” today could become the source of key insights that a business is searching for tomorrow. In the past, for instance, retailers had no use for voluminous “time of day” sales data and scrubbed it from their analytic systems; however, with the advent of mobile phones and real-time promotions, that data is now crucial to omnichannel success. Don’t throw away any data! Instead, load it in its entirety and preserve it for future analysis.

THE VALUE OF “MESSY” DATA

LOADING ALL THE DATA

With 1010data, source data is loaded in its entirety and its rawest, most granular form. It isn’t necessary to structure or transform the data ahead of loading – this can be done anytime afterwards, in direct service to users’ specific analytic needs. This approach enables 1010data to flexibly support today’s business questions as well as the questions that will be asked in the future. A number of technical innovations in the 1010data Insights Platform make this possible.

1010data’s analytics are powered by a purpose-built, cloud-native, massively parallel (MPP), columnar analytical database. The system is optimized for ingestion of new data and ad-hoc analysis of the biggest, most complex data. Unlike the relational databases employed in BI systems, 1010data does not require a pre-defined schema or data model for source data to be loaded. Furthermore, because the 1010data platform delivers industry-leading big data query performance, there is no need to summarize, pre-aggregate or sample data prior to loading it into the analytical database.

With 1010data, an extract, load and then transform (ELT) architecture is employed. The requirement for aggregation or “cubing” is eliminated. The overall need for transformations is greatly reduced, with transformations primarily aimed at cleaning, unifying and organizing data to serve specific analytics. Transformations within the 1010data Insights Platform are performed “just-in-time” in the same fell swoop as an end-user query.

8

In this way, all fully granular structured and semi-structured data can be loaded into the 1010data system in its raw format.

Benefits of this approach include:

Maximum flexibility for unifying data based on current and future requirements Ability to load new types of data into the analytic system immediately, without the need for an IT projectNo bounds placed on the scope of analytic questions that can be asked of that data

ENABLING IMMEDIATE ANALYSIS

As soon as data is loaded, 1010data makes it immediately available to analysts. Using self-service tools, they can explore, unify, analyze, and determine which data is of greatest value to the business. This capability exists in stark contrast to expert-focused technologies that require heavy programming or the intensive design of data models and semantic layers.

A number of innovations within the 1010data Insights Platform make this possible. First, the platform provides analysts with an intuitive spreadsheet-style interface – called the Trillion Row Spreadsheet® – for exploring, manipulating and analyzing big data.

In many BI tools or platforms, data preparation is a stand-alone component or even a separate offering that involves different interfaces, techniques and expertise than that used in subsequent analysis. With the Trillion Row Spreadsheet, data preparation is simply a particular use case of the interface. The same drag-and-drop and point-and-click conventions as well as function libraries that are used for analysis are also used for data preparation. This includes all data transformations and the functions that support joining, appending and merging datasets.

The 1010data Insights Platform offers data loading tools designed for both end-users and IT professionals:

TenUp is a single-step tool enabling IT professionals to perform operational data loads – incrementally or in bulk – from flat files and ODBC-compliant data sources

Front-end interfaces including the Custom Uploader, Quick Uploader, Table Editor, and Excel Add-in give end-users a range of options for incorporating new data into their analytics, on-the-fly

Tools for Loading Data into 1010data

9


As such, analysts, who understand both the context of the data as well as the business questions to be asked of that data, can do much of the data preparation themselves, in real-time. They can also seamlessly move between data preparation and analytic tasks without breaking train-of-thought.

Benefits of this approach include:

Immediate extraction of business benefit from new data by analystsQuick testing of the hypotheses about the value of new dataUnburdened IT and empowered business people

BUILDING HYBRID DATA STRUCTURES

Leveraging the work and underlying code generated by analysts in the previous phase, 1010data enables data unification processes to be productionalized using flexible hybrid data structures built using the most granular levels of data.

In implementing these hybrid data structures, data specialists are afforded a head start by the ad- hoc analytical work previously performed by analysts. As those analysts perform the operations that unify disparate data sets, the 1010data Trillion Row Spreadsheet generates behind-the-scenes 1010data XML Macrocode that serves as the single unifying programming language underlying every data operation.

IT can leverage this code as a building block for production-grade data unification. IT can also leverage extensive data harmonization functions and capabilities within the platform that range from basic to highly sophisticated – including fuzzy matching, string manipulation, linguistic algorithms such as metaphone, and many more.

The 1010data Trillion Row Spreadsheet provides intuitive graphical tools that let analysts explore newly loaded datasets of any size, prepare and transform disparate data sets, and then unify those data sets. Visual cues – such as color coding – allow data unification tasks that were once performed exclusively by developers to be executed by business analysts. Within the same interface, unified data can immediately be analyzed in a spreadsheet-like interface.

Data Unification and Analytics with a Single, Analyst-Friendly Interface

10

Centralized Governance for Efficient Enterprise Deployment

The core principles for creating optimal hybrid data structures on the 1010data platform include:

Unifying data at the fact level, for maximum flexibilityCollaborating with business analysts, to ensure business requirements are met in productionLeveraging the work already done by analysts during ad-hoc data exploration

SCALING ACROSS THE BUSINESS

1010data enables unified data representing a “single version of the truth” to be shared and scaled across the business using centralized management, governance and access controls.

The 1010data Insights Platform employs centralized data governance and user & group permissioning at the data-level. Each user may only access the tables and data – down to the cell level – for which they’ve been granted permissions.

The advantage of this centralized approach is that regardless of the interface or application used to access the underlying data, the exact same data-level permissions apply to the end-user. Aside from the obvious governance benefits, this approach also makes it easier to deploy a range of applications and interfaces – each tailored and personalized to the specific user type and skill level – without the additional overhead of configuring and maintaining permissions per deployment.

Governance and permissioning implemented centrally, at the data-level, enables single-step management and control applied across all access points including 1010data interfaces as well as 3rd party interfaces integrated with the platform via pre-built connectors and APIs.

Centralized Data Governance &Permissioning

Executives

Business Users & Analysts

Data Scientists

IT & Developers

Dashboards & Reports

Analytical Apps & Excel Add-in

Trillion Row Spreadsheet®

Integrations, SDKs, APIs

11


Additionally, the 1010data Insights Platform makes it easy to deploy custom hierarchies that operate atop the fully granular, fact-level data. These hierarchies enable users in different roles, departments and organizations to analyze the same underlying data using the dimensional structures and rollups that make the most sense to them – including custom calendars, product categories, customer segments, geographic zones, and much more.

UPDATING DATA STRUCTURES

Changes in business processes, the marketplace, or in the way a business measures performance will necessitate the updating of analytical processes and data structures. The flexibility of 1010data’s data unification approach keeps data in tune with changing business requirements.

With 1010data, data mappings and structures can be updated quickly and efficiently, so that the newest – and potentially most relevant – data and business processes can be analyzed immediately.

Some of the business and technological factors that can precipitate the need for the updating of existing data structures include:

Mergers and acquisitionsNew partnershipsNew business channels and processesNew KPIsEvolving market trends

Changes in corporate policiesRegulations and compliance requirementsUpgrade of source systemRip-and-replace of ERPBusiness reorganization

Historically, any one of these business changes would necessitate a new or modified data architecture – or even a wholesale reload of all historical data directly from the source systems – requiring an expensive and lengthy project to execute. But because the 1010data approach to data unification and transformation bypasses complex architecture and operates on the original raw data at the most granular levels, modifications to data, data mappings and transformations are far easier to incorporate.

Changes to the data structure in a source system due to an upgrade or system replacement can be handled with a few enhancements to the hybrid data structure, with all downstream analytic applications enabled to inherit the enhancements transparently.

12

The 1010data approach to data unification differs from traditional approaches as follows:

1010data’s approach enables the broadest range of users to ask powerful analytic questions leveraging both new and existing data. As greater volumes and varieties of valuable data are generated both inside and outside the enterprise, it is the answer to such questions that can deliver a key competitive edge.

Businesses that seek to rapidly gain deeper insights from big data – and to apply those insights to make immediate improvements in business performance – should consider employing a similar approach to data unification.

CONCLUSION

1010data Data Unification Traditional Data Integration

Data Loading • Loading of all raw granular data• Selective or sampled loading• Data altered and aggregated

prior to loading

Data Transformation• Collaboration between business

and IT• Integrated technology• Easily cross-trained resources

• IT-driven • Multiple technologies• Specialized resources

Data Analysis • Immediately upon data load • Enabled after extensive data modeling exercises

Enterprise Deployment• Centrally managed• Single version of the truth • Departmental silos and datamarts

Enhancements & Updates• Agile• Incremental• Business-driven • Lengthy, full-lifecycle IT projects

13

+1 [email protected]

Gaining actionable insight requires the best analytical tools and access to all relevant data. 1010data is a complete solution that provides both. We provide the only out-of-the-box, self-service, cross-enterprise insights platform. More than 875 of the world’s largest companies trust 1010data to manage, share and analyze over 34 trillion rows of data because of our proven ability to deliver results more quickly, easily and accurately than any other solution. Please visit www.1010data.com for more information.

TO GET STARTED, CONTACT 1010DATA TODAY.

ABOUT 1010DATA

v 06.01.17

© 2017 1010data Inc. All Rights Reserved. 1010DATA is a trademark of 1010data Inc. and is registered in the United States and other countries. All trademarks identified by ®, TM or SM, including the 1010DATA logo, are registered marks, trademarks, and service marks, respectively, of 1010data Inc. All other trademarks are the property of their respective owners.

white paper unifying data with the 1010data insights platform · unifying data with the 1010data...

Documents