Why Your Institution Needs a Data Management Strategy

Brad BronschData ArchitectEastern Washington University

Building BridgesApril 13, 2016Spokane, WA

...or How I Learned to Stop Worrying and Love Data Integration...

Introduction

Introduction

About Our Group

❖ Belong to the Business Intelligence group in IT; one manager, 3 report developers, 3 for data architecture, integration, maintenance

About Me

❖ From spreadsheets to data warehouses❖ Background in Financial, Retail, Utility & Education sectors with exposure

to a wide variety of data models & how to get data in and out❖ Role as “Data Evangelist” - a Data Architect’s job is fifty percent technology

and fifty percent communication.

Introduction

❖ Premise: An institution’s success is either hampered or realized by the the lack of or availability of accurate, timely information. Without a comprehensive, enterprise approach to data management, it’s difficult to meet this need.

❖ Discussion: A synopsis of the challenges of our educational environment will be presented, where educational services and technology are heading and how we’re dealing with it at EWU from a data management & data integration perspective.

❖ Disclaimer....

State of the Union

State Of The Union

There’s a reason they call it…

...IT...

...Information Technology…

State Of The Union

At a high level there are two types of information IT has to contend with:

❖ Operational Data - Application Data❖ Strategic Data - Business Intelligence

State Of The Union

The Student Information System (SIS)

❖ The SIS is the Enterprise Resource Planning (ERP) system for Education❖ Ellucian - Banner & Colleague❖ Peoplesoft - Campus Solutions❖ THE systems of record….or are they?

❖ How many institutions get all they need from their Student Information System?

State Of The Union

Ancillary Systems at EWU

❖ Learning Management System (LMS) - Canvas❖ Customer Relationship Management (CRM) - Hobson’s Radius❖ Degree Audit & Academic Planning - u.achieve/u.direct❖ Content Management System (CMS) - Word Press, Ingenuix, SharePoint(?)❖ Student Housing - StarRez❖ Facilities Management - AiM❖ And the list goes on…❖ Each may be the system of record of information...

State Of The Union

External & Internal Data Feeds

❖ Federal & State Reporting❖ Other third parties❖ Transmission via internal file share, Secure File Transfer Protocol (sFTP)

or manual upload via website.

Business Intelligence

❖ Reporting Platforms - Jaspersoft, Oracle Discoverer, SQL Server Reporting Services.

❖ Primary Data Source - Banner Operational Data Store (ODS).

State Of The Union

Support Challenges

❖ We have about 10 tech analysts❖ We’re often one deep with a single tech analyst supporting a

department and one or more ancillary system.❖ Each of these tech analysts wear several hats; business analyst,

application support, application administration…. and system (data) integration.

❖ Question for HGTV addicts...would your framer, plumber and electrician typically all be the same person?

❖ We have the DIY model versus a general contractor model.

State Of The Union

You’re a data architect, why aren’t you off building a data warehouse, instead of pestering me about data integration

and my system…?

State Of The Union

Data Warehouse Challenges

❖ Access to data in those ancillary systems (data islands).❖ Understanding of how data is structured in those systems.❖ Lack of standardization in data access (data integration).❖ Each system may or may not have some sort of API (application

programming interface) for data integration provided by the vendor.❖ Direct database access? The database implementation - Oracle, SQL

Server, MySQL?❖ Web services?

State Of The Union

Case in Point

Canvas

❖ Requires mission critical data feed from Banner, our SIS.❖ Data feed produced and transmitted using Oracle PL/SQL, a Linux Bash

shell script to execute the SQL script & then make a command line CURL call to a Canvas web service to push the the files to Canvas.

❖ Author of the process is no longer with us which makes supporting it challenging.

Just one example...many others.

Data Integration

State Of The Union

Enterprise data integration is the backbone of a good data management strategy. A data integration platform...

❖ Provides standardization - a single approach to across the organization and all systems instead of multiple services and languages cobbled together.

❖ Is maintainable, extensible, scalable and most importantly - supportable.❖ Can be monitored for success, failure and provides job statistics.❖ Provides built in notifications for communication of success or failure.

Data Integration

Choosing a Platform

❖ Consider a platform that is database independent and avoid vendor specific platforms. Microsoft, Oracle, IBM all have their own data integration tools, but vary in how well they integrate with others.

❖ Avoid platforms that are specific to a single system or business sector. In other words, don’t pick a platform that is specific to Education.

Data Integration

Why we chose Talend

❖ High functionality - integrates well with any data source or target; any flavor of database, any file type, web service, FTP, LDAP (Active Directory), etc.

❖ Cost - highest amount of functionality for the dollar.❖ Open Source Based - in addition to excellent vendor documentation, there

is a wealth of information available in user forums, developer websites.

Data Integration

Case Study - Retention & Student Success

❖ Potentially requires integration of information from multiple platforms.❖ Student Data - Banner (a given).❖ Housing Data - StarRez (how does a student’s living situation affect

success?)❖ Admissions Data - Hobson’s Radius (was there something about the

admissions, enrollment, registration process that adversely affected the student’s experience?)

Data Integration

How About Weather Data?

❖ Does the average daily temperature affect student success?❖ Probably not, but it allows me to demonstrate data integration without

violating FERPA or HIPAA restrictions…❖ Also, demonstrates the trend of systems towards de-centralization

(challenging) and the good news (cloud-based).

Data Integration

Weather Data from NOAA

❖ National Oceanic & Atmospheric Association❖ Cloud-based, API is a REST-based web service & very well documented❖ The existence of an api & documentation are two things you should

seriously consider when choosing 3rd party applications. Often vendors don’t consider you might actually want to get your data out of their systems, and if they do consider it, they like to charge you for it.

❖ Using Community (Open Source) versions of Talend and other tools for this demo.

Data Integration

NOAA Data: https://www.ncdc.noaa.gov/cdo-web/webservices/v2#gettingStarted

Data Integration

NOAA Data - Method: Browser Plug-In (personal token redacted)

Data Integration

NOAA Data (Method 1): JSON unformatted data in a text editor (Notepad++)

Data Integration

NOAA Data - Method 1: JSON formatted data

Data Integration

NOAA Data - Method 2: Using Excel with Power Query

Data Integration

NOAA Data - Method 2: Using Excel with Power Query

Power Query How-To: http://blog.crossjoin.co.uk/2014/03/26/working-with-web-services-in-power-query/

Data Integration

NOAA Data - Method 3: Using the Talend Platform, Job Design & Execution

Data Integration

NOAA Data - Method 3: Using the Talend Platform, Data Mapping

Data Integration

NOAA Data - Method 3: Using the Talend Platform, Results in Database

Data Integration

NOAA Data - Method 3: Using the Talend Platform, Administration

Conclusion

Conclusion

Session Summary

❖ Regardless of the systems you buy or build, you need a strategy for efficiently moving information between those systems. Enterprise Data Integration is the core of a good Data Management Strategy.

Conclusion

Brad Bronsch

Data Architect

Business Intelligence

Email: bbronsch@ewu.edu

Phone: (509) 359-6163