building a data warehouse with aws redshift, matillion and yellowfin

Building a Data Warehouse on AWS

Amazon S3

Amazon Redshift

CollectCollect ProcessProcess AnalyzeAnalyzeStoreStore

Data Answers

Visualize

@Lynn Langit

AWS MarketplaceEnterprise software store for business users who need simplified procurement

•2000+ product listings •to browse, test and buy software

•1-click deployment •to launch, in multiple regions around the world

•Pay-as-you-go pricing •to use on demand

Advanced Analytics

Data Enablement

Business Intelligence

Building a Data Warehouse on AWS

Move data into Redshift from S3 for analysis

Amazon S3

Amazon Redshift

AWS Marketplace Partners

Matillion

Visualize

Yellowfin

CollectCollect ProcessProcess AnalyzeAnalyzeStoreStore

Data Answers

Setup

Our Scenario and Source FilesFile Types

-- Text - .csv-- Compressed - .gz

File CategoriesDetails / Events -- Flights

-- WeatherMetadata

-- Airports -- Carriers

“In this scenario we will use Matillion ETL for Redshift to prepare two separate data sources ready for analysis. The sample data is US airport flight information from 1995 -> 2008. Every flight to or from a US airport (and whether it left on time or not) is included.

The second data set is weather data, taken from NOAA, including the daily weather readings for each US Airport.”

Loading data from S3 in to Redshift

Using Matillion ETL for Redshift• Create Instance (AMI/EC2) of Matillion/AWS Marketplace• Connect Matillion to Redshift

Loading Data in Redshift

Table distribution stylesDistribution Key All

Node 1

Slice 1

Slice 2

Node 2

Slice 3

Slice 4

Node 1

Slice 1

Slice 2

Node 2

Slice 3

Slice 4

key1

key2

key3

key4

All data on every nodeSame key to same location

Node 1

Slice 1

Slice 2

Node 2

Slice 3

Slice 4

EvenRound robin distribution

Sort Keys• Single Column - [ SORTKEY ( date ) ]

• Queries that use 1st column (i.e. date) as primary filter

• Compound - [ SORTKEY COMPOUND ( date, region, country) ]

• Queries that use 1st column as primary filter, then other columns

• Interleaved - [ SORTKEY INTERLEAVED ( date, region, country) ] • Queries that use different columns in filter

Time Series Data – Vacuum Operation

Unsorte

dRegion

Sorte

dRegio

n Sorte

d

Sorte

d

Sorte

d

Append in Sort Key Order

Sort Unsorted Region

Merge

Visualizing with Yellowfin

Automate – https://github.com/lynnlangit/AWSDataWarehouse