index [events.static.linuxfound.org] · awesome big data projects and tools at stratio. skills...

Index

1 Presentation

JORGE LÓPEZ-MALLA@jorgelopezmalla

After working with traditionalprocessing methods, I started todo some R&S Big Data projects

and I fell in love with the Big Dataworld. Currently I’m doing some

awesome Big Data projects and tools at Stratio.

SKILLS

Presentation

MARCOS PEÑATE @marcosmi5

Coming from the Dark Side I found a place to grow and innovate at Stratio.

I am passionate about astrophysics, a compulsive SciFi consumer and I really

enjoy automating and Dockerizing everything around me!

SKILLS

Presentation

Our Company

Our product

2 Use Case

DC/OS

➢

➢

➢

DC/OS

➢

➢

➢

➢

➢

DC/OS Use Case: preconditions

3 Spark, Kerberos & Mesos

➢

➢

➢

Spark <3 Mesos

Mesos Master

Agent-2

Executor-1 Executor-2

Task-0 Task-1Task-2 Task-3

HDFS

Agent-3

Executor-0 Executor-1

Agent-1

Executor-0

Main {…..….sc = new SparkContext()rdd1 = sc.textFile.map()rdd1.saveAs….}

Main {…..….sc = new SparkContext()rdd1 = sc.textFile.map()rdd1.saveAs….} Spark dispatcher

Driver

Driver

Mesos - Cluster

Adding Kerberos to the equation

➢

➢

➢

➢

○

○

○

➢

➢

➢

➢

Adding Kerberos to the equation

tgstgt

read.format(parquet).path(“/user1/b”).option(“user”, “user2”)read.format(parquet).path(“b”).option(“user”, “user2”)

toDF.saveAsParquet(“b”)textFile(“a”)saveAsTextFile(“a”)

Driver Executor

sparkUser

user1

user2

HDFS

tgttgs

Stratio’s Solution

4 Dynamic authentication

➢

➢

➢

KMS

Dynamic authentication

➢

➢

DCS cluster

Spark - KMS

Spark - KMS

DCS cluster

Spark - KMS Dynamic Authentication

★

★

★

Spark - KMS Dynamic Authentication

5 Mutual TLS

Long handshakes are always bad!!

Mutual TLS Datastores

Mutual TLS Datastores

Mutual TLS Datastores

Mutual TLS Datastores

Mutual TLS Datastores

Spark - Mutual TLS Datastores

6 Postgres

Postgres

➢

➢

➢

➢

➢

➢

Postgres

7 Network isolation

Are you sure that your cluster is secure?

DCS cluster

➢

➢

➢

SDN

DCS cluster

kafkakafka

Are you sure that your cluster is secure?

7 Live Demo

8 Q&A