index [events.static.linuxfound.org] · awesome big data projects and tools at stratio. skills...
TRANSCRIPT
Index
1 Presentation
JORGE LÓPEZ-MALLA@jorgelopezmalla
After working with traditionalprocessing methods, I started todo some R&S Big Data projects
and I fell in love with the Big Dataworld. Currently I’m doing some
awesome Big Data projects and tools at Stratio.
SKILLS
Presentation
MARCOS PEÑATE @marcosmi5
Coming from the Dark Side I found a place to grow and innovate at Stratio.
I am passionate about astrophysics, a compulsive SciFi consumer and I really
enjoy automating and Dockerizing everything around me!
SKILLS
Presentation
Our Company
Our product
2 Use Case
DC/OS
➢
➢
➢
DC/OS
➢
➢
➢
➢
➢
DC/OS Use Case: preconditions
3 Spark, Kerberos & Mesos
➢
➢
➢
Spark <3 Mesos
Mesos Master
Agent-2
Executor-1 Executor-2
Task-0 Task-1Task-2 Task-3
HDFS
Agent-3
Executor-0 Executor-1
Agent-1
Executor-0
Main {…..….sc = new SparkContext()rdd1 = sc.textFile.map()rdd1.saveAs….}
Main {…..….sc = new SparkContext()rdd1 = sc.textFile.map()rdd1.saveAs….} Spark dispatcher
Driver
Driver
Mesos - Cluster
Adding Kerberos to the equation
➢
➢
➢
➢
○
○
○
tgstgt
read.format(parquet).path(“/user1/b”).option(“user”, “user2”)read.format(parquet).path(“b”).option(“user”, “user2”)
toDF.saveAsParquet(“b”)textFile(“a”)saveAsTextFile(“a”)
Driver Executor
sparkUser
user1
user2
HDFS
tgttgs
Stratio’s Solution
4 Dynamic authentication
➢
➢
➢
KMS
Dynamic authentication
➢
➢
DCS cluster
Spark - KMS
Spark - KMS
DCS cluster
Spark - KMS Dynamic Authentication
★
★
★
Spark - KMS Dynamic Authentication
5 Mutual TLS
Long handshakes are always bad!!
Mutual TLS Datastores
Mutual TLS Datastores
Mutual TLS Datastores
Mutual TLS Datastores
Mutual TLS Datastores
Spark - Mutual TLS Datastores
6 Postgres
Postgres
➢
➢
➢
➢
➢
➢
Postgres
7 Network isolation
Are you sure that your cluster is secure?
DCS cluster
➢
➢
➢
SDN
DCS cluster
kafkakafka
Are you sure that your cluster is secure?
7 Live Demo
8 Q&A