Blue Whale in an Enterprise Pond– Docker experiences from the real world (in Finland)

Tero Niemistö

Who am I?

• Tero Niemistö

• Group Manager @Digia

• 18 years in the industry, past 6 years in Digia

• Cloud/Devops/Docker enthusiast

• Father of 3 (1 hairy, 2 hairless)https://www.linkedin.com/in/teroniemistohttps://twitter.com/tero_niemisto

DOCKER HUB

Your Typical Docker Architecture

DOCKER HUB

…with Finnish ICT Requirements…

VAHTI

KATAKRIFinnish Communications

Regulatory Authority

- Order 54

DOCKER HUB

…results in this!

VAHTITreasury

Order 54Finnish

Communications Regulatory Authority

KATAKRIMinistry of Foreign

Affairs

Obstacles in building a Docker solution

• Authorative requirements to all ICT competitive biddings in Finnish public sector

• Auditing tool for Vahti• Often a mandatory

requirement

• Order gives set of requirements aimed to secure Finnish communications network

Reform of EU data protection rules

European Union

• Privacy by design• Privacy by default

How do we solve server (or even Docker) compliance to

CIS Benchmarks?

Solutions often have SLA demands (ie. 99.9%). How can this be quaranteed?

Servers need to reside in 2 different data centers (or in same data center but in 2 rooms with different fire compartments)

Servers often need to reside in Katakri-audited data centers

People who operate servers need to have been cleared by SUPO

Data needs to reside in EU or in some cases only in Finland

Open source licenses. Do we have any GPL components?

Are we using any blacklisted open sourcce components?

Cyber Security responsibility in the end, supplier is responsible for all issues related to cyber security

(Just) Some issues for consideration

Checklist with Docker in and Enterprise Pond

Build our own secure containers1

Maintain own environment for CI pipeline

Double-check security on that the CI pipeline

Automate Docker server compliance

Duplicate entire system into 2 different server rooms

2

3

4

5

6

Automate container vulnerability scan on every level of the CI pipeline

Simple Container Creation Process

Developer Dockerfile Git Jenkins Sonatype Nexus

Crea

ting

cont

aine

rs

Commit Dockerfile to Git Repository

Jenkins detects changes from Git

Application container is built according to dockerfile

Container file is uploaded to Sonatype Nexus

Container stored and served from a private

Docker Registery

Application container is inspected by Blackduck

plugin

Define container contents with Dockerfile config

Slightly More Advanced Creation Process

Developer Docker Compose Git Jenkins Sonatype Nexus

Crea

ting

cont

aine

rs

Commit Docker Compose file to Git Repository

Jenkins detects changes from Git

Application container is built according to docker

compose file

Container file is uploaded to Sonatype Nexus

Container stored and served from a private

Docker Registery

Application container is inspected by Blackduck

plugin

Define service connections of the containers

Dependencies are retrieved from private registry

Application Build Process

Developer YAML Git Jenkins Sonatype Nexus

Crea

ting

appl

icati

on c

onta

iner

s

Commit yaml-file and application code to Git

repository

Jenkins detects changes from Git

Inspect code quality with Sonarqube + Blackduck

Application container is built according to yaml

configuration file

Container file is uploaded to Sonatype Nexus

Container stored and served from a private

Docker Registery

Application container is inspected by Blackduck

plugin

Create application yaml-file with dependencies

Container stored and served from a private

Docker Registery

Dependencies are retrieved from private registry

Docker containers

Infra level containers (middleware )

Application level containers

Base level containers

Application 1

Alpine Linux

Java 8 Ruby

Tomcat

Ubuntu

Python

Application 2 Application 3

Served from private repository

JBoss

MySQL

RabbitMQ

Jenkins

Deployment Process

Operations Jenkins Kontena Sonatype Nexus Docker Server

Depl

oyin

g ap

plic

ation

con

tain

ers

Kontena Master starts deployment process

Kontena retrieves containers from private

registry

Operations starts deployment process with

Jenkins

Container is deployed according to strategy and

load balancers are updated

Kontena deploys container to target runtime

environment

Jenkins connects to Kontena Master

Continuous Compliance To CIS Benchmarks

Our Typical Docker Architecture

Our Typical Docker Architecture

Real Life Issues: Docker Push through proxy

• The problem is that docker sends PATCH request with over HTTP but not HTTPS while pushing image.

• If nginx (or any) proxy is tuned to redirect any HTTP requests to HTTPS, then docker receives “Method not Allowed” response and push fails

• Hint: Configure your proxy to add request header X-Forwarded-Proto

DOCKER PUSH (HTTP) HTTPS

”METHOD NOT ALLOWED”

Real Life Issues: No access to frontend proxy

• Often access to customers HTTP internet proxy is very limited or it takes 3 days to change it.

• Hint: Even if your system has loadbalancing by service provider, use your own. Makes everything so much easier and you can actually have blue-green setup

HLB”TAKES 3 DAYS…”

HAProxy”TAKES 3 SECONDS …”

Real Life Issues: Attacking with SSH container

• We lost access to Docker server due to raising filehandlers too high which crashed ssh process and we couldn’t ssh in anymore

• We hijacked the server by deploying SSH container with mount into server’s filesystem.

• We then used SED to fix the issue

• Hint: Your CI server or local docker orchestration client becomes a new attack vector. Secure it!

DevOps & CyberSecurity Meetup 14.9. @Digia

+ We are hiring!

Kiitos!#digiarki

www.digia.com