www.tugraz.at ◼
WISSEN
TECHNIK
LEIDENSCHAFTExample Architectures –
Internet of Things &
Kubernetes
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Agenda
• Internet of Things (IoT)
• communication
• protocols
• student lab at the ISDS
• sensors platforms
• architecture
• virtualisation and containers
• Kubernetes
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Internet of Things (IoT)
History and Background
• problem
• connect sensors and actuators to digital networks
• enable control by computer systems
• automate (industrial) processes
• general term: Supervisory Control and Data Acquisition (SCADA)
• yesterday’s solutions
• point-to-point links (e.g., RS-232)
• fieldbuses (e.g., Modbus, CAN, PROFIBUS)
• Many started as proprietary by a single company but are now standards
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Internet of Things (IoT)
Gamechangers
• IP became the dominant communication protocol
• integrated computing power in sensors and actuators became powerful
enough to do IP-based communication
• Ethernet and variants of it became the dominate networking
technologies
• Industrial Ethernet
• WiFi
• Power over Ethernet (PoE)
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Internet of Things (IoT)
The Gamechanger's Consequences
• use Ethernet (a variant) almost everywhere
• use IP-based communication almost everywhere
• with IP also TCP or UDP became the dominant
• Layer 1 to 4 in the ISO OSI model are covered by standard technologies
• every device can communicate with every other device
• remote communication via the Internet is possible
• relatively high data rates are achievable (if necessary)
• standardised and cheap network components can be used
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Don’t take only my word for it…
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
source: Eclipse Foundation IoT developer survey 2019
www.tugraz.at ◼
How to transmit data on IoT?
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
source: Eclipse Foundation IoT developer survey 2019
www.tugraz.at ◼
How to transmit data on IoT?
HTTP – HyperText Transfer Protocol
• properties
• widely available: implementations in different languages
• easy to use: simple communication model
• almost never blocked and routed everywhere
• designed for something completely different: client-server hosting of
(static) content
• usage
• very common
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
How to transmit data on IoT?
WebSocket
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
• properties
• WebSocket is a full protocol on its own and not an add-on to HTTP
• BUT: WebSocket keeps combability to HTTP wherever possible
• offers full-duplex communication over single TCP connection
• Server can send messages to clients without prior requests by clients
once a connection has been established
• standardized by Internet Engineering Task Force (IETF)
• usage
• gaining popularity as we speak
www.tugraz.at ◼
How to transmit data on IoT?
MQTT – Message Queue Telemetry Transport
• properties
• simple on clients but requires central broker
• implementation in many languages exist
• standardised by the Organization for the Advancement of Structured Information Standards (OASIS)
• name is confusing, since there is no message queue in the protocol, its a historic reminiscence to the original creation at IBM
• usage
• version 3.1.1 from 2014 commonly used
• version 5 standardised in 2019
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
How to transmit data on IoT?
MQTT Publish and subscribe model
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
broker
sensoring
equipment
publish to topic
• publish-subscribe
communication
• sender publishes a
messages to a so called
topic
• every client subscribed to
the topic gets the
messages relayed by the
broker
www.tugraz.at ◼
What have we got in the ISDS student lab?
Sensor Platform – Tinkerforge
• complete sensor platform
• OpenSource hard- and software
• API to many programming languages
• offers embedded Linux system (RED Nod) and Raspberry PI integration
• MQTT enabled communication over Ethernet
• https://www.tinkerforge.com/
• Rolled out in six rooms collecting room climate data
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
What have we got in the ISDS student lab?
Sensor Platform – WaspMote & Meshlium
• WaspMote nodes and Meshlium base station
• nodes are based on Arduino, e.g., no operating
system
• Meshlium runs Linux and forwards data to other
services, e.g, via MQTT
• http://www.libelium.com/
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
What have we got in the ISDS student lab?
Sensor Platform – Power Meter
• Proprietary solution
• Ethernet interface
• Measurements are parsed from the web
interface
• http://energenie.com/
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Serverserver01 (10.0.100.248)
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
tinkerforge-01 – 08
(10.0.100.220-227)egm-pwm-lan01 – 13
(10.0.100.200-212)
http://openweathermap.org
WiFi + BT Scanmeshlium01
(10.0.100.245)
MQTT Brokerserver01:1883
server01:5432
SQL to RESTTap
http://server01:8080
光Grafana
http://server01:8085
https://server01:8443
Performance
ZAMG TAWEShttp://www.zamg.ac.at/ogd/
Student Lab Software Architecture• Devices
• Hardware
• Software
• Docker
• Communication
• MQTT
• HTTP
• SQL
• Bash
www.tugraz.at ◼
Student Lab Software Architecture
Key Take-Away Messages
• there is no one-fits all solution
• network and communication protocols
• devices and operating systems
• requirement might differ on parts of the architecture
• real time constraints
• reliability
• IoT, microservices and containers go well with each other
• independent modules in hardware or hard- and software
• connected by defined network interfaces
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Virtualisation
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
host OS + bins / libs
hardware
applic
ation 1
applic
ation 3
applic
ation 2
host OS
hardware
hypervisor
gu
est
OS
+
bin
s / lib
s
guest
OS
+
bin
s / lib
s
guest
OS
+
bin
s / lib
s
applic
ation 1
applic
ation 3
applic
ation 2
host OS
hardware
container hypervisor
bin
s / lib
sapplic
ation 1
applic
ation 3
ap
plic
ation
2
host OS
hardware
container hypervisor
ap
plic
atio
n 1
+
bin
s / lib
s
applic
ation 3
+
bin
s / lib
s
applic
ation 2
+
bin
s / lib
s
bin
s / lib
s
bin
s / lib
s
no virtualisation virtual machines container
virtualisation
unikernel
www.tugraz.at ◼
Virtualisation
When There is no Virtualisation
• no virtualisation overhead
• separation of applications is done by the OS
• allocation of resources are based on users and groups
• resources are files, CPU time, memory, network bandwidth
• binaries and libraries are shared via the OS
• package management
• combine common binaries and libraries in packages
• problems occurs with version mismatch:
• application 1 needs version x of package A
• application 2 needs version y of package A
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Virtualisation
Virtual Machines
• hypervisor re-creates the behaviour of hardware, i.e., hardware level virtualisation
• each virtual machine is completely separated from each other
• resources are allocated by the hypervisor to the virtual machines
• inside the virtual machines a full operating system is running
• the OS in the virtual machine runs the application with the required binaries and libraries
• most virtualisation overhead of all solutions
• Hypervisors
• VirtualBox, VM Ware Fusion / Player / Workstation / Server, Proxmox VE, KVM, Microsoft Hyper-V, Parallels Workstation, Xen, …
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Virtualisation
Container Virtualisation
• hypervisor re-creates the behaviour of OS, i.e., OS virtualisation
• CPU & memory management
• filesystem
• network
• container engine might offer additional separation over normal OS
• user separations
• devices
• inside the containers on a minimal OS with kernel, required binaries and libraries to run the application is available
• considerable less virtualisation overhead than full virtualisation
• most common product: Docker
• alternatives: Linux Containers (LXC), Solaris Containers, FreeBSD jail, …
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Virtualisation
Unikernels Virtualisation
• objective: achieve same separation like containers but decrease the
virtualisation overhead
• solution: include library operating systems directly into the application so
that it can run on a hypervisor directly
• lowest possible virtualisation overhead
• separation of running containers still done by the hypervisor
• examples: MirageOS, ClickOS, UniK, Rumprun
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Docker Containers
How to Run and Manage Containers
• directly, i.e. Docker CLI: too cumbersome if you’ve got hundreds running
• Docker Compose: can only manage singe hosts set-ups
• Docker Swarm: hardly used in the industry
• Kubernetes:
• currently the tool of choice in the industry
• handles deployment, scaling, and management
• designed to handle multi-host setups, i.e., clusters
• https://kubernetes.io/
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Kubernetes Architecture
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Kubernetes Terminology
• nodes: hosts or hardware
• pod: one or more containers running on the same node
• pods get IP address and volumes assigned
• service: set of pots creating an application
• ConfigMaps and Secrets:
• configuration input to the instances running of the containers
• can be anything from individual properties up to JSON / XML files
• available as environment variables or on the container’s file system
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Kubernetes Architecture
Master
• master / slave architecture
• distributed architecture
• ready for automated commission, running and decommission of services
• master controllers the cluster
• master offers API to control the complete setup either by other services (e.g., built automatisation) or by users / developers
• master runs the controller manager service
• more or less the central core or kernel of a Kubernetes Architecture
• scheduler: knows witch nodes are available and what in running where
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Kubernetes Architecture
Nodes
• nodes are the slaves in the architecture
• Kube-Proxy handling the network Communication with other nodes
• Kubelet does what ever the master tells it on the node
• cAdvisor: monitoring
• Pods: run the actual software that proving the service to the users
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
Virtualisation and Kubernetes
Key Take-Away Messages
• virtualisation in any of its forms is now common practice in running software systems
• containers offer a good compromise between overhead and separation
• Kubernetes is currently to tool of choice to manage large distributed container-based software systems
• Kubernetes is designed to manage large and distributed setups
• reason for its inherent complexity
• only pays of in distributed setups
• Kubernetes are provided by may cloud provides like Amazon AWS or Microsoft Azure
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS
www.tugraz.at ◼
WISSEN
TECHNIK
LEIDENSCHAFTExample Architectures –
Internet of Things &
Kubernetes
Software Architecture VO/KU (706.706/706.707) – 2019-10-16
Heimo Gursch & Roman Kern – ISDS