bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce
TRANSCRIPT
![Page 1: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/1.jpg)
SC3 Energy WorkshopTechnical overview4 Oct 2016
![Page 2: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/2.jpg)
Platform goals
◎ Low total cost of ownership
◎ Simple to get started with Big Data
◎ Cater for widely varying use cases
◎ Embrace emerging Big Data technologies
◎ Simple integration with custom components
![Page 3: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/3.jpg)
Key actors
![Page 4: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/4.jpg)
Big Data is
◎ Volumeo Quantity of data
◎ Velocityo Speed at which data is provided
◎ Varietyo Different formats/models in which data is provided
◎ Veracityo Accuracy/truthfulness of the data
Why did we need all this?
![Page 5: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/5.jpg)
Platform architecture
![Page 6: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/6.jpg)
Platform architecture
![Page 7: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/7.jpg)
Platform architecture
![Page 8: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/8.jpg)
Semantic Big Data
ongoing research!
◎ Semantic Data Lake
o from data swamp to data lake
o query contents in the data lake
◎ SANSA stack
o Big Data analytics on semantic graph
![Page 9: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/9.jpg)
Support layer
◎ Swarm UIo Launch, install and manage pipelines
◎ Pipeline daemon & monitoro Determine order in which steps are executedo eg: Upload files before running computations
◎ Integrator UIo Present dashboards in a unified interface
![Page 10: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/10.jpg)
Platform architecture
![Page 11: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/11.jpg)
Key actors
![Page 12: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/12.jpg)
Platform installation
![Page 13: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/13.jpg)
Platform installation
◎ Manual installation guide
◎ Using Docker Machine
o On local machine (VirtualBox)
o In the cloud (AWS, DigitalOcean, Azure)
o Bare metal
◎ Screencast
![Page 14: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/14.jpg)
Platform development
![Page 15: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/15.jpg)
◎ High level pictureo docker-compose.yml describes pipeline topology
◎ Common componentso extend template image with your code
◎ New componentso build a Docker image for your componento this is your own little Virtual Machine for your component
◎ Sharingo publish topology as git repositoryo publish new components on docker hub
Platform development
![Page 16: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/16.jpg)
Platform development
![Page 17: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/17.jpg)
Deployment
![Page 18: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/18.jpg)
Swarm UI
![Page 19: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/19.jpg)
Swarm UI
![Page 20: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/20.jpg)
Deployment
![Page 21: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/21.jpg)
Swarm UI
![Page 22: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/22.jpg)
Swarm UI
![Page 23: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/23.jpg)
Integrator UI
![Page 24: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/24.jpg)
Workflow UI
![Page 25: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/25.jpg)
More monitoring
This topic is ongoing, many interesting options
◎ Visualise logs with Kibana?
◎ Combine logs for large overview?
◎ Monitor node load?
◎ Provide autoscheduling?
![Page 26: Bde sc3 2nd_workshop_2016_10_04_p04_bde_platform_tenforce](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58ef00d61a28ab8b4f8b468d/html5/thumbnails/26.jpg)
Concluding remarks
◎ Used in practice
◎ Easy to get started
◎ Improving as we speak