hpc-sig ecosystem validation renato golin baptiste ... · vendors to rely on linaro for base oss...
TRANSCRIPT
![Page 1: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/1.jpg)
HPC-SIG Ecosystem ValidationJan. 14 2019
Baptiste GerondeauRenato Golin
![Page 2: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/2.jpg)
For more info visitlinaro.org/hpc
HPC-SIG Lab and Validation Matrix
Aggregate machines in the same infrastructure, and validate their performance using a Validation Matrix
● Validation Matrix must be applicable to every machine● Validation Matrix dimensions are software configurations
To generate as few tests as possible, we need to simplify the matrix without losing information
![Page 3: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/3.jpg)
For more info visitlinaro.org/hpc
HPC-SIG Lab’s Infrastructure
The infrastructure needs to :● Dispatch jobs (tests, provisioning, benchmarks)● Provide DHCP/TFTP services● Provide Package Cache services● Provide a secure file/results storage service● Be Low Maintenance● Be able to be replicated anywhere else
![Page 4: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/4.jpg)
For more info visitlinaro.org/hpc
Simplifying InfrastructureIdentifying the different dimensions
A Vertical Slice of the Stack
Principal dimensions :➔ Application➔ HPC environment stack➔ Machine provisioning
● HPC Stack : OpenHPC
● Validation Application : OpenHPC’s testsuite
![Page 5: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/5.jpg)
For more info visitlinaro.org/hpc
Simplifying InfrastructureIdentifying the different dimensions
The Stack from the Lab’s point of view
Machine provisioning :
➔ Network configuration➔ Kernel➔ OS➔ HPC Stack
● Multiple ways to do the provisioning
![Page 6: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/6.jpg)
For more info visitlinaro.org/hpc
Simplifying InfrastructureIdentifying the different dimensions
Provisioning Method Variations
Multiple ways to provision :
➔ Warewulf Stateless (VNFS)➔ Warewulf Stateful (OS image)➔ Ansible
![Page 7: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/7.jpg)
For more info visitlinaro.org/hpc
Simplifying InfrastructureIdentifying the different dimensions
Different Network Layouts
● Flat : Machines reachable from anywhere
● Tree: Machines reachable from cluster head node only
● Root : Master with DHCP/TFTP server
![Page 8: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/8.jpg)
For more info visitlinaro.org/hpc
Simplifying InfrastructureIdentifying the different dimensions
Different Kernels
● Upstream from OS
● ERP : Entreprise Reference Platform
● Contains support for platforms in the process of being upstreamed
![Page 9: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/9.jpg)
For more info visitlinaro.org/hpc
Simplifying InfrastructureIdentifying the different dimensions
Different Operating Systems
● 3 OSes available to the user
● No Debian support in OpenHPC
![Page 10: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/10.jpg)
For more info visitlinaro.org/hpc
Simplifying InfrastructureAbstractions, and the user’s environment
Abstracting Network Variations
● Invisible to the user● Handled by the lab installer● Dependent on hardware
![Page 11: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/11.jpg)
For more info visitlinaro.org/hpc
Simplifying InfrastructureAbstractions, and the user’s environment
Abstracting Provisioning Variations
● Multi-staged provisioning● Coexistence● Dependent on hardware
![Page 12: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/12.jpg)
For more info visitlinaro.org/hpc
Simplifying InfrastructureAbstractions, and the user’s environment
Abstracting Environment Variations
● Control over HPC Stack● Common OS configuration● Idempotency● Package Caches
![Page 13: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/13.jpg)
For more info visitlinaro.org/hpc
Simplifying InfrastructureAbstractions, and the user’s environment
Accounting for extra HPC services
● Infiniband Support● Lustre server support● Future additional features
(additional hardware)
![Page 14: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/14.jpg)
For more info visitlinaro.org/hpc
Simplifying InfrastructureWhat the User sees, configures
The Lab’s Interface
➔ Choose Application
❖ Lab picks default configuration❖ User fine tunes configuration
![Page 15: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/15.jpg)
For more info visitlinaro.org/hpc
Validation matrixCluster Deployment
![Page 16: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/16.jpg)
For more info visitlinaro.org/hpc
Validation matrixDistributed Applications Enablement
![Page 17: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/17.jpg)
For more info visitlinaro.org/hpc
Validation matrixToolchain Benchmarking
![Page 18: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/18.jpg)
For more info visitlinaro.org/hpc
Validation matrixLibrary Enablement and Enhancement
![Page 19: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/19.jpg)
For more info visitlinaro.org/hpc
Future● Vendors to rely on Linaro for base OSS validation
○ We have multiple vendors available○ On a standardised infrastructure
![Page 20: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/20.jpg)
For more info visitlinaro.org/hpc
Future● Vendors to rely on Linaro for base OSS validation
○ We have multiple vendors available○ On a standardised infrastructure
● Share our work○ OpenHPC Ansible recipes (with the OpenHPC community)○ SDI (MrP, Jenkins, Ansible) helping members to replicate our work○ Community CI (OpenHPC test-suite, MPI MTT, OpenMP tests, OpenBLAS CI)
![Page 21: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/21.jpg)
For more info visitlinaro.org/hpc
Future● Vendors to rely on Linaro for base OSS validation
○ We have multiple vendors available○ On a standardised infrastructure
● Share our work○ OpenHPC Ansible recipes (with the OpenHPC community)○ SDI (MrP, Jenkins, Ansible) helping members to replicate our work○ Community CI (OpenHPC test-suite, MPI MTT, OpenMP tests, OpenBLAS CI)
● Allow our engineers to develop the ecosystem○ Internal tests and benchmarks (via Jenkins, no infrastructure knowledge needed)○ Testing new packages, libraries, compilers (comparison jobs, CI results, statistic analysis)
![Page 22: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/22.jpg)
For more info visitlinaro.org/hpc
Future● Vendors to rely on Linaro for base OSS validation
○ We have multiple vendors available○ On a standardised infrastructure
● Share our work○ OpenHPC Ansible recipes (with the OpenHPC community)○ SDI (MrP, Jenkins, Ansible) helping members to replicate our work○ Community CI (OpenHPC test-suite, MPI MTT, OpenMP tests, OpenBLAS CI)
● Allow our engineers to develop the ecosystem○ Internal tests and benchmarks (via Jenkins, no infrastructure knowledge needed)○ Testing new packages, libraries, compilers (comparison jobs, CI results, statistic analysis)
HPC Lab Setuphttps://github.com/Linaro/hpc_lab_setup
Ansible OpenHPC installation recipe : https://github.com/Linaro/ansible-playbook-for-ohpc
![Page 23: HPC-SIG Ecosystem Validation Renato Golin Baptiste ... · Vendors to rely on Linaro for base OSS validation We have multiple vendors available On a standardised infrastructure Share](https://reader033.vdocuments.mx/reader033/viewer/2022050504/5f96191e7b773b196d71cdbc/html5/thumbnails/23.jpg)
Thanks!