Building, Deploying and Testing

an Industrial Linux Platform

SZ Lin (林上智)

MXcore, Software Supervisor

06/01 2017

Open Source Summit Japan 2017

Confidential

About Me

SZ LIN (林上智)

Software Engineer at Moxa

• Industrial Grade Linux Distribution

Debian Developer

Blog - https://szlin.me

2

Confidential

Industrial Linux Platform

3

Confidential

Industrial/ Harsh Environments Including smart rail, smart grid, intelligent transportation,

factory automation, oil & gas, marine, and more

Confidential 4

Confidential

Smart Grid

Smart Rail

Smart Oil Field

Smart Factory

Smart Transportation

Smart Marine

Application

Device

Industrial Routers

Industrial Ethernet

Industrial Wireless

LAN

Network Infrastructure

Confidential

Embedded Computers

Industrial Computing

Industrial Linux Platform

Serial Connectivity

I/O Connectivity

Video Connectivity

Edge Connectivity

5

Confidential

Smart Grid

Smart Rail

Smart Oil Field

Smart Factory

Smart Transportation

Smart Marine

Application

Device

Industrial Routers

Industrial Ethernet

Industrial Wireless

LAN

Network Infrastructure

Confidential

Embedded Computers

Industrial Computing

Industrial Linux Platform

Serial Connectivity

I/O Connectivity

Video Connectivity

Edge Connectivity

6

Hardware

Software

Confidential

Bug fixes

Security

fixes

New userspace

program backports

New kernel

features backports

Industrial Linux Distribution

Over 10+ years

Maintenance release

7

Confidential

Industrial Linux

Distribution

CI

CD LT

8

Ecosystems for Industrial Linux Distribution

CI : Continuous Integration

CD : Continuous Delivery/ Deployment

LT : Long-term Test

Confidential 9

Continuous Integration

and

Continuous Delivery/ Deployment

Confidential

Industrial Linux

Distribution

CI

CD LT

10

Ecosystems for Industrial Linux Distribution

CI : Continuous Integration

CD : Continuous Delivery/ Deployment

LT : Long-term Test

Confidential

Master

Develop

Feature ..

Merge

Pull Request

Fork

Local

Branch

Patches

Notification

Internal/ External

Developers

CI/ CD Automatic

Release Pipeline

Maintainer

Approval

Pass

Pass

Y

Y N

N

11

Confidential

CI/ CD Automatic Release Pipeline

Building Testing

Deploying

4 3 2 1

Release

12

Confidential

Code

Continuous

Integration

CI/ CD Automatic Release Pipeline

Building Testing

Deploying

4 3 2 1

Release

13

Confidential

Server

Slave

node

Webhooks

Patches

Internal/ External

Developers

Local

Branch

Pull Request

Slave

node Slave

node

Static Program

Analysis Coding style

OWASP [2]

Infer [3]

Sonarqube [4]

…

14

Confidential

Test Cases Management - Jenkins

15

Static analysis

#1

Static analysis

#2 …

Static analysis

#n

Confidential

Image: https://c1.staticflickr.com/5/4030/4438139050_04604b4908.jpg

16

Confidential

Distributed Compiler

• Software

– Icecream/ IceCC was created by SUSE based on distcc

[5][6]

• Improve performance of compile jobs in parallel

• Add dynamic scheduler of the compilation jobs

• Support multiple platform

• Support cross compiling

• Hardware - for each node

– SSD

– Large capacity memory

– Gigabit LAN

17

Confidential

Client Client Client

ICECC - Distributed Compiler Server

Patches

Internal/ External Developers

Local

Branch

Pull Request

3. Send

log & image

18

Webhooks

1. Send command

2. Get source

Confidential

CI/ CD Automatic Release Pipeline

Building Testing

Deploying

4 3 2 1

Release

Code

Continuous

Integration

Continuous Delivery

19

Confidential

Continuous Delivery – LAVA [7][8]

20

21

Server

Master

Worker Worker Worker

1. Send job file via XML-RPC

5. Deployment via TFTP

2. Dispatch job via ZMQ

3. Download image

via curl or wget

4. Boot up via Ethernet remote I/O

DUT Clusters

Dynamic Program

Analysis ● gcov [9]

● valgrind [10]

● profiling tools [11]

● ...

Platform Test ● LTP [12]

● Security testing

[13]

● Kselftest [17]

● ...

6. Send test command

Confidential

CI/ CD Automatic Release Pipeline

Building Testing

Deploying

4 3 2 1

Release

Continuous Delivery Continuous

Deployment Code

Continuous

Integration

22

Confidential

Master

Develop

Merge

Notification

CI/ CD Automatic

Release Pipeline

Maintainer

Approval

Pass

Pass

Y

Y N

N

23

Image

Deployment

24/ 7 Long-term

Platform Test

Confidential

Long-term Test

24

Confidential

Industrial Linux

Distribution

CI

CD LT

25

Ecosystems for Industrial Linux Distribution

CI : Continuous Integration

CD : Continuous Delivery/ Deployment

LT : Long-term Test

Confidential

Notification

Test Results

26

* Test cases are managed by LAVA

Confidential

24/ 7 Long-term Platform Test

Robustness

Robustness is the ability

of a computer system to

cope with errors during

execution and cope with

erroneous input [18]

Reliability

Reliability is enhanced

by features that help to

avoid, detect and repair

hardware faults [1]

Security

Quick response in

resolving CVE/

vulnerabilities and

attacks in platform

Longevity

Long-term support at

least 10 years life cycle

with bug fixes, new

features and new

hardware components

27

Endurance test

Compatibility test

…

Confidential

24/ 7 Long-term Platform Test

Robustness

Robustness is the ability

of a computer system to

cope with errors during

execution and cope with

erroneous input [18]

Reliability

Reliability is enhanced

by features that help to

avoid, detect and repair

hardware faults [1]

Security

Quick response in

resolving CVE/

vulnerabilities and

attacks in platform

Longevity

Long-term support at

least 10 years life cycle

with bug fixes, new

features and new

hardware components

28

Fuzz testing

[14][15][16]

…

Confidential

24/ 7 Long-term Platform Test

Reliability

Reliability is enhanced

by features that help to

avoid, detect and repair

hardware faults [1]

Security

Quick response in

resolving CVE/

vulnerabilities and

attacks in platform

29

Power failure test

Reboot test

…

Confidential

24/ 7 Long-term Platform Test

Security

Quick response in

resolving CVE/

vulnerabilities and

attacks in platform

30

Daily test for CVE

…

Confidential

CI/ CD/ LT are

concepts of software engineering

instead of

tools or procedures

31

Confidential 32

Collaboration with

upstream developer Bootloader test integration

Performance/ Unit test

integration Test cases framework

enhancement Keep eyes on Fuego

Future Work

Confidential 33

© Moxa Inc. All rights reserved. 34

Thank You

Confidential

References

[1]https://en.wikipedia.org/wiki/Reliability,_availability_and_servic

eability

[2] https://wiki.jenkins-ci.org/display/JENKINS/Plugins

[3] http://fbinfer.com/

[4] https://www.sonarqube.org/

[5] https://github.com/icecc

[6] https://www.slideshare.net/szlin/distributed-compiler-icecc

[7] https://www.linaro.org/initiatives/lava/

[8] http://elinux.org/images/3/35/LAVA_Project_Update.pdf

[9] https://gcc.gnu.org/onlinedocs/gcc/Gcov.html

[10] http://valgrind.org/

[11] https://perf.wiki.kernel.org/index.php/Main_Page

35

Confidential

References

[12] http://linux-test-project.github.io/

[13] http://www.openvas.org/

[14] https://github.com/google/syzkaller

[15] http://codemonkey.org.uk/projects/trinity/

[16] https://github.com/google/oss-fuzz

[17] https://kselftest.wiki.kernel.org

[18] https://en.wikipedia.org/wiki/Robustness_(computer_science)

36

Confidential

Meet Our Team

Elvis Yao

Software engineer

at Moxa System.

Fero Zhou

Software engineer

at Moxa System.

Gavin Lai

Software engineer at

Moxa Networking.

37