sonic ansible test automation… · sonic ansible test automation taskin ucpinar senior director of...
TRANSCRIPT
SONiC Ansible Test Automation
Taskin UcpinarSenior Director of SW DevelopmentEdgecore Networks
EW: Networking / SONiC
SONiC Development & Testing: Goals
• End-to-end testing in development workflow• Coverages and Automation• HW platform validations• Pre-SI RAS, Performance testing • Sonic ecosystem integrations
NETWORKING
TestedConfigurations
Edgecore’s Dedication to SONiC
AS7716-32XB
AS7326-56X
AS5712-54X
AS7116-54X
AS7212-54X
AS7312-54X
AS7312-54XS
AS7816-64XAS6712-32X
AS7712-32X
AS7716-32X
AS7712-32X
AS9716-32D
AS7312-54XSAS4222-28PE
MiniPackAS7512-32X
NETWORKING
TestedConfigurations
Edgecore’s Dedication to SONiC
• SONiC Feature Set Growth• Rapid Development Environment• Ensure Stability/Reliability• Testing• Interoperability• Regression• CI/CD
NETWORKING
TestedConfigurations
Regression Testing
• SONiC and Branching• Getting the full picture• Completing the coverage• Hardening • Community Services
NETWORKING
TestedConfigurations
SONiC Branches
And How Edgecore Handles Them…
NETWORKING
TestedConfigurations
1803 1807 1811 1904
1 2 3
1 2 3 4 5
SONiC and Releases
NETWORKING
TestedConfigurations
1811 1904
1
EC1811
2 3 4
1
EC1904
2 3 4 5
Edgecore’s Release Test Branches
NETWORKING
TestedConfigurations
Testing Releases
(and some more)
NETWORKING
TestedConfigurations
Getting The Full Picture
SONiC201705 BGP
ECMP
LAG
LLDP
QoS - ECN
One Image
...
SONiC201709 VLAN
ACL Per/Den
IPv6
Tunnel Decap
Mirroring
BGP MP
...QoS - RDMA
NETWORKING
TestedConfigurations
Getting The Full PictureRelease SAI Version No of Features No of Ansible Tests
SONiC201705 0.9.4 18 9
SONiC201709 0.9.4 8 4
SONiC201712 1.0 7 2
SONiC201803 1.2 5 2
SONiC201807 1.3 3 0
SONiC201811 1.3 6 2
SONiC.201904 1.4 16 5
Total 63 24
NETWORKING
TestedConfigurations
Testing From All Angles
• Test units/components/functions/functionality• Not only for testing, but for training purposes• Independent on features/platforms• Unit Tests: Typically implemented by developers• Functional Tests: Implemented by developers and testers• Test Frameworks
• gtest• pyunit
NETWORKING
TestedConfigurations
TestedConfigurations
100% Coverage
• All Features with Tests
• Coverage % Unknown
• Stability Unknown• Stability Improvements
• Measure Coverage %
• New tests
Hardening
Edgecore Community Labs
• Multiple Community Labs Across Globe
• SONiC Devices and Packet Generators
• Available to Edgecore Partners, Customers, Community
AAutomation
• Auto detect and execute
• Regression Testing
62%
• Features Missing Tests
• Coverage % Unknown
• Stability Unknown
Completing The Picture
NETWORKING
Development Environment
Edgecore
Cloud
Build
Tools gcc/cc/gdb
make/autoconfig/pkg-config
cmake
Source Codesonic-swss
sonic-swsscommon
…
Test(s)
Dependencieslibnl
python
…
PyUnitDevEnv
CI/CD PyUnit
Code Review
Regression UT Regression FT Test(s)
Static Analysis Performance …
NETWORKING
TestedConfigurations
Automated Testing Environment
NETWORKING
TestedConfigurations
Automated Testing Environment
NETWORKING
TestedConfigurations
Automated Testing Environment
NETWORKING
TestedConfigurations
Automated Testing Environment
0
10
20
30
40
50
60
70
80
1 2 3 4
No of Testbeds vs Execution Time (hrs) *
• Target AST7816-64X
• Release 1904
• 234 Test Cases
• Clean build before tests
• Cleanup Between Runs
• Reset Image & Config
• Reboot
NETWORKING
TestedConfigurations
Edgecore’s Contribution on 1904
In Progress Plan to
Submit
Submitted Merged Total
unit test 5 12 11 11 38
SONiC 6 4 10
sonic-mgmt
(test case)
10 7 17
Total 5 12 27 22 65
NETWORKING
TestedConfigurations
Automated Testing Environment
• Target AST7816-64X
• Release 1904
• 234 Test Cases
• Clean build before tests
• Cleanup Between Runs
• Reset Image & Config
• Reboot
0
50
100
150
200
250
Failed Passed Total
NETWORKING
TestedConfigurations