network operating system harmonization architecture and automated sanity test demo
TRANSCRIPT
Cisco Confidential© 2012 Cisco and/or its affiliates. All rights reserved. 1
Network Operating System Harmonization Architecture and Automated Sanity Test
DemoCisco Research Center Training
Vikram Hosakote, Sr. Software Developer, NOSTG
Cisco Systems, Boxborough MA, [email protected]
04/18/2014
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
OSH Manageability Test Architecture
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
Python show display templates
Testing for templates is initially done in internal dev env in a standalone fashion, independent of cli parser. Cli parser is not available in internal dev env.
Released sanity includes basic template testing of the entire test yang, on internal dev env. Additional tests are being added for python template api feature/functional testing, and negative python template api tests.
Initial integration with cli parser will occur on NXOS.
Target is to use same python based test framework for sanity testing and end-to-end testing with CLI parser. Test framework is being implemented to handle standalone vs. cli parser execution, & xmascon vs. rtr config mode for config data setting.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
CLI parser and Nvgen
Parser/Nvgen Test plan positive testcases EDCS-1293270
Parser/Nvgen Test Plan negative testcases EDCS-1295063
TIMS test report:http://tims.cisco.com/warp.cmd?ent=Tzm1557d
Bug Scrubber:http://wwwin-ottawa.cisco.com/tfoggoa/Scrubber/showquery.html?query=akalambu-7
DE-DT weekly Sync-up
CLI sanity test script
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
XTF test agent
Agent side APIs are tested using the test framework implemented in XTF
Agent side APIs are documented here - http://wwwin-ensoft.cisco.com/osh-mgbl/apis/xmas/latest/internal/xmas__agent_8h.html#a515d63f9bfc0301005b581e91017d738
The XTF Agent tests cover get/set all objects in test-comp.yang model, register APIs, action APIs, singleton write, bulk write, lookup, utility, bag encode and decode APIs
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
NETCONF
NETCONF is an IETF defined protocol to provide mechanisms to install, manipulate, and delete the configuration of network devices
The NETCONF server application resides on the IOS devices and accepts connections from external NETCONF client applications. The connected session then behaves as an initiator or listener
NETCONF capabilities & protocol operation feature test includes <get-config> , <edit-config>, <lock/unlock> <subtree filtering> and can be found at http://wwwin-eng.cisco.com/Eng/NMTG/Device_Instrumentation/NETCONF_sw_func_spec.doc
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
YANG compiler and southbound code
The test YANG and XMAS files are compiled using the OSH YANG / XMAS compiler.
Once the YANG and XMAS files are compiled, the DevEnv / Polaris / NXOS workspace is built.
The southbound code is the backend application C code that has the implementation for all the backend generated XMAS callbacks for registration, config, oper, RPC, notifications, and PCA.
YANG compiler uses pyang, Cython (Python to C translator integrated with CPython), and Jinja2
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
XMAS datatype translation
File.pyx Cython Compiler File.c C compiler File.so
File.py
App.py
import
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
OSH manageability sanities
XMAS YANG compiler sanity script
Python show display template sanity script
XTF test agent sanity script
CLI compiler sanity script
Daily sanities available in DevEnv (EnXR) and NXOS (osh_ual_72 and Jenkins) that run a total of 340 tests everyday
Info about these sanities on OSH manageability wiki - http://wikicentral.cisco.com/display/GROUP/NOSTG+OSH+UAL+Manageability+Test+Wiki
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
Sample OSH mgbl daily sanity email report
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
OSH manageability P&S testing
Below are the OSH performance and scaling numbers tested config lists with 5000 keys - mac address, uint32, int64, string oper lists with 5000 keys - mac address, uint32, int64, string 5000 leaves in config xmas:item list – mac address, uint32, int64, string 5000 leaves in xmas:item container - mac address, uint32, int64, string Send 1000 RPC results – sync and async Generate 1000 notifications
OSH manageability P&S test plan - EDCS-1380817
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
OSH manageability HA testing
HA switchover testing done on Polaris (ASR 1006), and NXOS (N7K SUP2)
Process restart tests done – test_comp_main, sysdb_svr, xyang_svr, xosds
The following HA tests were done: Python show templates XTF agent tests YANG XMAS compiler tests CLI tests
OSH manageability HA test plan - EDCS-1354815
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
OSH manageability test wiki
http://wikicentral.cisco.com/display/GROUP/NOSTG+OSH+UAL+Manageability+Test+Wiki
The OSH manageability wiki has information about OSH mgbl test plans and the OSH mgbl test strategy Bug scrubber links (XMAS, NETCONF, and xos-parser bugs) Weekly status meeting minutes TIMS links (EnXR, Polaris, and NXOS) OSH mgbl sanity scripts OSH mgbl training
OSH mgbl test mailing alias – osh-mgbl-test
Thank you.