dealing with telecommunication product verification challenges …€¦ · 1830 pss-24x/12x/8x otn...
TRANSCRIPT
© 2019 Nokia1
We create the technology to connect the world
Guido BauerOctober 2019
Dealing with Telecommunication Product Verification Challenges @ Nokia Nuremberg
© 2019 Nokia2
Agenda
• Nokia in General
• Nokia in Germany
• Optics Business Unit
• System Verification Tasks
• System Verification Challenges
• Large Product Family
• Complex Products
• Release Schedule & Content
• SW Load Deliveries
• Management Interfaces
© 2019 Nokia3
Unrivalled track record of innovation
Foundations of …
• The entire electronics industry
• The internet, networking & optics
• Mobile and fixed communications
Nobel Prizes9 Turing Prizes3 Grammys2 Emmys2 Oscar1
Satellite commsTransistors Unix/C/C++Laser/fiber optics
Solar cells Charge-coupled devices
Super-resolution microscopy
Coherent optics
© 2019 Nokia4
Nuremberg Highlights
▪ Center of Excellence for High Speed Optical interface andOptical Cross Connect switching systems development
▪ Home of the PSE-3 (Photonic Service Engine)
▪ Customer Demo Center for Optical Solutions
▪ Development Hub of the Optics BU
▪ Service Integration, Field Force and Procurement on site
Nokia Nuremberg
Employees▪ ~400 employees in Nuremberg▪ Highly skilled, motivated and multicultural▪ Very experienced team with low attrition rate▪ Agile development teams▪ Fully integrated in the international Nokia community
Nokia in Germany
Stuttgart
Nürnberg
Hannover
Hamburg
Berlin
Mannheim
Leipzig
UlmMünchen
Düsseldorf
Bonn
• about 4.000 employees
• 2 large multi-functional sites, 2 specializes development sites,Nokia Bell Labs research
© 2019 Nokia5
Optics Global R&D & Support Presence
OTTAWA
MURRAY HILL
CHENGDU
NA
ASIA
EMEA
VILLARCEAUX (FRANCE)
Nuremberg / Stuttgart (Germany)
VIMERCATE (ITALY)
WESTFORD
BANGALORE
SHANGHAI
R&D Centers
ONGOING GLOBAL INVESTMENTS IN OPTICS TECHNOLOGIES
29.000+Active Patents
2000+R&D Engineers
© 2019 Nokia6
WW Competence Center for Hi-Speed Photonics, OTN & GMPLSOptics in Germany
Stuttgart
Nürnberg
~300 R&D/PLM engineers and scientists
• Product Line Management
• Bell Labs Research
• System Engineering & Architecture
• High Speed Photonics Development
• Hardware & ASIC/FPGA Development
• Software Development
• System Verification
Key competencies
• Optical Switching Systems
• Photonic Technologies (WDM)
• GMPLS Control Plane, MRN and
Transport SDN
© 2019 Nokia7
1830 PSS-24x/12x/8x OTN Switches (Core & Metro)Optical Innovation from Germany
PSE-2 powered
cards
Flexible 100G-400G Super Coherent transport
wavelengths
Optimized for 100G interface density
1830 PSS-12x
1830 PSS-8x
1830 PSS-24x
Scalability
Capacity
4.8T
48T
24T
4T
1.6T
12T
9.6T
Flexibility
Programmable
Interfaces
SPQPSK
BPSK
QPSK
8QAM
16QAM
64QAM
Efficiency
0.8W/GbSpectral Efficiency
32T
21.6T
17.6T
10.8T
© 2019 Nokia8
Ultimate performance from subsea to metro powered by PSE-3Optical Innovation from Germany
500-600GMetro
300G-450GRegional / Long Haul
200G-250G Long Haul / Subsea
© 2019 Nokia10
Large Product FamilySystem Verification Challenges
Details Addressed by
Large diverse product family• in network configurations• with network planning & network management
support• requiring interoperability• with stringent load, robustness and stability
demands• Continuously growing
HW and SW asset development and reuse across products for effort minimization, cost control and quality improvement
Be prepared for product transfers within the Optics BU• Global organizations with global discipline leads
ensuring efficient worldwide operation• Global tooling with assigned engineers for
• Ticket system• Document repositories and reviews• Test automation environment
Dedicated System Verification sub-teams for• Core test automation• Interoperability testing• Cross product Load & Robustness
© 2019 Nokia11
Large Product Family (cont’d)System Verification Challenges
Details Addressed by
See previous slide Verification in various configurations• simple functionality in single NE• Network features in multi-NE configs of single
and mixed product types
© 2019 Nokia12
Complex ProductsSystem Verification Challenges
Details Addressed by
Complex products comprising• High end HW with latest CPUs, FPGAs/ASICs
at technology limits and high speed electrical and optical connections
• Distributed, embedded SW with demanding real time requirements
• Highly sophisticated network planning tool and network management system
Invest in testers‘ system white box know-how to• be able to determine and contact correct
responsible developer• provide right log information for fast root
cause analysis in case of product issues
Investment in “faking” functionality to improve product verification coverage
• SW debug and faking (e.g. for alarm creation)
• HW fault faking by using a system boundary scan
• Power cycle equipment
© 2019 Nokia13
Complex Products (cont’d)System Verification Challenges
Details Addressed by
Demanding Lab Environment• Space• Power supply
• AC for measurement equipment and IT HW• 48/60V DC for network elements (NEs)• PSS-24x in 9T6 configuration: at maximum
14.4kW and typical < 8kW• Cooling• Noise
Extensive investment in and usage of VMWare based simulation environment (HIL: HW-in-the-loop) to• Start testing very early before product HW is
available• Improve costs by reducing needed target
configurations• Increase testing speed
© 2019 Nokia14
Release Schedule & ContentSystem Verification Challenges
Details Addressed by
Very demanding release schedules regarding• number of releases per year
• release content size (amount and magnitude of new features)
• release date and quality adherence
Automation is key• Develop and use automation already for initial
new feature testing• For regression testing available latest end of
release• There is no time to add automated
regression functionality at the beginning of the next release
• Optics System Verification core automation team
• System Verification Nuremberg automation core team
• Each tester develops automation for functional area (s)he is owning• No separation between manual and
automated testing resources
Rx.0.0
May year Y
Content + bug fixes
Rx.0.1
Jul year Y
bug fixesonly
Rx.1.0
Nov year Y
Content + bug fixes
Rx.0.1
Jan year Y+1
bug fixesonly
© 2019 Nokia15
Release Schedule & Content (cont’d)System Verification Challenges
Details Addressed by
• rapidly increasing amount of “legacy features”• Testers are more and more busy with
regression test result evaluation
• Started recently investment in AI/ML based automatic test result assessment
© 2019 Nokia16
Release Schedule & Content (cont’d)System Verification Challenges
Details Addressed by
Last major release contained ~18000 test cases• ~4000 new feature test cases• ~14000 regression test cases
Ensure continuous product quality• Day-by-day
• Daily load delivery with quality verification• When approaching the release date (CP)
• Start of so-called bug scrub process 2 months before CP• Load line split (dedicated release load
line)• Control of each fix going to release
load line by System Verification team• Fix pre-test in next release or private
load before merge to release load line• Final Quality checks right before CP
• 1 week soak test• Final regression and upgrade tests
© 2019 Nokia17
SW Load DeliveriesSystem Verification Challenges
Details Addressed by
SW Load Delivery Handling• “Continuous Integration” leading to daily loads
for up to 2 load lines (main and release endgame after bug scrub start)
• Additional private loads on demand for• bug fix retest in bug scrub phase• dedicated problem investigations
• Global load build
Load healthiness assurance by Sanity testing• 2h basic verification that load is not broken• in simulation per functional area and on target
per product
Ensure “legacy” feature quality• Daily (overnight) test in simulation and on
target• Weekend Regression Test on target• Feature coverage
• functionality of previous release• already verified features of current release
• Varying content of target regression runs due to amount of tests
• Unique test case run time tracking• in August ~9000h, i.e. 375 days
© 2019 Nokia18
Management InterfacesSystem Verification Challenges
Details Addressed by
Management interface richness• Customers want to manage their products and
network differently• With Nokia management capabilities• By using own management systems
• Therefore large amount of management interfaces implemented and to be verified• SNMP
• Network Management System• Craft Terminal (WebUI)
• CLI• TL1 (for dedicated NAR customers)• NetConf & gRPC for SDN application
Test environment supporting verification of all interfaces in manual and automated operation
Integration of management interface switching into automation environment• For products owned by Nuremberg currently
CLI, TL1 and WebUI• Automated sanity and regression tests can
switch the interface run by run