iot testing and quality assurance indicthreads
TRANSCRIPT
IoT Testing and Quality Assurance
Girish KolapkarSAS Research and Development (India)
Agenda• Introduction to Internet of Things (IoT)
• Industries approaching the opportunity of IoT
• Emerging IoT Applications
• Typical IoT Technology Stack
• Challenges testing software’s supporting IoT
• Software QA’s Role Redefined
• IoT Testing Areas
• IoT Testing Types
• Strategy for effective IoT software testing
• Q&A
Introduction• The Internet of Things (IoT) is the network of physical
objects or “things” embedded with electronics, software, sensors, and network connectivity, which enables these objects to collect and exchange data.
• Software industry is blending the physical and logical cyber worlds with the Internet of Things.
• IoT isn’t merely a buzzword. It’s where we’re headed.
• Offers potential for businesses worldwide to unlock value and benefits from devices, systems and services communicating with each other, using the internet as a backbone.
Industries approaching the opportunity of IoT
InternetOf Things
Smart Cities and Homes
Connected Customer
RetailHealthcare
Energy
Connected Car/ Transportation
Communications
Manufacturing
Emerging IoT Applications
• Information & Analysis
• Tracking Behaviour
• Examples- SCM, Trucks with sensors for predictive maintenance
• Enhanced Situational Awareness
• Examples- Weather Warning Systems
• Sensor-driven Decision Analytics
• Examples- Oil and gas company optimizing oilfield production
Emerging IoT Applications (Contd.)
• Automation & Control
• Process Optimization
• Examples-Manufacturing Assembly Line
• Optimized Resource Consumption
• Examples-Energy Distribution Networks
• Complex Autonomous Systems
• Examples- Collision Avoidance Systems
Typical IoT Technology Stack
Data Streams at the EdgeData Streams at the Edge
Thing\Device\Sensor\Actuator
Controller, Agent,
Short Haul\Long haul Communication
InfrastructureInfrastructure
Network, Compute ,Data Storage
Application PlatformApplication Platform
Event Management, Intelligent Filter\ Transform, Steaming Model Execution
ApplicationApplication
Websites\Mobile Apps\Industry Specific appliances & consoles, Alerts\Reports\ Decision Dashboards
Device Hub\ Gateway, Device Management
API Design\Build
Challenges testing software’s supporting IoTBig Data-Edges generates astronomical volume of data, speed & variety
Challenges testing software’s supporting IoT (Contd.)
• Complex mix of modern IT & Web services with embedded systems
• Security
• Expensive test environment setup
• Too Many- networks, protocols, gateways, device hubs
Challenges testing software’s supporting IoT (Contd.)
• Multi-user Applications & distributed environment deployment
• Applications require decision support in real-time, batch, hybrid modes
• Debugging requires understanding of multiple technologies
• Difficult to establish point of failures
• Limited User Interface\console
Software QA’s Role Redefined
• More focus on user centric testing
• Just not find bugs,prevent them
• Quality advocates, influencing both development and operational processes
• Software testers who do not have experience testing hardware or embedded systems may need to develop some new skills
• Testers with domain knowledge
IoT Testing Areas
• Connectivity
• Security
• Performance
• Functionality
• Compatibility
• Localization
• User Experience
IoT Testing Areas-Connectivity
• Seamless connectivity
• Data restore when no connectivity
• Sample Test cases
• Verify that all the gadgets involved in the Internet of things testing are able to register to the network.
• Verify that application ensure data security and storage so that whenever the connection is restored, data is back to shape as it was.
IoT Testing Areas-Security
• More possibilities and avenues for cyber attacks
• More threats with financial data. E.g. RFID based highway toll collection systems
• IoT ecosystem provides multiple entry point for attacks
• Lost or stolen devices
• Sample Test cases
• Verify no unauthorized access to device or information.
• Verify data on compromised IoT devices can be remote wiped out.
IoT Testing Areas-Performance
• Device- to-Device communication
• Device to Server communication
• Server To Server communication
• Network bandwidth, latency, packet loss
• Sample Test cases
• Verify response time against benchmarked time with defined connectivity conditions.
IoT Testing Areas-Functionality
• Web\Mobile applications
• Functional\business requirements
• Access control
• Data Storage
• Identity\Roles management
• Sample Test cases
• Verify medical monitoring system returning correct outputs.
IoT Testing Areas-Compatibility
• Device-to-device communication protocols are not compatible, forcing the user to select a family of devices that employ a common protocol
• Lack of standardization leads to incompatibility issues• Compatibility needs to be validated for combinations of
devices, protocols, operating systems
• Sample Test cases
• Verify IoT software supports defined set of decives.
IoT Testing Areas-User Experience• Refers to a person's total experience using a particular
product, system or service
• Familiar
• Clear & Concise
• Feedback
• Consistent
• Attractive
• Efficient
• Forgiving
IoT Testing Types • Compatibility Testing
• Functional Testing
• Performance Testing
• Security Testing
• Exploratory Testing
• Load Testing
Strategy for effective IoT software testing• Automation rules-Develop automation systems, through
code, that can ensure quality standards at each level and which assures -
• Response Time
• Data Validation
• Real Time Data Accuracy
• Secure Access control
• Early detection of issues at design time
• Needs more detailed Acceptance Criteria
• Robust Backend – NoSQL ?
• Minimal manual end to end testing
• Virtualization
• Simulators for various industries
• Remote Testing
• Mock up the API
Strategy for effective IoT software testing (Contd.)
• Elevate a test approach rather than simply executing it
• Analyse Test Results and Test effectiveness
• Measure quality of Test
• Timely review testing tool set as per user feedback, remote debugging and quick turnaround
Strategy for effective IoT software testing (Contd.)
• Generate Realistic Test data
• Data Generator Tool
• Test Analytics
• Understand signals in data to derive new patterns based test cases
Strategy for effective IoT software testing (Contd.)
• Effective Test Data Management Tool
• Data on Demand
• Data Integration
• Data Security
• Data validation
Strategy for effective IoT software testing (Contd.)
• Test suite of source adapters
• Predefined adapters e.g. Hadoop (+YARN), RabbitMQ
• Test for publish(read) only- e.g. Sniffer,Twitter, SYSLOG, HTTP RESTful Interfaces
• Test for subscribe(write) only- e.g. SOAP,SMTP
• Customizable publish\subscribe API’s
Example- Test Event Stream Processing Engine (Contd.)
• Test visual interface which define\refine event stream models
Example- Test Event Stream Processing Engine (Contd.)
• Deploy & Test On Workstation
• E.g. Test for iPhone sensor logs
• Deploy & Test on Edge Gateway
• E.g. Test for physical raspberry-pi
Example- Test Event Stream Processing Engine (Contd.)
• Connect iPhone and Laptop to wireless network
Example- Test Event Stream Processing Engine (Contd.)
• Step 1: Install and check SensorLog app on iPhone, get ip address.
• Step2: Start ESP server and put ESP model in test mode
• Step3:Run SensorLog and start streaming data through socket
• Step4: Run adapter script to read iPhone sensor data
• Step5: Open visualization to see streaming data
• Choose line graph to show sensor data and moving average
• Shake your iPhone to see graph changes
Example- Test Event Stream Processing Engine (Contd.)
QUESTIONS?
