quality team strategy
TRANSCRIPT
My Target• Suggest test Automation plan including the process
and suggested tools• Highlight how you are going to handle the stress/ load
test?• Security is a key issue, what is your plan for checking
security vulnerabilities in our platform and associated apps
• How you can consider the UX testing as an integral part of your test plan and test Automation plano What’s the international standards you can use as bench mark KPI for platform
and applications performance as well as the UXo Do you suggest any a test automation tool for the focused groups? What tools?
Aspects to consider• Our platform and associated apps are targeting
multimillions of users across 35 different country• Our technology is GIS oriented where map is
representing a common control in all apps associated to our platforms
• We crave open source technologies:o Dynamics programing language for frontend and middleware (Ruby on
Rails)o Java is running on the backendo Open source GIS server (Geography)
• Distributed PostSQL is our main database
Aspects to consider• Our sources control is GIT and we manage it
using Stash integrated with JIRA system• We use Native mobile programing languages for
(IOS and Android)• Our platform is totally hosted as a cloud service • We are an advertising platform so that we 75% of our Apps is mobile based (IOS and Android)• Our software Engineering SOP is based on Agile-
Scrum methodology and we manage our sprints using JIRA
SOFTWARE DEVELOPMENT DONE RIGHT
IT Company
Enough is not enoughTest Strategy for Mobile
Mobile application testing strategy
• Key elements to consider for effectively testing applications are :o Licensed Tools or open sourceo Target Device Selection – Create an optimal mix of simulator
testing and physical device testing on different models to maximize test coverage.
o Test Automation – Select an effective test automation tool and maximize the use of automation to reduce the cost of regression testing.
o Network Environment – Consider testing primarily on Wi-Fi networks and using network simulation tools to simulate cellular connectivity and various network conditions.
o Types of Testing – Consider different types of testing required (functional, performance, security, and compliance)
Types of testing• Usability • Performance • Security • Interface Testing • Services Testing • Low level resource testing • Compatibility • Operational
Performance Testing• Need for speed • Get real • Business impact • 3rd party integrations • Performance tuning
Security Test• Code scanning
o Can help enforce coding rules• Avoid poor coding styles• Enforce use of mandated functions
o Can reliably find a number of data flow & semantic issues e.g.:• Format String• Injection Attacks – SQL, Process, XML• Certain classes of buffer overflow
• Automated Testingo Three areas of automated tests:
• Targeted • Generated• Leverage
Security Testing• Implementing security in the unit testing cycle
means o investing more in developer awareness of security and how to test for
security issues, and less in specialized external resources.o A long-term investment that can vastly improve the overall quality of
softwareo Reduce the number of vulnerabilities in web applications, and
consequently, the associated risks.• Security vulnerabilities are often introduced
through software failures under precisely these exceptional conditions.
Integrated Security Test
Security Standards Example
Security Standards Example
Testing Security Tools• Http Unit, j Web Unit,
Html Unit, Canoo Web test and the Ruby based WATIR
• Testing HTML injection with j Web Unit
• Testing for SQL injection in a login form
• Testing Access Control• Testing for XSS
Select Automation - iOS
• Frank • KIF (Keep It Functional) • UIAutomation • Zucchini
Select Automation Tools
• Robotium• Calabash• MonkeyTalk• Sikuili• Native Driver• Android Instrumentation• Bot-Bot • Monkeyrunner • DroidPilot • AndroidDriver
(Selenium/ WebDriver)
Select Automation - cross platform
• Appium • Calabash • NativeDriver • iPhone Driver• Watir• And many more…
Automation Tools• MobiTest• Google Page Speed• Load Runner• Jmeter
Emulator• All applications can be deployed and tested in
emulator without investing in mobile handset for mobile OS
• Emulator are mostly available free and we can also perform UI , stress and performance test on that
• 30 : 40 % of test
Weighted Device Platform Matrix
Mobile UX Standard• Mobile Web Best Practices 1.0" is a W3C Web Standard that helps
people design and deliver content that works well on mobile devices.• This set of cards summarizes the guidelines of that document in ten
themes. • By following these guidelines, content producers will broaden their
audience, create effective Web sites and applications, and make browsing the Web convenient on more devices.
• For more information: http://www.w3.org/TR/mobile-bp/• Also HTML5 as a standard of building mobile-optimized sites/apps.• Note: It's really hard to define a standard because there are so many
types of phones out there (smart phones, feature phones), and so many operating system versions (iOS, Android, Blackberry, Nokia, etc...).
Mobile UX Standard• 10 ways to mobilize
o Design for One Webo Rely on Web standardso Stay away from known hazardso Be cautious of device limitationso Optimize navigationo Check graphics & colorso Keep it smallo Use the network sparinglyo Help & guide user inputo Think of users on the go
Agile
Agile Test
Test Driven Development
• Write the unit test case• Execute the build
Test fails• Write just enough code
to cover the test• Execute the build
Test passes• Refactor
Write the Test
Refactor
Execute the build
Coding
Execute the build
Fail
Pass
Pass
Fail
A software development technique which ensures that the software being developed is 100% covered by unit test cases
Continuous Integration
• Continuous Integration is a software development practice where members of a team integrate their work frequently; usually each person integrates at least daily - leading to multiple integrations per day.
• Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible – Martin Fowler
• Pre-release builds (hourly/nightly) help developers to improve the quality of the source code and release builds help in automating the release process
• It is essential that code repository is centralized and unified tools are used
Role of Tester in Agile• Testers are integral part of the team• Participate in Release/Iteration planning• Start Testing activities from the day 1•Collaborates with the customer to define the acceptance test criteria Validates that the system is doing exactly what it is supposed to do• Tests Stories once they are complete• Focuses on test automation • Focuses more on exploratory testing• Practices pair testing (similar to pair programming)•Collaborates with Development team• Provides continuous feedback to the team
• Starts by looking at prioritized personal work queue
• Identifying the work items for the day
• Attending the standup meeting• Collaboration with Dev Team
(seeking and offering support)• Depending on the type of work item
o Writing acceptance Criteriao Writing the test caseo Automating a test caseo Executing a test case (manually)
Three-tier Mobile Application Testing
Framework
Cloud Testing Services• Device Anywhere• Perfecto Mobile
Additional test cases should be considered
• How much battery life does the application use? What good is a mobile device that has to be supplied with electricity just to power the application?
• How does the application function with limited or no network connectivity? • Minimally the application should not crash; ideally the user should not even
notice a difference. • How fast is the application? Even with slower processors and networks,
users still expect desktop speeds out of their mobile devices. • How quickly can users navigate the application? With limited attention
spans, mobile devices need to be highly intuitive. • How much data will the application need? Will users without unlimited data
plans or devices without large internal storage be able to use the application?
• Will peripheral devices affect the application? Whether or not the application uses peripheral devices, these devices affect the processes running in the background, in turn affecting the application.
Mobile App Automated
Testing Cucumber
• Cucumber Provideso a framework for writing software specificationso a tool for executing those specifications.
• Specifications are written DSLs
• Works with Ruby, Java, .NET, Flex and many more
• http://cukes.info/
Example – Feature
Step Definition• Make Cucumber Tests come alive.
• Written in various Programming Language (mostly Ruby)
Calabash• One Interface Cucumber for iOS and Android
o Reuse of Test Cross Platform (iOS and Android)
• Targets Native and Hybrid apps for Android and iOS
• Runs on Physical Device• Support for Hybrid Apps (embedded web-views)• Open Source Tool• Cross Platform Testing
Calabash iOS - architecture
• Calabash tests are executed with the Cucumber tool and consist of two parts:
• Client Library written in Ruby
• Calabash.framework, a server framework written in Objective-C
Calabash iOS• To use Calabash we make a special test target in
Xcode that links with calabash.framework• The Server Framework starts HTTP Server inside
the App, that listens to the request from Client Library.
• The Cucumber tool executes the feature file that represents the intended behavior of the App Under Test.
• The Feature Files contains steps (Predefined or Custom)
Calabash Android- architecture
• Features
• Step-Definitions
• Instrumentation Test Server: This is another app which will be installed and executed into device.
Calabash iOS• App is built for Testing by linking a static library (calabash-server)
o Uses a combination of UIAutomation and other APIs
• Supports Interactive test-development
• Based on Frank but changes some core parts.o Setting up Frank Requires modification of the app source,
inclusion of static resources and source files manually, where as Calabash has Automated Set Up. 3 commands and we are done
• Touch Synthesis supports multi-touch, gestures
• Can use device accessibility for identifying views
Queries• Explore the App Interactively• Run Command ‘calabash-ios console’• It gives us a Calabash console (it is just a Ruby irb with calabash loaded)• Example -
irb(main):001:0> query("button")
[0] { "class" => "UIButton", "frame" => { "y" => 20, "width" => 101, "x" => 20, "height" => 101 }, "UIType" => "UIControl", "description" => "<UIButton: 0x4cc1120; frame = (20 20; 101 101); opaque = NO; autoresize = RM+BM; layer = <CALayer: 0x4c89cc0>>" },
Returns an array with objects that are descriptions of the buttons in the current screen in your app
Queries• The query function
o takes a string query as an argument.o the query argument is similar to a css selector
• The syntax for queries is based on UIScript, o but is a new implementation with additional features added (and some
removed).• UIScript gives a nice "CSS-selector" like approach to
finding view objects in your app screens.irb(main):001:0> query("tabBarButton").count => 4
irb(main):002:0> query("tabBarButton")[0] => {"class"=>"UITabBarButton", "frame"=>{"y"=>1, "width"=>76, "x"=>2, "height"=>48}, "UIType"=>"UIControl", "description"=>"<UITabBarButton: 0x856a820; frame = (2 1; 76 48); opaque = NO; layer = <CALayer: 0x856d210>>" }
Prototype gestures• Calabash iOS has a number of built-in event
sequences that can be played back.o (touch, swipe, pinch, etc)
• Events can be relocated (translated) to different views, and optionally offsets.
• Extensible: you can record your own gestures if none of the built in suits you.
Mobile Web Automation using
Selenium2.0 • Selenium, is an open Source tool from OpenQA.• The tool is considered to be one of the best tools
available in the market to automate the web applications testing.
• Selenium offers quick and powerful way to write functional tests for applications and an environment where they can be run across different hosts.
• One of Selenium’s key features is the support for executing one’s tests on multiple browser platforms.
An Approach of Using Se
Integration of Selenium2.0 with TestNG and ANT.• This framework is created using ANT, TestNG and Selenium 2.0.• Through this framework, an user will be able to create an automated test case that
can be executed in a single command.
Parameters
Report_Template.xls Reports.xls(*)+TestNG Report
Temp Folder
Report Results to a File
Mobile Web
Appending values
ANT
Return's Result
Selenium2.0+TestNG Script
How Works for Mobile Web(Android Devices) ?
• The Android Driver uses the RemoteWebDriver. • The client side is your test code(Java Code). • The server side is an application that gets
installed on the android emulator or device.(android-server.apk)
• The client and the server communicate using the Wire Protocol which consists of REST requests using JSON over HTTP.
• The application executes the tests in the context of an Android Web View.
How Works for Mobile Web(Android Devices) ?
• Step 1:Install Android SDK• Step 2:Set up the Emulator or
Android Device.• Step 3:Install the Android
WebDriver Application using the command
adb -s <serialId> -e install -r android-server.apk• Step 4: Setup Port Forwarding
using the command adb -s <serialId> forward tcp:8080 tcp:8080• Step 5:Start the Web Driver
server on the Emulator or Android device.
• Step 6:Run the Test Script.
• The Android Driver uses the RemoteWebDriver.
• The client side is your test code(Java Code).
• The server side is an application that gets installed on the android emulator or device.(android-server.apk)
• The client and the server communicate using the Wire Protocol which consists of REST requests using JSON over HTTP.
• The application executes the tests in the context of an Android Web View.
Test Automation Tools & Frameworks
• Android Testing Framework
Android Testing Framework
• Android Instrumentation Framework
Android Testing Framework
• Positron Frameworkpublic class AddRecord extends TestCase { @Before public void runBeforeEveryTest() { //Start the activity in test mode startActivity("org.demo.Demo","org.demo.demo2.Demo2"); pause(); //Wait for the requested answer press("Name", DOWN); //Simulate typing a word click();//Simulate click event in a focused UI element } @Test public void addPerson() throws InterruptedException { /*Assert state*/ assertTrue( "not click", booleanAt("save.isPressed")); field1=stringAt("listView.1.0.text");//Get string } }
Sikuli GUI testing Framework
JaBUTi/ME White Box Testing
• “white box testing criteria use a representation known as Control Flow Graph (CFG) to abstract the structure of the program or of part of the program, as a procedure or method”
• JaBUTi (Java Bytecode Understanding and Testing Tool) is a complete tool suite for understanding and testing Java programs and Java-based components.
Mobile TestBlack Box Testing
MobileTest architecture
Basic usage scenario of Mobile Test