selenium camp 2016 - kiev, ukraine
TRANSCRIPT
![Page 1: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/1.jpg)
The Mobile Grid - Getting Started Android & iOS
Kiev, Ukraine - 26 Feb 2016
![Page 2: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/2.jpg)
How will you test all of these?!The Challenge!
![Page 3: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/3.jpg)
And each one of these cases?
• Validate every locale / languages your app supports.
• Validate every OS version your app supports.
• Validate different resolutions.
• Validate different screen sizes (phones, tablets, phablets etc..).
• Validate different device manufacturers.
![Page 4: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/4.jpg)
You could try the single test approach!Not recommend…
![Page 5: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/5.jpg)
Or the multi-threaded approach!Also, not recommend…
![Page 6: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/6.jpg)
Though, this guy doesn’t seem to mind.Still not recommend, however…
![Page 7: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/7.jpg)
Hire an army of zombie testers?!Nah, definitely not recommended!!!
![Page 8: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/8.jpg)
So what options do we have?!
• Test Automation of course!
• But how, why, and where???
![Page 9: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/9.jpg)
Run Tests Single threaded?Still, not recommend…
![Page 10: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/10.jpg)
Parallelization!
![Page 11: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/11.jpg)
Selenium Grid + Appium
BFFS 4EVER
![Page 12: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/12.jpg)
Test On The Cloud• The Pros:
• It’s the present and the future of automated testing.
• Ease of setup and integration.
• Save on costs of buying mobile devices or machines.
• Some test services provide you with excellent analytics and reporting.
• Some test services integrate well with cloud CI services. e.g. Travis CI.
• You expose your app to a much larger range of devices and OS’s.
• The Cons:
• It can be difficult sometimes to narrow down issues.
• Network problems.
• Latency issues.
• Outages can occur.
• Bugs in their software.
• Acquired by another company and then shutdown.
• Costs can become significant when running CI on every commit.
![Page 13: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/13.jpg)
Cloud Test Services• Sauce Labs
• Test Object
• Amazon Test Cloud (was Appthwack)
• Google Test Lab (was Appurify)
• Perfecto Mobile
• Testdroid
• TouchTest
• Keynote (was Device Anywhere)
Just to name a few…
![Page 14: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/14.jpg)
The Mobile Grid• The Pros:
• You can run tests an unlimited amount without worrying about costs.
• You’re not handcuffed to a cloud test services 24/7 availability.
• You have all the critical test data local for debugging.
• Have full access to every connected device via adb or instruments.
• No latency issues.
• You don't expose your app or sensitive data to the cloud.
• The Cons:
• Can be tough finding documentation or good examples specific to your bindings.
• Takes a bit of time to setup and smooth things out.
• Require some degree of ops knowledge.
• You might not be able to expose your app to a large range of devices and os’s.
• Build and maintain the entire setup (software & hardware).
• Devices can be a real pain! (I’ll cover this later.)
![Page 15: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/15.jpg)
What I’ll demo…• Getting all of the connected devices programmatically.
• Launching Grid hub & Appium nodes programmatically.
• Capturing screenshots, logs, video and logcat for reporting.
• Running single threaded.
• Running tests distributed.
• Running tests in parallel.
• Leveraging cloud test services such as Sauce Labs.
• Reduce your runtime costs by only executing selected scenarios.
• Generating an Allure report with all above metadata attached.
• The grid setup at Wunderlist.
• and lets briefly look at the core code…
![Page 16: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/16.jpg)
Get Connected Device Data
• We retrieve connected device data and insert into an environment variable. • The “thread” value will then match our “TEST_NUMBER” env var for
parallelization. TEST_NUMBER is created from the parallel gem.
![Page 17: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/17.jpg)
• Capture device data for every connected device.
• At a minimum, you need the UDID to connect appium to a grid node.
• However, it is important to know the OS, and SDK versions too.
![Page 18: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/18.jpg)
Node Config Method
• Programmatically create your node based on the connected devices.
![Page 19: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/19.jpg)
Generated Node Config
![Page 20: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/20.jpg)
Appium Server Setup
• Method to setup appium server and pass in any arguments. • The wonderful Appium developers baked a grid node into the
appium server.
![Page 21: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/21.jpg)
Launch Hub & Appium Server Methods
• We set a “THREAD” variable. This will tell our parallel library how many processes it should run.
• Launch the nodes in parallel so all start at the same time.
![Page 22: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/22.jpg)
ADB Methods (Video and Logcat)
• Video records up to 180 seconds for real devices. • Capture logcat data.
![Page 23: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/23.jpg)
Spec Helper Methods
• Parse the DEVICES env variable and match it’s thread to the running process, then initialize appium.
![Page 24: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/24.jpg)
Test Setup and Teardown
• Set the UDID environment variable for your tests.
• Before each test, start video and logcat.
• After each test, stop video and logcat.
• Only attach output to report on failures!
![Page 25: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/25.jpg)
Test Example
• Here is where the UDID environment variable is used. This will let us know which device ran a specific test.
![Page 26: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/26.jpg)
![Page 27: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/27.jpg)
Android Setup
• Download every SDK version your app supports. • Install Intel HAXM Accelerator!
![Page 28: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/28.jpg)
Create Android Emulators
• Create an emulator for each sdk you installed.
![Page 29: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/29.jpg)
Single Thread Test Example
• As you can see, this example was pretty slow. 48 seconds in total.
![Page 30: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/30.jpg)
![Page 31: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/31.jpg)
• Distributed means we spread each test to an available node. e.g. device.
• This example only has two tests (specs). So in theory we only need two devices, with each running a test.
• In this example we have one emulator (left) and one real device (right).
• Our runtime goes from 48 seconds single-threaded to 27 seconds multi-threaded.
Distributed Test Example
![Page 32: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/32.jpg)
Allure Report Distributed
![Page 33: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/33.jpg)
Parallel Test Example• Parallel in this case means we
run same tests on every available node. e.g. device.
• This example has two tests (specs). All tests will run on all devices/emulators.
• We have five emulators, each running a different SDK version, and one real device (bottom right).
• By testing on all SDK versions, this gives us a warm fuzzy feeling our app works correctly on all versions.
• As you saw, emulator-5554 (GRID5) crashed. Lets look at the report!
![Page 34: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/34.jpg)
Allure Report Parallel
![Page 35: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/35.jpg)
Run On The Cloud
• In the example tests theres only one spec tagged with “sauce”. If you we had more, we could run in parallel for faster test execution in Sauce Labs. e.g rake android[dist,sauce].
![Page 36: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/36.jpg)
Allure Report Sauce Labs
• If needed, you can also download the test assets and attach to the allure. Example: https://github.com/isonic1/appium-mobile-grid/blob/master/common/dowload_sauce_assets.rb
![Page 37: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/37.jpg)
![Page 38: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/38.jpg)
iOS Setup
• Enable Developer Mode on your devices.
• Pair your devices to your machine.
• Enable UI Automation in the Developer menu.
• Ideally, connect a device running each os version your app supports. iOS is generally pretty good at backwards compatibility of os versions.
Real devices…
![Page 39: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/39.jpg)
Parallel iOS
![Page 40: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/40.jpg)
iOS Report
![Page 41: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/41.jpg)
Wunderlist Mobile Grid
![Page 42: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/42.jpg)
• Mac Pro - 3.5 GHz 6-Core - 32 GB Ram.
• OSX Server with performance mode enabled.
• Two Anker 9 Port USB Hubs.
• Energenie programmable power strip. (Reboots USB hubs, router, Mac)
• Blink(1) programmable usb light to display CI status.
• Jenkins - builds and runs smoke tests on every commit.
CI Machine Specs
![Page 43: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/43.jpg)
![Page 44: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/44.jpg)
![Page 45: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/45.jpg)
That is great but reporting is key!
![Page 46: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/46.jpg)
Challenges• Adb disconnects.
• I programmatically rebooting USB hubs before every test run. This reconnects adb.
• WiFi issues.
• I toggle airplane mode on/off prior to every test run.
• Other Possible Solutions:
• Reverse USB tethering. (requires rooting.)
• Using multiple routers on different broadcasting channels.
• Alternatives to real devices:
• Android Studio emulators.
• Cannot video record. *I will open a source a solution to this.
• A big plus IMO is that you have access your apps files (database) without rooting.
• Can use up a significant amount of memory depending on specs and the number running.
• Using Genymotion emulators. It’s the closest vm software to a real device.
• Has video recording but no API or CLI to start the recording. :/
• Some devices just behave erratically. (random reboots, adb connection issues, wonky text output, battery issues).
![Page 47: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/47.jpg)
Things to be excited about!• A replacement for Apple's xcodebuild that makes it easier to build and test iOS or OSX apps.
• https://github.com/facebook/xctool
• WebDriverAgent is a WebDriver server for iOS that runs inside the Simulator and is written entirely in Objective-C. Can run multiple iOS simulators in parallel.
• https://github.com/facebook/webdriveragent
• Run iOS tests on multiple simulators in parallel at the same time.
• https://github.com/plu/parallel_ios_tests
• Appium iOS driver, backed by Apple XCUITest.
• https://github.com/appium/appium-xcuitest-driver
• Remote control all your Smartphone devices From the comfort of your browser.
• http://openstf.io/
![Page 48: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/48.jpg)
![Page 49: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/49.jpg)
Lastly, some thank you’s
• Thank you to the Selenium open source developers and the community whom make WebDriver happen.
• Also Dan Cuellar, the creator of Appium, whom is here at the conference with us.
• If you see Dan, please give him a big bearhug! He’ll love it!
• Without Dan, people like him, and all the open source developers we would never have these amazing tools to help do our jobs more efficiently, and for free.
![Page 50: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/50.jpg)
Thank You!
• email: [email protected]
• twitter: @isonic1
• https://github.com/isonic1/appium-mobile-grid
![Page 51: Selenium Camp 2016 - Kiev, Ukraine](https://reader033.vdocuments.mx/reader033/viewer/2022061307/587271f01a28ab31498b6c1d/html5/thumbnails/51.jpg)
• Android Distributed
• https://www.dropbox.com/s/eidrtd71dbmv7uh/android-distributed.mov?dl=0
• https://www.dropbox.com/s/gildgq9oossdp4k/android-dist-report.mov?dl=0
• Android Parallel
• https://www.dropbox.com/s/fjla64pggg3cbr6/android-parallel.mov?dl=0
• https://www.dropbox.com/s/14ow5q2joikne1s/android-parallel-report.mov?dl=0
• Android Sauce Labs
• https://www.dropbox.com/s/n8gx9lsoxrn77te/android-saucelabs.mov?dl=0
• https://www.dropbox.com/s/4umdim23cvheevj/android-saucelabs-report.mov?dl=0
• iOS Parallel
• https://www.dropbox.com/s/161azld3lq0lc2f/ios_parallel.mov?dl=0
• https://www.dropbox.com/s/pxlzxx2ljl6q1wd/ios_allure_report.mov?dl=0
• Wunderlist Mobile Grid
• https://www.dropbox.com/s/y5xsmnxfldguzr0/grid-trimmed.mov?dl=0
• https://www.dropbox.com/s/675nonhixzq0ct7/emulators.mov?dl=0
• https://www.dropbox.com/s/a8g0erkabpa5zny/signup_failure.mov?dl=0
Slide Videos