Download - Yelp Tech Talks: Mobile Testing 1, 2, 3
@YelpEngineering
YelpEngineers
engineeringblog.yelp.com
github.com/yelpyelp.com/careers
Building Yelp for Apple WatchBill Meltsner
[email protected]@billmeltsner
Today
Initial Scoping / Planning
Yelp.app on Apple Watch Deep Dive
Lessons Learned
Who I Am
iOS Technical Lead
Yelp on Apple Watch project lead
Worked largely on Watch app logic
Who We Were
Designer Engineer Product Manager Engineer (Intern)
Why Build Yelp Watch App?
Yelp everywhere
Day-one advantage
Knew we could build a great experience
Initial Scoping
time = features
Initial Scoping
unlimited time* = unlimited features
*this never happens
Initial Scoping
finite time = finite features
Initial Scoping
Product team defined features
Engineering team estimated available time
Worked together to define MVP as cost of time versus feature
Initial Scoping
UI & Logic could be parallelized- 1 engineer for each
Milestones for MVP, MVP+1, MVP+2, …
Flexibility in our schedule to add / remove features and stay agile
Demo
Yelp Watch AppTechnical Overview
WatchKit - A Changing Landscape
Brand new platform with docs & APIs changing drastically between betas
No defined best practices
Focused on making best-effort technical decisions with the ability to refine later
Overview of a WatchKit app
Parent App
API Requests
Watch App
Storyboard
WatchKit Extension
Location
Logic
Interface Control
Images
iPhone Apple Watch
Interface Controllers
View Controller analog
Interface hierarchy is fixed
YPWKSearchResultsInterfaceController
Storyboard Overview
Before After
Networking
API requests owned by parent app
Image loading owned by extension
Location
Owned by extension – runs in foreground, parent app runs in background
We only request foreground access
Location
Permissions belong to parent app, must be granted on phone
Phone ↔ Watch Communications
All calls in one iteration of the run loop coalesced together
Communication between watch and phone is rate-limited serial queue
Overhead is high – batch your calls!
Images
Key part of our search UI
Naive approach: send each image to the watch as it’s loaded
Result: traffic jam of communications, unresponsive app
Images
Solution: Wait x seconds, send all images loaded in that timeframe at once
Problem: that can be a lot of data
Solution Part 2: crush the heck out of ‘emUIImageJPEGRepresentation(image, 0.0) // max
compression
Lessons Learned
Think like a Startup
Priority 1 was being there on launch day
MVP comes first, everything else can wait
Technical debt is not inherently bad
Plan Ahead
Define designs and scope before writing any code
New platforms are hard to predict effectively
Bend but don’t break
Questions?
Testing The Yelp AppiOS & Android
Who We Are
Mason Glidden● iOS Engineer● iOS Testing
o KIF & Jenkins● [email protected]
Tim Mellor● Android Engineer● Android Testing
o Espresso & Jenkins
How we develop new mobile APIs
iOS & Android - Tests & Testing Strategy
Today
Building New Mobile APIsHow we use our documentation to test new APIs
Mobile APIs @ Yelp
● API shared by iOS & Android● New APIs start with documentation and
examples● Client and API can be developed simultaneously● API team manages backwards compatibility tests
/*h2. Photo (full)
|_. Name |_. Type |_. Description || id | string | Identifier || time_created | time | Timestamp for when photo was uploaded || url_prefix | string | Prefix for image url || user_passport | "Passport":{{site.url}}/v1/objects/passport/index.html | Passport for user who uploaded photo (not provided for photos added by biz owners to their own biz from their biz owner account) (Optional) || caption | string | Caption || feedback_positive_count | integer | Number of likes for the photo ||*/{ "id": "PHOTO_ID123", "time_created": 1370985832, "user_passport": {% include_jsondoc "v1/objects/passport/default.json" Passport %}, "url_prefix": "http://s3-media4.ak.yelpcdn.com/bphoto/_Xwa-lWTpivnlB1tgX-sJw/", "caption": "Korean tacos ($5.75)", "feedback_positive_count": 19,}
Documentation
/*h2. Photo (full)
|_. Name |_. Type |_. Description || id | string | Identifier || time_created | time | Timestamp for when photo was uploaded || url_prefix | string | Prefix for image url || user_passport | "Passport":{{site.url}}/v1/objects/passport/index.html | Passport for user who uploaded photo (not provided for photos added by biz owners to their own biz from their biz owner account) (Optional) || caption | string | Caption || feedback_positive_count | integer | Number of likes for the photo ||*/{ "id": "PHOTO_ID123", "time_created": 1370985832, "user_passport": {% include_jsondoc "v1/objects/passport/default.json" Passport %}, "url_prefix": "http://s3-media4.ak.yelpcdn.com/bphoto/_Xwa-lWTpivnlB1tgX-sJw/", "caption": "Korean tacos ($5.75)", "feedback_positive_count": 19,}
Documentation
Textile
/*h2. Photo (full)
|_. Name |_. Type |_. Description || id | string | Identifier || time_created | time | Timestamp for when photo was uploaded || url_prefix | string | Prefix for image url || user_passport | "Passport":{{site.url}}/v1/objects/passport/index.html | Passport for user who uploaded photo (not provided for photos added by biz owners to their own biz from their biz owner account) (Optional) || caption | string | Caption || feedback_positive_count | integer | Number of likes for the photo ||*/{ "id": "PHOTO_ID123", "time_created": 1370985832, "user_passport": {% include_jsondoc "v1/objects/passport/default.json" Passport %}, "url_prefix": "http://s3-media4.ak.yelpcdn.com/bphoto/_Xwa-lWTpivnlB1tgX-sJw/", "caption": "Korean tacos ($5.75)", "feedback_positive_count": 19,}
Documentation
JSONDoc
Textile
Documentation/*h2. Photo (full)
|_. Name |_. Type |_. Description || id | string | Identifier || time_created | time | Timestamp for when photo was uploaded || url_prefix | string | Prefix for image url || user_passport | "Passport":{{site.url}}/v1/objects/passport/index.html | Passport for user who uploaded photo (not provided for photos added by biz owners to their own biz from their biz owner account) (Optional) || caption | string | Caption || feedback_positive_count | integer | Number of likes for the photo ||*/{ "id": "PHOTO_ID123", "time_created": 1370985832, "user_passport": {% include_jsondoc "v1/objects/passport/default.json" Passport %}, "url_prefix": "http://s3-media4.ak.yelpcdn.com/bphoto/_Xwa-lWTpivnlB1tgX-sJw/", "caption": "Korean tacos ($5.75)", "feedback_positive_count": 19,}
Documentation -> JSON
● Included as submodule in client repos● Build step to flatten documentation into JSON
(e.g. v1--objects--photo+full.json)● Code requests specific mocks
Why This Approach Works for Us
● API & client contract● Fewer dependencies for developers & Jenkins● Improved test speed & reliability on iOS &
Android
iOS Testing @ Yelp Mason Glidden
● Prevent Regressions● Give developers confidence● Run quickly● Reliable results● Easy to write
Test Goals
● Unit Tests● Integration Tests● Acceptance Tests
Test Types
● Prevent UI & Logic Regressions● ~150 logic unit tests● ~100 network request contract
tests● ~650 view tests● Continuous Integration on Jenkins
Unit Tests
● Generally pretty simple● Test-Driven Development● Super fast to run
Logic Tests
Example: Business Hours Logic- (void)testOpensSoon { // Test that "Opens soon" appears with the correct time interval [NSDate yk_setDate:[NSDate dateWithTimeIntervalSince1970:1356040364]]; NSArray *openHoursArray = @[@[@5160, @5460]]; OpenHours *openHours = [OpenHours openHoursFromJSON:openHoursArray timeZoneString:@"America/Los_Angeles"]; STAssertEqualObjects(@"Opens in 8 min", [openHours openOrClosedStringUsingMinutes:YES], nil);}
Logic Tests
● Makes sure client can still parse documented API changes
● Example: ReviewsListRequestTest- (void)testList { ReviewsListRequest *request = [[ReviewsListRequest alloc] init]; [OHHTTPStubs yp_receiveFromPath:@"v1--reviews+reviews.json" statusCode:200 MIMEType:@"application/json" afterDelay:0.1]; [request listWithBusinessId:@"BIZID" selectedReviewId:nil offset:0 limit:10 delegate:self]; [self waitForStatus:YPAsyncTestWaitStatusSuccess timeout:10.0 requestToCancelOnTimeout:request];}
Parsing Tests
View Tests
● View with mock data
● Screenshot of view● Compares with
previous versions● Based off GHUnit
View Tests
View Tests
● Example: contribution buttons view test- (void)testBasicButtons { Business *business = [Business businessFromJSONDictionary: [YPDebug JSONFromResource:@"v1--objects--business+full.json"] request:nil context:nil]; YPBusinessContributeButtons *buttons = [[YPBusinessContributeButtons alloc] init]; [buttons setBusiness:business]; YPVerifyView(buttons);}
View Tests
● Pros:○ Easy way to catch regressions○ Invaluable when refactoring or updating to
new OS versions● Cons:
○ Slow: ~¾ seconds per test○ Lots of false-positive failures
Integration Tests
● Testing that application behaves as expected● Interaction between view controllers● Primary signals of a problem:
○ Non-visual - analytics & network requests○ Visual - button or label
● ~225 Integration tests
KIF
● ~150 KIF tests● Uses accessibility labels to navigate● Custom hooks for analytics,
requests● Continuous integration on Jenkins● Separate iPad and iPhone tests
github.com/kif-framework/KIF
Integration Test Example
Integration Test Example● Example: ReviewCompositionIntegrationTest
- (void)testReviewWrite { [YPAPI yp_addMockSessionConfirmed:YES sendNotification:YES]; [self yp_openBusinessViewController];
[tester tapViewWithAccessibilityLabel:@"Write a Review"]; [tester yp_waitForModalViewControllerOfClass:[ReviewComposeViewController class]]; [tester yp_waitForExpectedAnalytics:@[@{@"iri": kAnalyticsReviewWriteIRI, @"params": @{@"intended_compose_type": @"add"}]];
[tester tapViewWithAccessibilityLabel:@"Rating"]; [tester clearTextFromAndThenEnterText:@"My review" intoViewWithAccessibilityLabel:@"Write your review here."]; [tester tapViewWithAccessibilityLabel:@"Post"]; [tester yp_waitForRequestWithPath:@"/review/save" queryParams:nil postParams:@{ @"text": @"My review", @"rating": @"3" }];}
Integration Test Example● Example: ReviewCompositionIntegrationTest
- (void)testReviewWrite { [YPAPI yp_addMockSessionConfirmed:YES sendNotification:YES]; [self yp_openBusinessViewController];
[tester tapViewWithAccessibilityLabel:@"Write a Review"]; [tester yp_waitForModalViewControllerOfClass:[ReviewComposeViewController class]]; [tester yp_waitForExpectedAnalytics:@[@{@"iri": kAnalyticsReviewWriteIRI, @"params": @{@"intended_compose_type": @"add"}]];
[tester tapViewWithAccessibilityLabel:@"Rating"]; [tester clearTextFromAndThenEnterText:@"My review" intoViewWithAccessibilityLabel:@"Write your review here."]; [tester tapViewWithAccessibilityLabel:@"Post"]; [tester yp_waitForRequestWithPath:@"/review/save" queryParams:nil postParams:@{ @"text": @"My review", @"rating": @"3" }];}
Integration Test Example● Example: ReviewCompositionIntegrationTest
- (void)testReviewWrite { [YPAPI yp_addMockSessionConfirmed:YES sendNotification:YES]; [self yp_openBusinessViewController];
[tester tapViewWithAccessibilityLabel:@"Write a Review"]; [tester yp_waitForModalViewControllerOfClass:[ReviewComposeViewController class]]; [tester yp_waitForExpectedAnalytics:@[@{@"iri": kAnalyticsReviewWriteIRI, @"params": @{@"intended_compose_type": @"add"}]];
[tester tapViewWithAccessibilityLabel:@"Rating"]; [tester clearTextFromAndThenEnterText:@"My review" intoViewWithAccessibilityLabel:@"Write your review here."]; [tester tapViewWithAccessibilityLabel:@"Post"]; [tester yp_waitForRequestWithPath:@"/review/save" queryParams:nil postParams:@{ @"text": @"My review", @"rating": @"3" }];}
Integration Test Example● Example: ReviewCompositionIntegrationTest
- (void)testReviewWrite { [YPAPI yp_addMockSessionConfirmed:YES sendNotification:YES]; [self yp_openBusinessViewController];
[tester tapViewWithAccessibilityLabel:@"Write a Review"]; [tester yp_waitForModalViewControllerOfClass:[ReviewComposeViewController class]]; [tester yp_waitForExpectedAnalytics:@[@{@"iri": kAnalyticsReviewWriteIRI, @"params": @{@"intended_compose_type": @"add"}]];
[tester tapViewWithAccessibilityLabel:@"Rating"]; [tester clearTextFromAndThenEnterText:@"My review" intoViewWithAccessibilityLabel:@"Write your review here."]; [tester tapViewWithAccessibilityLabel:@"Post"]; [tester yp_waitForRequestWithPath:@"/review/save" queryParams:nil postParams:@{ @"text": @"My review", @"rating": @"3" }];}
Sandboxing
Mocked During Tests:● Networking● Date, Time, Timezone● Device permissions● Singletons
Between Test Runs:● Clean caches● Reset user defaults● Reset navigation stack● Device orientation
Other Tooling
● OHHTTPStubs to block & mock network requests
● OCMock for mocking● XCTool to run our tests
Acceptance Tests
● Test overall look and feel● ~50 manual test cases
○ Moving some to KIF● iOS7 & 8, iPad & iPhone● Run by Engineers + PM during release
process
Closing Thoughts
● API mocks make it easy for us to reliably grow our testing suite
● Different types of tests for different problems● Sandboxes to create consistent environments● KIF <3
Android Testing @ YelpTim Mellor
tests = tools + code
Simple, right?
● AndroidTestCase● InstrumentationTestCase● ApplicationTestCase● ActivityTestCase● ActivityUnitTestCase● ActivityInstrumentationTestCase● ActivityInstrumentationTestCase2
More decisions
Problem: Devices
● Devices are necessary
● Devices suck● Virtual devices
are bearable
Solution: Devices
● Genymotion’s gmtool● Clone image into new device● Speed of Genymotion
$ python gmtool_wrapper.py start \ --vms '{"18":1, "19":1, "21":1}'
Problem: Flakes!
● Part 1: Instrumentation + Device● Part 2: Test library
Android Instrumentation
Instrumentation consequences
● Uncaught exceptions halt test suite● Activities/Services/etc. stay open
Solution: Instrumentation Flakes
● One test per instrumentation run!$ adb shell pm clear com.yelp.droid
Flakiness and Test libraries
Robotium and its solo.waitFor* methods
Android test kit to the rescue!
Main Thread
click()
Espresso
blocked
Main Thread
Test thread
assertionsTest
click()
Espresso
blocked
Main Thread
Test thread
assertionsTest
Task ThreadBackground task
Problem: slow test suites
● Consequence of needing devices● Long = impractical
Solution: test sharding
$ adb shell am instrument -w \ -e numShards 4 \ -e shardIndex 1● github.com/shazam/fork● Resources are the limit!
Yelp Testing Process
● ~300 unit tests● ~100 integration tests● ~150 UI integration tests● Manual testing against production● Beta group● 50% roll-out in Play Store
UI Integration test toolkit @ Yelp
● Espresso!● Home-rolled MockHttpClient
o MockResponseo MockRequestMatcher
● Spoon
public void test_ClickingBookmark_SendsAddRequestAndUpdatesUi() { mBusiness.setBookmarked(false); setActivityIntent(ActivityBusinessPage.intentForBusiness(getYelpContext(), mBusiness)); getActivity(); takeScreenshot("business detail unbookmarked");
// Check that the bookmark button has the text “Bookmark” and click on it // This should trigger a bookmark add request. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.action_bookmark)))) .perform(click()); takeScreenshot("bookmarked");
// Make sure we hit bookmarks/add with the proper params. assertThat(mAddBookmarkResponse.getRequestCount(), is(1)); // Make sure the "Bookmarked" button now shows. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.bookmarked))));}
public void test_ClickingBookmark_SendsAddRequestAndUpdatesUi() { mBusiness.setBookmarked(false); setActivityIntent(ActivityBusinessPage.intentForBusiness(getYelpContext(), mBusiness)); getActivity(); takeScreenshot("business detail unbookmarked");
// Check that the bookmark button has the text “Bookmark” and click on it // This should trigger a bookmark add request. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.action_bookmark)))) .perform(click()); takeScreenshot("bookmarked");
// Make sure we hit bookmarks/add with the proper params. assertThat(mAddBookmarkResponse.getRequestCount(), is(1)); // Make sure the "Bookmarked" button now shows. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.bookmarked))));}
public void test_ClickingBookmark_SendsAddRequestAndUpdatesUi() { mBusiness.setBookmarked(false); setActivityIntent(ActivityBusinessPage.intentForBusiness(getYelpContext(), mBusiness)); getActivity(); takeScreenshot("business detail unbookmarked");
// Check that the bookmark button has the text “Bookmark” and click on it // This should trigger a bookmark add request. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.action_bookmark)))) .perform(click()); takeScreenshot("bookmarked");
// Make sure we hit bookmarks/add with the proper params. assertThat(mAddBookmarkResponse.getRequestCount(), is(1)); // Make sure the "Bookmarked" button now shows. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.bookmarked))));}
public void test_ClickingBookmark_SendsAddRequestAndUpdatesUi() { mBusiness.setBookmarked(false); setActivityIntent(ActivityBusinessPage.intentForBusiness(getYelpContext(), mBusiness)); getActivity(); takeScreenshot("business detail unbookmarked");
// Check that the bookmark button has the text “Bookmark” and click on it // This should trigger a bookmark add request. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.action_bookmark)))) .perform(click()); takeScreenshot("bookmarked");
// Make sure we hit bookmarks/add with the proper params. assertThat(mAddBookmarkResponse.getRequestCount(), is(1)); // Make sure the "Bookmarked" button now shows. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.bookmarked))));}
public void test_WriteTip_SendsProperRequest() { setMockSession(); MockResponse tipSaveResponse = mMockHttpClient.addDefaultMock( ApiPath.QUICKTIPS_SAVE, getTipSaveParams());
openTipPage();
onView(withId(R.id.edit_text)).perform(typeText(TIP_TEXT)); takeScreenshot("tip typed");
onView(withId(R.id.done_button)).perform(click()); takeScreenshot("business page");
// We should show a "Thanks for the tip!" dialog on the business page. onView(withText(R.string.thanks_for_the_tip)).check(matches(isDisplayed())); assertThat(tipSaveResponse.getRequestCount(), is(1));}
public void test_WriteTip_SendsProperRequest() { setMockSession(); MockResponse tipSaveResponse = mMockHttpClient.addDefaultMock( ApiPath.QUICKTIPS_SAVE, getTipSaveParams());
openTipPage();
onView(withId(R.id.edit_text)).perform(typeText(TIP_TEXT)); takeScreenshot("tip typed");
onView(withId(R.id.done_button)).perform(click()); takeScreenshot("business page");
// We should show a "Thanks for the tip!" dialog on the business page. onView(withText(R.string.thanks_for_the_tip)).check(matches(isDisplayed())); assertThat(tipSaveResponse.getRequestCount(), is(1));}
public void test_WriteTip_SendsProperRequest() { setMockSession(); MockResponse tipSaveResponse = mMockHttpClient.addDefaultMock( ApiPath.QUICKTIPS_SAVE, getTipSaveParams());
openTipPage();
onView(withId(R.id.edit_text)).perform(typeText(TIP_TEXT)); takeScreenshot("tip typed");
onView(withId(R.id.done_button)).perform(click()); takeScreenshot("business page");
// We should show a "Thanks for the tip!" dialog on the business page. onView(withText(R.string.thanks_for_the_tip)).check(matches(isDisplayed())); assertThat(tipSaveResponse.getRequestCount(), is(1));}
public void test_WriteTip_SendsProperRequest() { setMockSession(); MockResponse tipSaveResponse = mMockHttpClient.addDefaultMock( ApiPath.QUICKTIPS_SAVE, getTipSaveParams());
openTipPage();
onView(withId(R.id.edit_text)).perform(typeText(TIP_TEXT)); takeScreenshot("tip typed");
onView(withId(R.id.done_button)).perform(click()); takeScreenshot("business page");
// We should show a "Thanks for the tip!" dialog on the business page. onView(withText(R.string.thanks_for_the_tip)).check(matches(isDisplayed())); assertThat(tipSaveResponse.getRequestCount(), is(1));}
● Library choices matter● Address the issues at the source!● Tests don’t have to be a pain
Lessons learned
Questions?