sauce labs beta program overview
TRANSCRIPT
![Page 2: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/2.jpg)
DimDim tip
04/15/23 Sauce Labs Confidential 2
Choose Fit Screen
![Page 3: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/3.jpg)
The Problem
Many, many functional tests…
04/15/23 Sauce Labs Confidential 3
Hundreds of test flows- Happy path- Corner cases - Negative tests- Before check-in, after check-in
![Page 4: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/4.jpg)
Dozens of test platforms
Hundreds of test flows- Happy path- Corner cases - Negative tests- Before check-in, after check-in
… and test platforms
04/15/23 Sauce Labs Confidential 4
6, 7 (and soon 8)
9.5 (and soon 9.6)
1.0
3.0, 3.1 (and soon 4)
2, 3 (and soon 3.1)
Add-ons
Browser configurations
Languages/locales
![Page 5: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/5.jpg)
… and test data
04/15/23 Sauce Labs Confidential 5
Lots of test data Correct data, Incorrect data, Missing data, Data overflow
Dozens of test platforms
6, 7 (and soon 8)
9.5 (and soon 9.6)
1.0
3.0, 3.1 (and soon 4)
2, 3 (and soon 3.1)
Add-ons
Browser configs
Languages/locales
![Page 6: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/6.jpg)
…run on minimal test hardware.
04/15/23 Sauce Labs Confidential 6
Test jobs In-house test lab
Minutes
10 20 30 40 50 60 70 80 90
Test results
Can’t run all scripts + browsers
Can’t run all scripts + browsers
Test lab ratholeTest lab rathole Too long to find and fix bugs
Too long to find and fix bugs
![Page 7: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/7.jpg)
How many tests?
Which browsers/OS combos?
How much data?
04/15/23 Sauce Labs Confidential 7
![Page 8: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/8.jpg)
Concurrent Test Execution
04/15/23 Sauce Labs Confidential 8
Minutes
0 5
Test jobs
2000 nodes Test results
Run all scripts + browsers
Run all scripts + browsers
Find and fix bugs fast
Find and fix bugs fast
No configuration
needed
No configuration
needed
![Page 9: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/9.jpg)
How many test machines?
How long to run today?
How long, ideally?
How often run today?
How often, ideally?
04/15/23 Sauce Labs Confidential 9
![Page 10: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/10.jpg)
User Experience
04/15/23 Sauce Labs Confidential 10
Agile developer / tester
Source control
1 Check in code & test scripts
5 Run scripts
HTTP(s)
7 View results
Web, RSS
REST
8
Deletescripts & results
6Download results
REST
4 Upload scripts
Sauce LabsSauce Labs
FTP
App Under Test
App Under Test
3 Promote new build to cloud-accessible location
(Your infrastructure)
Start new build
Polling or trigger
Build System
2
cronrake
![Page 11: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/11.jpg)
How does this map to your workflow?
Which CI or build system?
Build script language?
Test script language?
04/15/23 Sauce Labs Confidential 11
![Page 12: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/12.jpg)
What’s in our cloud?
04/15/23 Sauce Labs Confidential 12
Sauce LabsSauce Labs
ScriptsSelenium test scriptsWritten in Python, Ruby (now)Selenese, PHP, Java, C#, Perl (future)
ConfigsWhich browser, addons, & OS thata job should run on
JobsSimilar to a print jobwaiting, running, done…
ResultsOutput of ScriptStdout, stderr, and more…
UsersCan access certain Scripts, Configs, Jobs, and Results
![Page 13: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/13.jpg)
What’s in our cloud?
04/15/23 Sauce Labs Confidential 13
Sauce LabsSauce Labs
Waiters (small number)
Listen to API requests
Dispatch Job execution to Chefs
Handle other requests directly
Chefs (many, up to 2000)
Selenium RC + browser
Run Job and return Results
![Page 14: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/14.jpg)
What’s the app under test?
Internet-accessible?
Max # of Selenium RCs?
04/15/23 Sauce Labs Confidential 14
![Page 15: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/15.jpg)
04/15/23 Sauce Labs Confidential 15
![Page 16: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/16.jpg)
require 'rubygems'require 'couchrest' # http://github.com/jchris/couchrest
# Link to user's databasedb = CouchRest.database("http://jdoe.saucelabs.com")
# Upload a script# -----------------------------------------------------------------------# A SauceLabs script is simply a regular ruby script wrapped in JSONresponse = db.save( { "Type" => "script", "Name" => "Google Search", "Language" => "ruby", "Content" => "open ('google.com')\ntype ('q', 'selenium')\
nclick'btnG')\nwaitForPageToLoad()\nverifyTextPresent('openqa.org')" })
script_id = response['id’]
04/15/23 Sauce Labs Confidential 16
4 Upload script
![Page 17: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/17.jpg)
# Run a job
# ------------------------------------------------------------------
# You run a job by uploading a new "job" document, linking to the
# script id.
# A job is instantly queued for execution upon upload.
response = db.save(
{ "Type" => "job",
"ScriptId" => script_id,
})
job_id = response['id’]
04/15/23 Sauce Labs Confidential 17
5 Run job
![Page 18: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/18.jpg)
# Check job status# ----------------------------------------------------------------------------status = db.get(job_id)['Status']
# 'status' will be one of:# "new"# "queued"# "in progress"# "complete"# "error”
# Download result# ----------------------------------------------------------------------------# Once the job is complete, a "result" document is created. The "job" document# is updated with a reference to the result.
# First, retrieve the ResultId from the jobresult_id = db.get(job_id)['ResultId']
# Second, retrieve the complete result document:result = db.get(result_id)
# You can now introspect the results# View standard outputstandard_output = result['Stdout']
# View standard errorstandard_error = result['Stderr’]
04/15/23 Sauce Labs Confidential 18
6 Download result
![Page 19: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/19.jpg)
# Delete result
# ------------------------------------------------------------------
result = db.get(result_id)
db.delete(result)
# Delete script
# ------------------------------------------------------------------
script = db.get(script_id)
db.delete(script)
04/15/23 Sauce Labs Confidential 19
8 Delete result and script
![Page 20: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/20.jpg)
What we’ll do for you
Provide free access to product
Provide coaching on:
How to integrate our service into your build scripts
How to parallelize your test scripts
Demonstrate prototypes for your feedback
Notify you of downtime and updates (planned or not)
Take precautions to ensure that your information remains secure
04/15/23 Sauce Labs Confidential 20
![Page 21: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/21.jpg)
What we need from you
Information about your existing testing environment
Perform actual integration of Sauce Labs into your test environment
Use Sauce Labs' product in your testing process
Report any bugs or feature shortcomings you find
Assign a primary point of contact
Participate in weekly check-in calls
Provide a product review (good or bad) that we may publish
Speak with potential Sauce Labs investors about your experiences
04/15/23 Sauce Labs Confidential 21
![Page 22: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/22.jpg)
Next steps
04/15/23 Sauce Labs Confidential 22
![Page 23: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/23.jpg)
xUnit, TestNGFit(nesse), RSpec
HudsonContinuumCruiseControlTeamCityBambooGit(hub)SubversionCronEclipse
Internet ExplorerFirefox SafariChromeOperaFlash/FlexiPhoneAndroid
Selenium outputScreencastsScreenshots
DOMshotsJavaScript state
YslowHTML errorsAccessibilityHTTP traffic
Dtrace… all tracked over time
PythonRubyJavaPerl
SeleneseC#
PHPSelenium on Rails
Where do we go from here?
04/15/23 Sauce Labs Confidential 23
Platforms Languages
Build Systems Telemetry
Frameworks
![Page 24: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/24.jpg)
# CouchDB library from http://code.google.com/p/couchdb-python/
from couchdb import Server
# Link to user's database
server = Server('http://sauceapp.com/')
db = server['jsmith’]
# Upload a script
# ----------------------------------------------------------------------
# A Sauce Labs script is simply a regular python script wrapped in JSON
script_id = db.create({
"Type" : "script",
"Name" : "Google Search",
"Language" : "python",
"Content" : "open ('google.com')\ntype ('q', 'selenium')\nclick'btnG')\nwaitForPageToLoad()\nverifyTextPresent('openqa.org')"
})
04/15/23 Sauce Labs Confidential 24
4 Upload script
![Page 25: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/25.jpg)
# Run a job
# ----------------------------------------------------------
# You run a job by uploading a new "job" document, linking to the script id.
# A job is instantly queued for execution upon upload.
job_id = db.create(
{ "Type" :"job",
"ScriptId" :script_id,
})
04/15/23 Sauce Labs Confidential 25
5 Run job
![Page 26: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/26.jpg)
# Check job status# --------------------------------------------------------------------------status = db[job_id]["Status"]
# 'status' will be one of:# "new"# "queued"# "in progress"# "complete"# "error”
# Download result# --------------------------------------------------------------------------# Once the job is complete, a "result" document is created. The "job" document# is updated with a reference to the result.
# First, retrieve the ResultId from the jobresult_id = db[job_id]['ResultId']
# Second, retrieve the complete result documentresult = db[result_id]
# You can now introspect the results.# View standard outputstandard_output = result['Stdout']
# View standard errorstandard_error = result['Stderr']
04/15/23 Sauce Labs Confidential 26
6 Download result
![Page 27: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/27.jpg)
# Delete result
# -----------------------------------------------------------------
del db[result_id]
# Delete script
# -----------------------------------------------------------------
del db[script_id]
04/15/23 Sauce Labs Confidential 27
8 Delete result and script
![Page 28: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/28.jpg)
# Using CouchDB library by Michael Hendricks (not in CPAN, yet!)# http://git.ndrix.com/?p=Net-CouchDB use Net::CouchDB; # Link to user's databasemy $server = Net::CouchDB->new('http://localhost:5984');my $db = $server->db('perltest'); # Upload a script# --------------------------------------------------------------------------# A Sauce Labs script is simply a regular perl script wrapped in JSONmy $script = $db->insert( { Type => "script", Name => "Google Search", Langage => "perl", Content => "open ('google.com')\ntype ('q', 'selenium')\nclick'btnG')\
nwaitForPageToLoad()\nverifyTextPresent('openqa.org')" }); my $script_id = $script->id;
04/15/23 Sauce Labs Confidential 28
4 Upload script
![Page 29: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/29.jpg)
# Run a job
# ----------------------------------------------------------
# You run a job by uploading a new "job" document, linking to
# the script id.
# A job is instantly queued for execution upon upload.
my $job = $db->insert(
{ Type => "job",
ScriptId => $script_id
});
my $job_id = $job->id;
04/15/23 Sauce Labs Confidential 29
5 Run job
![Page 30: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/30.jpg)
# Check job status# --------------------------------------------------------------------------my $status = $db->document($job_id)->{'Status'}; # 'status' will be one of:# "new"# "queued"# "in progress"# "complete"# "error” # Download result# --------------------------------------------------------------------------# Once the job is complete, a "result" document is created. The "job" document# is updated with a reference to the result. # First, retrieve the ResultId from the jobmy $result_id = $db->document($job_id)->{'ResultId'}; # Second, retrieve the complete result document:my $result = $db->document($result_id); # You can now introspect the results.# View standard outputmy $standard_output = result->{'Stdout'}; # View standard errormy $standard_error = result->{'Stderr'};
04/15/23 Sauce Labs Confidential 30
6 Download result
![Page 31: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/31.jpg)
# Delete result
# ------------------------------------------------------------------
my $result = $db->document($result_id);
$result->delete;
# Delete script
# ------------------------------------------------------------------
my $script = $db->document($script_id);
$script->delete;
04/15/23 Sauce Labs Confidential 31
8 Delete result and script
![Page 32: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/32.jpg)
Curl command:
curl -X POST -u jdoe:password -d '
{
"Type":"script",
"Name":"Google Search",
"Language":"python",
"Content":"open ('google.com')\ntype ('q', 'selenium')\nclick'btnG')\nwaitForPageToLoad()\nverifyTextPresent('openqa.org')",
"CreatedBy":"jdoe",
"CreatedDate":"Mon, 18 Aug 2008 14:26:10 +0000",
"LastModifiedBy":"jdoe",
"LastModifiedDate":"Mon, 18 Aug 2008 14:26:10 +0000"
}' http://saucedb.com/jdoe/ 04/15/23 Sauce Labs Confidential 32
4 Upload script
![Page 33: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/33.jpg)
Curl returns:
{"ok":true,"id":"4749f6c4159ae5a097e586f3e5040cbe","rev":"668367075"}
To view in browser: (You'll probably need to provide name/password)
http://jdoe.sauceapp.com/4749f6c4159ae5a097e586f3e5040cbe
04/15/23 Sauce Labs Confidential 33
4 Upload script
![Page 34: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/34.jpg)
Curl command:
curl -X POST -u jdoe:password -d '
{
"Type":"job",
"Status":"new",
"ScriptId":"3ece08113906d84325f9341e794caefa",
"ResultId":null,
"CreatedBy":"jdoe",
"CreatedDate":"Mon, 18 Aug 2008 14:54:55 +0000"
}' http://saucedb.com/jdoe/
04/15/23 Sauce Labs Confidential 34
5 Run job
![Page 35: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/35.jpg)
Curl returns:
{"ok":true,"id":"3c4c1f2f21ea4e346aebc244f21dde76","rev":"1359430379"}
To view in browser: (You'll probably need to provide name/password)
http://jdoe.sauceapp.com/3c4c1f2f21ea4e346aebc244f21dde76
04/15/23 Sauce Labs Confidential 35
5 Run job
![Page 36: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/36.jpg)
Partnering with Sauce Labs
• Sauce focuses on hosted testing
• Looking for independent consultants, and consulting firms, who use our service during their engagements
• Guru pricing, training, etc. available
04/15/23 Sauce Labs Confidential 36
![Page 37: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/37.jpg)
Agile is now mainstream
Source: Dr. Dobb’s Journal. 5/7/08. http://bit.ly/4EjcDH
04/15/23 37Sauce Labs Confidential
![Page 38: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/38.jpg)
Continuous Integration is now mainstream
04/15/23 Sauce Labs Confidential 38
Source:VersionOne. 8/22/08. http://www.versionone.com/pdf/3rdAnnualStateOfAgile_FullDataReport.pdf
![Page 39: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/39.jpg)
Functional tests tend to run at end of day
04/15/23 Sauce Labs Confidential 39
Source: Viewtier Systems. http://bit.ly/2qQ5LX
![Page 40: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/40.jpg)
Our current thinking about pricing(Subject to change)
04/15/23 Sauce Labs Confidential 40
$ per test minute
Tests queued no later than…
Five minutes
Express: Relatively high per test minute
8 hours
Economy: Low amount per test minute
![Page 41: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/41.jpg)
Economy
Chef #
minute
Express entails lower chef utilization(We’ll need beta usage to quantify)
04/15/23 Sauce Labs Confidential 41
Expressscript 1 script 6 script 10
script 2 script 7 script 11 script 12
script 3 script 8 script 13
script 4 script 9 script 14
script 5
10 20 30 40 50 60
1
2
3
4
5
Chef #
10 20 30 40 50 60
script 1 script 2 script 3 script 4 script 5 script 61
minute
![Page 42: Sauce Labs Beta Program Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062308/55bec368bb61eb095c8b458a/html5/thumbnails/42.jpg)
Thinking about pricing the right way?
How much Express would you use?
How much Economy?
04/15/23 Sauce Labs Confidential 42