how salesforce built a scalable, world-class, performance engineering team
DESCRIPTION
TRANSCRIPT
![Page 1: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/1.jpg)
How Salesforce built a Scalable,
World-Class, Performance
Engineering Team
September 18th, 2012
Kasey Lee, Salesforce, VP Performance Engineering
in/leekasey
![Page 2: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/2.jpg)
Safe Harbor
Safe harbor statement under the Private Securities Litigation Reform Act of 1995:
This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if
any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-
looking statements we make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of
product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of
management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments
and customer contracts or use of our services.
The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our
service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth,
interruptions or delays in our Web hosting, breach of our security measures, the outcome of intellectual property and other l itigation, risks associated
with possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain,
and motivate our employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling
non-salesforce.com products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the
financial results of salesforce.com, inc. is included in our annual report on Form 10-Q for the most recent fiscal quarter ended July 31, 2012. This
documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site.
Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may
not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently
available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.
![Page 3: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/3.jpg)
Welcome! What brings you here?
![Page 4: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/4.jpg)
A. I’m curious how PerfEng can excel in an Agile Environment
B. I’m curious how to utilize a Performance Engineer's time
C. I’d like to understand how to better articulate the value of
Performance Engineering
D. I thought this was a great place to take a break and check my
social feeds before dinner
E. A, B, or C
F. All of the above
![Page 5: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/5.jpg)
What do typical Performance Teams start as?
“Performance Engineering is run as a Shared Services
model so your charter is the entire organization with
maximum visibility. Everything flows through PerfEng
because it’s so critical. Dev, QE, Technical
Operations, Level II and III Support, and Professional
Services wants the most out of your engineers by
leveraging your talent across projects to scale mission
critical applications”
![Page 6: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/6.jpg)
What it sounds like
PerfEng
![Page 7: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/7.jpg)
What it actually feels like ;)
PerfEng
![Page 8: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/8.jpg)
Top Ten Signs Your Team Needs Help 1. You laugh when asked to signoff at Feature Freeze (and Release Freeze)
2. Your engineers work on 6-12 parallel projects (others work on 1-2 projects serially)
3. If you attended each of your assigned scrum teams’ daily 15 min standup you’d
never sit down (the entire week)
4. When you can’t signoff on a feature, everyone wants to raise the goals instead of
fixing the performance problem
5. Every day you answer “How did you decide to prioritize my feature? How can I
escalate this?” (even after you had agreement)
6. You’re told to commit to a plan for the next release while your team is busiest in the
current release and has no time to plan
7. Your team wants to influence the product or hardware architecture but can’t find the
time to even write up their analysis
8. Developers discount poor results due to variance without looking at the data (even
though the results of the latest release are always worse)
9. IT always asks “Why do you need isolated labs? Dev and QA don’t need them”
10.Devs ask your engineers to do manual tasks at all hours
![Page 9: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/9.jpg)
That sounds like my
situation… How did
Salesforce approach this?
![Page 10: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/10.jpg)
What’s in store?
Introduction
The Unique Challenge at Salesforce
How the Team Scales
Workloads
Automation, Tools, Environments
Closing Thoughts and Tips
![Page 11: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/11.jpg)
Q: Who is Kasey?
![Page 12: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/12.jpg)
Brief Background
VP @ Salesforce
Performance Engineering
Sr. Director / Tech Lead @ Wily Technology
Performance Engineering, Software Tools, QA,
R&D Lab
Architect @ Event Zero
Developer, consultant for startups
Developer @ Ziff-Davis Benchmark Operation
Industry Standard Software Benchmarks
iBench, WebBench, ServerBench, NetBench
![Page 13: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/13.jpg)
What drew me to Salesforce?
• Performance and Scalability is one of the
top three core values of the company
• One of the most complex Enterprise
scalability challenges anywhere
• As of today one of the best funded teams in
the industry and growing as quickly as we
can find the best people
![Page 14: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/14.jpg)
What are some key challenges at Salesforce?
1. Mission Critical Enterprise Apps Customers pay for
No perf testing in production on unwary customers
No tolerance for downtime or slow response times which
immediately impact customers’ bottom line
2. Security is Paramount
Extremely difficult to access production systems / data
Can’t easily examine load and data shapes in detail
3. True Multi-Tennant Architecture
Every customer can create completely different load / data
characteristics at a moment’s notice
![Page 15: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/15.jpg)
Noteworthy Milestones
Mid 2006 – “System Test” Team created from HA crisis
April 2008 – Kasey Lee joins a struggling team of 7
Sept 2008 – Automation & Tools Team Created
Sept 2009 – Team averts162 R1 Load Balancer Disaster
Jan 2010 – Leads solution to Capacity Planning crisis
Sept 2010 – Team predicts GC Heap 168 R1 Regression
Sept 2010 – Team leads solutions to NA6 Perf
Nov 2011 – Team helps reduce production CPU >60%
May 2012 – Team Triages 178 R1 Bytecode Regression
June 2012 – Team size rises to 60
Jan 2013 - Target size: 80+
Traffic & complexity
continues to increase to
~60B / Quarter, but response
times have decreased!
![Page 16: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/16.jpg)
Major Accomplishments ex. – “CPU 15”
• SWAT team optimization /
tuning efforts saved the
company ~$150 Million dollars
• Optimizations include potential
to change the JVM spec
directly to benefit everyone
• Great example of ROI
Not only in dollars, but helps
build the credibility that you can
leverage to do even more
![Page 17: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/17.jpg)
Performance
Daily Dashboard
– 10/24/2011 –
RED!
![Page 18: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/18.jpg)
Performance
Daily Dashboard
– 11/15/2011 –
Look at all that
GREEN!
![Page 19: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/19.jpg)
How do we accomplish this?
• Baseline Functionality & Benchmarking
• New Feature Benchmarking
• Patches / Production Support
• Hardware / Infrastructure Analysis
• Special Studies / Research / POC
• Production Visualizations
• Capacity / Sizing Guides
• Architecture Expertise
• Profiling Concepts and Training
• Automation Frameworks
• Self Service Frameworks
• Data Analysis, Creation, Visualization Tools
• Load Generation Tools
• Environment Design
• Optimization
![Page 20: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/20.jpg)
What We Continually Focus On Blazing fast performance delivered by Cloud teams and PerfEng through
collaboration, innovation and transparency
Empowered and engaged PerfEng inspired by the real world impact of
their work and widely recognized as industry thought leaders
Quick and accurate test results, effective testing, seamless scheduling
and flexible environments
Frequent assessment, optimizations, and deep visibility into feature
performance during development and in production
Fully integrating PerfEng into product development as beneficial and
essential members of Cloud teams
Performance built in by Cloud teams and able to catch obvious
performance issues themselves
![Page 21: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/21.jpg)
What really makes us so effective?
1. Our Perf/Dev ratios have been adopted (after numerous “discussions” )
2. We have a Software Development Team
3. We have a Product Owner (Prod. Mgr) for our Labs
4. We have a dedicated TechOps team “PerfInfra” for Labs
5. We have a substantial lab for testing
6. We have a Program Manager focused on cross functional project strategy,
visibility, and communications
![Page 22: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/22.jpg)
Performance
• Sales/Service/Data
• Features
• Workloads
• Chatter
• Features
• Workloads
• Platform/Mobile/UI
• Features
• Workloads
• Core/Search/Analytics
• Features
• Workloads
Performance Engineering Team Structure
Automation & Tools & Env
• Software Tools Developers
• Environments
• Product Owner
• Special Projects Lead
Architect
Program Manager
![Page 23: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/23.jpg)
The Importance of Early Starts…
![Page 24: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/24.jpg)
PerfEng Historical Lag per Release
Product Development Sprints
Jan Feb Mar Apr May Jun
Planning Final Plans Due Feature Freeze
Release Sprint Release
Sandbox R1 Release
Freeze
Accumulated performance bug debt
Cost of
finding &
fixing bugs
PerfEng Begins Testing
![Page 25: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/25.jpg)
Why Start Early
and Profile
upstream?
Avoid obvious
problems earlier
![Page 26: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/26.jpg)
• Late starts with
minimal
workloads
• Increased
workloads and
decreased time
to bring online
• No longer need
to track
![Page 27: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/27.jpg)
PerfEng Starts vs. Release Timeline
Product Development Sprints
Jan Feb Mar Apr May Jun
Planning Final Plans Due Feature Freeze
Release Sprint Release
Sandbox R1 Release
Freeze
![Page 28: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/28.jpg)
Q: How do we do scale PerfEng to meet
demands of a larger organization?
![Page 29: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/29.jpg)
Ratios are Key to Establish and Socialize
PerfEng established a 1:8 ratio of Perf/Dev IC
No more than two scrum teams or three projects / release
Does not include workload engineers (min of two per cloud)
Does not include Managers or Software Tools Engineers
Perf Managers/IC ratio may need to be higher than 1:8
Managers may require 1:3 or 1:5 due to the additional teams
managers interact with cross functionally
Find a ratio that enables PerfEng
Factor early participation, deep dives, optimization work to provide
meaningful contributions
Support discussion with velocity points, automation and efficiency
examples, ROI Examples
![Page 30: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/30.jpg)
Gap is closing today, but still
haven’t reached the target
>2x
>2x
>1.2x
![Page 31: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/31.jpg)
Self Service to the Rescue!
![Page 32: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/32.jpg)
Embed Performance Mindset into Every Team
“Closely partner with scrum teams to
provide early, fast, continuous
architecture engagement / results /
analysis for complex scenarios and
enable scrum teams to catch obvious
performance issues with self service
tools, automation, and processes before
they reach Performance Engineering”
![Page 33: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/33.jpg)
How We Interact with Scrum Teams
• Each scrum team appoints one Dev and QE engineer
who are mapped to a single PerfEng
• Teams must co-develop their release plans and sign off
criteria up front
• Teams are accountable for their features (complete
ownership coming back to PerfEng as team scales up)
• Teams must characterize obvious performance criteria
themselves every sprint (Cadence, PTest)
• Teams must deliver their features on time or accept
testing into the release sprint or beyond
![Page 34: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/34.jpg)
PerfEng Rep Mapping
PerfEng
Rep
Dev/QE
Reps
![Page 35: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/35.jpg)
Embedding Performance – A Tiered Approach
80% Scrum Team +
0% PerfEng
15% Scrum Team +
40% PerfEng
5% Scrum Team +
60% PerfEng
Single user transactions on
Desktops/Local Builds
Single user transactions on Corsa Single user transactions on IST
Single user transactions in PTests High Load, High Concurrency on
Corsa
High Load, High Concurrency on
IST
Increasing Test Complexity and Feature Risk
Scrum Teams focus on catching obvious low-hanging
fruit; PerfEng focuses on difficult to construct, high
load/concurrency scenarios requiring highly specialized
knowledge to detect and analyze
![Page 36: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/36.jpg)
86 GB of meta data primarily from PerfEng workload tests!
![Page 37: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/37.jpg)
1.4 TB of meta data from tests created by Devs and outside teams!
![Page 38: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/38.jpg)
Q: What are the key Agile Release
Milestones and activities for PerfEng?
![Page 39: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/39.jpg)
Release Timeline and PerfEng Activities
Product Development Sprints
Jan Feb Mar Apr May Jun
Planning Final Plans Due Feature Freeze
Release Sprint Release
Sandbox R1 Release
Freeze
•Appoint Liaisons
•Complete Release Plans
•Double Check Exit
Criteria
•Signoff on all
Features and
Workloads
•Continue Workload
Optimizations
Milesto
nes
•Monitor Sandbox
•Final Optimizations
•Initial visibility into
all features
•Signoff on ¾ of
features
•Get workloads
green
![Page 40: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/40.jpg)
How Do We Allocate Engineer’s Time?
70% Velocity Points Open
Feature or Workloads work for a specific cloud
30% Velocity Points Reserved
PTOn (9 days/year to work on whatever they want)
External Training Classes (e.g. SQL Tuning)
Other Cloud’s projects they are interested in
Conferences (e.g. HBase, Hadoop)
Foundation events (1:1:1) We Leverage Agile and
ADM to enable People’s
Changing Interests
![Page 41: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/41.jpg)
Requirements/Arch Strategy/Test Plan Analysis/Results
Templates Cover Most Important Phases of a Project
![Page 42: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/42.jpg)
Release Signoff Criteria and Team Dynamics
• PerfEng will only sign off on features we
worked on directly (or have thoroughly
reviewed the plans and results)
• Scrum Teams may sign off on features by
themselves at their own risk for any feature
with Medium or Less Risk (if PerfEng is
short of resources)
![Page 43: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/43.jpg)
Quick Tip – Negotiating Release Criteria
Bring in teams from operations and support
Quote examples of consequences of releasing
without adequate throttles and caps in place
Cite examples from your company or other
leading companies of the cost of reduced
customer credibility
![Page 44: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/44.jpg)
Workloads
![Page 45: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/45.jpg)
What is a “Workload”?
• A repeatable test simulation or benchmark that provides a
meaningful result by utilizing specific inputs into the system under
test while recording numerical metric data, which is subsequently
analyzed and weighted to perform a qualitative assessment
• Changing a variable in the workload and re-running provides a
meaningful comparison
• Baseline Workloads are automated and enhanced release over
release wherever possible
![Page 46: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/46.jpg)
Workloads Map
DB Workloads
Grinder
Force.com
Apex
VisualForce
Chatter
Search
MQ
UI
Sharing
![Page 47: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/47.jpg)
“Shape” Terminology
Load Shape – The distribution, rate, and
type of requests injected into the system
under test (SUT)
Data Shape - The size, skew, and type of
data, files, etc. accessed during the test
![Page 48: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/48.jpg)
Categories
Playback tests take production traffic logs and replay traffic against the cut of data from
that time period
• This enables Salesforce.com to properly capture data skews, volumes, and transactions that customers have
run at a particular time and cover features that are heavily customizable
Synthetic tests involve utilizing custom tools to profile production load and data shapes
and then use custom tools to create workloads that mimic the desired characterisitics
• Synthetic tests enable the team to create data and load shapes that may be far greater or more accentuated
than in production, in a deterministic and precise fashion that enables granular studies of linearity,
bottlenecks, and resource utilization
• In most situations different versions of Salesforce are compared against one another, although absolute
performance metrics are used for new features or situations where it is too difficult to make meaningful
comparisons
![Page 49: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/49.jpg)
Workload Highlights
Name Summary Load Shape Data Shape
DB Workloads A workload the replays real production requests against
customer data in a precise fashion to meticulously identify
proper DB stats, tuning
100,000 complex
reports and filters
Sanitized copy of real world production
Data with emphasis on massive data sets
for reports
Grinder A large scale, high load, high concurrency test that simulates an
hour of peak production traffic by replaying transactions
400 RPS, target
production
steady state
utilization of 35%,
peaks of 80%
Sanitized copy of real world production
Data
Force.com Simulates traffic against a standard Ideas sites / base them
application.
Requests are
generated across
40 different URLs /
operations
Synthetic data based on real world force.com
app “Ideas”
Visual Force A read-only targeted test isolating specific components of VF at high
request rates. Apex components are designed to be constant across
all requests so regressions are to pure VF
32 concurrent
requests across 10
orgs
Small Synthetic VF classes. Viewstates,
Wrapperless / Wrapped nested data
presentation and Namespaces
Apex A targeted test that exercises the components of Apex Cache, CPU
consumption, Memory
64 threads across
16 organizations
Synthetic set of classes that exercise Apex
Cache, CPU Use of Apex L1, Maximal
number of lines of apex, creation of
temporary objects
Sharing A workload that performs DML Operations on Sharing Enabled Orgs,
Performs Sharing Rule Maintenance Operations on Various Entities,
Territory Management Operations and Accounts/Opportunity
reassignments, Routes bulk processing via Message Queues.
2 app servers, 10
concurrent Users,
7 Thread groups
3 bulk processing
Synthetic Orgs (One Territory Managed, One Regular)
![Page 50: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/50.jpg)
Workload Highlights (continued)
Name Summary Load Shape Data Shape
Search High load, high concurrency test that simulates peak production traffic
by replaying searches and concurrently simulating incremental
indexing. Monitors and reports metrics on entire stack [Indexers, DB,
Query Servers, App Servers, Memcached]
Replay production searches
and performs incremental
indexing at peak load. Issues
searches at 55 RPS
Sanitized copy of real world
production Data
MQ Workload:
QPID (transport in
isolation)
A workload which enqueues messages into QPID on an IST using
multiple IST app servers. Tests QPID (the MQ transport service) in
isolation. Suitable for acceptance testing an upgrade.
20 app servers x 20 threads
enqueue messages of
varying sizes for 10min-6hr
synthetic; configurable message
size
MQ Workload:
Hydra
(integrated)
A workload which creates load on the integrated SFDC MQ framework
using the SFDC MQ API library. Uses synthetic asynchronous handlers
running on the app servers to simulate message and resource
consumption. Suitable for running with every release, and for
simulating the impact of a new asynchronous handler.
20 app servers x 20 threads
enqueue messages of
varying sizes for 10min-6hr
synthetic; configurable message
size
Mobile Workloads simulate user actions over a real 3G network. Captures
metrics to measure end-user perceived response times on slow
networks and real devices
Real Device & Emulator. On
Real 3G networks Sanitized copy of real world
production Data
UI Workloads simulate user actions in a real browser. Captures metrics to
measure end-user perceived response times. Org with Chatter data is
very large.
6 Browsers – Nightly tests Synthetic user data – across all
standard pages. 3 Different orgs
to test across different
skins/chatter.
![Page 51: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/51.jpg)
Workload End to End Coverage* (At a Glance)
UI Network App Search Indexer FFX Batch DB SAN
DB Wkld 8 1
Grinder 1 8 3 3 3 3 7 3
Force.com 1 8 1
VF 7
Apex 7
Sharing 7 2
Search 6 6 6 4 3
MQ 6 2
Mobile 5 4 1
UI 8 5 4 1
Batch 6
*Higher numbers indicate better coverage in a given tier
![Page 52: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/52.jpg)
Daily DB, Appserver, and UI Performance Tests!
UI - End User Response
Time Workloads
Appserver Workloads
Database Workloads
![Page 53: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/53.jpg)
168 – Performance Bugs ROI
290 Total = 78 Workloads (27%), 211 Feature Testing (73%)
Note that >50% of P0 bugs were
found by baseline workloads!
![Page 54: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/54.jpg)
Automation and Tools
![Page 55: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/55.jpg)
Tools (Analysis, Monitoring, Automation)
Custom
Michelangelo / Caliper
StatsForce
Leonardo
LightHouse
Suzuki
Cadence
SUIT / CSP
PTest
ReplayForce
DataForce
Off the shelf
JMeter
STAF
JProfiler
Dynatrace Ajax
Splunk
Shunra
Jiffy
HTTP Analyzer
Selenium
Fiddler, YSlow, PageSpeed, Firebug
![Page 56: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/56.jpg)
Michelangelo – Results Viewer
• Provides single point of
entry into all automated
tests
• Dynamic Test vs. Test
views
• Automatic Averaging of
test runs and filtering of
outliers
• Compare baseline to
results trends
![Page 57: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/57.jpg)
Michelangelo Changelist Trend Example
• Dramatically shows
changes in performance
to the changelist
• Row and Column
highlighting
• Color Coding
• Annotations
• Compare baseline to
results trends
• Absolute and Relative
difference comparisons
Specific
Changelist fix
results in 33%
more GC
activity
![Page 58: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/58.jpg)
Michelangelo Demo
![Page 59: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/59.jpg)
StatsForce – High Resolution
Time Correlated
Visualizations
• OS Statistics
• Application Statistics
• JVM Statistics
• Errors
• Mix and match
representations and chart
types on demand
Notice the
benefits of
time
correlation!
Notice different
chart types
(scatter) on
same timeline!
Notice
how Full
GCs affect
Response
Times!
![Page 60: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/60.jpg)
This looks odd!
Statsforce Example - Force.com Workload Load
Balancer Regression 166 164
![Page 61: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/61.jpg)
166 164
This looks odd!
Statsforce Example – Errors Per Second
![Page 62: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/62.jpg)
Statsforce
Example -
Custom Test vs.
Test Views!
![Page 63: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/63.jpg)
Environments
![Page 64: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/64.jpg)
Name Description Size
IST (Integration
System Testing)
• Large scale pod. Closest to production in both software and hardware
configuration (load balancers, 8 node RAC database, etc.)
• Primarily uses production data
CST (Comparison
System Testing)
• Small environments focused on Database workloads
• Primarily uses production data
DB Load (Prod,
Synthetic)
• Small environment with large sized DBs (4TB – 20TB)
• “Prod” uses production data , “Synthetic” uses synthetic data
Corsa (“Race”)
• Small environments with hardware vertically identical to production
• Fewer horizontal nodes, focused on a particular SUT (Search, DB)
• Does not utilize production data
VMs / Autobuilds Dedicated environment for each engineer for development purposes
Desktops / Adhocs Dev local machines or Adhocs for PerfEng – dedicated for each
engineer for local tests or development
Environment Types
![Page 65: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/65.jpg)
Continuous Data Refresh System
•Enables teams to access latest
production / synthetic data with
minimal downtime
•Performance tests can modify / delete
TB of data and rollback in minutes
Details
Production snapshots and corsa
images are taken periodically and
stored on SAN Refresh
A “jukebox” server prepares snaps
into “green” database images
The jukebox applies schema updates
and keep them “green”
The “green” images are always ready
to use and rsynched directly to the
environments
![Page 66: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/66.jpg)
Closing Thoughts and Tips
![Page 67: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/67.jpg)
Where is Salesforce.com PerfEng Today? 30,000 ft. view
• Team has evolved from seven “Systest” engineers who
struggled to produce meaningful analysis, to a world class
Performance Engineering organization of >60 engineers with
no significant production issues the day after release for
almost three years
• Active participation in features, provides visibility and risk
assessment at critical milestones and averts major
degradations, helps triage and mitigate production issues,
delivers optimizations across the stack, and whose skills and
headcount are now lobbied for by Development teams
• Automation has increased from two workloads which ran a
handful of times late in the release, to over 15 sophisticated
workload suites that run every day and are critical to signoff
![Page 68: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/68.jpg)
Top Ten Tips for Scaling Your Team
1.Socialize your ratios for PerfEng to Developers to eventually embed into teams
2.Propose a dedicated model over a shared service model
3.In a pinch, provide teams the velocity points they have funded, and ask them to prioritize
4.Build out your management team at every opportunity
5.Develop meaningful automated workloads with low variance and show the ROI regularly
6.Create a tools team that spends >=75% of their time developing automation and tools
7.Make your Labs and Test Frameworks self service
8.Develop production monitoring tools to collect relevant data for workloads and exit criteria
9.Create frameworks to enable staged work from Dev desktop to large scale Perf environments
10.Develop training classes for perfeng, new hires, Dev/QE liaisons – smaller population first
![Page 69: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/69.jpg)
What else could be
responsible for this
dramatic optimization?
![Page 70: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/70.jpg)
Could Increasing PerfEng
continue this trend…?
![Page 71: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/71.jpg)
Bonus Tips for a Happy Team
• Contribute to a positive atmosphere that
promotes Autonomy, Mastery, and Purpose
with interesting projects to tackle in depth
• Focus on your strengths and strive to
improve at every opportunity
• Set a bold vision with achievable
milestones, and celebrate progress
![Page 72: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/72.jpg)
“Is anything truly impossible? Perhaps it is
temporarily impractical or unlikely” – Kasey Lee
Ex. Human Exoskeletons (2:05)
What will you take from today?
What will you change starting next week?
![Page 73: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/73.jpg)
Kasey Lee
VP Performance
Engineering,
in/leekasey
![Page 74: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/74.jpg)
![Page 75: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/75.jpg)
Turn your PerfEng team from this… Into this…
Manual Black Box Testers Architecture / Analysis /
Simulation / Optimization /
Visualization / Automation /
Monitoring Experts
![Page 76: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/76.jpg)
Lines of Defense
1. Single user requests in PTest on VMs
2. Single user requests / high load on Corsa
3. Concurrent / high load on Corsa
4. Single user requests on DB Load
5. Concurrent / high load on DB Load
6. Single user requests on IST
7. Concurrent / high load on IST
![Page 77: How Salesforce built a Scalable, World-Class, Performance Engineering Team](https://reader034.vdocuments.mx/reader034/viewer/2022051610/5481cfcbb47959050d8b4614/html5/thumbnails/77.jpg)
Questions