10/7/2013 #bbcon 1

BBCRM Performance From Cobblestones to Autobahnen

PRESENTED BY LEN WYATT

10/7/2013 #bbcon 3

Cobblestones? Autobahnen?

Why such a silly subtitle?

10/7/2013 #bbcon 4

WHAT IS NEEDED TO BUILD A

COBBLESTONE ROAD

10/7/2013 #bbcon 5

WHAT IS NEEDED TO BUILD AN AUTOBAHN

10/7/2013 #bbcon 6

WHAT IS NEEDED TO USE A

COBBLESTONE ROAD

10/7/2013 #bbcon 7

WHAT IS NEEDED TO USE AN AUTOBAHN

10/7/2013 #bbcon 8

• To schnell fahren auf der Autobahn, there is a lot behind the scenes!

• The same is true for complex multi-user systems like BBCRM

• So when something goes wrong, why is software so much harder to

troubleshoot?

LAYERS OF INFRATRUCTURE,

KNOWLEDGE AND SUPPORT

10/7/2013 #bbcon 9

• Browser

• Operating System

• Client computer

• CPU, Memory, Disk

• Local LAN

• Internet

• Web server

• CPU, Memory, Disk

• Data center LAN

• Database server

• CPU, Memory, Disk

• Database schema

• Database contents

• Stored procedures

• Customizations

LAYERS OF ABSTRACTION

What the user did

What everybody else is doing

What is stored in the systems

How the systems have been configured

How the systems have been maintained

Developers learn to use these abstractions

to hide the behavior of other layers

Performance issues can occur in any layer.

Everything is subject to examination.

10/7/2013 #bbcon 10

• Because abstraction is so easy in software, we do it a lot!

• All the tools, infrastructure and support are built on a

digital background that we can’t physically see – we have

to hold all the concepts in mind

• Evolution has not prepared us for this

• We do not have hundreds of years of shared experience

LAYERS OF ABSTRACTION

10/7/2013 #bbcon 11

• “It’s taking a long time” does not point to the problem

• This is not the user’s fault

- The layers of abstraction hide the issue

• We have to ask a lot of questions, and use tools to collect information

WHAT THE USER CAN TELL US

10/7/2013 #bbcon 12

ENTER THE ENTERPRISE PERFORMANCE TEAM

10/7/2013 #bbcon 13

ENTERPRISE PERFORMANCE TEAM

WORK AREAS

• Tactical improvements. Some customers have

performance issues today, and we need to

address key ones. Tactical work focuses on

carefully chosen areas that are common to

multiple customers, and seeks solutions.

Recommendations are made to product teams.

• Strategic improvements. Long term, we must

find ways to improve the product design and

engineering processes to focus on performance

early and continually.

• Practices. Deliver information to help

customers as well as our own customer-facing

teams (SDO, Prof. Svcs, Support, Sales) do

their jobs better.

10/7/2013 #bbcon 14

THIS IS A NEW APPROACH

• Yep, we have had some

performance problems with BBCRM

• No, we haven’t always handled them well

• It feels like we have been playing

Whack-a-Mole

• Now we have…

• A focused team

• Working on root causes

• Being scientific

• Bringing new thinking about

performance to the organization

10/7/2013 #bbcon 15

Performance work across the organization…

Product performance deliverables:

• Ad-Hoc Query Improvements

• Revenue Batch Improvements

• Security Model Improvements

• Blackbaud Direct Marketing

segmentation on the DW

Driven by development teams

• Planned, scheduled work

BBCRM PERFORMANCE INITIATIVES

Research-based approaches:

• Tactical improvements

• Strategic improvements

• Practices information

• All based on representative

workloads

Driven by the

Enterprise Performance Team

• Exploratory & opportunistic work

10/7/2013 #bbcon 16

ONE TACTICAL IMPROVEMENT

• Individual Revenue Summary View Data Form

• Found issue by observing Slow Web Requests in SDM

• Issued patch for 2.93 (later versions were already improved)

• The impact on AHA alone:

• Before (6/1/2013 – 6/30/2013)

• 4,889 SWRs

• Average 60 seconds per SWR (55 seconds over threshold)

• After (7/1/2013 – 8/5/2013)

• 113 SWRs

• Average of 7 seconds per SWR (2 seconds over threshold)

• Nearly ½ of a full time person no longer spent waiting!

10/7/2013 #bbcon 17

• Postings on the BBCRM blog

- http://www.blackbaudknowhow.com/blog/blackbaud-crm

EARLY PRACTICES RESULTS

10/7/2013 #bbcon 18

• Who has seen those blog postings?

• What is the most effective way to get information out to you?

• Is it different depending on the type of information?

- User tips

- Database administrator recommendations

- Configuration & deployment guidance

QUESTIONS!

10/7/2013 #bbcon 19

WE WORK BY INVESTIGATION

WE WORK BY INVESTIGATION: WE DO SCIENCE

10/7/2013 #bbcon 20

• We study and learn from customer experience

- Review and analyze Work Items related to performance

- Study available instrumentation

• SDM data for hosted customers

• We rarely work directly on a specific customer case

- Because we are looking for the root causes that affect

many customers

• We examine the workings of a system in operation

- To see what’s really happening, and not speculate

• We work with engineering teams to implement changes

- Code ownership stays where it belongs

- Developers and QA people learn from the process

SCIENCE STARTS WITH OBSERVATION

10/7/2013 #bbcon 21

• To study a working system, a testbed is required

- Does it help if we make this change?

- Are there side-effects from changing this?

- Is alternative A better than B?

- Is this configuration optimal?

- What if we have more (of fewer) CPUs? Memory? Disks?

- Could we support more users?

- What is the source of database growth?

- How does index fragmentation change performance?

- Would database compression help performance?

- … and on and on…

• Using the testbed requires a workload

- Models the activities the system is supporting

- Models the data in the system

- Allows measurement of response times

TESTBED

10/7/2013 #bbcon 22

• Both the state of the SUT and the applied work must be meaningful in

order to get a meaningful result

- Mostly this means the database contents and the user actions

• Representative workloads will allow us to do science instead of guessing

MEANINGFUL WORKLOADS ARE VITAL

FOR PERFORMANCE WORK

Database

Reflects the rich

complexity of real

customer

databases

Web Tier

User Actions

Reflect what

users do and

action rates

Test

Driver

System Under Test

10/7/2013 #bbcon 23

• Workloads model user actions

- What do users do?

- What parameters are filled in?

- How much time between actions?

- How many users?

• Workloads model the database contents

- How many constituents? Addresses? Contributions? Reports? Tributes?

- The patterns in the data are important…

• Do some constituents make many contributions and others only a few?

• Are there some for whom we have lots of address changes recorded?

• What is the ratio of general contributions vs. directed ones?

• Creating a workload like this is expensive; we can’t do it too often!

- So it has to be representative of many customers

WORKLOADS

10/7/2013 #bbcon 24

• To create lab workloads that are highly

representative of real customer workloads,

we are modeling from systems run by

Performance Partners

- If we do our job well, the lab workloads will be

almost doppelgangers of the originals

• Performance Partners…

- Are BBCRM customers

- Chosen because they have issues representative of multiple others

- Agree to work with us

- Let us work with a (masked) copy of their database

- Let us monitor user activity for an extended time (a month)

- Agree to let us instrument systems from time to time, if needed

- Want to try updates when applicable

- Have willingness and capability to test, measure and try some things

PERFORMANCE PARTNERS

10/7/2013 #bbcon 25

• Heifer International

• The University of Iowa Foundation

• They are different…

- Usage models

- Sizes of databases

- Patterns of performance issues

- Hosted by Blackbaud vs. self-hosted

• They are similar to others in their categories

PERFORMANCE PARTNERS

10/7/2013 #bbcon 26

• What system configurations do our customers use?

- For hosted customers, we know

- For self-hosted customers, we have only spotty information

• How can we know if our testbed is representative of self-hosted sites?

• Help us help you…

- We have sent a survey through the TAMs

- It asks about physical equipment

- Accompanied by a script that collects

statistics on the database

• If you haven’t seen this, ask your TAM

- Have them ask me if they don’t know

TESTBED, REDUX

10/7/2013 #bbcon 27

THE SELF-HELP SECTION

10/7/2013 #bbcon 28

• End Users

- Watch for tips like we have posted on the BBCRM blog

• We have posted about Constituent Search and Querying by Merging

Selections

- Try not to run large batch operations or slow reports during prime time

• System administrators

- Make sure maintenance tasks are performed regularly

• Update statistics

• Rebuild indexes

- Eliminate unnecessary data

• Excessively old audit data

• Unused export tables

- Know your baseline performance

• Monitor PerfMon counters

MAKING THINGS BETTER YOURSELF

10/7/2013 #bbcon 29

• Developers

- Familiarize yourselves with SQL query tuning techniques

- Follow known best practices

• Learn them at talks tomorrow: “Blackbaud CRM Boot Camp” Parts 1 & 2

• Support staff

- Be familiar with what Blackbaud support will need to know about

performance issues

• You can start to ask the same questions

MAKING THINGS BETTER YOURSELF

10/7/2013 #bbcon 30

HELP BB SUPPORT HELP YOU!

10/7/2013 #bbcon 31

• Was any of that helping-yourself material a surprise?

• Which of those things are you doing today?

• What tools do you use for routine maintenance tasks?

DISCUSSION

10/7/2013 #bbcon 32

FIN

• We’re taking a whole new approach to performance

• This is going to take time

• The benefits are going to be big

• We are improving BBCRM

• We are changing the organization

10/7/2013 #bbcon 33

WHEW! THAT WAS A LOT OF MATERIAL.