Behind the Wizard’s Curtain: Scalability and Security at Zuora

Subscribed 2013

Thomas Fou Information Security &

Compliance

Levon Stepanian Performance Engineering

&

Key 9

•  Key 9 •  A Day in the Life of Zuora •  Behind the Curtain: The Zuora Cloud & Platform •  Zuora’s Investment in Performance •  Security & Compliance •  Q/A

AGENDA

Key 9 Key 9 •  9 Keys to Subscription Success •  Key 9: Mission Critical, Reliable, Scalable & Secure

“Subscription businesses want a reliable ‘enterprise-grade’ system with services that are built on a secured, mission-critical, and scalable infrastructure. The SaaS Subscription Infrastructure must have reliable 7x24x365 operations, regular new feature deployments, and well-formed, predictable business continuity”

Key 9 Disclaimer Pt. 1 •  This  session  may  contain  product  features  that  are  currently  under  

development.  •  This  session/overview  of  new  technology  represents  no  commitment  

from  Zuora  to  deliver  these  features  in  generally  available  product.  •  Customers  who  purchase  services  should  make  the  purchase  

decisions  based  upon  features  that  are  currently  available.    •  Technical  feasibility  and  market  demand  will  affect  final  delivery.  •  Pricing  and  packaging  for  any  new  technologies  or  features  discussed  

or  presented  have  not  been  determined.  

Key 9 Disclaimer Pt. 2 Zuora Confidential

Not for distribution beyond the intended audience at Subscribed 2013

thebillablehour.com  

Key 9 A Day in the Life of Zuora

•  Monthly Synchronous Transaction Volumes –  SOAP and REST APIs

•  Subscription Management –  550K created –  1M amended

–  Half a Billion API calls a month (15M/day) –  > 80% are queries

•  Top 3 Objects: Subscriptions, Products, RatePlans

Key 9 A Day in the Life of Zuora

•  Monthly Asynchronous Transaction Volume (Aug/2013) •  39M  total  asynchronous  transac0ons  

•  ~50%  during  1st  day  &  last  4  days  of  month  •  Transac0on  mix  –  small  &  large  •  Monthly  varia0on  

6M  

1M  

6K  

?K  

336M   15M  

INVOICE  

Key 9 Performance Data •  Benchmark Data

–  375 orders/sec for a single tenant •  2012 Amazon Cyber Monday Peak Rate: 306 items/sec

–  150+ payment authorizations/sec for a single tenant

•  Production Data –  50% of our tenants -> 70K invoices per hour –  Tenant generating 1.7M invoices in a single bill run

Key 9

LB

AMQ  DB  (M)  

Behind the Curtain: The Zuora Cloud

LB

Billing  &  Payment  Servers  

UI/API  Server  

Global  (S)  

PDFGen  Servers  

Web  Server   Zuora  for  

Salesforce  Servers  

Messaging  Infrastructure  

File  Storage  Global  (M)  

Tenant  Shard  

Tenant  Shard  (M)  

Tenant  Shard  (M)  Tenant  Shard  (S)  

File  Storage  

AMQ  DB  (S)  

(M)aster/(S)lave  RO  Replicas  (Not  Shown)  

FW Security  

Appliance  

Key 9

LB

AMQ  DB  (M)  

Scaling Zuora

LB

Billing  &  Payment  Servers  

UI/API  Server  

Global  (S)  

PDFGenServers  

Web  Server   Zuora  for  

Salesforce  Servers  

Messaging  Infrastructure  

File  Storage  Global  (M)  

Tenant  Shard  

Tenant  Shard  (M)  

Tenant  Shard  (M)  Tenant  Shard  (S)  

File  Storage  

AMQ  DB  (S)  

FW Security  

Appliance  

Key 9 Infrastructure Scalability

–  Enterprise Ready Tier 1 Data Center •  Switch SuperNAP (Las Vegas) •  High density, state of the art infrastructure •  Super beefy hardware, storage and networking gear •  7x24x365 resource monitoring and alerting

10  

90  

Avg.  Produc0on    Capacity  U0liza0on  

uMlized  

idle  

–  Plenty of standby spare capacity to accommodate growth

•  Max utilization ~ 30%

Key 9 Zuora’s Investment in Performance

•  Bottlenecks are everywhere!

•  Zuora’s massive & continual investment in performance –  Search & Destroy philosophy adopted by all teams –  Refactoring/optimizing code –  Production-like environment profiling/analysis –  Better aligning s/w and h/w architectures –  Investment in state of the art technology

Key 9 Customer Facing Performance Improvements

•  Rating & Billing Engine (RBE) TurboBooster •  1.6X to 25X Bill Run speedup in production

•  Optimizing & minimizing # of queries •  More charges/subscription = Larger speedups

•  Zuora for Salesforce 360 TurboSync •  Up to 50X 360 Sync speedup in production

•  Exploiting parallel pipelines, Bulk Salesforce APIs •  More objects to sync = Larger speedups

Key 9 Customer Facing Performance Improvements

•  Zuora for Salesforce 360 TurboSync

50X  

Key 9

?  

Customer Facing Performance Improvements

•  Orders/sec Capacity Improvements •  Continuous infrastructure improvements •  Code re-factoring, optimizations

Orders/sec  

(Peak  Cap.)  

Key 9 Customer Testimonials •  “After directly engaging with Zuora Engineering on a looming requirement to

support a large increase in scale, I was reassured by how quickly they moved to support the stated 100tps for creation of hosted payment methods. I was impressed that Zuora Engineering then applied the 100tps requirement to the other API calls, in anticipation of downstream increases in volume. If a difficult requirement arises in the future, I will have no hesitation in directly engaging Zuora Engineering again, knowing that it will be a job well done.” – Architect

•  "We're really excited about Zuora's improvements to Bill Run execution times. As a business that bills hundreds of thousands of transactions in each bill run, it's important for Hosting.com to generate invoices rapidly and get paid as quickly as possible" said Rick Moore, VP Finance & Business Operations at Hosting.com. "The latest performance improvements have significantly reduced our scheduled bill run times by over 50%--that's a huge improvement, and attests to the fact that as our business grows, Zuora continues to scale to accommodate that growth.”

•  “We named it TurboSync because of how fast it copied 3million records into our system. What would normally would have taken days instead took a few hours!” said Cathy MacDonald, Executive Vice President of IT at Xplornet Communications Inc.

Anonymous    Zuora  Customer  

Key 9 Zuora Compliance •  PCI Level 1 Compliant •  SSAE16 SOC 1 Type 2 Compliant •  TRUSTe Certified •  US-EU Safe Harbor

Key 9 Zuora Security •  Physical Security

–  World-class primary and backup datacenters –  Switch SuperNAP – PCI and SSAE16 SOC1/SOC2/SOC3 –  CoreSite – SSAE16 SOC1 Compliant

•  Network Security –  Production environment completely separate –  Firewall and network zone segregation –  Two-factor authentication remote access

•  Application Security

–  HTTPS for all incoming/outgoing data transfer –  CC data encrypted using AES-256 SafeNet FIPS certified hardware encryption –  Application security testing

Key 9 Zuora Security •  Vulnerability Management

–  Qualys Internal/External Network Scans –  WhiteHat Security Application Scans –  Coalfire Web Application Penetration Testing –  Monitor CVE, NIST, vendor vulnerability lists –  Apply critical patches monthly

Key 9 Zuora Data Flow

Key 9 How To Reduce PCI Scope

•  Each entity responsible for how it uses data •  Limit where PCI data is stored, processed, transmitted •  Segment cardholder data network from other networks •  Use effective encryption •  Implement strong key management practices •  Limit scope of Cardholder Data Environment (CDE) •  Zuora Hosted Payment Method (HPM)

Key 9 End - QA

Key 9 Appendix

Key 9 Scalability Case Study •  Customer Profile

–  Leading global news company –  Top UK newspaper publisher

•  Subscription Launch –  Digital paper (smartphones, tablets, online) –  Access to breaking news (24x7) –  Fantasy soccer –  Apps for streaming soccer matches

Key 9 Scalability Case Study •  Performance Requirement

–  100 transactions per second customer acquisition rate •  360,000 customers an hour!

•  End to end testing uncovered bottlenecks –  Internal and External to Zuora

•  Outcome –  Configuration tuning –  Horizontally scaled back-end servers –  Enhanced monitoring + alerting around launch dates