ebay architecture

46
eBay Architecture Scalability with Agility Tony Ng Director, Systems Architecture October 2011

Upload: tony-ng

Post on 01-Nov-2014

9.216 views

Category:

Technology


4 download

DESCRIPTION

 

TRANSCRIPT

Page 1: eBay Architecture

eBay  Architecture  Scalability  with  Agility  

Tony  Ng  Director,  Systems  Architecture  October  2011  

Page 2: eBay Architecture

About  Me  

•  eBay  –  Systems  Architecture  and  Engineering  

•  Yahoo!  –  Social,  Developer  PlaEorms,  YQL  

•  Sun  Microsystems  –  J2EE,  GlassFish,  JSRs  

• Author  of  books  on  J2EE,  SOA  

 

 

2  

Page 3: eBay Architecture

eBay  Stats  

•  97  million  acQve  users  

•  62B  Gross  Merchandise  Volume  in  2010  

•  200  million  items  for  sale  in  50,000  categories  

•  A  cell  phone  is  sold  every  5  seconds  in  US  

•  An  iPad  sold  every  2.2  minutes  in  US  

•  A  pair  of  shoes  sold  every  9  seconds  in  US  

•  A  passenger  vehicle  sold  every  2  minutes  

•  A  motorcycle  sold  every  6  minutes  

3  

http://www.ebayinc.com/factsheets

Page 4: eBay Architecture

eBay  Scale  

•  9  Petabytes  of  data  storage  •  10,000  applicaQon  servers  •  44  million  lines  of  code  

•  2  billion  pictures  •  A  typical  day  

–  75B  database  calls  –  4B  page  views  –  250B  search  queries  –  Billions  of  service  calls  – Hundreds  of  millions  of  internal  asynchronous  events  

4  

Page 5: eBay Architecture

History  of  Technology  

Inno

vatio

n Po

tent

ial

Agi

lity

/ TTM

Arc

hite

ctur

e M

atur

ity

1995

·∙ Perl/C++·∙ Inline HTML·∙ Monolithic·∙ Vertical Scale·∙ Walled Garden

1999

2001

·∙ Java·∙ XSL·∙ Layered·∙ Horizontal Scale·∙ Some APIs

2005

2009+

·∙ Java·∙ V4 Components·∙ Services·∙ Internal Cloud ·∙ Platform

Page 6: eBay Architecture

eBay  Scalable  Architecture  

•  ParQQon  everything  – Databases,  applicaQon  Qer,  search  engine  

•  Stateless  preference  – No  session  state  in  app  Qer  

• Asynchronous  processing  – Event  streams,  batch  

• Manage  failures  – Central  applicaQon  logging  – Mark  downs  

6  

Page 7: eBay Architecture

Next  Challenges  

•  To  stay  compeQQve,  we  need  to  deliver  quality  features  and  innovaQons  at  acceleraQng  paces  

• Complexity  as  our  codebase  grows  

•  Improve  developer  producQvity  

•  Enable  faster  Qme-­‐to-­‐market  while  maintaining  site  stability  

7  

Page 8: eBay Architecture

Scalability  with  Agility  

•  Strategy  1:  AutomaQon  with  Cloud  

•  Strategy  2:  Next  Gen  Service  OrientaQon  

•  Strategy  3:  Modularity  

8  

Page 9: eBay Architecture

Automa=on  with  Cloud  

9  

Page 10: eBay Architecture

Hardware  Acquisi=on  

10  

request { servers,

model, app }

order receive & rack & wire Label (app)

deliver

1 w 2-3 w

repurpose

“several” weeks

request {servers, model }

order Receive pre-racked Pre-wired

deliver to cache

1 day 2-3 w

repurpose

minutes

request {servers,

model, app }

deliver

quarterly

Page 11: eBay Architecture

Improving  U=liza=on  

11  

Number of servers required based on utilization for 8 pools

DR

Page 12: eBay Architecture

Infrastructure  Virtualiza=on  

Infra Infra Infra Infra

Spare spare spare spare

Application App App App

Shared infrastructure

Global resource pool

Application App App App

Page 13: eBay Architecture

eBay  Cloud  

Self Service Portal

Automation

pool provisioning in minutes

Hardware Acquisition

Resource Allocation

Virtualization

Spare Capacity

Capacity Management

Improved Time to Market

Page 14: eBay Architecture

Architecture  Decision  

14  

Page 15: eBay Architecture

Infrastructure  &  PlaIorm  as  a  service  

15  

Virtualized & Common Infrastructure

Automated Operations

Infrastructure As A Service

Front End, Search Back End, Generic Platform

Automated Life Cycle Management

Platform As A Service

Enables innovation on new platforms

Infrastructure

level automation

Higher developer productivity

Full application level

automation

Page 16: eBay Architecture

Model  Driven  Deployment  Automa=on  

16  

•  Desired configuration is specified in the expected state and persisted in CMS

•  Upon approval, the orchestration will configure the site to reflect the desired configuration.

•  Updated site configuration is discovered based on detection of configuration events

•  Reconciliation between the expected and current state allows to verify the proper configuration.

•  On going validation allows the detection of out of band changes.

LB Pool

Server Server Server

Current State

Site

Discovery

Comparison

Expected State

Reconciliation

Orchestration

LB Pool

Server Server Server

Page 17: eBay Architecture

Open  Source  Integra=on  

IaaS/PaaS API orchestrat

ion Resource Allocation

Distributed

State

Compute Controller

Cluster Controller

Pool Controller

Compute Mgt.

DNS Mgt.

LB Mgt.

Network Prov

Image/Pkg Repo

Software Dist.

AuthN/AuthZ

Application

Controller

Access Point

Controller

IaaS/PaaS API

Open  Source  SoluQon  

(openstack  /  Cloudstack)  

orchestration

Resource Allocation

Distributed

State

Compute Controller

Cluster Controller

Pool Controller

AuthN/AuthZ

Application

Controller

Access Point

Controller

Monitoring

Page 18: eBay Architecture

Applica=on  Architecture  

Ongoing “Cloud

Friendly”

Future ‘Cloud ready’

Before

Page 19: eBay Architecture

Next  Gen  Service  Orienta=on  

19  

Page 20: eBay Architecture

Services  @  eBay  •  It’s a journey !

•  History •  One of the first to expose APIs /Services •  In early 2007, embarked on service orienting our entire

ecommerce platform, whether the functionality is internal or external

•  Support REST style as well as SOA style •  Have close to 300 services now and more on the way •  Early adopter of SOA governance automation (Discovery

vs. control)

20

Page 21: eBay Architecture

Architecture  Vision  

Application Platform Services

Login   Iden=ty   Catalog   Search   List   Pricing   Offer   ADs  Messages   Cart   Coupons  Payment  Shipping  CS  

Customer Experience

Core  Experience   Custom  Experiences   Channels  

Technology Platform

App  Stack  

Data  Access  Layer  

Dev  Tools   Presenta=on  

Messaging   SOA   Cloud  

Operations Infrastructure Layer

Power   Data  Center   Hardware   Network   Database   Opera=ons  Tools  

Page 22: eBay Architecture

Challenges  

• MulQple  data  formats  

•  Latency  

•  Service  consumer  producQvity  

22  

Page 23: eBay Architecture

Challenge  1:  Mul=ple  Data  Formats  

• Mix  of  user  preferences  –  SOAP  –  XML  /  HTTP  –  JSON  – Name-­‐Value  Pair  (NV)  

•  Service  developers  don’t  want  to  write  extra  code  to  do  conversions;  too  much  maintenance  impact  

•  Key  observaQons:  – Users  ask  for  whatever  data  format  they  want.  –  Anything  you  can  express  in  XML,  you  can  express  in  other  formats  

–  Complete  mapping  from  XML  structures  to  NV  and  JSON  

23  

Page 24: eBay Architecture

24  

Solu=on:  Pluggable  Data  Formats  Using  JAXB  

XML

Other formats

JSON

NV

A single Instance of Service Impl

JAXB Java objects

Passed to pi

pelin

e XML NV

JSON

Directly deserialized into

SOA framework

others Ser

/Des

er m

odul

e

Uniform interface

No intermediate format, Avoids extra conversion

Pluggable formats

Page 25: eBay Architecture

Challenge  2:  Latency  

•  For  large  datasets,  there  can  be  nasty  latencies.  – Not  fixed  by  compressing  or  using  Fast  Infoset  

25  

0 50

100 150 200 250 300 350

2MB structured response payload

Wire Time (msec)

Page 26: eBay Architecture

Solu=on:  Binary  Formats  •  Evaluated  binary  formats:  

•  Google  Protocol  Buffers,  Avro,  ThriY  

•  Numbers  look  promising  (serializaQon,  deserializaQon)  

•  New  challenges  with  these:  •  Each  has  its  own  schema  (type  definiQon  language)  to  model  types  and  messages  

•  Each  has  its  own  code  genera=on  for  language  bindings  •  NOT  directly  compaQble  with  JAXB  beans  

•  eBay  SOA  plaEorm  uses  WSDL/XML  Schema  (XSD)  data  modeling,  and  JAXB  language  bindings  

26  

Page 27: eBay Architecture

Compare  Popular  Binary  Formats  

27  

Protobuf Avro Thrift •  Own IDL/schema •  Sequence numbers for each

element •  Compact binary representation on

the wire •  Most XML schema elements are

mappable to equivalents, except polymorphic constructs

•  Versioning is similar to XML, a bit more complex in implementing due to sequence numbers

•  JSON based Schema •  Schema prepended to the message

on the wire •  Compact binary representation on

the wire •  Most XML schema elements are

mappable to equivalent, except polymorphic constructs

•  Versioning is easier

•  Own IDL/schema •  Sequence numbers for each

element •  Compact binary representation on

the wire •  Most XML schema elements are

mappable to equivalents, except polymorphic constructs

•  Versioning is similar to XML, a bit more complex in implementing due to sequence numbers

Complex  Types  

Unions  (Choice  Type)  

Self-­‐References  (Trees)   Enums  

Inheritance/Polymorphism   Inline  A`achment  

Protobuf   Yes   No   Yes   Yes   No   No  

Avro   Yes   Yes  Yes  (with  workaround)   Yes   No   No  

ThriY   Yes   No   No   No   No   No  

XML   Yes   Yes   Yes   Yes   Yes   Yes  (MIME-­‐TYPE)  

Page 28: eBay Architecture

Comparison  of  Data  Formats  

0

20

40

60

80

100

120

140

160

180

200

JSON XML Fast Infoset Protobuf

Size (KB) Wire time (msec)

28  

Response data: 50 items x 75 fields (about 8000 objects)

Page 29: eBay Architecture

Latency  Improvements  

0 20 40 60 80

100 120 140 160 180 200

XML XML no poly

XML flat PB no poly

PB flat

Wire Time(msec)

29  

Page 30: eBay Architecture

Challenge  3:    Service  Consumer  Produc=vity  

•  Large,  complex  requests  and  responses  

•  Get  exactly  what  they  want  in  data  returned  from  services  

•  Lack  of  consistency  in  service  interface  convenQons  and  data  access  pajerns  

•  Real  client  applicaQons  make  calls  to  mulQple  services  at  a  Qme  –  Serial  calls  increase  latency.  Managing  parallel  calls  is  complex  

•  Impedance  mismatch  between  service  interface  and  client  needs  –  Too  much  data  is  returned  –  1  +  n  calls  to  get  detailed  data  

30  

Page 31: eBay Architecture

Sneak  Preview:    

• New  technology  from  eBay  

•  Plan  to  open  source  soon  

•  SQL  +  JSON  based  scripQng  language  for  aggregaQon  and  orchestraQon  of  service  calls  

•  Filtering  and  projecQons  of  responses  

• Async  orchestraQon  engine  – AutomaQc  parallelizaQon,  fork  /  join  

31  

Page 32: eBay Architecture

What  ql.io  Enables  

•  Create  consumer-­‐controlled  interfaces    -  fix/patch  APIs  on  the  fly  

•  Filter  and  project  responses    -  use  a  declaraQve  language  

•  Bring  in  consistency    -  offer  RESTful  shims  with  simpler  syntax  

•  Aggregate  mul=ple  APIs  -  such  as  batching  

•  Orchestrate  requests  -  without  worrying  about  async  forks  and  joins  

32  

Page 33: eBay Architecture

ql.io  Examples  

•  Simple  Select  – select  *  from  ebay.finding.items  where  keywords=‘ipad’  

•  Field  ProjecQons  – select  Qtle,  itemId  from  ebay.finding.items  where  keywords=‘ipad’  

•  Sub-­‐Select  – select  e.Title,  e.ItemID  from  ebay.item.details  as  e  where  e.itemId  in  (select  itemId  from  ebay.finding.items  where  keywords  =  ‘ipad’)  

33  

Page 34: eBay Architecture

ql.io  Batch  Example  

itemId  =  select  itemId  from  ebay.finding.items  where  keywords  =  'ferrari'  limit  1;  

item  =  select  *  from  ebay.shopping.singleitem  where  itemId  =  '{itemId}';  

user  =  select  *  from  ebay.shopping.userprofile  where  userId  =  'sallamar';  

tradingItem  =  select  *  from  ebay.trading.geQtem  where  itemId  =  '{itemId}';  

bestOffers  =  select  *  from  ebay.trading.bestoffers  where  itemId  =  '{itemId}';  

bidders  =  select  *  from  ebay.trading.getallbidders  where  itemId  =  '{itemId}';  

return  {    

     "user"  :  "{user}",  

     "item"  :  "{item}”,  

     "tradingItem"  :  "{tradingItem}",  

     "bidders"  :  "{bidders}",  

     "bestOffers"  :  "{bestOffers}"  

};  34  

Page 35: eBay Architecture

ql.io  Demo  

35  

Page 36: eBay Architecture

Modularity  

36  

Page 37: eBay Architecture

Key  modularity  concepts  for  soYware  

• Building  blocks  

• Re-­‐use  

• Granularity  

• Dependencies  

•  EncapsulaQon  

• ComposiQon  

• Versioning  

37

Source: http://techdistrict.kirkk.com/2010/04/22/granularity-architectures-nemesis/ Author: Kirk Knoernschild

Page 38: eBay Architecture

Challenges  for  Large  Enterprises  

•  Some  stats  on  the  eBay  code  base  – ~  44  million  of  lines  of  code  and  growing  – Hundreds  of  thousands  of  classes  – Tens  of  thousands  of  packages  – ~  4,000+  jars  

• We  have  too  many  dependencies  and  Qght  coupling  in  our  code  – Everyone  sees  everyone  else  – Everyone  affects  everyone  else  

 38

Page 39: eBay Architecture

Challenges  for  Large  Enterprises  

•  Developer  producQvity/agility  suffers  as  the  knowledge  goes  down  –  Changes  ripple  throughout  the  system  –  Fallouts  from  changes/features  are  difficult  to  resolve  –  Developers  slow  down  and  become  risk  averse  

39  code size

knowledge complexity

Page 40: eBay Architecture

Our  Goals  with  Modularity  Efforts  

•  Tame  complexity    

•  Organize  our  code  base  in  loose  coupling  fashion  – Coarse-­‐grained  modules:  number  majers!  – DeclaraQve  coupling  contract  – Ability  to  hide  internals  

•  Establish  clear  code  ownership,  boundaries  and  dependencies  

•  Allow  different  components  (and  teams)  evolve  at  different  speeds  

•  Increase  development  agility  

40

Page 41: eBay Architecture

Modularity  Solu=ons  Evalua=on  

•  Evaluated  OSGi,  Maven,  Jigsaw  and  JBoss  Module  

• Criteria  include:  – Modularity  enforcement  – End-­‐to-­‐end  development  – MigraQon  concerns  – AdopQon  – Maturity  

•  Selected  OSGi  

41  

Page 42: eBay Architecture

OSGi  @  eBay  

•  Modularize  plaEorm  into  OSGi  bundles  with  well-­‐defined  imports  and  exports  

•  Challenges:  split  packages,  Classloader  contructs  

•  Source  to  binary  dependencies  

•  Refresh  end-­‐to-­‐end  development  life  cycle  

42  

Repository

IDE

SCM

Server runtimeDeployment

publish/consumeconsume

packaging deploy

Command line

build (CI)

pull/push pull

Page 43: eBay Architecture

Lessons  Learned  

•  OSGi  learning  curve  is  sQll  fairly  steep  –  large  group  of  developers  with  varying  skill  levels  

•  End-­‐to-­‐end  development  lifecycle  –  Tools  may  not  work  well  together.  Leverage  OSGi  tools  like  bnd  

•  Conversion/migraQon  of  exisQng  code  base  –  Not  starQng  from  vacuum  –  Cost  to  rewrite  /  refactor  code  – We  cannot  afford  disrupQon  to  business  meanwhile:  “change  parts  while  the  car  is  running”  

•  SemanQc  versioning  adopQon  is  important  

 

 

43

Page 44: eBay Architecture

Overall  Summary  

•  Strategies  – Deployment  Agility:  AutomaQon  with  Cloud  – Development  Agility:  Next  gen  Service  OrientaQon  – Taming  complexity:  Modularity  

•  Systems  quality  &  scalable  architecture  as  key  foundaQon  

• Complexity  management  and  developer  producQvity  becomes  increasingly  important  

•  Strike  balance  between  agility  and  stability  

44  

Page 45: eBay Architecture

eBay Open Source

45

(Coming soon)

•  eBay has been a strong supporter of Open Source model and community

•  Check out http://eBayOpenSource.org •  Mission is to open source some of the best of breed

technologies that were developed originally within eBay Inc.

•  Under a liberal open source license. •  These projects are generic technology projects and

several years of development effort has gone into them to mature them.

Page 46: eBay Architecture