osdc.tw - building a platform from open source

Building a platform from open source

SYMFONY-PROJECT.COM | DEVELOPER.YAHOO.COM

Overview

Why symfony?

symfony vs. ysymfony

Social Search: Delicious and Answers

International: ShopGenie.co.uk, FoxyTunes, Digu

Yahoo! Open Strategy Developer Tools

YUI, Design Patterns, Tutorials

Data & Social APIs YQL: Yahoo! Query Language

Who am I?

Working with symfony since open source symfony Core Team Member

Responsible for the development and support of ysymfony at Yahoo! Y! Answers, Delicious, Y! Widgets, Y! Application Platform

Developer advocacy for Yahoo! Developer Network

Contact [email protected]

@dustinwhittle

DEVELOPER.YAHOO.COM EXAMPLES | TUTORIALS | CODE SAMPLES

YAHOO! EMBRACES OPEN STANDARDS

W3C | MICROFORMATS | OAUTH | OPENID | OPENSOCIAL

YAHOO! GIVES BACK TO OPEN SOURCE

YUI | BROWSER PLUS | DESIGN PATTERNS | R3 | YSLOW + PERFORMANCE RULES

YAHOO! ENGAGES COMMUNITIES WITH OPEN HACK EVENTS AROUND THE WORLD

Conferences | Hack Days | HackU | Tech Talks | YDN Theater

Users

Load Balancers

Frontend

PHP APC, PEAR, PECL, Custom Extensions

Linux

ysymfony / YUI Apache Custom Modules

Backend

MySQL/Memcache YQL / Web Services Ad API User API

Why a frontend platform?

Rasmus says “frameworks are not well suited for Yahoo!” Build applications to requirements

Do exactly what you need: no more, no less

Understand that frameworks add a lot of overhead

Choosing functional components is a better t

Despite choosing open source or building your own Everyone uses a framework

If you use open source, use only the pieces you need

Yahoo! needs from a frontend platform Fit existing environment (RHEL/PHP5/Apache) Development Cycle – How easy to develop, test, and

deploy? Clean separation between data, logic, and display (MVC) Independent model layer to t service oriented

architecture Extensible and pluggable Internationalization and localization support Detailed documentation and active community of

support Open source and ability to contribute back

Why a framework at all?

Another software layer (ysymfony, yphp, yapache) Factors out common patterns

Code Layout Conguration URL Routing Authentication / Security Form Validation / Repopulation Internationalization / Localization

Encourages good design Abstraction > Consistency > Maintainability

The choice to adopt symfony?

Philosophy Full-stack framework for building complex web

applications Adopt best ideas from anywhere, using existing code if

available (Mojavi, Prado, Rails, Django)

Design Clean separation between Model, View, and Controller Controller using modules and actions Views using templates in straight PHP with helpers Easy to reuse view modules to compose a page

Layouts, Components, Partials, Slots

The choice to use symfony

Congurability / Flexibility Features we do not want are easily disabled

Use of factories for easy customization

Documentation / Support Community The Denitive Guide to symfony

Askeet, Jobeet, Cookbooks, Advents

Active community with wiki, mailing lists, forums, irc channel

Why ysymfony for Yahoo! teams?

Eliminate common patterns by adding a layer on PHP Code layout/structure (MVC)

Conguration

Internationalization

ysymfony is just a toolkit Learn one set of tools

Shift between multiple projects

Consistency

Long term maintainability through platform

How Yahoo! uses symfony Full Stack

Yahoo! Bookmarks Yahoo! Answers Delicious Yahoo! Widgets Yahoo! Application Platform ShopGenie.co.uk, Foxytunes, Digu Internal Dashboards + Reporting Tools

Components Dependency Injection Forms Caching (APC/Memcache) Routing

A look at Yahoo! Answers

http://answers.yahoo.com

Yahoo! Answers is the largest collection of human knowledge on the Web with more than 135 million users and 515 million answers worldwide (Yahoo! Internal Data, March 2008).

Yahoo! Answers is the 2nd ranked education & reference site on the web (comScore)

Available in 26 markets and 12 languages

Yahoo! Answers at the beginning

Started as a small development team on PHP4 from a fork of Yahoo! Taiwan Knowledge+

Launched December 2005 by December 2006 there were 60 million users and 65 million answers

The code base eventually became difcult to maintain and iterate new features

Large distributed development teams (US / UK)

The big picture

A complete platform for building web applications from frameworks PHP Framework

JavaScript Framework

CSS Framework

UI Design Patterns + Best Practices

Development Tools (logger, proler, debugger, docs)

Unit + Functional Testing Frameworks (LIME / YUI Test)

Deployment Tools (rsync deployment system)

What does Yahoo! change? Minor changes to t our environment

Most changes are easily implemented via factories/plugins

Dropped the ORM and pushed down the stack (SOA) Added a parallel API Dispatcher (ysfAPIClientPlugin)

Added dimensions to congurations (ysfDimensionsPlugin) Integrated R3 translation/template management (ysfR3Plugin)

R3 - http://developer.yahoo.com/r3/

Created a build and deployment solution (ysfBuildPlugin) Uses internal tools for packaging/deployment

Integrate support for Y! User Interface libraries (ysfYUIPlugin)

Propel or Doctrine or ???

No ORM for large projects

Doctrine for medium sized projects

Service Oriented Architecture Platforms/Backends as services (reusable to all)

Thin Controller/Fat Model (where model == services)

Use PHP as the frontend glue No heavy lifting in PHP = Push down the stack

Java/C++/Erlang + JSON/XML

Localizing with dimensions

Cascading Conguration based on YAML Framework -> Project -> Application -> Module

Extending the cascade to be based on dimensions Dimensions can be anything (and can be chained together)

Data Center + Environment for customizing congurations

Culture for localizing user interface + data

Theme for customizing look and feel

User info (is user on corporate intranet?)

Caching

A build and deployment system

Aggregate and minify stylesheets and javascripts Rewrite templates, css, js for CDN (Akamai, S3, …) Generate translations for congurations + templates Generate conguration cache Aggregate core classes + remove debug statements Run lint, unit, functional tests Package applications as .tgz Deployment via packages Integrated with Yahoo! internal tools

Debug Toolbars

Hello World Performance

Hello world benchmarks are generally not useful You don’t use a framework to write hello world

die(‘hello world’);

Performance is relative to the features you use ORM

I18N

Output Escaping

Forms/Validation

Ability to scale != performance

What does it mean to scale?

A system whose performance improves after adding hardware, proportionally to the capacity added, is said to be a scalable system.

High Availability + Scalability + Performance Bigger dataset, more trafc, maintainable Not about performance

PHP is slow, but it is very rarely your bottleneck

Languages do not scale, architectures do Planning to grow and planning to fail

Capacity Planning Business Continuity Planning

Scaling – The basics in PHP PHP is rarely the bottleneck (even though it can be slow)

“Most performance comes not from the language, but from application design” – Rasmus

Share Nothing Architecture Independent, self-sufcient, no single point of contention No local storage = No PHP Sessions

Use a database (works for distributed) Use a small signed cookie (ideal)

Important data in database Individual expiration on session objects Small data items

Use a distributed cache Memcache

Forget about small efciencies Premature optimization is the root of all evil.

“ vs. ‘, echo vs. print

Scaling Databases – The basics

Drop the ORM It’s a choice of convenience over performance

Master/Slave Replication First steps Helps with reads, writes are still bottleneck

Partitioning Segmenting data

Sharding (horizontal partitioning) Segmenting data onto different physical machines Make problems smaller, easier to grow

Improving latency with Caching

Always use PHP opcode cache (APC, Xcache, etc) Use for routing and i18n cache

Memcache (distributed cache) Use for view cache

Distributed invalidation can be a pain sfViewCacheManager makes this easy!

Be intelligent about cache_keys (uri, user, state)

There is a ne line to caching At what point do you spend more time managing the

cache, than reading from it?

Tweaking Performance

Do not use .htaccess (move to real apache cong) Set a minimal include path Increase realpath_cache_size + realpath_cache_ttl Use apc.stat=0 Don’t use features you do not need

Disable in settings.yml / factories.yml

Use core_compile to aggregate classes to reduce le i/o Remove debug statements + optimize le lookups

sfOptimizerPlugin / project:optimize

Use @routeName and use caching for factories

Symfony (v2)

Symfony 2 is a set of cohesive yet decoupled components This makes it much easier to use single component to solve

a single problem Which makes it easier to build micro frameworks that solve

very specic problems Yahoo! Teams generally prefer solutions that are specic to

their exact problem Selling the full stack can be difcult when a team only wants a few

components

Symfony 2 is the right direction, even if it breaks backwards compatability

symfony-reloaded.org

Do it yourself for cheap

Open source software = Free Apache PHP MySQL Memcache / Perlbal / MogileFS / Squid / Gearman symfony / Doctrine / Propel / Swift Nagios / Cacti

Amazon Shared Infrastructure = Cheap EC2 Cloud S3 Storage + Cloudfront SimpleDB

What is Yahoo! Developer Network? The Yahoo! Developer Network offers open source tools

and open data APIs to make it easy for developers to build applications and mashups.

50+ APIs / Web Services Developer Dashboard to create/manage Oauth applications Tutorials + Code Samples on using our apis Complete API Documentation

Yahoo! User Interface libraries ASTRA Flash Components Design Patterns Library Evangelism: Conferences / Theater / Blogs / Events

JavaScript Framework Utilities - YAHOO, Dom, Event, Animation, Browser History

Manager, Connection Manager, Cookie, DataSource, Drag and Drop, Element, Get, ImageLoader, JSON, Resize, Selector, Loader

Controls / Widgets - AutoComplete, Button, Calendar, Charts, Color Picker, DataTable, ImageCropper, Rich Text Editor, Slider, Uploader

Container (Module, Overlay, Panel, Tooltip, Dialog), Layout Manager, Menu, TabView, TreeView

Debug – Logger, Proler, Test

Y! Developer Network – YUI Javascript

CSS Foundation Reset - Neutralizes browser CSS styles

Base - Applies consistent style foundation

Fonts - Foundation for typography and font-sizing

Grids - Thousands of wireframe layouts

User Interface Design Patterns Library Proven solutions to common interfaces

http://developer.yahoo.com/ypatterns/

Grade Browser Support / Progressive Enhancement

Y! Developer Network – YUI CSS

More than 275 functional examples http://developer.yahoo.com/yui/examples/

YSlow + Performance Rules http://developer.yahoo.com/performance

YUI Blog http://yuiblog.com/

Mailing List @ Yahoo! Groups http://tech.groups.yahoo.com/group/ydn-javascript/

Y! Developer Network – Documentation

A Query Language for the Web

SELECT * FROM INTERNET

Thousands of web services that provide valuable data

Require developers to read documentation and form URLs/queries.

Data is isolated and can not be combined

Needs combining, tweaking, shaping even after it gets to the developer.

Before YQL

SQL-Like Language Synonymous with Data access

Familiar to developers

Expressive enough to get the right data

Self Describing - show, desc table

Allows you to query, lter and join data across Web Services.

Y! Open Stack – YQL

developer.yahoo.com/yql/console/

YQL – Open Tables

Twitter Google Facebook Friendfeed Wesabe Whitepages Zillow Search Weather

Flickr Upcoming Delicious Dopplr Github New York Times Shopping …any web service can be

as easy as SQL

Available on github - http://github.com/spullara/yql-tables/

YQL – Open Tables

select * from social.connections

select * from delicious.feeds.popular

select * from ickr.photos.interestingness

select * from friendfeed.status

select * from github.checkins

YQL - Examples

OAuth Endpoint

http://query.yahooapis.com/v1/yql?q=...

Public Endpoint

http://query.yahooapis.com/v1/public/yql?q=

YQL Console

http://developer.yahoo.com/yql/console

YQL – API End Points

Open Source Communities

http://www.symfony-project.org/jobeet/1_2/

Documentation

http://www.symfony-project.org/doc

Practical symfony - Jobeet

The Denitive Guide to symfony

The symfony Reference Guide

symfony Forms in Action

Doctrine ORM for PHP

Cookbooks + Cheat sheets + Wiki

API Documentation

Books worth reading

QUESTIONS? (I will be around, so please say hello and ask questions)

DEVELOPER.YAHOO.COM EXAMPLES | TUTORIALS | CODE SAMPLES

WANT TO JOIN YAHOO? WE ARE HIRING!

osdc.tw - building a platform from open source

Technology