Pascal Alberty@zorrobiwan

CTO Managing Partner

History

More than 50 web specialistsAbout 20 companies or independent people

RueDuWeb

Where ?

Demo

Main Architecture

● Front (Akimedia)

○ Public Website○ Private Website○ Flash Animations○ Sessions configuration management

● Back (Tesial)

○ Sessions content management

Front - Key components (Akimedia)

● CMS Akiwi (PHP) & MySQL database (MyISAM)○ Pubic Website○ Private Website (users, eCommerce

● Flash animations○ Displaying messages and other features

● “Manager” in PHP○ Communication between Animations/Front/Back

Back - Key Components (Tesial)

● Engine○ Java web application○ API used by Front for data management

■ sessions, tweets, moderation, tweetos,...

● Batch○ Java web application○ Formerly grabbing Tweets○ Grabbing pictures (avatars & medias)

Technical Infrastructure

● Front/Back Server

● DB Servers○ Front / Back○ Replication (fail over)

● Dev Server○ dev (front / back)○ pré-prod (front / back)

SVN

● KISS

● dev = trunk● pre-prod = branch● prod = tag on branch or trunk

● front = release by switch● back = release by packaging

Back Software Architecture

Maven 2Spring 3.1Hibernate 3.2ehcache 1.5Twitter4J 3Maya 0.2...

Back Software Architecture

● DTO & DAO○ Data Structure & persistence

■ session, tweet, tweetos, hashtags, ...● Services

○ Business functionalities○ Transaction management

● Clients○ Engine○ Batches

Engine

● Java Web Application

● API REST○ CRUD operations

● Used by Front for data management

● Call Services layer

Batches

● Java Web Application

● Quartz Jobs○ Formerly Tweets Grabbing ○ Avatars Grabbing○ Pictures Grabbing

● Call Services Layer

Challenges

● Scalability○ SaaS and Agency are growing

● Twitter○ New technical and business constraints

● Product○ Must evolve by integrating other social networks

Twitter API

● Let’s talk about Twitter API

Twitter APIs

● APIs○ Search API○ REST API○ Streaming API○ https://dev.twitter.com/docs/history-rest-search-api

● Versions○ API version 1 (deprecated)○ API version 1.1

Rate limiting

● Rate limit window duration○ 15 minutes

● Requests allotted○ per user○ via application-only auth

● Example○ GET statuses/user_timeline○ per user: 150 requests per 15 minutes○ via app: 300 requests per 15 minutes

https://dev.twitter.com/docs/rate-limiting/1.1/limits

Other technical limits

● Direct messages○ 250 per day

● Tweets○ 1,000 per day○ Retweets are counted as Tweets.

● Changes to account email○ Four per hour

● Following○ 1,000 per day

● Following (account-based)○ up to following 2,000 other users

Platform objects

● Tweets can be found alone, within user objects, but most often within timelines

● Users can be found tweeting, following, and favoriting on Twitter

● Entities are most often found within Tweets● Places can be found throughout the natural universe,

but typically only appear attached to Tweets on Twitter

Objects type

● JSON !

● XML output will be dropped soon○ Only JSON output with API 1.1

● Be careful with ids (64 bits)○ use String version (especially if using Javascript)○ Twitpocalypse

■ https://dev.twitter.com/docs/twitter-ids-json-and-snowflake○ Idpocalypse

■ https://dev.twitter.com/blog/64-bit-twitter-user-idpocalypse

Tweet Example

https://api.twitter.com/1/statuses/show.json?id=341545927401680897&include_entities=true

User example

https://api.twitter.com/1/users/show.json?screen_name=zorrobiwan&include_entities=true

Entities / Entity Example

● Hashtags, Medias, URLs, User mentions, Symbols

● No need to parse● Data used for display also (ex: display_url)

Place example

https://api.twitter.com/1/geo/id/247f43d441defc03.json

Libraries

● PHP○ Twitter-async (https://github.com/jmathai/twitter-

async)○ tmhOAuth (https://github.

com/themattharris/tmhOAuth)○ Zend Framework (http://framework.zend.

com/manual/1.12/en/zend.service.twitter.html)● Java

○ Twitter4J (http://twitter4j.org/)

https://dev.twitter.com/docs/twitter-libraries

Twitter4J

● Unofficial Java library for the Twitter API● 100% Pure Java (5+)● Android platform and Google App Engine

ready● Zero dependency : No additional jars

required● Built-in OAuth support● Out-of-the-box gzip support● 100% Twitter API 1.1 compatible

Tools

● Twurl○ https://github.com/marcel/twurl○ "curl" for Twitter API○ Manage access tokens (authentication)

● Apigee Console○ https://apigee.com/console○ Free console to execute APIs

Search API

● Similar to http://search.twitter.com

● Criteria○ terms○ geocode○ language ("Language detection is best-effort")○ count, until, since-id, max-id○ result type: mixed, recent, popular○ include entities

● Paginated !○ you have to manage this by yourself

REST API

● Timelines● Tweets● Direct Messages● Friends & Followers● Users● Suggested Users● Favorites● Lists● Saved Searches● Places & Geo● Trends● Spam Reporting● OAuth● Help

Streaming API

● Persistent HTTP connection○ until you decide to close it

● Could be reeeeaaaaaaally huge !○ Think about you architecture (hard and soft) !○ Separate storage and consumption

● Only one stream opened at a time

● Could be tricky to managed○ Use a library

Streaming API endpoints groups

● Public Streams○ GET statuses/sample○ POST statuses/filter○ GET statuses/firehose (limited access)

● User Streams○ Data and events for a specific user

● Site Streams (beta and limited access)○ Real time update for large number of users

About the challenges

● Scalability● Twitter constraints● Product evolution

=> Use of Twitter Stream API=> Other Social Networks integration

=> Maya

Maya

Abstraction layer for feed andpost

Used in TWP as the input sourcefor media

Maya - General architecture

Listener

Listener

SessionSession

Maya - concepts

● Post: content emitted on a (social) network by a user

● Feed: a collection of posts matching a set of filters (PostFilter)

● FeedOperator: the service to register, manage stop and restart feeds

Maya twitter plugin

● Backed by Twitter4J● Rely on the live stream twitter API● Challenges

○ Adding new feeds on the fly without losing any history

○ Single channel for all feeds, need to identify to which feed(s) a tweet belongs to

● Demo!

Front Evolutions

● From Pull to Push○ NodeJS

● Manager from PHP to JS○ NodeJS

● HTML5 Animations

Next Improvements

● Queuing traitement with SI○ Message traitement○ Medias grabbing and/or treatment

● Adding other social networks

● Better scalability - reliability - performance○ load balancing○ replication○ nginx○ ...

Q/A