unlocked london - technical track

No 11 cavendish

square

11 july 2013

london

Thursday, July 11, 13

developer VOCAB

Alex brandtsr developer

MODULES

LIBRARIES

FRAMEWORKS

•The building blocks of an application - a set of common software functions that take the work out writing all code from scratch.

LANGUAGES

What is a data structure?

•images• url

• stars

• *title

• caption

• variants

•small

•medium

•big

•tags

•*_id

•rolestars•*image_id

•*role_id

•roles• *name

• privileges

• login

• description

• location

• child_roles

• _id

API tier/Application

STORAGE

APPLICATION

PRESENTATION

API tier/Application

STORAGE

APPLICATION

PRESENTATION

APPLICATION/API

Bindings

EVERYTHING IS CODE

wayne wallscloud evangelist

alex brandtsr developer

Infrastructure as code

Chef Server

Chef Workstation

Chef Client

environment

•Nodes: Servers

•Environments: Groups of nodes (Dev/QA/Prod)

•Recipes: Describes the desired state of a node

•Cookbooks: Group of Recipes

•Roles: Group of Cookbooks (Web)

•Attribute: information about the current state of a node

PuppetCA

PuppetMaster

PuppetAgent

•Nodes: Servers

•Manifests: Describes the desired state of a node

•Catalog: Group of Manifests

•Facts: information about the current state of a node

SaltMaster

SaltMinion

•Nodes: Minions

•States: Describes the desired state of a node

•Grains: information about the current state of a node

Thinking like a developer30

scrum Development

Product Backlog

Sprint Backlog Sprint Working increment

of the software

1. Design

5. QA 2. Build

4. Release 3. Configure

scrum Development

1. Design

5. QA 2. Build

4. Release 3. Configure

TYPES OF DEVELOPMENT

Test Driven Development

Behavior Driven Development

Behavior Driven Development Domain Driven Development

Application Bliss

Domain Driven Development

Model View Controller (MVC)

UPDATES MANIPULATES

SEES USES

VIEW CONTROLLER

DEv tools

Gerrit

CODE REVIEW PROCESS

Our hero, the developer, submits code for review.

The code is reviewed in Gerrit by other developers, who approve the change.

Jenkins runs automated tests against the code change.

Jenkins pushes the code to GitHub so others can build on the change.

POLYGLOT PERSISTENCE

technical track

A database, in the most general sense, is an organized collection of data.

Database Defined

http://glenn-packer.net

•Atomicity: refers to a database’s ability to either fully process or fully roll back a transaction

•Consistency: database should ensure that all data written therein follows all rules and constraints specified in the database

•Isolation: transactions must be processed securely and independently, without interfering with each other

•Durability: the database must ensure that all committed transactions are saved permanently and cannot be accidentally erased, even in a database crash

•A relational database is essentially a group of tables or, to use the technical name, entities.

•Each table is made up of rows (tuples) and columns (attributes).

relationaldatabases

Some companies needed something more to accommodate their growing data

rdbms may not be the best fit

Enter Non-Relational database

management systems

INTRODUCTION TO NOSQL

• NoSQL Database (1998)• NoREL• “Not only SQL”• NoSQL (2009)

NoSQL is difficult to define but most NoSQL databases will share common characteristics

1. Non-relational

2. Distributed

3. Open-source

4. Horizontally Scalable

5. Schema-less

An “aggregate” is a collection of related objects that we wish to treat as a unit

orders

customers

order lines

credit cards

ID: 1001

line items:

customer: Ann

0321293533

0321601912

0131495054

$48 $48

payment details:

Card: AmexCC #: 12345expires: 04/2001

aggregate

There are several different types of data stores that are

considered to be NoSQL

•Key/Value Databases

•Document Databases

•Column-Family Stores

•Graph Databases

Key/Value Database•Primarily constructed of aggregates that have a Key or ID that’s used to access the data

•The database cannot see structure within the aggregate

Key/Value Database

•Good Use Cases:

• Storing Session Information

• User Profiles & Preferences

• Shopping Cart Data

• Bad Use Cases:

• Data Relationships

•Multi-operation Transactions

•Query by Data

•Like a Key-Value database, document databases primarily store and access data using aggregates

•In contrast to the Key-Value database, document data bases can see structure within the aggregate.

•You are able to retrieve the whole aggregate or specific parts of it

document Database

document Databases

•Good Use Cases:

• Event Logging

• Blogging Platforms

• Real-time Analytics

• Bad Use Cases:

• Complex Transactions

•Queries against Varying Aggregate Structures

• Can be looked at as a two-level aggregate structure

• The first key is often described as a row ID picking up the aggregate of interest

• The second level values of the aggregates are referred to as columns

• You are able to access a row as a whole or operations will allow you to pick out a particular column to get specific data

COLUMN FAMILY

COLUMN-FAMILY datastores

•Good Use Cases:

• Event Logging

• Blogging Platforms

• Counters (page visits)

• Expiring Usage

• Bad Use Cases:

• ACID transactions required

• Early prototype applications

•Unique compared to most NOSQL types because it was not developed with the intention of scaling horizontally into a cluster

•Data is stored as “nodes” that are connected by edges

•Edges are directional in nature and must be specified

•Queries need a starting point and them will traverse the edges to gather the data requested

graph database

Amanda

Robert

BUILTBUILT

Christie

Destiny

Jessica

graph database

graph databases

•Good Use Cases:

• Connected Data

• Location-based services

•Recommendation engines

• Bad Use Cases:

• Ability to update specific aggregates

Companies using nosql

DATABASE COMPANIES

THE FUTURE IS POLYGLOT

Neo4Jdatabase

Cassandradatabase

Couchdatabase

MongoDBdatabase

RELATIONALDATABASE

Graph-structureddomain rules

Columnar dataaccess with

decentralization

Documentstructures

Document structureswith offlineprocessing

(Actors) (Actors)

(Actors)(Actors)

Asynchronous message passing

MODULE 1

MODULE 2

MODULE 3

MODULE 4

IEEE Software Sept/October 2010 - Debasish Ghosh / Twitter @debasishg

POLYGLOT PATTERN: cache

ORDERTAKING

RESTAURANTMANAGEMENT

REDISCACHE

MYSQL DATABASE SECONDFIRST

CONSUMERRESTAURANT

POLYGLOT PATTERN: write to sql and NOSQL

NEWTWEET

INSERT

LPUSHXLTRIM

TWEETS

FOLLOWS

FOLLOWERS

Redis: Timeline

FOLLOWER 1

FOLLOWER 2

FOLLOWER 3

Avoidexpensive

MySQL joins

POLYGLOT PATTERN: REPLICATE FROM MYsql to NOSQL

QUERYSERVICE

UPDATESERVICE

REDISMYSQL

DATABASE

SYSTEMOF RECORD

MATERIALIZED VIEW

READER WRITER

replicate and

denormalize

query() update()

Atomic

Consistent

Isolated

Durable

ACID to base

Basically Available

Soft state

Eventually consistent

5 PILLARS OF CLOUDINESS

technical track

FIVE PILLARS OF CLOUDINESS

1 2 3 4 5

Parallel Modular Horizontal Agile Secure

Parallel

Parallel - what

1MRecords

Answer

Parallel - how

1MRecords

Answer

Node1-250K

Node251K-500K

Node501K-750K

Node751K-1M

Parallel - how

1MRecords

Answer

Node1-250K

Node251K-500K

Node501K-750K

Node751K-1M

Parallel - how

1MRecords

Answer

Node1-250K

Node251K-500K

Node501K-750K

Node751K-1M

Parallel - how

CALCULATIONMAP

CALCULATIONREDUCE

SHUFFLE

Avoid Deadlock

Thread 1

Deadlock

Thread 2

Thread 1

Deadlock

Thread 2

File A

Thread 1

Deadlock

Thread 2

File A

Thread 1

Deadlock

Thread 2

File A File B

Thread 1

Deadlock

Thread 2

File A File B

Thread 1

Deadlock

Thread 2

File A File B

Thread 1

Deadlock

File B

Thread 2

File A File B

Thread 1

Deadlock

File B

Thread 2

File A File B

File A

Thread 1

Deadlock

File B

Thread 2

File A File B

File A

Thread 1

Deadlock

File B

Thread 2

File A File B

File A

Thread 1

Deadlock

File B

Thread 2

File A File B

File A

Thread 1

Deadlock

File B

Thread 2

File A File B

File A

Thread 1

Deadlock

File B

Thread 2

Solution? Scheduling

Parallel - why

1M records

Answer

1M records

Answer

Node1-250K

Node251K-500K

Node501K-750K

Node750K-1M

Parallel - why

1M records

Answer

1M records

Answer

Node1-250K

Node251K-500K

Node501K-750K

Node750K-1M

Parallel - why

modular

ONE PERSON BUILDS A CAR

Modular - whatExample: Assembly Line

CHASSIS ENGINE BODY INTERIOR

DecoupledSelf-contained unitsData Communication

Modular - whatExample: Assembly Line

Modular - what

Modules within Modules

Nested Modularity

Modular - what

Modules within ModulesNested Modularity

Modular - what

SEATS FLOOR

DASHBOARD

GASAND

BRAKE PEDALS

Modules within ModulesNested Modularity

Small units are more maintainableAllows us to tolerate some partitions (but not all)

Allows us to be parallel!

modular - Why

Direct communication (1:1)Topic communication (1:N)

Fanout communication (1:ALL)

modular - Why

•Decouple requests

•Message Queue

•Make small units of work

•Define tiers by function

modular - how

Potential for Upstream Complexity

defect

defect defect

defect defect defect

defect defect defectdefect

defect defect defectdefect FAIL

Mandelbugs

horizontal

2 GB2 CORES

Horizontal - What

8 GB4 CORES

2 GB2 CORES

VERTICAL

Horizontal - What

8 GB4 CORES

2 GB2 CORES

VERTICAL

Horizontal - What

8 GB4 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

8 GB4 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

8 GB4 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

8 GB4 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

8 GB4 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

8 GB4 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

8 GB4 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

8 GB4 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

8 GB4 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

SHARED NOTHING THROUGH STATELESSNESS

Horizontal - What

8 GB4 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

8 GB4 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

2 GB2 CORES

8 GB4 CORES

Horizontal - why

ONE SERVER NEAR

CAPACITY

Horizontal - why

ONE SERVER NEAR

CAPACITY

ADD ANOTHER SERVER

Horizontal - why

ONE SERVER NEAR

CAPACITY

ADD ANOTHER SERVER

Horizontal - why

1234 MESSAGE QUEUE

Horizontal - HOW

1234MESSAGE QUEUE

Horizontal - HOW

MESSAGE QUEUE

Horizontal - HOW

56789101112 MESSAGE QUEUE

Horizontal - HOW

5678 9101112MESSAGE QUEUE

Horizontal - HOW

5678 9101112

ADD MORE SERVERSBASED ON

MESSAGE QUEUE

Horizontal - HOW

MESSAGE QUEUE

13141516

MESSAGE QUEUE

Horizontal - HOW

5678 9101112

13141516

MESSAGE QUEUE

Horizontal - HOW

5678 9101112

13141516

MESSAGE QUEUE

Horizontal - HOW

5678 9101112

13141516

MESSAGE QUEUE1234

MESSAGE QUEUE

Horizontal - HOW

5678 9101112

13141516

MESSAGE QUEUE1234

MESSAGE QUEUE

Horizontal - HOW

DECREASE SERVERSBASED ON

MESSAGE QUEUE

5678 9101112

MESSAGE QUEUE1234

Horizontal - HOW

DECREASE SERVERSBASED ON

MESSAGE QUEUE

Agile - WHAT

the power of moving quickly and easily; nimbleness

Features In Progress Blocked Tabled Testing Complete

Feature A

Feature B

Feature C

Agile - why Monday

Agile - why

Feature A

Feature B

Feature C

Tuesday

Agile - why

Feature A

Feature B

Feature C

Feature E

Feature D

Tuesday

Feature AFeature BFeature C

Feature E

Feature G

Agile - why

Wednesday

Feature AFeature BFeature C

Feature E

Feature D

Feature F

Feature G

Agile - why

Wednesday

Feature A

Feature B

Feature C

Feature E

Feature D

Feature F

Feature G

Agile - why

Friday

Feature A

Feature B

Feature C

Feature E

Feature D

Feature F

Feature G

Feature H

Feature I

Agile - why

Friday

Feature A

Feature B

Feature C

Feature EFeature D

Feature F

Feature G

Feature H

Feature I

Agile - why

Friday - WK 2

Agile - why

Failed Successful Challenged

Waterfall Agile

Source: The CHAOS Manifesto, The Standish Group, 2012

• CI/CD when appropriate

• Quality is in the eye of the beholder

• Release fast, release slow

Agile - How

IMAGE: blogs.bu.edu

Agile - How

• Increase quality by reusing code from partners and libraries

Agile - How

Lint Testing

Functional Testing

Black Box

Integration Testing

Unit Testing

White Box

Smoke Testing

Acceptance Testing

ENSURES QUALITY

Agile - How

Lint Testing

Functional Testing

Black Box

Integration Testing

Unit Testing

White Box

Smoke Testing

Acceptance Testing

ENSURES CONFIDENCE

Agile - How

Lint Testing

Functional Testing

Black Box

Integration Testing

Unit Testing

White Box

Smoke Testing

Acceptance Testing

secure

Security - what

•All about Trust•Provider•Customer•User

SECURITY - HOW•Security at every layer

PRESENTATION

APPLICATION

STORAGE

SECURITY - HOW•Security at every layer

PRESENTATION

APPLICATION

STORAGE

INTERFACESECURITYACROSS ALL LAYERS

SECURITY - HOW

•Use industry standards

•SSL

•Encryption

•At rest

•In-transit

SECURITY - HOW•Authentication

•User authentication

•Manage yourself

•Offload to 3rd party

•Host-based Authentication

•Security Groups

•Firewalls

•Configuration Management Systems

SECURITY - HOW•Hybrid Cloud

•Store different types of data in the right place

•Sensitive data in on-premise cloud

•Non-sensitive on public cloud

SECURITY - WHY

•Trustworthiness as an organization

•Cloud security is too important to leave up to the end-user alone

•Ownership gaps that leads to process and governance issues

•Compliance

•Logging and auditing

•Regulatory requirements

3:00 - 3:15

APPLICATION dissection

technical track

Data Schema

•images• url

• stars

• *title

• caption

• variants

•small

•medium

•big

•tags

•*_id

•rolestars•*image_id

•*role_id

•roles• *name

• privileges

• login

• description

• location

• child_roles

• _id

Requirements•List stars

•List roles’ images

•CRUD

•Tag searching

•Who liked what

•Social media

•Image resizing

•Added by tag

•Hot tagged images

application decisions

API TIER

UT UR RSWORKERTIER

OBJECTSTORAGE

SOCIALMEDIA

INTERNET

•Load Balancer•Infrastructure•Authentication•Message Queue•Database•Content Delivery•Social Media

API TIER

UT UR RSWORKERTIER

OBJECTSTORAGE

SOCIALMEDIA

INTERNET

API TIER

UT UR RSWORKERTIER

OBJECTSTORAGE

SOCIALMEDIA

INTERNET

API TIER

UT UR RSWORKERTIER

OBJECTSTORAGE

SOCIALMEDIA

INTERNET

API TIER

UT UR RSWORKERTIER

OBJECTSTORAGE

SOCIALMEDIA

INTERNET

API TIER

UT UR RSWORKERTIER

OBJECTSTORAGE

SOCIALMEDIA

INTERNET

API TIER

UT UR RSWORKERTIER

SOCIALMEDIA

INTERNET

API TIER

UT UR RSWORKERTIER

INTERNET

API TIER

UT UR RSWORKERTIER

INTERNET

PARALLEL

API TIER

UT UR RS

INTERNET

WORKERTIER

•API Tier

PARALLEL

API TIER

UT UR RS

INTERNET

WORKERTIER

API TIER

•API Tier

PARALLEL

API TIER

UT UR RS

INTERNET

WORKERTIER

API TIER

•Works in tandem

•API Tier

PARALLEL

API TIER

UT UR RS

INTERNET

WORKERTIER

API TIER

•Works in tandem•Doesn’t wait for workers

•API Tier

PARALLEL

API TIER

UT UR RS

INTERNET

•Update Tags, Update Relations, Resize Workers operate in tandem

WORKERTIER UT UR RS

•API Tier

PARALLEL

API TIER

UT UR RS

INTERNET

•Update Tags, Update Relations, Resize Workers operate in tandem

•Non-blocking actions

WORKERTIER UT UR RS

API TIER

MODULAR

API TIER

UT UR RSWORKERTIER

INTERNET

•Separate tiers by functionality

MODULAR

API TIER

UT UR RSWORKERTIER

API TIER

UT UR RS

INTERNET

MODULAR

API TIER

UT UR RS

•Use MQ to decouple API from worker

WORKERTIER

INTERNET

MODULAR

API TIER

UT UR RS

•Use MQ to decouple API from worker

•Separate workers to simplify units of work

WORKERTIER UT UR RS

INTERNET

Horizontal

API TIER

UT UR RSWORKERTIER

INTERNET

•Restful API for shared nothing scale

Horizontal

API TIER

UT UR RSWORKERTIER

API TIER

INTERNET

Horizontal

API TIER

UT UR RS

•Workers share no state, simply work on input

WORKERTIER

INTERNET

Horizontal

API TIER

UT UR RS

•Workers share no state, simply work on input

•Scale data across nodes (even DCs)

WORKERTIER UT UR RS

API TIER

INTERNET

API TIER

UT UR RSWORKERTIER

INTERNET

•Don’t re-invent ability or hosting

API TIER

UT UR RSWORKERTIER

INTERNET

API TIER

UT UR RS

•Split processes that are individually testable

WORKERTIER

API TIER

UT UR RS

INTERNET

API TIER

UT UR RS

•CI/CD is achievable by component

WORKERTIER

INTERNET

API TIER

UT UR RS

WORKERTIER

INTERNET

API TIER

UT UR RS

WORKERTIER

INTERNET

API TIER

UT UR RS

WORKERTIER

INTERNET

API TIER

UT UR RS

WORKERTIER

INTERNET

API TIER

UT UR RS

WORKERTIER

API TIER

INTERNET

SECURE

API TIER

UT UR RSWORKERTIER

INTERNET

•Use HTTPs for secure data transmission

SECURE

API TIER

UT UR RSWORKERTIER

API TIER

INTERNET

SECURE

API TIER

UT UR RS

•No host logins; managed by Chef

WORKERTIER

API TIER

UT UR RS

INTERNET

SECURE

API TIER

UT UR RS

•No host logins; managed by Chef

•StormPath for user authentication

WORKERTIER

INTERNET

thank you!@rackspaceUK@Racker_beighton@waynewalls@alunduil@jennbolestweets

unlocked london - technical track

types of development

nosql nosql database

code change

node cookbooks

database isolation

database crash

desired state

node catalog

Technology

"unlocked: the hybrid cloud" business track

unlocked london - case studies

fdh unlocked

london olympic games 2012 belgian track cycling evaluation...

capitulo2 unlocked

hypnodate - unlocked

nonlinear unlocked

redoxhandbook unlocked

preparatoriagrallazarocardenasenlaciudaddezinapecuaromichoacan...

pdfpirate.org unlocked

dbd unlocked

1008ha unlocked

unlocked mobile

nfce unlocked

west london transit – one track mind or broad vision?

hyperkalemia unlocked

dinamicas unlocked

unlocked cells

124 unlocked

09e02664 unlocked