APEREO OAEBootcamp, Miami 2014

WHAT WOULD YOU LIKE TO LEARN?

OAE @ MIAMI• Sunday: Development Bootcamp

• Monday @ 10am: State of the Project

• Monday @ 11am: Project talks

• Monday @ 1pm: Architectural Overview

• Tuesday @ 1pm: The FOLD

• Tuesday @ 2pm: Stories from the Frontlines of innovation

• Wednesday @ 11.45am: Apereo & ESUP: Brothers in arms

TOPICS 1. OAE Project

2. Hilary System Architecture

3. Node.js Exercise

4. Back-end Exercise

5. UI Architecture

6. Front-end Exercise

Support for academic collaborationand networking

https://oae.oae-qa0.oaeproject.org

PROJECT GOALS• User-led

• Multi-tenant platform

• Cloud-ready

• SaaS

• Cross-institutional

• Used at large scale

INTEGRATION

• Integrate OAE with other systems

• Integrate other systems with OAE

MULTI-TENANCY• Support multiple institutions on same installation

• Easily created, maintained and configured

• Each institution has its own URL, branding, skinning and users

• Share infrastructure

• Institutions can share and collaborate with other institutions

TOPICS 1. Project Goals

2. Hilary System Architecture

3. Node.js Exercise

4. Back-end Exercise

5. UI Architecture

6. Front-end Exercise

OAE ARCHITECTUREThe Apereo OAE project is made up of 2 distinct source code platforms:

• “Hilary”

• Server-side RESTful web platform that exposes the OAE services

• Written entirely using server-side JavaScript in Node.js

• “3akai-ux”

• A client-side / browser platform that provides the HTML, JavaScript and CSS that make up the UI of the application

OAE ARCHITECTURE

APPLICATION SERVERS• Written in server-side JavaScript, run in Node.js

• Light-Weight

• Single-Threaded

• Event-Driven

• Non-Blocking

• Uses callbacks/promises and an event queue to stash work to be done after I/O or other heavy processes complete

• App servers can be configured into functional specialization:

• User Request Processor

• Activity Processor

• Search Indexer

• Preview Processor

APACHE CASSANDRA• Authoritative data source

• Provides high-availability and fault-tolerance without trading away performance

• Regarding CAP theorem, Cassandra favours Availability and Partition Tolerance over Consistency, however consistency is tunable on the connection-level (we always use “quorum”)

• Uses a ring topology to shard data across nodes, with configurable replication levels

• Used by: Netflix, eBay, Twitter

ELASTICSEARCH• Lucene-backed search platform

• Built for incremental scaling and high-availability

• Exposes HTTP RESTful APIs for indexing and querying documents

• RESTful query interface uses JSON-based Query DSL

• Powers all of Hilary search functions (Library Search, Members / Memberships Search)

• Used by: GitHub, FourSquare, StackOverflow, WordPress

RABBITMQ• Light-weight message queue platform written in

Erlang

• Used for distributing tasks to specialized application server instances

• Supports active-active queue mirroring for high availability

• Used by: Joyent

REDIS• REmote DIctionary Server

• Fills a variety of functionality in OAE:

• Broadcast messaging (could move to RabbitMQ)

• Locking

• Caching of basic user profiles

• Holds volatile activity aggregation data

• Comes with no managed clustering solution (yet), but has slave replication for active fail-over

• Some clients manage master-slave switching, and distributed reads for you

• Used by: Twitter, Instagram, StackOverflow, Flickr

ETHERPAD• Open Source collaborative editing application written

in Node.js

• Originally developed by Google and Mozilla

• Licensed under Apache License v2

• Powers collaborative document editing in OAE

• Collaborative documents sharded by ID to dedicated Etherpad instances

NGINX

• HTTP and reverse-proxy server

• Used to distribute load to application servers, etherpad servers and stream file downloads

• Useful rate-limiting features based on source IP

• Proxy web socket connections to Hilary and Etherpad

• Used by: Netflix, WordPress.com

TOPICS 1. Project Goals

2. Hilary System Architecture

3. Node.js Exercise

4. Back-end Exercise

5. UI Architecture

6. Front-end Exercise

IDE• OAE Team uses mostly Sublime Text 2 and Aptana Studio 3

• http://www.sublimetext.com/2 • http://www.aptana.com/products/studio3/download

• Useful things in an IDE:

• Syntax highlighting • Automatic “linting” • Variable assistance (don’t expect much from dynamic

scripting language) • File-system navigation

GITHUB• OAE Team uses it for :

• Version control • Issue Tracking • Code Integration Workflow (Pull Requests)

• TravisCI Integration for :

• Running unit tests on all commits and pull requests • Packaging up and uploading all commits to Amazon S3 so they can be easily pulled

down and deployed • Packaging up and uploading all tagged releases to Amazon S3 for production releases

• Coveralls Integration for :

• Reporting on trends in unit tests code coverage (e.g., indicates if test coverage drops from 90.2% to 90.15%)

TOPICS 1. Project Goals

2. Hilary System Architecture

3. Node.js Exercise

4. Back-end Exercise

5. UI Architecture

6. Front-end Exercise

GOAL

NPM• Used to download packages and manage versions

• Packages can be installed from:

• Central NPM repo: npm install oae-tasklist • Git: npm install git+https://github.com/oaeproject/oae-tasklist • File-system directory: npm install ../oae-tasklist

• package.json file describes metadata about a module and its dependencies

• When a package is installed, they are placed either locally in a ./node_modules directory or globally (npm install -g) in a shared location such as /usr/local/share/npm/lib/node_modules

EXTENDING OAE WITH NPM• NPM module represents a logical set of functionality (e.g., a

back-end REST API, or a set of related widgets)

• NPM modules in 3akai-ux are searched for custom widgets

• NPM modules in Hilary (that start with oae-) are searched for init.js and rest.js to integrate to the application container

• New dependencies can be added to package.json file and then fetched / installed running “npm install”

PACKAGE.JSON

• Describes metadata about a package:

• Version • Author • Unique package id

• Required for installing a package as a module

TOPICS 1. Project Goals

2. Hilary System Architecture

3. Node.js Exercise

4. Back-end Exercise

5. UI Architecture

6. Front-end Exercise

UI ARCHITECTURE

UI ARCHITECTURE

UI ARCHITECTURE

• In essence

• HTML • CSS • JavaScript

• Talks to Hilary via REST APIs

WIDGETS• Modular UI components

• HTML Fragment • JavaScript • CSS • I18n bundles • Manifest file

• Loaded into DOM

• Re-usable

WIDGETS

TOOLBOX

JS frameworks

CSS frameworks

3rd party plugins

OAE UI API

OAE CSS components

TOOLBOX

JS frameworks

CSS frameworks

3rd party plugins

OAE UI API

OAE CSS components

JS FRAMEWORKS

• jQuery

• RequireJS

• underscore.js

• DOM manipulation

• Cross-browser abstraction

• Events

• Pretty much everything

• Utility toolbelt

• Manipulate objects, arrays, etc.

• File and module loader

• Load files on the fly

• Necessity to keep things modular

• Optimisation built-in

TOOLBOX

JS frameworks

CSS frameworks

3rd party plugins

OAE UI API

OAE CSS components

CSS FRAMEWORKS

• Twitter Bootstrap

• Font Awesome

TWITTER BOOTSTRAP

TWITTER BOOTSTRAP

• Most popular CSS framework

• Responsive (!)

• Documentation

• Basic components, styles, etc.

• Override where necessary

BOOTSTRAP GRID• Predefined classes make up rows and columns

• 12 available columns to span

• e.g., 3 x `.col-xs-4` columns

• 4 media-queries

Device Class WidthPhones .col-xs- < 768pxtablets .col-sm- ≥ 768px

Desktops .col-md- ≥ 992pxDesktops .col-lg- ≥ 1200px

BOOTSTRAP GRID

FONT AWESOME

FONT AWESOME• Icon font

• No more images

• Style with CSS

• Skinning

• Easy

TOOLBOX

JS frameworks

CSS frameworks

3rd party plugins

OAE UI API

OAE CSS components

3RD PARTY PLUG-INS

• jQuery plug-ins

• Bootstrap plug-ins

3RD PARTY PLUG-INS• Autosuggest

• History.js

• Fileupload

• Validation

• Markdown

• etc.

INCLUDE YOUR OWN

TOOLBOX

JS frameworks

CSS frameworks

3rd party plugins

OAE UI API

OAE CSS components

OAE UI API• Wrapper for REST requests

• Authentication • Content • Comments • Discussions • Following • Groups • Users • Admin

OAE UI API• Utilities

• i18n • l10n • Template rendering • Widget loading • Notifications • XSS escaping • etc.

OAE UI API

TOOLBOX

JS frameworks

CSS frameworks

3rd party plugins

OAE UI API

OAE CSS components

OAE CSS COMPONENTS

• Re-usable HTML fragments

• Consistency

• Design guide

OAE CSS COMPONENTS• Clips

• Visibility icon

• Large options

• Thumbnails

• Clip

• Left hand navigation

• Tiles

• List items

• List header

• Skinnable Panel

WIDGET BEST PRACTICES

• Namespacing

• i18n

• UI Templating

NAMESPACING• Widgets share same container

• Avoid clashes

• Namespace:

• HTML IDs • CSS classes • jQuery selectors

I18N

• UI available in multiple languages

• Standard .properties files

• 2 types of bundles

• Core bundles • Widget bundles

I18N

Translation priority

1. Widget user language file

2. Widget default language file

3. Container user language file

4. Container user language file

I18N

__MSG__TRANSLATION_KEY__

I18N

Complete

Partial

I18N

L10N

• API methods for localizing:

• Number formatting • Date formatting

UI TEMPLATING

• TrimPath

• Avoids lots of DOM manipulation

• Pass in JSON data

• Supports if statements, for loops, etc.

UI TEMPLATING

• Templates are defined in between <!-- -->

• oae.api.util.template().render(...)

UI TEMPLATING• Template

<div id="example_template"><!--

<h4>Welcome {firstName}.</h4> You are ${profile.age} years old--></div>

• Input oae.api.util.template().render($("#example_template"), {

“firstName”: “John”,“profile”: {

“placeofbirth”: “Los Angeles”,“age”: 45

}});

• Result<h4>Welcome John.</h4> You are 45 years old.

UI TEMPLATING• Template

<div id="example_template"><!--

{if score >= 5}<h1>Congratulations, you have succeeded</h1>

{elseif score >= 0}<h1>Sorry, you have failed}

{else}<h1>You have cheated</h1>

{/if}--></div>

• Input oae.api.util.template().render($("#example_template"), {

“score”: 6});

• Result <h1>Congratulations, you have succeeded!</h1>

UI TEMPLATING• Template

<div id="example_template"><!--

{for conference in conferences}<div>${conference.name} (${conference.year})</div>

{forelse}<div>No conferences have been organized</div>

{/for}--></div>

• Input oae.api.util.template().renderTemplate($("#example_template"), {

“conferences”: [ {“name”: “Apereo Miami”, “year”: 2014}, {“name”: “Apereo San Diego”, “year”: 2013}, {“name”: “Sakai Atlanta”, “year”: 2012}]

});

• Result <div>Apereo Miami (2014)</div> <div>Apereo San Diego (2013)</div> <div>Sakai Atlanta (2012)</div>

UI RELEASE PROCESSES• Grunt

• Task-based build system implemented in JavaScript

• Similar in theory of operation to Make, Rake

• Rich ecosystem of plug-ins to do most tasks

• Easy to implement new task when a plugin doesn’t exist yet

• Used for running test suites, production builds, linting tools

UI RELEASE PROCESSES• Production Build

• Optimizes the static assets to reduce throughput, request frequency, and optimize caching across versions

• Require.js Optimization:

• Concatenate JavaScript dependencies (reduces number of web requests significantly) • Minify / Uglify JavaScript files (reduces payload sizes significantly, even when gzip enabled

on web server)

• Hash optimization:

• Hash the contents of static assets and append result to the filename, then cache them indefinitely on the browsers

• When the files change, the hash in the filename changes to force reloading of the updated asset

• If files never change across version, client never reloads file until their cache is cleared

TOPICS

1. Project Goals

2. Hilary System Architecture

3. Node.js Exercise

4. Back-end Exercise

5. UI Architecture

6. Front-end Exercise