Guided MEAN Stack Hackathon

Valeri KarpovSoftware Engineer, MongoDBwww.thecodebarbarian.com

www.slideshare.net/vkarpov15github.com/vkarpov15

@code_barbarian

Building (and Testing) a Single Page App in 2 Hours

*

Who is this guy?

•Coined the MEAN stack in April ‘13

•CI/Tools/NodeJS/Go Engineer at MongoDB

•Maintains mongoose, omni-di

•AngularJS since 0.9.4 in 2010

•Hackmaster General at Bookalokal

•Former CTO, LevelUp

*

General Outline

•MEAN = MongoDB, ExpressJS, AngularJS, NodeJS

•Beginner talk: high-level concepts of MEAN

•Emphasis on testing and workflow

•Building a single page app - Axl

•NPM-inspired Package Manager for Golang

•Browser interface only

•No Go knowledge required

*

What You’re Building

*

What You’re Building

•Server for semver-aware package manager

*

Concepts You’ll Be Learning About

•Streams

•API testing

•DOM integration tests with Sauce

•Build systems and architecture

•Configuration + workflow

•General principles for writing NPM packages

*

NPM Packages You’ll Be Using

•mocha: unit tests and API integration tests

•browserify: compile Node JS into browser JS

•mongoose: schema validation for Node + MongoDB

•karma: browser automation for testing

•gulp: general-purpose build workflow

•And several more

*

Step By Step, the Server Side

•Step 1: Define the problem + brief overview of Go

•Step 2: Define schema and models (mongoose)

•Step 3: Build an API (express)

•Step 4: API-level testing (mocha)

•Step 5: Extending the API to support search

*

Step By Step, the HTML Client Side

•Step 6: Create AngularJS SPA client (browserify)

•Step 7: Testing the Single Page App (karma)

•Step 8: CDN for templates and JS (gulp)

*

Step 1: A Brief Overview of Go

•Language developed by Google

•Cute toy language with some niche use cases:• Fast-compiling small native binaries

• Tight integration with C

• “Agent” programs like MongoDB MMS agents

•Has some bad limitations• Generics

• Package management

*

Hello, World in Go

*

Why Build a Go Package Manager

•npm gets package management mostly right

•Go doesn’t and could use some help

*

What npm Does Right

•Mostly semantic versioning (no v20130722)• Will my code break if I upgrade from v2.1.3 to v2.1.4?

•2 commands to install and test• npm install

• npm test

• And you’re ready to hack

•No need to change PATH or env variables

•Result: a package for every use case

•Go doesn’t have any of this

*

Goals for Axl Server

•Upload/download releases of projects

•Projects hosted on Amazon Cloudfront (streams!)

•Query for projects using semver syntax:• version 1.2.0 matches “~1.2”, “>=1.2.0”, “<=1.3.0”, etc.

• Thankfully, there’s an npm module for that

•Ability to search projects by keyword

•Sane build system and workflow

•Take advantage of a single page app

*

Step 2: Design Schemas

“Smart data structures and dumb code works a lot better than the other way around.”

- Eric S. Raymond

“Bad programmers worry about the code. Good programmers worry about data structures and their relationships.”

- Linus Torvalds

*

A Brief Overview of Mongoose

•MongoDB is schema-less

•But your schema design still matters!

•Mongoose = NodeJS schemas for MongoDB• Validation

• Casting for objects and queries

• Promises and other syntactic sugar

• Object document mapper (ODM)

•Users include Tinder and McDonalds

*

Simple Example: A User Schema

*

Model = Schema + DB Connection

*

About the User Model

•_id and username will be pulled from Twitter oauth

•Most real data will be tracked in other models

*

The Project Schema

•In Go, project names are strings like “github.com/vkarpov15/mgo”

•Releases a list of semver strings

*

Why the data Field?

•Simple and intuitive access control

•data contains fields user can edit

*

Project Schema Indexes

•MongoDB has indexes for speeding up queries

•Tradeoffs: may make writes slower

•Project schema indexes on:• Project name

• Maintainers usernames (multi-key!)

• Name and keywords text index (more in search section)

*

The Release Schema

•Semver release of a project (with download info)

*

Validators in Mongoose

•validate ensures name exists and version is valid semver

*

Virtuals in Mongoose

•stringify can be accessed as a property

*

Indexes for Release Schema

•Indexes can be ordered to help with sorting

•The -1 means higher releaseNumbers come first

•Indexes define how you’ll query your data

*

Last Schema: DownloadHistory

•Because you want to highlight popular packages

*

DownloadHistory Indexes

•For the two common use cases of DownloadHistory• How many downloads did a package have this month?

• What are the most popular packages this month?

*

Schema Design Takeaways

•Design schemas carefully

•Use indexes that fit your use cases

•Don’t go overboard on indexing everything!

•Use Mongoose for seamless validation + casting

*

Step 3: REST-ish API with Express

•Express: powerful lightweight web framework

•Highly customizable (> 6900 packages on npm)

•Rapidly growing user base:• NBC

• SegmentIO

• MySpace

*

Whats in an API?

•API = actions on top of objects

•REST principles: use HTTP error codes and verbs

•Key concerns for API scalability:• Load search results piece-by-piece from DB

• Hit indexes

• Don’t preclude horizontal scalability (multiple servers)

•MEAN stack allows you to be very lazy

*

API Basics: Load a Project By Name

•GET /api/project?project=gopkg.in/mgo

*

A Slide Dedicated to a Bad Pun

•Only place where Axl can tolerate a Slash :)•(Slash was the guitarist for GNR. He and Axl don’t like each other)

*

Brief Aside on Dependency Injection

•Using my lightweight DI helper, omni-di

•Tool for managing and constructing dependencies

•Create objects once, pass them in to functions

•Injector inspects parameter names to find correct dependencies

*

Brief Aside on Dependency Injection

*

Routes With Dependency Injection

*

Using the Express Router

*

Filling out the Project API

*

Brief Aside: Authentication

•Twitter Oauth with redirect

•Passport: Express-compatible login middleware

*

Brief Aside: Authentication

*

POST = Create a Project

•POST /api/project - Mongoose is magical

*

PUT = Modify an Existing Project

*

More Sophisticated: Semver Query

•Find me a Release that matches “~1.2”

•GET /api/release?project=gopkg.in/mgo&version=~1.2

•Perfect application of streams

•Hard to index semver queries

*

More Sophisticated: Semver Query

*

Streaming Document by Document

•Like a for loop, but don’t load whole array in RAM

*

And If You Don’t Find a Document...

•Return a nice, handy 404

*

POST /api/release

•The most important and most sophisticated

•Subtleties:• Race conditions between multiple servers

• Error handling

• CDN (or, why the “scale npm” campaign happened)

• Authorization

*

Pushing a new Release

•POST /api/release?project=gopkg.in/mgo&version=1.2.0

*

MongoDB for Multi-server Concurrency

•MongoDB operations are ACID per document

•ACID stands for• Atomic

• Consistent

• Isolated

• Durable

•FindAndModify underlies Mongoose findOneAndUpdate()

*

Multi-Server POST /api/release

•findOneAndUpdate is ACID means a version gets added to a project only once

•Multiple servers => only one wins

*

The isNew Flag

•isNew = false means “return document before update”

*

POST /api/release Checks, Part 1

*

POST /api/release Checks, Part 2

•Check if release existed before findOneAndUpdate

•If so, return handy conflict error

*

Pushing To Amazon S3

•Easy to set up Cloudfront on top of Amazon S3

•Node lets you stream files to S3

•No more worrying about server HD space

*

Pushing To Amazon S3

*

Using PipeToS3

*

Step 3: Testing the API

•Writing API is easy, testing can be hard

•But easy with Express + Mocha + Node!

•Stub out authentication

•Event-driven nature of JS: no interrupts

•Single process responsible for:• Running Express server

• Sending HTTP requests to Express server

• Using Mocha to test the results of HTTP requests

*

Step 4: Testing the API

•Writing API is easy, testing can be hard

•But easy with Express + Mocha + Node!

•Stub out authentication

•Event-driven nature of JS: no interrupts

•Single process responsible for:• Running Express server

• Sending HTTP requests to Express server

• Using Mocha to test the results of HTTP requests

*

Why One Process?

•Minimal setup

•Run tests with one command

•Use real configuration

•Can assert on state of DB as well as result

*

High Level Mocha Structure

*

General Idea for Release API Tests

*

First Case: Load Exact Version

*

Test Loading Version with Semver

*

Test Uploading a Release

*

Testing Workflow

•Systems like Gulp are sometimes overkill

•Don’t ask people to “npm install mocha -g”!• Version management

• Incompatibilities with different projects versions

• Breaks the npm install promise

*

Using Makefile for Testing

•Makefile surprisingly common in NodeJS

•Usually as a command shortener

•Use “make api-test” to run tests

*

Brief Demo of Tests

*

Step 5: Search API

•Use testing framework to do some TDD

•Test-driven Development: tests first, then code

•Usually done with unit tests

•Faster and more fun with API integration tests

*

TDD First Step

*

Assumptions from Test

•GET /api/search?q=test

•Search project name and keywords

•Return list of projects in JSON

*

MongoDB Text Search

•Introduced in MongoDB 2.6 (beta feature in 2.4)

•Does exactly what this search API needs

•First need to create a text index

*

Filling Out The Search API

*

Notes on Text Search

•Requires mongoose >= 3.8.9

•The $meta keyword used to get “text score”

•Higher text score => more relevant

•Usually want to sort by text result

•Supports stemming in 15 languages

*

What is Stemming?

*

What is Stemming?

•“Test” matches both “test” and “testing”

*

Step 6: AngularJS SPA Client

•REST API is only the beginning

•What about building a client?

•AngularJS is the best tool for the job (IMO)• MVC-ish client-side templating framework

• Two-way data binding

• Dependency injection, elegant structure

•Writing client-side JS is painful - use Browserify

•Won’t need very sophisticated AngularJS for Axl

*

Book Announcement

•My book on AngularJS is coming out Dec 22

•Pre-order on Amazon for a more detailed dive into AngularJS

*

Overview of AngularJS

•Client-side templates (Two-way data binding)

•Client-side routing (Single Page Apps)

*

Why a Single Page App?

•Clean separation of data and views

•Ship HTML separately from data

•Enables you to store HTML as static asset (Cloudfront)

•Leaves your server free to serve up JSON

•Cleaner separation of concerns

*

Overview of Browserify

•Compiles NodeJS JS into one browser-friendly file

•Use same dependencies on client and server:• moment

• underscore

• mongoose (schemas + validation only)

• other isomorphic JS

•Easy to upload JS to Cloudfront as static asset

*

Using AngularJS and Browserify

•npm has an angular package

•Only includes “core angular”

•You may still need other packages

*

Word of Warning

•npm angular package depends on contextify

•Contextify is notoriously picky about install reqs

•You need right version of python and a C++ compiler

•Possible, but tricky, to install on Windows

•I prefer to list angular as a devDependency in package.json

*

Writing AngularJS in NodeJS

•Create an AngularJS “module”

•Axl also needs the angular client-side routing module

*

Setting up Client Side Routing

*

How Client-Side Routing Works

•Only modify hash portion of the URL

*

What Does a Template Look Like?

•Written in Jade:

•“ng-” and “{{ }}” are tie-ins to AngularJS

*

Ok, So What’s a Controller?

•Defines an in-HTML API for the template to use

•Responsible for loading and organizing data

•Attaches properties to template “scope”

*

And What’s $projects?

•A service

•Typically a convenience wrapper around a resource loaded from the server

*

How Do Controllers and Services Get Added to AngularJS?

*

Compiling with Browserify

•Output ./bin/javascript/ui.js with all JavaScript

*

Using Compiled JS

•In base jade file, layout.jade, include ui.js:

•Browserify downside: need to compile before use

*

Compiling Templates

•Jade downside: need to compile to HTML

•Jade has significant upsides over static HTML:• Readability

• Can include conditionals on server configuration

• Handy for testing - next section

•So we also need to compile Jade!

*

Why Compile Templates

•Jade downside: need to compile to HTML

•Jade has significant upsides over static HTML:• Readability

• Can include conditionals on server configuration

• Handy for testing - next section

•So we also need to compile Jade!

*

Idea For Compiling Jade

*

Compiling Templates Wrapup

•AngularJS templates need to be HTML

•Very easy to compile Jade

•Take advantage of server configuration

*

Step 7: Testing the AngularJS Client

•Trickiest part of any project: users will run your code in different OS, browser, machine, etc.

•My job at MongoDB = this for the mongodb server

•Great tools for this in web-dev world:• Sauce: think Amazon EC2 for browsers

• Karma: Popular browser automation tool

• ngScenario: AngularJS E2E testing framework

*

DOM Integration Tests

•Test the whole browser side of your code

•That is, test AngularJS’ integration with the DOM

•Stub out REST API calls (speed and ease of setup)

•Setting up data for E2E tests is often complex

•Sometimes known as a “midway test”

*

Karma Overview

•Interface for launching a browser and running tests

•Rich set of plugins

•Handles tunneling for remote browsers

•Plugins for launching• local browsers (chrome, firefox, IE, etc.)

• browsers in the Sauce cloud

•My answer to “if I were stuck on a desert island with only one npm module”

*

Tradeoffs, Local vs Sauce

•I recommend doing both

•Sauce allows you to spawn any browser:• Chrome on Linux

• IE6 on WinXP

• Android 4.3 (yes, mobile included)

•But is slow and flakey

•Local is difficult to set up, but:• Easier to debug: you can actually see the tests run

• Fast and reliable

*

Workflow and Local vs Sauce

•Make sure tests run with local config on Chrome

•Then use Sauce to run tests on different browsers

•Karma makes this easy: different configs

*

Karma Plugin Ecosystem

•Karma itself is a lightweight core

•Need plugins to make it do anything useful

•Yet another npm package with its own npm package ecosystem (~170 plugins)

*

Generating a Karma Config

•Karma configs are pretty straightforward

•Karma has some handy tools:

• “node_modules/karma/bin/karma init” creates a new config after asking you a few questions

•Or just copy/paste examples

*

Local Karma Config for Axl

*

Sauce Karma Config - Browser List

*

Sauce Karma Config - The Rest

*

Karma Sauce Environment Vars

•karma-sauce-launcher relies on 2 env variables:• SAUCE_USERNAME: your Sauce username

• SAUCE_ACCESS_KEY: Sauce API key

•Make sure these are set or your tests will fail

*

Writing the Actual DOM Tests

*

Couple Words of Warning

•The api() function is home-baked

•The AngularJS equivalent, $httpBackend, is weird

•More about $httpBackend in Chapter 9 of Professional AngularJS :)

•ngScenario is quirky and technically deprecated

•Its replacement, Protractor, is more quirky and doesn’t support DOM integration tests

*

How Does the API Function Work?

•ngScenario DSL: domain specific language

*

window.AxlAPI

•Use a different layout.jade for running tests:• Don’t load bootstrap

• Expose methods to stub out API requests

•ngScenario runs your tests in an iframe, so it can actually interact with the code under test

*

window.AxlAPI

*

Makefile Rules for Running Tests

•Remember, need env vars for Sauce tests

*

Demo of Running DOM Tests

*

Step 8: Deploying to S3 with Gulp

•With SPA, your HTML is static

•Might as well push it out to a CDN

•Last step, stay focused!

*

Overview of Gulp

•Simple build tool for web apps

•Another core package with a lot of plugins

•Grunt is its more well-known competitor

•Advantage of Gulp: its plain NodeJS

•List tasks in gulpfile.js

•Re-use your existing code and use streams!

*

Including Gulp in package.json

*

Minifying JS with Gulp

*

Uploading to S3 with Gulp Overview

*

Uploading Templates to S3 with Gulp

*

Configuring AngularJS to Load Templates Remotely

*

Configuring Jade to Load Templates Remotely

*

And the End Result

*

And that’s a wrap! Time to Review

•Single page app with MEAN stack

•Mongoose Schema

•Express API

•AngularJS routing

•Browserify for building client code

•Karma for testing

•Gulp for pushing HTML+JS to S3+Cloudfront

*

Thanks for Listening!

•Slides on:• Twitter: @code_barbarian

• Slideshare: slideshare.net/vkarpov15

•Repo on github: https://github.com/vkarpov15/axl-server

•Professional AngularJS on Amazon