functional programming with streams in node.js
DESCRIPTION
My talk at HTML5DevConf && FluentConf covering how I combined some basic functional programming techniques with node.js streams to abstract a streaming VFS zipping layer that can also (theoretically / eventually) run in the browser.TRANSCRIPT
Functional Programming with Streams in node.js
Who Am I?
HP webOS Framework Team
@CrabDude (Twitter, Github, etc…)
Organizer BayNode & Dallas.node
Author:
trycatch – async try/catch in node.js
stepup – step node.js control flow + trycatch
tiki – in-browser package manager
The Problem VFS
hermesClient
hermesBuild 3rd Party API
hermesDeploy
Ares (browser)
VFS VFS
hermesClient hermesClient
hermes
The Goal
Don’t buffer any data
Don’t write to disk
Minimize latency
Allow reuse of functionality (e.g., readdir)
Functionality may be local or remote (e.g., getFolder)
Why node.js?
“Node's goal is to provide an easy way to build scalable network programs.” – nodejs.org
node.js === JavaScript
JavaScript is the future of the web, and code reuse is its unfair advantage.
Basics
EventEmitter
Stream
Traditional streams (aka non-node.js lazy streams)
EventEmitter: Example
EventEmitter: Extend
Streams
“Streams are to time as arrays are to space.” – Jed Schmidt @ JSConf.eu/2010
Readable Streams
Events: data, end, error, close
Methods: pause, resume, end, destroy
Writable Streams
Events: drain, error, close, pause, resume
Methods: write, end, destroy
readable.pipe(writeable)
Readable streams can be piped to writable streams (and vice-versa!)
on(‘data’) => write()
on(‘end’) => end()
on(‘drain’) => resume()
on(‘close’) => destroy()
on(‘error’) => on(‘error’)
on(‘pause’) => pause()
readable.pipe(writeable): Example
Lazy Streams
Streams in functional programming…
Lazy Streams: Higher Order Functions
A higher-order function is a function that does at least one of the following:
Take one or more functions as input output a function
Lazy Streams: Lists
Got All That?
Hermes: Revisited FTP Dropbox Box.net Filesystem
hermesFtp hermesDropbox hermesBox hermesFilesystem
hermesBuild 3rd Party API
hermesDeploy
Ares (browser)
Still Got All That?
We need…
We need to recursively zip a directory…
Let’s start with abstract implementation non-specific functional versions of…
Filter paths from readdir
Filter directories from paths
Filter files from paths
Recursively return all file paths in a directory
We need…
More “abstract implementation non-specific functional versions” (errr…) of…
ls (aka readdir)
getFile (aka readFile)
getFolder (stream of gets)
We need…
Yet More AINSFV of...
Zip a stream of files
Allow node.js streams to be piped to lazy streams
pipe(): lazy stream => node.js stream
pump(): node.js stream => lazy stream
Almost there!
Now we can…
Zip a folder up using an arbitrary…
getFile / readFile
ls / readdir
… And stream the response! Yeeeeehaw!
Sweeeeeet.
Now What?
Since all the implementation specific functions are passed in, we can…
Port this to the browser!
Say whaaat?
node-browserify by @Substack does a lot of this already
Stream, Buffer, Path, etc…
We need…
Complete buffer implementation (browser has TypedArrays & ArrayBuffer)
fs (browser has FileSystem API)
zlib…
inflate(), deflate(), gzip(), gunzip(), unzip()
Stay Tuned.
I’m currently porting the fs and buffer modules to the browser, and eventually zlib.
http://github.com/crabdude/fs-browserify http://github.com/crabdude/buffer-browserify http://github.com/crabdude/zlib-browserify
We’re HIRING! webOS Framework Team
If you think this is pretty cool… Or you think I’M pretty cool…
Stop by the Enyo booth or talk to me afterwards.
Thank you.
[email protected] https://github.com/Gozala/streamer/blob/master/readme.js http://en.wikipedia.org/wiki/Higher-order_function https://github.com/mikeal/request http://felixge.s3.amazonaws.com/11/nodejs-streams.pdf https://github.com/substack/node-browserify
Functional Programming in JavaScript: http://igstan.ro/posts/2011-05-02-understanding-monads-with-
javascript.html http://ndc2011.macsimum.no/mp4/Day2%20Thursday/
Track6%201500-1600.mp4 http://drboolean.tumblr.com/ http://osteele.com/sources/javascript/functional/