Polishing RubyZenHacks & Friends

by: Ryan Davis, Seattle.rb

Setting ExpectationsThis Presentation is...

• ...58 slides in 45 minutes - very dense

• ...about the Shinies

• ...“Hard Core”, not for everyone

Driving Philosophy• Simplicity & Clarity

• Otherwise, you don’t fully understand.

• Make it Right

• Make it Work

• Make it Fast

• Speed comes last, and only objectively.

• Balance size and speed.

• Accelerate Maturity by splitting from parents.

• Have fun! Otherwise, what’s the point?

Standard Example

• I’ll be using factorial throughout most of the code in order to make things easier to track.

• Most of the time, it’ll be implemented as:

def factorial(n) f = 1 n.downto(2) { |x| f *= x } return fend

Overview

ZenHacks

ruby2ruby

sexp as method

auto-refactoring

obfuscator

profiler

zenoptimize

Advanced

Dependencies

ABC Metrics

Colored Sexp

Graphs

Overview

Current Status

Step by Step

Factorial

Future Plans

Intermediate

Architecture

& extensibility

+ A Sekrit

Overview

factorial

node printer

code2test

test2code

unit_diff

RubyInline ruby2c

Beginner

Overview

RI::C - hello world

factorial_c

ZenTestParseTree/

SexpProcessor

Newb

Overview

factorial

node printer

code2test

test2code

unit_diff

Beginner

Overview

RI::C - hello world

factorial_c

RubyInlineZenTestParseTree/

SexpProcessor

• Generates missing tests.

• Generates missing methods under test (MUT).

• Helps illuminate failing tests.

ZenTest

ZenTestGenerating Tests

class TestFactorial < Test::Unit::TestCase def test_factorial raise NotImplementedError, 'Need to write test_factorial' endend

class Factorial def factorial(n) f = 1 n.downto(2) { |x| f *= x } return f end end

ZenTestGenerating Tests

ZenTestGenerating MUTs

class Factorial def factorial raise NotImplementedError, ‘Need to write factorial’ endend

class TestFactorial < Test::Unit::TestCase def test_factorial @f = Factorial.new assert_equal(1, @f.factorial(1)) assert_equal(2, @f.factorial(2)) assert_equal(6, @f.factorial(3)) endend

ZenTestGenerating MUTs

& ./test.rb... 1) Failure:test_conditional4(TestTypeChecker) [./r2ctestcase.rb:1068:in `test_conditional4' ./r2ctestcase.rb:1055:in `test_conditional4']:<t(:if, t(:call, t(:lit, 42, Type.long), :==, t(:arglist, t(:lit, 0, Type.long)), Type.bool), t(:return, t(:lit, 2, Type.long), Type.void), t(:if, t(:call, t(:lit, 42, Type.long), :>, t(:arglist, t(:lit, 0, Type.long)), Type.bool), t(:return, t(:lit, 3, Type.long), Type.void), t(:return, t(:lit, 4, Type.long), Type.void), Type.void), Type.void)> expected but was<t(:if, t(:call,

ZenTestIlluminating Issues

Which would you rather read/debug???

This:

& ./test.rb | unit_diff... 1) Failure:test_conditional4(TestTypeChecker)12c10< :>,---> :<,

Or this:

ZenTestIlluminating Issues

• Allows you to write C functions within your ruby classes.

• No compiling/linking/packaging phase required.

• Only recompiles when needed.

• Incredibly accelerated development time.

• Write & Run. There is no step 3.

• You aren’t limited to C.

RubyInline

RubyInlineObligatory Hello World

class Hello inline do |builder| builder.include ‘<stdio.h>’ builder.c ‘ void hello() { puts(“hello world”); } ‘ endend

Hello.new.hello

% ./hello.rbhello world

RubyInline

RubyInlineFactorial

class FastMath inline do |builder| builder.include ‘<math.h>’ builder.c ‘ long factorial_c(int max) { int i=max, result=1; while (i >= 2) { result *= i--; } return result; }’ endend

math = FastMath.new10000.times do math.factorial(20); end

RubyInlineFactorial

ParseTree & SexpProcessor

“ParseTree is a little brown stinky ferret that digs down a hole and violently rips the AST away from the warm bosom of ruby”

ParseTree & SexpProcessor

• ParseTree:

• Extracts ruby’s AST internals and makes them digestible by mortals.

• SexpProcessor:

• The framework for digesting them.

Basic Vocabulary

• Sexp = s-expression = An S-expression (S stands for symbolic) is a convention for representing data structures in a text form. (wikipedia)

• AST = Abstract Syntax Tree = What the parser makes when it parses and what the interpreter runs.

ParseTree

• ParseTree is the library that converts ruby into sexps.

[:defn, :factorial, [:scope, [:block, [:args, :n], [:lasgn, :f, [:lit, 1]], [:iter, [:call, [:lvar, :n], :downto, [:array, [:lit, 2]]], [:dasgn_curr, :x], [:lasgn, :f, [:call, [:lvar, :f], :*, [:array, [:dvar, :x]]]]], [:lvar, :f]]]]

mmmmm, factorial…

ParseTreeparse_tree_show

SexpProcessorClass Catalog

class QuickPrinter < SexpProcessor def initialize super self.strict = false self.auto_shift_type = true end def process_class(exp) puts “class #{exp.shift}” exp.shift # superclass process exp.shift # body return s() end def process_defn(exp) puts “ def #{exp.shift}” exp.shift # body return s() endend

class QuickPrinter def initialize def process_class def process_defn

=>

SexpProcessor

Dependencies

ABC Metrics

Colored Sexp

Graphs

Overview

Current Status

Step by Step

Factorial

Future Plans

Intermediate

Architecture

& extensibility

+ A Sekrit

RubyInline ruby2cParseTree/

SexpProcessor

Intermediate

Eric Hodel’sCar!

It’s famous!Have you had Matz in your

car?

RubyInlineGeneral Architecture

• RubyInline has a totally open architecture making it easy to extend with other languages.

• Uses ducktyping, an RI extension only needs to implement:

• load_cache

• build

• load

builder = builder_class.new self

yield builder

unless options[:testing] then unless builder.load_cache then builder.build builder.load endend

• You thought jruby was wrong? HA!

• Ugly, yeah... but there is a LOT out there.

• Originally by Yoshida Masato for ruby 1.4!

• 20 minute conversion into RubyInline.

• Sooo easy. Goodbye setup.rb!

• Seems to run complex PERL just fine.

RubyInlineInline::PERL

I’m really sorry for this.

Honest.

RubyInlineInline::PERL

ParseTree & SexpProcessor

• Tools you easily make with SexpProcessor:

• Dependency Reporting

• Complexity Analysis

• Parser Visualization

• All of the Above

(and many more)

SexpProcessorDependency Analysis

def process_defn(exp) name = exp.shift @current_method = name return s(:defn, name, process(exp.shift), process(exp.shift)) end

def process_const(exp) name = exp.shift const = (defined?($c) ? @current_class.name : “#{@current_class}.#{@current_method}”) is_class = ! (Object.const_get(name) rescue nil).nil? @dependencies[name] << const if is_class return s(:const, name) end

SexpProcessorDependency Analysis

SexpProcessorComplexity Metrics

• ABC Metrics• # of Assignments• # of Branches• # of Calls

% parse_tree_abc printer.rb|ABC| = Math.sqrt(assignments^2 + branches^2 + calls^2)

1) QuickPrinter.process_class = 0 + 1 + 7 = 7.07 2) QuickPrinter.process_defn = 0 + 0 + 4 = 4.00 3) QuickPrinter.initialize = 2 + 0 + 0 = 2.00 4) Total = 2 + 1 + 11 = 13.07

SexpProcessorComplexity Metrics

:defn

:factorial :scope

:block

:args :lasgn :iter :return

:n :f :lit

1

:call :dasgn_curr :lasgn

:lvar :downto :array

:n :lit

2

:x :f :call

:lvar :* :array

:f :dvar

:x

:lvar

:f

SexpProcessorVisualization

SexpProcessorVisualization

Ruby2cThe Problem

Takahashi Method:

C Sucks

• Automatic translation of static ruby subset to C.

• Only translate if it has native support in C.

• Open architecture will allow it to translate to both Ruby internals C and generic ANSI C.

• Uses type inference to help with translation to generic C types.

Ruby2cThe Basics

Ruby2cBasic Design

SexpProcessor

ParseTree Rewriter TypeChecker Ruby2CCompositeSexp!

Processor

Sexpprocessesprocessors

ParseTree

Rewriter

TypeChecker

R2CRewriter

RubyToC

!P

ip

el

in

e!

Ruby2cStep-by-Step

• Ruby parses code

• ParseTree extracts AST

• Rewriter doesn’t touch it

• (SexpProcessor.process converts it to a Sexp)

• TypeChecker unifies it

• Ruby2C translates to C

Ruby

false and true

[:and, [:false], [:true]]

case NODE_AND: add_to_parse_tree(current, node->nd_1st); add_to_parse_tree(current, node->nd_2nd); break;

ParseTree

s(:and, s(:false), s(:true))

(no code for :and)

Rewriter

[:and, [:false, Type.bool], [:true,

Type.bool], Type.bool]

def process_and(exp) rhs = process exp.shift lhs = process exp.shift rhs.sexp_type.unify lhs.sexp_type rhs.sexp_type.unify Type.bool return t(:and, rhs, lhs, Type.bool)end

TypeChecker

Qfalse && Qtrue;

def process_and(exp) lhs = process exp.shift rhs = process exp.shift return "#{lhs} && #{rhs}"end

Ruby2C

Ruby2cFactorial

• For plain algorithmic code, ruby2c works quite well.

• Type inference determines the types of the args and automatically translates from Ruby types to C types.

• Ruby iterators are converted to equivalent while loops in C.

def factorial(n) f = 1 n.downto(2) { |x| f *= x } return fend

static VALUE factorial (VALUE self, VALUE _n) { long n = NUM2INT(_n); long f, x; f = 1; x = n; while (x >= 2) { f = f * x; x = x - 1; }; return INT2NUM(f);}

• For 1.0 final:

• Clean

• Document

• Differentiate between generic C and Ruby C

• More Ping-Pong against Metaruby

Ruby2cFuture Plans

Advanced

ruby2ruby -

factorial

sexp as method

auto-refactoring

obfuscator

profiler

zenoptimize

Advanced

ZenHacks

ZenHacksComing Full Circle

• A cornucopia of hackery: Toys, Tricks and Tools that have spawned out of our other projects but don't exactly fit there.

• Integrative: They almost always depend on multiple projects.

• Stuff that others might want to play with, but that I didn’t want to officially “support”.

• Anything I wrote that made me wince.

ZenHacksRubyToRuby - Building Blocks

• First SexpProcessor instance translates Sexps back to ruby.

• Provides a powerful foundation for a number of tools that follow a similar pattern:

• Extract Sexp

• Analyze and Modify Sexp

• Translate to Ruby

• Feed back to Ruby via eval

sexp = [:defn, :factorial, [:scope, [:block, [:args, :n], [:lasgn, :f, [:lit, 1]], [:iter, [:call, [:lvar, :n], :downto, [:array, [:lit, 2]]], [:dasgn_curr, :x], [:lasgn, :f, [:call, [:lvar, :f], :*, [:array, [:dvar, :x]]]]], [:lvar, :f]]]]

RubyToRuby.new.process sexp=> “def factorial(n) f = 1 n.downto(2) {|x| f = (f * x) } fend”

ZenHacksRubyToRuby - Building Blocks

ZenHacks• Back in June _why posted a blog entry about

having a ruby-lisp hybrid.

• It was just ruby w/ extra parens...

• ZenHacks lets you do nasty stuff like this:

require ‘sexp2ruby’

class Foo _ [:defn, :example, [:args], [:call, [:lit, 1], :+, [:array, [:lit, 1]]]]end

Foo.new.example #=> 2

...not that you’d want to.

RubyToRubyauto-refactoring examples

• User contribution by Rudi Cilibrasi.

• RubyToRuby allows sexps to be user friendly.

• We can analyze ruby mechanically, but then report what we find using ruby again.

• It allows us to write “lint-like” tools with nice reports like the following:

Suggest refactoring HastilyWritten#weirdfunc from:

def weirdfunc() thingy() while condition do thingy() endend

to:

def weirdfunc() begin thingy() end while conditionend

RubyToRubyauto-refactoring examples

Ruby Obfuscator

• ParseTree

• + SexpProcessor

• + ruby2c

• + a new tail-end

• = RubyObfuscator

def factorial(n) f = 1 n.downto(2) { |x| f *= x } return fend! ! ! ! ! becomes:

static VALUErrc_cF_factorial(VALUE __self, VALUE n) { VALUE f; VALUE x; f = LONG2FIX(1); x = n; while (RTEST(rb_funcall(x, rb_intern(“>=”), 1, LONG2FIX(2)))) { f = rb_funcall(f, rb_intern(“*”), 1, x); x = rb_funcall(x, rb_intern(“-”), 1, LONG2FIX(1)); }; return f;}

Ruby Obfuscator

zenprofile

• Standard profiler is 65 lines long, but dead slow.

• Shugo modified the event system to bypass set_trace_func.

• Massively fast, but 701 lines of C.

• zenprofile is 190 lines long, 114 are inlined C.

• 24x faster than ruby, ⅓rd slower than shugo’s.

• zenprofile’s goal is to balance size and speed.

0

187.5

375.0

562.5

750.0

original zenprofile shugo

16

24

576 701

190

65

zenprofilesize vs speedLoC Time (*8)

zenprofile

ZenOptimizeFull Circle... on steroids

• Take profiler’s architecture• Make a simple fast

method call counter• Use the profiler to trigger

a threshold event.• Event pulls method’s sexp.• Pass to ruby2c• Pass C to inline• Now you have

zenoptimize!• in 153 lines of code...

% time ruby factorial.rb 5000000Iter = 5000000, T = 67.23166600 sec, 0.00001345 sec / iterreal 1m7.310suser 0m55.980ssys 0m0.280s

% time ruby -reallyfast factorial.rb 5000000*** Optimizing Factorial.factorialIter = 5000000, T = 13.30087900 sec, 0.00000266 sec / iterreal 0m14.382suser 0m12.550ssys 0m0.290s

ZenOptimizeFull Circle... on steroids

Other Possibilities

• Automated Refactorings

• Idiomatic Optimizations (e.g. always use the most efficient iterator)

• Duplicate code checker

• Many kinds of lint-like tools

• Deprecation rewriter, etc.

Thank You

http://www.zenspider.com/seattle.rb

Presenters: Send me your

slides!