wtf is a method cache?

BitLove

What the fuck is a method cache?

James Golick

Thursday, 17 July, 14

James Golickwriting: http://jamesgolick.com

code: https://github.com/jamesgolickshit talk: https://twitter.com/jamesgolick

podcast: http://realtalk.io


http://jamesgolick.com

http://jamesgolick.com

https://github.com/jamesgolick

https://github.com/jamesgolick

https://twitter.com/jamesgolick

https://twitter.com/jamesgolick

http://realtalk.io

http://realtalk.io

@jamesgolick


BitLove

DISCLAIMERC O M P U T O L O G Y A H E A D


Big “O” Notation


stuff.each do |thing| # ...end

Variable Time


a = stuff.popa = !!astuff.unshift a

Constant Time


BitLove

What the fuckis a method cache?


BitLove

1. Background


struct RClass { struct RClass super; struct st_table m_tbl;};


Method Resolution


class A def a puts 'Hi!' endend

class B < A; endclass C < B; endclass D < C; endclass E < D; endclass F < E; end

F.new.a


rb_method_entry_tvm_resolve_method(struct RClass klass, symbol_t method_name){ rb_method_entry_t ent = st_lookup(klass.method_tbl, method_name);

if (ent) { return ent; } else { if (klass->super) { return vm_resolve_method(klass.super, method_name); } else { return NULL; } }}


Module Inclusion


module A def a "Hello, World!" endend

module B; include A; endmodule C; include B; end

class D include Cend

D.new.a


module A ICLASS ICLASS ICLASS

module B ICLASS ICLASS

module C ICLASS

class D


irb> ActiveRecord::Base.included_modules.length=> 71


Summary

• Methods are stored in a hashtable on the class where they’re defined.

• Method resolution is a variable time algorithm whose complexity depends on the depth of your class hierarchy.

• Module inclusion substantially increases the depth of your class hierarchy, especially if those modules themselves include modules.

• Method resolution is expensive.


BitLove

What the fuck is a method cache?


BitLove

2. Method Cachingin the pre-

jamesgolick era


Instruction Caches


static uint global_vm_state = 0;


struct inline_cache { struct RClass klass; uint vm_state; rb_method_entry_t me;}


rb_method_entry_tvm_search_method(struct RClass klass, rb_symbol_t method_name, struct inline_cache ic){ rb_method_entry_t me; if (is_valid_cache_entry(ic, cache)) { me = ic.me; } else { me = vm_resolve_method(klass, method_name); ic.me = me; ic.vm_state = GET_VM_STATE(); ic.klass = klass; } return me;}


intis_valid_cache_entry(struct inline_cache ent, struct RClass klass){ return ent.klass == klass && ent.vm_state = GET_VM_STATE();}


Global Method Cache


instruction cache

instruction cache

instruction cache

global cache

method resolution


struct method_cache_entry { struct RClass klass; uint vm_state; rb_method_entry_t me;}


#define METHOD_CACHE_SIZE 2048

static struct rb_method_cache_entry method_cache[METHOD_CACHE_SIZE];


rb_method_entry_t *vm_resolve_method(struct RClass *klass, symbol_t method_name){ struct method_cache_entry ent; rb_method_entry_t *me; ent = method_cache[method_name % METHOD_CACHE_SIZE]; if (is_valid_cache_entry(ent, klass)) { me = cache_entry.me; } else { me = vm_resolve_method_without_cache(klass, method_name); cache_entry.me = me; cache_entry.vm_state = GET_VM_STATE(); cache_entry.klass = klass; } return me;}


intis_valid_cache_entry(struct method_cahe_entry ent, struct RClass klass){ return ent.klass == klass && ent.vm_state = GET_VM_STATE();}


Cache Invalidation


static uint64_t global_vm_state = 0;

#define INC_VM_STATE global_vm_state++

voidrb_define_method(struct RClass *klass, symbol_t name, rb_method_entry_t *me){ // ... INC_VM_STATE(); // ...}


Defining methods.

Aliasing methods.

Removing methods.

Setting or removing constants.

Defining a class.

Defining a module.

Including a module.

things that bust the cache


Extending a module.

Using a refinement. (Ruby 2.0)

Garbage collecting a class.

Garbage collecting a module.

Changing the visibility of a constant.

Marshal loading an extended constant.

Autoload.

Built-in non-blocking IO methods.



OpenStruct instantiation.



Summary

• Method resolutions are cached in two places.

• Instruction caches are structs attached to the send instruction.

• The global method cache is a hash table fixed at 2048 entries with no collision semantics and a random eviction policy.

• Method cache entries are valid if their `vm_state` property is the same as the current value of the `global_vm_state` counter.


Summary

• Method cache invalidation is always global, and happens frequently in most ruby code.

• Method cache invalidation is constant time.


Numbers


BitLove

3. jamesgolick Method Caching


struct RClass { struct RClass super; struct st_table m_tbl; struct st_table mc_tbl; uint64_t seq; subclass_list_entry_t subclasses;};


static uint64_t rb_vm_sequence = 0;

#define NEXT_SEQ() ++rb_vm_sequence


struct RClassclass_alloc(...){ struct RClass klass; // ... klass.seq = NEXT_SEQ(); // ... return klass;}


struct inline_cache { uint64_t seq; rb_method_entry_t me;}


rb_method_entry_t *vm_search_method(struct RClass klass, rb_symbol_t method_name, struct inline_cache ic){ rb_method_entry_t me; if (ic.seq == klass.seq) { me = ic.me; } else { me = vm_resolve_method(klass, method_name); ic.me = me; ic.seq = klass.seq; } return me;}


struct method_cache_entry { uint64_t seq; rb_method_entry_t me;}

rb_method_entry_t *vm_resolve_method(struct RClass klass, symbol_t method_name){ struct method_cache_entry ent; rb_method_entry_t me; ent = vm_get_method_cache_entry(klass, method_name); if (ent.seq == seq) { me = cache_entry.me; } else { me = vm_resolve_method_without_cache(klass, method_name); cache_entry.me = me; cache_entry.seq = klass.seq; } return me;}


voidrb_clear_cache_by_class(struct RClass klass){ subclass_list_entry_t ent; klass.seq = NEXT_SEQ(); ent = klass.subclasses; while(ent != NULL) { rb_clear_cache_by_class(ent.klass); ent = ent.next; }}


Object

User

ActionController::Base

ActiveRecord::Base

UsersController

SessionsController Group


Summary

• Both types of method cache entries now only need to store a seq and method entry.

• Method caches are now stored with the RClass structs and are !effectively" unbounded in size.

• Each RClass has a globally unique 64bit identifier.

• Method cache entries are tagged with the sequence of their target klass at the time the cache entry was filled.


Summary

• Entries are valid if their filled entry sequence is the same as the current sequence identifier of the klass that is the target of the invocation.

• Method caches are invalidated by assigning a new sequence value to a klass.

• When changes are made to a klass, we traverse all of its descendents and assign them new sequence values.

• This traversal is unfortunately a variable time algorithm, and can be quite expensive.


BitLove

4. rvm install jamesgolick


Dat Patch

• Top-down class hierarchy tracking.

• Class#subclasses

• Module#included_in

• Possible future bug fixes.

• Hierarchical method cache invalidation.

• Method cache instrumentation.


Instrumentation

• RubyVM::MethodCache.hits

• RubyVM::MethodCache.misses

• RubyVM::MethodCache.miss_time

• RubyVM::MethodCache.invalidation_time


Get The Code

• rvm install jamesgolick

• git clone git://github.com/jamesgolick/ruby.git

• https://github.com/jamesgolick/ruby


https://github.com/jamesgolick/ruby

https://github.com/jamesgolick/ruby

Questions?


wtf is a method cache?

Documents

t method

global method cache

entry method

state cache

method cache entries

global cache method

method caching

cache ic