Grow your own toolsdebugging & profiling story

Remigijus Jodelis - SameSystem

VilniusRB Vilnius Ruby Community

Part I

Debugging

binding.pry

Debugging

Debugging

puts

Debugging

Rails.logger.info

Debugging

p some, thing

Debugging

pp some, thing

Debugging#<ActiveSupport::Logger:0x007f1aa39426f8 @default_formatter=#<Logger::Formatter:0x007f1aa3942450 @datetime_format=nil>, @formatter= #<ActiveSupport::Logger::SimpleFormatter:0x007f1aa4c0c900 @datetime_format=nil>, @level=0, @logdev= #<Logger::LogDevice:0x007f1aa39422c0 @dev=#<File:/vagrant/samesystem/log/development.log>, @filename=nil, @mutex= #<Logger::LogDevice::LogDeviceMutex:0x007f1aa3942298 @mon_count=0, @mon_mutex=#<Mutex:0x007f1aa3942090>, @mon_owner=nil>, @shift_age=nil, @shift_size=nil>, @progname=nil>=> #<ActiveSupport::Logger:0x007f1aa39426f8 @progname=nil, @level=0, @default_formatter=#<Logger::Formatter:0x007f1aa3942450 @datetime_format=nil>, @formatter=#<ActiveSupport::Logger::SimpleFormatter:0x007f1aa4c0c900 @datetime_format=nil>, @logdev=#<Logger::LogDevice:0x007f1aa39422c0 @shift_size=nil, @shift_age=nil, @filename=nil, @dev=#<File:/vagrant/samesystem/log/development.log>, @mutex=#<Logger::LogDevice::LogDeviceMutex:0x007f1aa3942298 @mon_owner=nil, @mon_count=0, @mon_mutex=#<Mutex:0x007f1aa3942090>>>>

SHIT...

Debugging

We need to do better

Debugging

STEP 1

Better name

Debugging

# initializers/wtf.rb

def WTF?(*args) p argsend

Debugging def sync_records(resource, local_idx, remote_idx, key_name, options={}) options.reverse_merge!(@opt)

cross_idx, actions = diff_collection(local_idx, remote_idx) actions.each do |action, items| @cnt["#{resource.name}_#{action}"] = items.size end WTF? resource.name.to_s, *actions, :line, :file

actions.each { |action, items| items.clear unless options[action] }

if options[:create_many] resource.create_many(*actions[:create]) if actions[:create].any? else actions[:create].each do |record| remote = resource.create(record) or next if local = local_idx[remote[key_name.to_s]] cross_idx[local.id] = remote['id'] end

Debugging

STEP 2

More options

Debugging

def WTF?(*args) if args.last == :pp args.pop pp args else p args endend

WTF? my_struct, :pp

Debugging

WTF? my_struct, :pp

WTF? my_var, :yamlWTF? my_var, other_var, :pp, :fileWTF? *other_things, :json, :file, :time

But I want even more...

Debugging

def WTF?(*args) allowed = [:pp, :yaml, :json] options = {} while allowed.include?(args.last) options[args.pop] = true end

case when options[:yaml] puts args.to_yaml # ...

Debugging

data = "" data << "[%s] " % Time.now if options[:time] rx = %r{([^/]+?)(?:\.rb)?:(\d+):in `(.*)'$} # <- WTF is this? data << "WTF (%s/%s:%s)" % caller[0].match(rx).values_at(1,3,2)

data << ": " << case when options[:pp] args.pretty_inspect.gsub(/^\[|\]$/,'') # removes array marks when options[:yaml] YAML.dump(args) when options[:json] JSON::pretty_generate(args) when options[:text] args.map(&:to_s).join("\n")

Debugging

WTF (search_controller/browse:15): #<Paginator:0x007fd0199e6bf0 @count=169, @per_page=20,

def browse if params[:search].present? joins = search_joins conditions = search_conditions includes = search_includes count = klass.joins(joins).includes(includes).where(conditions).count pager = ::Paginator.new(count, search_config.per_page) do |offset, per_page| klass.joins(joins).where(conditions).includes(includes). endWTF? pager, :pp

Debugging

STEP 3

Moreoutput options

Debugging

case when options[:page] (Thread.current[:wtf] ||= []) << data when options[:file] time = Time.now.strftime('%m%d_%H%M%S') filename = "wtf_#{time}_#{rand(10000)}.txt" File.write("#{Rails.root}/tmp/#{filename}", data) when options[:raise] raise data when options[:redis] REDIS.rpush 'wtf', data REDIS.expire 'wtf', 30*60 else Rails.logger.info data end

Debugging

OK, I got it

But, where do we place all this code?

Debugging

module Kernel WTF_OPTIONS = [:time, # prefix :pp, :yaml, :json, :csv, :text, :line, # format :bare, # modifiers :page, :file, :raise, :redis, :log] # output

def WTF?(*args)

The same place as p

Debugging

Time to refactor!

Debugging

Object.class_eval do def WTF?(*args) WTF::Dumper.new(*args) endend

module WTF class Dumper OPTIONS = [:time, :nl, :none, # ...

Part II

Profiling

So we got a situation...

Profiling

class Overview # loading dependent data and calculations def load! load_departments

@budget_repo = Overviews::BudgetRepository.new(@period, @shops) @data = {}

columns.each do |col| calculate(col) end # … few more lines … end

# … 50 more methods …

The problem lurks somewhere here

Profiling

Let’s try the profiler

Profiling

Profiling

WTF am I looking at?

Profiling

Second shot

start with a simple thing

Profiling

# loading dependent data and calculations def load! st = Time.now

# ...

WTF? Time.now - st end

WTF (overview/load!:250): 180.3073...

Profiling

What if I tried this

on EVERY methodin the same class

Profiling

module MethodTracker class << self def included(base) methods = base.instance_methods(false) + base.private_instance_methods(false) base.class_eval do methods.each do |name| original_method = instance_method(name) define_method(name) do |*args, &block| MethodTracker.on_start(base, name) return_value = original_method.bind(self).call(*args, &block) MethodTracker.on_end return_value end end end end

Profiling

In Ruby 2.x we can do nicer

hint: prepend OverridesModule

Profiling

def override_method(base, name) %{ def #{name}(*args) WTF::MethodTracker.on_start(#{base}, :#{name}) return_value = super WTF::MethodTracker.on_end return_value end } end

Profiling

def prepare(base) methods = base.instance_methods(false) + base.private_instance_methods(false) compiled = methods.map do |name| override_method(base, name) end base.module_eval %{ module Tracking #{compiled.join} end prepend Tracking } end

Public interfacemodule WTF class << self def track(*objects) MethodTracker.setup(*objects) MethodTracker.reset_state end

def track_finish MethodTracker.finish end endend

class Overview def load! WTF.track(self, Overview::Helpers) # with additional classes # ... WTF.track_finish endend

Collecting data

require 'absolute_time'

def add_stats(at_start = nil) stat = stats[stack.last]

this_time = AbsoluteTime.now stat[:time] += this_time - last_time self.last_time = this_time

this_heap = GC.stat[:heap_length] stat[:heap] += this_heap - last_heap self.last_heap = this_heap

stats[at_start][:freq] += 1 if at_start end

Profiling

What about the output?

Profiling

Profiling

Profiling

Let’s look at one more problem.

I got some Rails logs...

Profiling

WTF is generating this query?

Profiling

I want to do this

# loading dependent data and calculationsdef load!

WTF.sql %(SELECT `weekly_balance_lines`.* FROM `weekly_balance_lines

load_shops

@budget_repo = Overviews::BudgetRepository.new(@period, @shops) @salary_repo = Overviews::SalaryRepository.new(@period, @shops, @c

Profiling

… and get the answer

SQL: "SELECT `weekly_balance_lines`.`date`, `weekly_balance_lines`.`context_id`, "(eval):4:in `log'" "/home/vagrant/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/activerecord-4.1 "/home/vagrant/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/activerecord-4.1 "/home/vagrant/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/activerecord-4.1 "/home/vagrant/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/activerecord-4.1 "/home/vagrant/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/activerecord-4.1 "/vagrant/samesystem/app/models/overview.rb:350:in `weekly_balance_lines_for_p "/vagrant/samesystem/app/models/overview.rb:342:in `weekly_balance_lines'" "/vagrant/samesystem/app/models/overview.rb:870:in `sales'" "/vagrant/samesystem/app/models/overview.rb:396:in `calculate'" "/vagrant/samesystem/app/models/overview.rb:241:in `block in

Profiling

ActiveRecord::ConnectionAdapters::AbstractAdapter.module_eval %( module TrackingSQL def log(sql, *args, &block) WTF::QueryTracker.on_sql(sql) # callback and WTF? caller super(sql, *args, &block) end end prepend TrackingSQL)

Easy! The key piece is here

Conclusion

Does it look like a gem?

YES!

github.com/remigijusj/wtf-tools

Please check out the repo!

Thank you!