modern black mages fighting in the real world
TRANSCRIPT
Modern Black Mages Fighting in the Real World
Sep 9, 2016 in RubyKaigi 2016
@tagomoris Satoshi "Moris" Tagomori
Satoshi "Moris" Tagomori (@tagomoris)
Fluentd, MessagePack-Ruby, Norikra, ...
Treasure Data, Inc.
https://github.com/tagomoris/msgpack-inspect
http://docs.fluentd.org/articles/logohttp://www.fluentd.org/
Fluentd
• What is Fluentd? • Open Source Log Collector • Pluggable, Reliable, Less resource usage, Ease to use
• Versions of Fluentd • v0.12: stable versions (2014/12 - Now) • v0.14: versions for next stable (2016/05 - Now)
http://docs.fluentd.org/articles/logo
What's about this session?
• Introduce some patterns of "Black Magic"s (a.k.a. meta programming) in Ruby
• Show you some PRAGMATIC use of Black Magics
Fluentd v0.14 Release
Fluentd v0.14 API Update
• Everything changed :)
• Plugin namespace • before: Fluent::* (Top level classes even for plugins!) • after: Fluent::Plugin::*
• Plugin base class for common methods • Inconsistent Output plugin hierarchy • Plugin must call `super` in common methods
http://www.slideshare.net/tagomoris/fluentd-v014-plugin-api-details
Classes hierarchy (v0.12)
Fluent::Input F::Filter
F::Output
BufferedOutput
ObjectBuffered
TimeSliced Multi
Output F::BufferF::Parser
F::Formatter
3rd party plugins
Classes hierarchy (v0.14)
F::P::Input F::P::Filter F::P::Output
Fluent::Plugin::Base
F::P::BufferF::P::Parser
F::P::FormatterF::P::Storage
both ofbuffered/non-buffered
F::P::BareOutput(not for 3rd party
plugins)
F::P::MultiOutput
copyroundrobin
diff v0.12 v0.14
F::P::Output
Fluent::Plugin::Base
both ofbuffered/non-buffered
F::P::BareOutput(not for 3rd party
plugins)
F::P::MultiOutput
copyroundrobin
F::Output
BufferedOutput
ObjectBuffered
TimeSliced Multi
Output
Super classes byhow to buffer data
All output pluginsare just "Output"
Basic Black Magics: Class and Mixin in Ruby
Class and Subclass in Ruby
class A
#bar
class B
#bar
super
B.new.bar
class A
#bar
class B
#bar
super
B.new.bar
module M
#bar
Introducing Methods by Mixin
class A
#bar
class B
#bar
super
B.new.bar
module M
#bar
Singleton Class of Ruby
#bar
B.new.singleton_class
class A
#bar
class B
#bar
super
b=B.new b.singleton_class.include M2 b.bar
module M
#bar
Adding Methods on An Instance (1)
B.new.singleton_class
#bar
M2
#bar
class A
#bar
class B
#bar
super
b=B.new b.extend M2 b.bar
module M
#bar
Adding Methods on An Instance (2)
B.new.singleton_class
#bar
M2
#bar
Back to Fluentd code :)
diff v0.12 v0.14
F::P::Output
Fluent::Plugin::Base
both ofbuffered/non-buffered
F::P::BareOutput(not for 3rd party
plugins)
F::P::MultiOutput
copyroundrobin
F::Output
BufferedOutput
ObjectBuffered
TimeSliced Multi
Output
Super classes byhow to buffer data
All output pluginsare just "Output"
Fluentd v0.12 Fluent::Output
class Fluent::Output
#emit(tag, es, chain)
MyOutput
Engine calls plugin.emit(tag, es, chain)
@buffer
Fluentd v0.12 Fluent::BufferedOutput (1)
class Fluent::Outputclass BufferedOutput
#emit(tag, es, chain, key)
MyOutput
#emit(tag, es, chain)
super(tag, es, chain, any_key)
Engine calls plugin.emit(tag, es, chain)
@buffer
#emit(key, data, chain)
#format(tag,time,record)
#format_stream(tag,es)
Fluentd v0.12 Fluent::BufferedOutput (2)
class Fluent::Outputclass BufferedOutput
#emit(tag, es, chain, key)
MyOutput
#emit(tag, es, chain)
super(tag, es, chain, any_key)
Engine calls plugin.emit(tag, es, chain)
@buffer
#emit(key, data, chain)
#format_stream(tag,es)
#format_stream(tag,es)
#format(tag,time,record)
Fluentd v0.12 Fluent::TimeSlicedOutput
class Fluent::Outputclass BufferedOutput
#emit(tag, es, chain, key)
MyOutput#emit(tag, es, chain)
Engine calls plugin.emit(tag, es, chain)
@buffer
#emit(key, data, chain)
#emit(tag, es, chain) class TimeSlicedOutput
#format(tag,time,record)
Fluentd v0.12 Fluent::ObjectBufferedOutput
class Fluent::Outputclass BufferedOutput
#emit(tag, es, chain, key)
MyOutput#emit(tag, es, chain)
Engine calls plugin.emit(tag, es, chain)
@buffer
#emit(key, data, chain)
#emit(tag, es, chain) class ObjectBufferedOutput
Fluentd v0.12 Fluent::BufferedOutput
class Fluent::Outputclass BufferedOutputMyOutput
@buffer calls #write in OutputThread
@buffer
chunk#write(chunk)
OutputThread
#pop
Fluentd v0.12 Fluent::TimeSlicedOutput
class Fluent::Outputclass BufferedOutput
@buffer
MyOutput
class TimeSlicedOutput
OutputThread
#write(chunk)
@buffer calls #write in OutputThread
#write calls chunk.keychunk
#pop
Fluentd v0.12 Fluent::ObjectBufferedOutput
class Fluent::Outputclass BufferedOutput
@buffer
MyOutput
class ObjectBufferedOutput
OutputThread
#write(chunk)
#write(chunk)
#write_object(chunk_key, chunk)
@buffer calls #write in OutputThread
chunk#pop
Fluentd v0.12 API Problems
• Entry point method is implemented by Plugin subclasses • Fluentd core cannot add any processes
• counting input events • hook arguments/return values to update API
• Fluentd core didn't show fixed API
• Plugins have different call stacks • It's not clear what should be implemented for authors • It's not clear what interfaces are supported for
arguments/return values
How can we solve this problem?
Fluent::Plugin::Output (v0.14)
Fluentd v0.14 Fluent::Plugin::Output
class Outputclass MyOutput
#process(tag, es)
Engine calls plugin.emit_events(tag, es)
@buffer
#write
#emit_events(tag, es)
#format(tag, time, record)
#write(chunk)
#try_write(chunk)
#emit_sync(tag, es)
#emit_buffered(tag, es)
Fluentd v0.14 Fluent::Plugin::Output
class Outputclass MyOutput
Output calls plugin.write (or try_write)
@buffer
chunk
#write(chunk)
#try_write(chunk)
flush thread
#process(tag, es)
#format(tag, time, record)
Fluentd v0.14 Design Policy
• Separate entry points from implementations • Methods in superclass control everything
• Do NOT override these methods! • Methods in subclass do things only for themselves
• not for data flow, control flow nor others
• Plugins have simple/straightforward call stack • Easy to understand/maintain
❤
How about existing v0.12 plugins?
Requirement:
(Almost) All Existing Plugins SHOULD Work Well WITHOUT ANY MODIFICATION
• Fluent::Compat namespace for compatibility layer
v0.14 Plugins & Compat Layer
F::P::Output
F::P::Base
v0.14 PluginsFluent::
Compat::Output
F::C::BufferedOutput
F::C::TimeSliced
Output
F::C::ObjectBuffered
Output
Fluent::OutputF::
BufferedOutput
F::TimeSliced
Output
F::ObjectBuffered
Output
v0.12 Plugins
Double Decker Compat Layer?
• Existing plugins inherits Fluent::Output or others • No more codes in Fluent top level :-(
• Separate code into Fluent::Compat • and import it into Fluent top level
Fluentd v0.14 Fluent::Plugin::Output
class Outputclass MyOutput
#process(tag, es)
Engine calls plugin.emit_events(tag, es)
@buffer
#write
#emit_events(tag, es)
#format(tag, time, record)
#write(chunk)
#try_write(chunk)
#emit_sync(tag, es)
#emit_buffered(tag, es)
Fluentd v0.12 Fluent::BufferedOutput (2)
class Fluent::Outputclass BufferedOutput
#emit(tag, es, chain, key)
MyOutput
#emit(tag, es, chain)
super(tag, es, chain, any_key)
Engine calls plugin.emit(tag, es, chain)
@buffer
#emit(key, data, chain)
#format_stream(tag,es)
#format_stream(tag,es)
#format(tag,time,record)
Fluent::Plugin::Outputclass MyOutput@buffer
#write
#emit_events(tag, es)
v0.12 Plugins via Compat Layer: Best case (virtual)
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain, key) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk) flush thread
Fluent::Plugin::Outputclass MyOutput@buffer
#write
#emit_events(tag, es)
v0.12 Plugins via Compat Layer: Best case (real)
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain, key) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk) flush thread
Fluent::Plugin::Outputclass MyOutput@buffer
#write
#emit_events(tag, es)
When plugin overrides #format_stream
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain, key) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk) flush thread
Fluent::Plugin::Outputclass MyOutput@buffer
#write
#emit_events(tag, es)
When plugin overrides #format_stream
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain, key) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk) flush thread
default implementation for calling "super"
Fluent::Plugin::Outputclass MyOutput@buffer
#write
#emit_events(tag, es)
When plugin overrides #emit
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk) flush thread
Fluent::Plugin::Outputclass MyOutput@buffer
#write
#emit_events(tag, es)
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk) flush thread
When plugin overrides #emit
Fluent::Plugin::Outputclass MyOutput@buffer
#write
#emit_events(tag, es)
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk) flush thread
This call doesn't happen, in fact
#emit doesn't return values!
When plugin overrides #emit
Fluent::Plugin::Outputclass MyOutput@buffer
#write
#emit_events(tag, es)
When plugin overrides #emit
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk) flush thread
#emit calls @buffer.emit → NoMethodError !
Fluent::Plugin::Outputclass MyOutput@buffer
#write
#emit_events(tag, es)
When plugin overrides #emit
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk) flush thread
Fluent::Plugin::Outputclass MyOutput@buffer
#write
#emit_events(tag, es)
When plugin overrides #emit
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk) flush thread
#emit
1. #emit calls @buffer.emit with data to be written in buffer
0. plugin calls @buffer.extend to add #emit
2. @buffer.emit stores arguments into plugin's attribute
3. get stored data
4. call @buffer.write with data
Fluent::Plugin::Outputclass MyOutput@buffer
#write
#emit_events(tag, es)
When plugin overrides #emit
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk) flush thread
Fluent::Plugin::Outputclass MyOutput@buffer
#write
#emit_events(tag, es)
Thinking about "chunk" instance ...
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk) flush thread
#write may call "chunk.key", but v0.14 chunk doesn't have #key !
Fluent::Plugin::Outputclass MyOutput@buffer
#write
Compat::BufferedOutput
#write(chunk) flush thread
"chunk" has #metadata, and values of #key can be created via #metadata
Let's "chunk.extend" !
Where to do so?
?
Thinking about "chunk" instance ...
Fluent::Plugin::OutputMyOutput@buffer
#write
C::BufferedOutput
#write(chunk) flush thread
Thinking about "chunk" instance ...
#write(chunk)
BufferedChunkMixin
plugin.extend BufferedChunkMixin in #configure
Similar hacks for TimeSlicedOutput and ObjectBufferedOutput ...
Controlling Plugin Lifecycle
Plugin Lifecycle Updated
Methods(v0.12) • #configure • #start
• #before_shutdown • #shutdown
v0.12 Plugins often doesn't call "super"!
Methods(v0.14) • #configure • #start • #stop • #before_shutdown • #shutdown • #after_shutdown • #close • #terminate
In v0.14, these methods MUST call "super"
• #configured? • #started? • #stopped? • #before_shutdown? • #shutdown? • #after_shutdown? • #closed? • #terminated?
For Example: shutdown compat plugins
Fluent::Plugin::Base
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
It doesn't call "super"! We want to call this...
What We Want To Do:
Fluent::Plugin::Base
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
1. call #shutdown anyway
0. Fluentd core calls #shutdown
2. call #shutdown? to check "super" is called or not
3. call #shutdown of superclass forcedly!
What We Want To Do:
Fluent::Plugin::Base
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
How to make this point?
One More Magic! Module#prepend
class A
#bar
class B
#bar
super
B.new.bar
Wrapping Methods on a Class (1)
B.new.singleton_class
#bar
class A
#bar
class B
#bar
super
B.new.bar
module M
Wrapping Methods on a Class (2)
B.new.singleton_class
#bar
#bar
Using extend is powerful, but it should be done for all instances
How about wrapping methods for all instances of the class?
class A
#bar
class B
#bar
super
module M;def bar;super;end;end B.prepend M B.new.bar
module M
Wrapping Methods on a Class (3): Module#prepend
B.new.singleton_class
#bar
#bar
module M wraps B, and M#bar is called at first
What We Want To Do:
Fluent::Plugin::Base
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
THIS ONE !!!
What We Got :-)
Fluent::Plugin::Base
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
1. call #shutdown anyway
0. prepend CallSuperMixin at first
2. call #shutdown? to check "super" is called or not
3. if not, get method of superclass, bind self with it, then call it
Thank you @unak -san!
Beating Test Code
Testing: Capturing return values by Test Driver
OutputMyOutput@buffer
#write
#emit_events
#format
#emit_buffered
Output Plugin Test Driver
Create plugin instances
Feed test data into plugin
Testing: Capturing return values by Test Driver
OutputMyOutput@buffer
#write
#emit_events
#format
#emit_buffered
We want to assert this return value!
Output Plugin Test Driver Feed test data into plugin
Using #prepend to capture return values
OutputMyOutput@buffer
#write
#emit_events
#format
#emit_buffered
Output Plugin Test Driver Feed test data into plugin
moduleM
#format
Store return value of "super"
Using #prepend ... doesn't work :-(
OutputMyOutput@buffer
#write
#emit_events
#format
#emit_buffered
Output Plugin Test Driver Feed test data into plugin
#format
Test code sometimes overwrites methods for many reasons :P
singletonclass
#format
😱
moduleM
One Another Magic: Stronger Than Anything
class A
#bar
class B
#bar
super
b=B.new b.singleton_class.module_eval{define_method(:bar){"1"}} b.bar
Another Study: How To Wrap Singleton Method?
B.new.singleton_class
#bar
module M
#bar
class A
#bar
class B
#bar
supermodule P
Another Study: How To Wrap Singleton Method?
B.new.singleton_class
#bar
b=B.new b.singleton_class.module_eval{define_method(:bar){"1"}} b.singleton_class.prepend P b.bar
#bar
module M
#bar
Singleton class is a class, so it can be prepended :)
It's actually done in Test Driver implementation...
Using #prepend on singleton_class: Yay!
OutputMyOutput@buffer
#write
#emit_events
#format
#emit_buffered
Output Plugin Test Driver Feed test data into plugin
#format
singletonclass
#format😃
moduleM
Prepending modules on singleton_class overrides everything!
IS BUILT ON A TOP OF BUNCH OF BLACK MAGICS :P
Do Whatever You Can For Users!
It Makes Everyone Happier!
... Except for Maintainers :(