카카오 전사 모니터링/로깅/알람플랫폼 kemi

�카카오�전사�모니터링/로깅/알람플랫폼�KEMI

issac�at�ccc�issac.lim@kakaocorp.com

Who�am�I?�

Issac(임성국)�

2007~2009:�Mobile�Health�Care�

2009~2010:�Open�API�서비스�플랫폼�

2010~2014:�KT�Open�API�플랫폼�

2014�~:�Kakao�Openstack,�CMDB,�Logging,�Monitoring,�…

KEMI?�

(Kakao�Event�Metering�&�monItoring)�

카카오의�모든�리소스(baremetal,�private�cloud,�public�cloud,�container)에서�발생하는�데이터를�수집하고,�필요한�이벤트를�만들어�필요한�곳으로�전달하자!

왜�KEMI를?�

구글�has�StackDriver�

Netflix�has�Genie�and�atlas�

kakao?

왜�KEMI를?�

for�Production�Service�on�the�openstack�

KAKAO�infra(CMDB)와도�자동�연동된�모니터링/알람이�필수!

Ceilometer,�monasca,�cacti,�nagios,�…��

CMDB기반으로�자동화된�툴을�만들어보자!

KEMI는?

수집�

조회/알람�

제어�

->�Realtime/Batch

RealTime

Batch

Set�by�API�&�UI

Log는?

Realtime�&�Batch

Issue…

save�&�query�large�time�series�data(more�than�year)�

-�slow�query�&�cluster�hang�

->�restructuring�

origin�hbase�key:�[salt]<metric_uid><timestamp><tagk1><tagv1>[…

<tagkN><tagvN>]�

->�

[salt]<resource_id><timestamp><metric_uid><metric�value1>[…<tagkN><tagvN>]

Elasticsearch�Log�Indexing:��

no�replica,�more�shards,�ES�on�Hadoop�

kafka�Msg�Corruption:�

invalid�message�exception�handling

Is�it�enough?

Automatic�Resource�Mgmt�

Prediction�

More�features��

&�Open�source…

Thanks�

(@hardy,�@jenny,�@joanne,�@andrew�&�@@ccc)