LINE의 Git 기반 고가용성 서비스 설정 저장소

Central Dogma

LINE+ | 개발4실 | 이희승 (@trustin)

2017-10-25

What is Central Dogma?

Case studies

Call for participation

01

02

03

Agenda

Any non-trivial service needs to store itsconfiguration somewhere, “safely.”

What is Central Dogma?

Central Dogma is ...

• Repository service for textual configuration– Primarily JSON

– YAML, XML, INI, JavaScript, …

• Highly available

• Version controlled

• Advanced query mechanism

• Change notification

• Mirroring from an external Git repository

Stores anything textual

• What’s fetched at start-time– Application parameters

– Bean properties

• What’s updated at run-time– User · IP blacklist

– Scheduled maintenance notice

– Roll-out & A/B experiment parameters

• What’s evaulated at run-time– Rule-engine scripts

Highly-available

• Multi-master

• Eventually consistent– Writing to A then reading from B → ?!

– Client-side load-balancing

• Fast read / Slow write

• ZooKeeper as a replication log queue

Version-controlled

• jGit as a back-end storage– History – diffs and authors

– Bigger than RAM

• Focus on simplicity– Integer revision numbers

– Linear history – no branches

Advanced query mechanism

• … thanks to the first-class JSON support

• JSON path

• JSON patch – RFC6902

Change notification

• Get notified on a new commit

Mirroring from an external Git repository

• Keep your settings in a GitHub / GitLab repository

• Send a pull request to modify the settings

• Get it reviewed and merged

• Let your services read from Central Dogma– Queryable

– Watchable

– Highly-available

To mirror or not to mirror

• Git-to-CD mirroring is optional.– You can commit to CD directly using:

● Client library● Web dashboard

• Do not commit directly to a mirrored repository

• Things that do not need mirroring:– Files updated by a non-human being

Putting it all together

One more thing

• We have a command-line client as well.– Easy to access from shell scripts

– Cross-platform: written in Go-lang

Real world use cases at LINE

Case studies

Scheduled maintenance

• Enter · Exit scheduled maintenance– “Our service is under maintenance until <endTime>.”

• An administrator updates maintenance.json using a Python script.

• Web application watches maintenance.json.

PlanOut integration

• Online field experimentation framework

• Implemented com.glassdoor.planout4j.config.Planout4jRepository which uses Central Dogma as a back-end.

– A/B testing

– Feature roll-out

Using as a directory service

• Using a JSON path query to find the information about a service:

$[?(@.hostname == 'TKSVR1234' && @.port == 8080)] [{"zoneId": 0, "zone": "JP", "groups": [{ "groupSet": "ROLE", "name": "Android" }], "projectId": "talk-server", "projectPhase": "RELEASE", "hostname": "TKSVR1234", "ip": "192.168.1.234", "port": 8080, "weight": 2000, "status": "NORMAL", "keepAlive": false }]

Wanna give it a try?

Call for participation

Current status

• Doing great at production– … with continuous improvements and new features

– Increasing # of internal customers

• We’re not “there” yet.– Your feed back

– Your contributions● Technically intriguing to tackle

Future work

• Fine-grained authorization

• Command-line client improvements

• Web UI improvements

• More metrics (with Armeria)

• Write performance

• Find more at our issue tracker:– https://github.com/line/centraldogma/issues

How to contribute

• Apache-licensed

• Visit the official web site– https://line.github.io/centraldogma/

• Fork or star the project at GitHub:– https://github.com/line/centraldogma/

Copyright ⓒ 2017 SAMSUNG ELECTRONICS. ALL RIGHTS RESERVED

THANK YOU