ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved.

Tuning Kafka for Fun and Profit

ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. 2

Zookeeper

5-node vs. 3-node Ensembles

Solid State Disks– Use good SSDs– Transaction logs only– Significant improvement in latency and outstanding requests

ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. 3

Kafka Broker Disks

Disk Layout

JBOD vs. RAID– JBOD and RAID-0 are similar– RAID-5/6 has significant performance overhead– RAID-10 still offers the best performance and protection

Filesystem– New testing shows XFS has a clear benefit– No tuning required– Will be continuing testing with more production traffic

ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. 4

Scaling Kafka Clusters

Disk Capacity

Network Capacity

Partition Counts– Per-Cluster– Per-Broker

Limitations– Topic list length

ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. 5

Topic Configuration

Retention Settings

Partition Counts– Balance over consumers– Balance over brokers– Partition size on disk– Application-specific requirements

ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. 6

Mirror Maker

Network Locality

Consumer Tuning– Number of streams– Partition assignment strategy

Producer Tuning– Number of streams– In flight requests– Linger time