A Tale of Three Tools: Kubernetes, Jsonnet, and Bazel

Miles YuchtSF Kubernetes MeetupJuly 26, 2017 1

Who am I?

Miles• CS @ Princeton• SWE on Growth team @ Databricks

since 2015• "Lead Developer" of kubecfg

2

Agenda

• Kubernetes @ Databricks • Why Jsonnet• Combining Bazel and Kubernetes• Q & A

3

Kubernetes

4

Kubernetes at Databricks

● Multiple Kubernetes clusters, hundreds of deployments● 6+ terabytes of memory, hundreds of cores● Many customizations for security● Everything new goes onto Kubernetes● All managed by a 7-person team

5

Kubernetes at Databricks

We're hiring! Come talk to Priya or me after the talks!

6

Combinatorial Configurations

6 clusters, 3 environments, 4 modes of deployment● Lots of parameters● Extremely verbose

○ >75k lines of checked-in YAML templates● YAML is not modular● Hard to manage, manipulate, and correctly refactor templates

7

Kubernetes development lifecycle

● Simple to test & develop in Kubernetes● Keep dev, test, staging, prod environments similar● Critical for testing kubernetes/cloud features

○ E.g. replication, PROXY protocol

8

Many-step Deployment

9

Build Docker Image

Tag Docker Image

Push Docker ImageEdit Template

Apply Template

kubecfg!

● Python wrapper around kubectl○ Still have all your favorite kubectl commands

● Preprocesses inputs and passes the result to kubectl● Augments kubectl

○ Easy to add new commands to kubecfg● Automates several common workflows

○ Simpler for developers○ Fewer chances to make mistakes

10

Jsonnet

11

Jsonnet

● DSL for defining JSON data● Break up monolithic configurations into composable parts● >75k lines of YAML → ~20k lines of Jsonnet● To learn about how we use Jsonnet:

○ https://databricks.com/blog/2017/06/26/declarative-infrastructure-jsonnet-templating-language.html

12

Jsonnet → JSON

When running kubecfg apply -f webapp.jsonnet:● jsonnet webapp.jsonnet > /tmp/aaa.json● kubectl apply -f /tmp/aaa.json

13

Original kubecfg architecture

14

Jsonnet Template Kubernetes spec Kubectl

Integrating build and deploy

Kubernetes template⇕

Tagged image

15

Integrating build and deploy

Kubernetes template⇕

How to produce the artifact that it depends on

16

Integrating build and deploy

1. What do you need to do to prepare the artifact?→ Ask your build tool.

2. How do you refer to the artifact?→ Use a tagged docker image.

17

Bazel

18

Bazel

● Open-source, language-agnostic build tool● Very customizable/configurable● Ported our build from SBT to Bazel mid-2016● To learn about how we use Bazel:

○ Blog post coming soon!

19

Kubectl Templates

{

image: "registry.databricks.com/webapp:latest",

...

}

20

Kubectl Templates

{

image: "registry.databricks.com/webapp:latest",

...

}

Kubernetes template⇕

Tagged image21

Kubecfg Templates

{

target: "//docker/webapp",

...

}

22

Kubecfg Templates

{

target: "//docker/webapp",

...

}

Kubernetes template⇕

Bazel target23

Integrating build and deploy

Build:● bazel build //docker/webapp:webapp

Tag & Push (new):● bazel run //docker/webapp:webapp_push -t <tag>

Deploy (new):● $(cat //docker/webapp:webapp_repository):<tag>

24

Tagging images

For clean checkouts:● <GIT_SHA>-<COMMIT_TIMESTAMP>

For dirty checkouts:● <GIT_SHA>-<COMMIT_TIMESTAMP>-DIRTY-<DIFF_SHA>

25

Current kubecfg architecture

26

Jsonnet Template “Kubernetes spec” with Bazel targets

Kubectl Kubernetes spec with images

Build docker image

Tag and push imagegit ref-log

Conclusion

● kubecfg makes deployment a two-step operation1. git checkout my-branch2. kubecfg apply -f the-template.jsonnet

● Speeds up development cycle● Helps with tracking down errors

27

Future of kubecfg

● We plan on open sourcing this library● Development in Bazel makes this easier

○ https://github.com/bazelbuild/rules_docker

28

Thank youWe're hiring! Talk to me or Priya if you're interested in joining our team.

29