Go @ Tokopedia Qasim Zaidi

About Tokopedia

• Launched in 2009

• 300K Active merchants

• 8 Million Products

• Indonesia’s biggest online marketplace

• Everyone pays shipping charges

• Shipping is a function of distance

• No discounting or cash-backs, neutral marketplace.

• Most payments are offline

nginxssl termination and

proxy pass

nginx-mod_perl

nginx-mod_perl

nginx-mod_perl

The Stack

postgres

mongo

redis

proxy_pass

What we already use go for

• Search & Discovery

• Image Uploads

• Analytics

60% of the time, it works every time.

nginxssl termination and

proxy pass

nginx-mod_perl

nginx-mod_perl

nginx-mod_perl

The Stack

postgres

mongo

redis

proxy_pass

mod_perl blocks nginx isolation is hard

cancellation is harder

Challenges of scaling with mod_perl

• Each request is synchronously processed in a worker

• If there is one bad request, it ends up queuing everything else.

• Cancellation is not easy - so even when the user moves away, the request continues to run.

In 2012, we bet on node.js

scaled well too - from 50K transactions in 2012 to over a million in 2015

node.js challenges

• Too easy to mess up (dynamic, no type checking)

• linting, code reviews

• Unbounded Concurrency is hard.

• async, promises

• Scale issues with http client for c10k

• even dns latencies can have big impact

Picking a new language in 2015.

the no QA philosophy

QA is not a different role.

Developers are responsible for testing their code.

Testing in Go

• Easy to write unit tests

func TestFoo(t *testing.T) {

...

}

• run as go test

• No separate frameworks

• Benchmarking is as easy as testing

the no Documentation philosophy

Code should be self explanatory.

Documentation outside of code is never updated.

Documentation with GoDoc• Comments in code - e.g.

preceding function definition

• It extracts the comments and presents them:

• $ godoc strings Joinfunc Join(a []string, sep string) stringJoin concatenates the elements of a to create a single string. The separator string sep is placed between elements in the resulting string.

// Join concatenates the elements of a to create a single string. // The separator string sep is placed between elements in the resulting string. func Join(a []string, sep string) string { //self explanatory blah blah blah //more of the same

}

godoc - examples and testing

func ExampleJoin() {

s := []string{"foo", "bar", "baz"}

fmt.Println(strings.Join(s, ", "))

// Output: foo, bar, baz

}

• Also integrated with the testing framework to provide testable example functions.

godoc - webview

Concurrency should be easy

And not yet forced.

Too much concurrency spoils the code.

Code should be hard to mess with

A compiler that catches bugs

and is not too slow

statically compiled binaries - no library mess

Institutional knowledge

Go brings google’s institutional knowledge to you, for free

- Groupcache- Expvars- Single-flights and Cancellations

Object Caching

• In node.js - we built in an object caching solution in house, that can easily make a function called cached without changing much code.

• var wrapper = cache.wrap(original);

• Can chose whether to cache in memory or redis.

• Can avoid thundering herds - single flighting

Enter group cache

• In memory

• distributed

• single flight

ELB

app groupcache

app app groupcachegroupcache

http http

expvars - server metrics over http• Export variables over http

• http://localhost:8123/debug/vars {"cmdline": ["/var/folders/bc/27hv15_d2zvcc3n3s9dxmfg00000gn/T/go-build421732654/command-line-arguments/_obj/exe/main","-l","debug","-s","i.sock","-c","realtest"], "counters": {"a": 10, "b": 10},"memstats": {"Alloc":1076016,"TotalAlloc":1801544,"Sys":5966072,"Lookups":209,"Mallocs":7986,"Frees":4528,"HeapAlloc":1076016,"HeapSys":2097152,"HeapIdle":327680,"HeapInuse":1769472,"HeapReleased":0,"HeapObjects":3458,"StackInuse":212992,"StackSys":917504,"MSpanInuse":21504,"MSpanSys":32768,"MCacheInuse":8800,"MCacheSys":16384,"BuckHashSys":1441160,"GCSys":1183744,"OtherSys":277360,"NextGC":1436032,"LastGC":1418102095002592201,"PauseTotalNs":2744531,"PauseNs":[480149,171430,603839,288381,494934,522995,182803,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"NumGC":7,"EnableGC":true,"DebugGC":false} }

Cancellations

• User moves away, nginx sends 499, but queries keep running.

• Not Abandoning work when user has moved away is often wasteful

• Go’s has advisory cancellations - cancel across go-routines

• facilitated by contexts

moving from ‘dynamic’ to ‘compiled’• Deployment changes

• from git clone to compile, build & deploy

• have a way of knowing binary version that is automatic BUILDHASH=$(shell cd src/$(SRCDIR) && git rev-parse --verify HEAD | cut -c 1-7)

go build —ldflags -X $BUILDHASH

appname —version

• config and code is separate

• make sure everything you may ever need to configure is in config/flags, can’t change binary appname —port appname —configtestappname —debug

• have good debugging (we have debug() statements that are turned on by —debug flag)

The go workflow

jenkins dpkg-buildpackagetriggers build

build dpkg using go get

ansible

srv1 srv1 srv1

elb

nginx binary, run via upstart

deploy dpkg restart

2

1

triggers deploy

3

No templates, no exceptions, weird and sometimes inconsistent.

Go is an opinionated language

“Instructions, registers, and assembler directives are always in UPPER CASE to remind you that assembly programming is a fraught endeavor.”

- Rob Pike

-Dick Gabriel, from the golang FAQ

Who would have guessed sophistication bought such noise?

Golang - missing a package manager?

• go get - but its not complete

• No dependency management, unlike npm, no easy hermetic builds

• gopkg.in is a solution - allows versioning

• if you have to change versions, edit every single import

Deployment Challenges

• No Fork() - forking with subroutines ain’t easy

• Facebook grace (https://github.com/facebookgo/grace)

• PID changes - no relationship

• problems with upstart / job monitor

• log rotation

Moving from a script to binary

• Deployment changes (from git clone to build, package and deploy)

• Need to know which version of code is running

• appname —version

• Need to debug easily

• appname —debug - enables debug() output

• Keep configuration flexible, because in case of issues, you can’t just edit and deploy

• appname —configtest

• appname —port (over ride config)

Go @ Tokopedia

• Go is a modern language, very suitable for web.

• Go is Performant, unless you are doing C.

• Go is mostly google still, and not 100% complete.

• Go is opinionated, like Plan9, and there will be a learning curve.

• Go is slower for prototyping, e.g. when compared to node.js

Questions?