4053 - whispering gophers

7/21/2019 4053 - Whispering Gophers

1/46

Whispering GophersNetwork programming in Go

Andrew GerrandFrancesc Campoy


2/46

Introduction

This code lab demonstrates network programming with Go.

The final goal is to build a "whispernet": a peer to peer mesh network for transmitting

text messages.


3/46

Design overview


4/46

Pre-requisites

You must have these tools installed and working:

Go 1.1 (see http://golang.org/doc/install (http://golang.org/doc/install))

Mercurial (see http://mercurial.selenic.com(http://mercurial.selenic.com))

This is not an introducion to the Go Programming Language;

some experience writing Go code is required.

To learn Go:

Take A Tour of Go(http://tour.golang.org)

Check out the Go documentation(http://golang.org/doc)
http://golang.org/dochttp://golang.org/dochttp://golang.org/dochttp://tour.golang.org/http://mercurial.selenic.com/http://golang.org/doc/install


5/46

Creating a workspace

To write Go code you need a workspace. Create one now if you have not already.

$ mkdir $HOME/gocode

$ export GOPATH=$HOME/gocode # or add this to ~/.profile

Build and install a helloexample program:

$ go get code.google.com/p/whispering-gophers/hello

This installs the hellobinary to $GOPATH/bin.

Check that this worked:

$ $GOPATH/bin/hello

It works!


6/46

The exercises

This code lab is divided into 8 exercises. Each exercise builds on the previous one.

The skeletondirectory in thewhispering-gophersrepository contains unfinished

programs that you should complete yourself.

There is also a solutiondirectory that contains the finished programsonly peek if

you really need to! :-)


7/46

Vital resources

Visit

code.google.com/p/whispering-gophers (http://code.google.com/p/whispering-gophers)

for code samples and support libraries.

These slides are available at:

whispering-gophers.appspot.com (http://whispering-gophers.appspot.com)
http://whispering-gophers.appspot.com/http://code.google.com/p/whispering-gophers


8/46

Part 1: reading and writing

Write a program that

reads lines from standard input (os.Stdin)

encodes each line as a JSON object, written to standard output (os.Stdout)

This line of input:

Hello, world

should produce this output:

{"Body":"Hello, world"}

This is our system's basic message format.


9/46

Readers and Writers

The iopackage provides fundamental I/O interfaces that are used throughout most Go

code.

The most ubiquitous are the Readerand Writertypes, which describe streams of data.

package io

type Writer interface {

Write(p []byte) (n int, err error)

}

type Reader interface {

Read(p []byte) (n int, err error)

}

Readerand Writerimplementations include files, sockets, (de)compressors, image

and JSON codecs, and many more.


10/46

Chaining Readers

package main

import (

"compress/gzip" "encoding/base64"

"io"

"os"

"strings"

)

func main() {

var r io.Reader r = strings.NewReader(data)

r = base64.NewDecoder(base64.StdEncoding, r)

r, _ = gzip.NewReader(r)

io.Copy(os.Stdout, r)

}

const data = `

H4sIAAAJbogA/1SOO5KDQAxE8zlFZ5tQXGCjjfYIjoURoPKgcY0E57f4VZlQXf2e+r8yOYbMZJhoZWRxz3wkCVjeReETS0VHz5fBCzpxxg/PbfrT/gacCjbjeiRNOChaVkA9RAdR8eVEw4vxa0Dcs3Fe2ZqowpeqG79L995l3VaMBUV/02OS+B6kMWikwG

51c8n5GnEPr11F2/QJAAD//z9IppsHAQAA

` Run


11/46

Buffered I/O

The bufiopackage implements buffered I/O. Itsbufio.Scannertype wraps an

io.Readerand provides a means to consume it by line (or using a specified "split

funcion").

const input = `A haiku is more

Than just a collection of

Well-formed syllables

`

func main() {

s := bufio.NewScanner(strings.NewReader(input)) for s.Scan() {

fmt.Println(s.Text())

}

if err := s.Err(); err != nil {

log.Fatal(err)

}

} Run


12/46

Encoding JSON objects

The encoding/jsonpackage converts JSON-encoded data to and from native Go data

structures.

type Site struct {

Title string

URL string

}

var sites = []Site{

{"The Go Programming Language", "http://golang.org"},

{"Google", "http://google.com"},

}

func main() {

enc := json.NewEncoder(os.Stdout)

for _, s := range sites {

err := enc.Encode(s)

if err != nil {

log.Fatal(err)

}

}

} Run


13/46

The Message type

Messages are sent as JSON objects like this:

{"Body":"This is a message!"}

Which corresponds to a Go data structure like this:

type Message struct {

Body string

}


14/46

Error checking

Many functions in Go return an errorvalue.

These values are your friends; they will tell you where you went wrong.

Ignore them at your peril!

Use log.Println (http://golang.org/pkg/log/#Println)to print log messages, and log.Fatal

(http://golang.org/pkg/log/#Fatal)to print a message and exit the program printing a stack trace.

func main() {

log.Println("Opening gzip stream...")

_, err := gzip.NewReader(strings.NewReader("not a gzip stream!")) if err != nil {

log.Fatal(err)

}

log.Println("OK!")

} Run
http://golang.org/pkg/log/#Fatalhttp://golang.org/pkg/log/#Fatalhttp://golang.org/pkg/log/#Fatalhttp://golang.org/pkg/log/#Println


15/46

Part 1: reading and writing (recap)

Write a program that

reads lines from standard input (os.Stdin)

encodes each line as a JSON object, written to standard output (os.Stdout)

This line of input:

Hello, world

should produce this output:

{"Body":"Hello, world"}

This is our system's basic message format.


16/46

Part 2: Send messages to a peer

Extend your program:

take an address string from the command line

make a TCP connection to the remote host

write the JSON-encoded messages to the connnection instead of standard output


17/46

Flag

The flagpackage provides a simple API for parsing command-line flags.

$ flag -message 'Hold on...' -delay 5m

package main

import (

"flag"

"fmt"

"time"

)

var (

message = flag.String("message", "Hello!", "what to say") delay = flag.Duration("delay", 2*time.Second, "how long to wait")

)

func main() {

flag.Parse()

fmt.Println(*message)

time.Sleep(*delay)

} Run


18/46

Making a network connection

The netpackage provides talk/code for network operations.

The net.Dial (http://golang.org/pkg/net/#Dial)function opens a nework connection and returns a

net.Conn (http://golang.org/pkg/net/#Conn), which implements io.Reader, io.Writer, and io.Closer

(or io.ReadWriteCloser).

(Usually you would use the net/httppackage to make an HTTP request; the

purpose of this example is to demonstrate the lower-level netpackage.)

func main() {

c, err := net.Dial("tcp", "www.google.com:80")

if err != nil {

log.Fatal(err) }

fmt.Fprintln(c, "GET /")

io.Copy(os.Stdout, c)

c.Close()

} Run
http://golang.org/pkg/net/#Connhttp://golang.org/pkg/net/#Dialhttp://golang.org/pkg/net/#Connhttp://golang.org/pkg/net/#Dial


19/46

Part 2: Send messages to a peer (recap)

Extend your program:

take an address string from the command line

make a TCP connection to the remote host

write the JSON-encoded messages to the connnection instead of standard output


20/46

Part 3: Serving network connections

Write a new program:

listen on a TCP port,

accept incoming connections and launch a goroutine to handle each one,

decode JSON messages from the incoming connections,

print each message Bodyto standard output.


21/46

Listen/Accept/Serve (1/2)

The net.Listen (http://golang.org/pkg/net/#Listen)function binds to a socket and returns a

net.Listener (http://golang.org/pkg/net/#Listener).

The net.Listener (http://golang.org/pkg/net/#Listener)provides an Acceptmethod that blocks until aclient connects to the socket, and then returns anet.Conn (http://golang.org/pkg/net/#Conn).

This server reads data from a connection and echoes it back:

func main() {

l, err := net.Listen("tcp", "localhost:4000")

if err != nil {

log.Fatal(err)

}

for {

c, err := l.Accept()

if err != nil {

log.Fatal(err)

}

fmt.Fprintln(c, "Welcome to the echo server!")

io.Copy(c, c)

}

} Run
http://golang.org/pkg/net/#Listenerhttp://golang.org/pkg/net/#Listenerhttp://golang.org/pkg/net/#Listenhttp://golang.org/pkg/net/#Connhttp://golang.org/pkg/net/#Listenerhttp://golang.org/pkg/net/#Listenerhttp://golang.org/pkg/net/#Listen


22/46

Goroutines

Goroutines are lightweight threads that are managed by the Go runtime. To run a

function in a new goroutine, just put "go"before the function call.

package main

import (

"fmt"

"time"

)

func main() {

go say("let's go!", 3*time.Second)

go say("ho!", 2*time.Second)

go say("hey!", 1*time.Second)

time.Sleep(4 * time.Second)

}

func say(text string, duration time.Duration) {

time.Sleep(duration)

fmt.Println(text)

} Run


23/46

Listen/Accept/Serve (2/2)

To handle requests concurrently, serve each connection in its own goroutine:

func main() {

l, err := net.Listen("tcp", "localhost:4000") if err != nil {

log.Fatal(err)

}

for {

c, err := l.Accept()

if err != nil {

log.Fatal(err)

} go serve(c)

}

}

func serve(c net.Conn) {

fmt.Fprintln(c, "Welcome to the echo server!")

io.Copy(c, c)

} Run


24/46

Decoding JSON

Decoding JSON from an io.Readeris just like writing them to an io.Writer, but in

reverse.

type Site struct {

Title string

URL string

}

const stream = `

{"Title": "The Go Programming Language", "URL": "http://golang.org"}

{"Title": "Google", "URL": "http://google.com"}

`

func main() {

dec := json.NewDecoder(strings.NewReader(stream))

for {

var s Site

if err := dec.Decode(&s); err != nil {

log.Fatal(err)

}

fmt.Println(s.Title, s.URL)

}

} Run


25/46

Part 3: Serving network connections (recap)

Write a new program:

listen on a TCP port,

accept incoming connections and launch a goroutine to handle each one,

decode JSON messages from the incoming connections,

print each message Bodyto standard output.


26/46

Part 4: Listening and dialing

Combine parts 2 to part 3 (Listen andDial)

Test by connecting two instances


27/46

Part 5: distributing the listen address

Add an Addrfield to the Messagestruct,

When sending messages, put the listen address in the Addrfield.


28/46

Add an Addr field to Message

Add an Addrfield to the Messagetype:


Addr string Body string

}

Now, when constructing Messagevalues, populate the Addrfield with the listen

address:

{"Addr":"192.168.1.200:23542,"Body":"This is a message!"}


29/46

Obtaining the listener address (1/2)

The net.Listenerinterface provides an Addrmethod that returns a net.Addr.

When listening on all interfaces, as specified by the empty hostname in the string

":4000"above, the net.Addrwon't be that of our public IP address.

To complete our program, we need to find that IP.

import (

"log" "net"

)

func main() {

l, err := net.Listen("tcp", ":4000")

if err != nil {

log.Fatal(err)

} log.Println("Listening on", l.Addr())

} Run


30/46

Obtaining the listener address (2/2)

The "code.google.com/p/whispering-gophers/util" (http://godoc.org/code.google.com/p/whispering-gophers/util)

package provides a Listenfunction that binds to a random port on the first available

public interface.

import (

"log"

"code.google.com/p/whispering-gophers/util"

)

func main() { l, err := util.Listen()

if err != nil {

log.Fatal(err)

}

log.Println("Listening on", l.Addr())

} Run
http://godoc.org/code.google.com/p/whispering-gophers/utilhttp://godoc.org/code.google.com/p/whispering-gophers/utilhttp://godoc.org/code.google.com/p/whispering-gophers/util


31/46

Part 5: sending the listen address (recap)

Add an Addrfield to the Messagestruct,

Import"code.google.com/p/whispering-gophers/util" ,

Use util.Listeninstead of net.Listen,

Store the listen address string in a global variable named self,

When sending messages, put the listen address in the Addrfield.


32/46

Part 6: separate reading and writing

Separate reading from standard input and dialing into separate functions that run

in separate goroutines.


33/46

Channels

Goroutines communicate via channels. A channel is a typed conduit that may be

synchronous (unbuffered) or asynchronous (buffered).

package main

import "fmt"

func main() {

ch := make(chan int)

go fibs(ch)

for i := 0; i < 20; i++ {

fmt.Println(


34/46

Part 6: separate reading and writing (recap)

Separate reading from standard input and dialing into separate functions that run

in separate goroutines.

Pass messages from one function to another using a channel.


35/46

Part 7: connect to multiple peers

As we see new peer addresses, make connections those peers.

For each line read from standard input, broadcast that message to all connected

peers.


36/46

Sharing state

Mutexes are a simple means to protect shared state from concurrent access.

var (

count int mu sync.Mutex // protects count

)

for i := 0; i < 10; i++ {

go func() {

for {

mu.Lock()

count++

mu.Unlock() time.Sleep(5 * time.Millisecond)

}

}()

}

time.Sleep(time.Second)

mu.Lock()

fmt.Println(count)

mu.Unlock() Run


37/46

Tracking peer connections (1/2)

Each peer connection runs in its own goroutine.

Each goroutine has its own chan Message. It reads messages from the channel, and

writes them to the connection as JSON objects.

A central peer registry associates each peer address with its correspondingchan

Message.

type Peers struct {

m map[string]chan


38/46

Tracking peer connections (2/2)

Before making a peer connection, ask the peer registry for a channel for this address:

// Add creates and returns a new channel for the given address.

// If an address already exists in the registry, it returns nil.func (p *Peers) Add(addr string)


39/46

Sending without blocking

If a peer connection stalls or dies, we don't want to hold up the rest of our program. To

avoid this problem we should do a non-blocking send to each peer when broadcasting

messages. This means some messages may be dropped, but in our mesh network thisis okay.

ch := make(chan int)

select {

case ch


40/46

Part 7: connect to multiple peers (recap)

As we see new peer addresses, make connections to those peers.

For each line read from standard input, broadcast that message to all connected

peers.


41/46

Part 8: re-broadcast messages

Add IDfield to Message,

When creating a Message, populate the IDfield with a random string,

Track the IDof each received message; drop duplicates,

For each received Message, broadcast it to all connected peers.


42/46

Add ID to Message

Add an IDfield to the Messagetype:


ID string Addr string

Body string

}

Now, when constructing Messagevalues, populate the IDfield with a random string:

{"ID":"a09d2abb1ad536ada",Addr":"192.168.1.200:23542,"Body":"This is a message!"}


43/46

Generating random strings

Use the util.RandomID (http://godoc.org/code.google.com/p/whispering-gophers/util/#RandomID)function to generate a

random string.

package main

import (

"fmt"

"code.google.com/p/whispering-gophers/util"

)

func main() {

id := util.RandomID()

fmt.Println(id)

} Run
http://godoc.org/code.google.com/p/whispering-gophers/util/#RandomID


44/46

Tracking message IDs

To track messages IDs, use a map[string]bool. This works as a kind of set.

seen := make(map[string]bool)

To check if an id is in the map:

if seen[id] {

fmt.Println(id, "is in the map")

}

To put an id in the map:

seen[id] = true

You should implement a function named Seen`make sure it is thread safe!

// Seen returns true if the specified id has been seen before.

// If not, it returns false and marks the given id as "seen".func Seen(id string) bool


45/46

Part 8: re-broadcast messages (recap)

Add IDfield to Message

When a Message, populate the IDfield with a random string

Track the IDof each received message; drop duplicates

For each received Message, broadcast it to all connected peers


46/46

Thank you

Andrew Gerrand

Francesc Campoy

4053 - whispering gophers

Documents