A Go Developer's Notebook
A Go Developer's Notebook
Buy on Leanpub
A Go Developer's Notebook

Table of Contents

Introducing Go

In this section we’re going to explore Go through a series of increasingly complex programs which we can interact with either through the command line or across a network.

Getting Started

If you’re of an impatient nature (my teenage self certainly was) temper that enthusiasm for a second and read Google’s install docs to get your Go install up-and-running. You might also like to install LiteIDE which is an excellent cross-platform development environment for Go which also happens to be written in Go.

This book is laid out as a series of parts focused on different topics I’m interested in and the order you read it in is up to you with one clear exception: Hello World is designed as a quick tutorial introduction to Go so if you don’t know the language that’s the place to start. Even if you do know Go you’ll probably find other stuff in Hello World which will be useful as it covers network communications and cryptography.

Now Go have fun messing with my code.

So what is Go?

Well obviously it’s a programming language, which means an artificial language intended for humans to read and write but which can be usefully translated into the 1s and 0s understood by digital computers.

But is it a good programming language?

Opinion on the internet has been bitterly divided on this question (which is hardly unusual) since Go was released in November 2009 and the language has many detractors who either see it as deficient in some crucial feature without which efficient programs can’t possibly be productively developed, or else as a wholehearted nostalgia trip back into the 1980s by a team who’ve failed to move with the times.

So let’s look at the reality and see if these criticisms are justified.

Open source

Go’s developed as an open source project so anyone with an interest can read through its source code, make modifications for their own purposes, or get involved in its future development. However it’s also a carefully curated project and unlike many popular languages there’s a tightly defined language specification which is intended to be read and understood by anyone working with Go.

These are pragmatic choices. Not only do many eyes make light work of code quality, a clear and readable language specification makes it very clear what Go is and isn’t trying to achieve.

Efficient compilation

Work started on the language at Google in 2007 as a way to address three key problems in systems programming.

First there’s efficient compilation, the process of taking the source code for a program and turning it into an executable form with a minimum of fuss. As increasingly complex programs are developed in traditional systems languages like C and C++ they become dependent on equally complex build tools and compilation times can stretch into minutes or hours.

This isn’t a particularly new phenomenon. The systems I was working on in the 90s often had compile times in the order of 10 to 30 minutes so builds were a great excuse to make a cup of tea whilst waiting to see if things compiled successfully. Still there’s only so many cups of tea you can make in a day and maintain any semblance of productivity.

Sadly one of the main reasons for the popularity of interpreted languages like Ruby and Python (both of which have other much more compelling reasons to explore) is the interactive nature of developing with them. This basically allows devs to embrace the same smugness the Lisp crowd exude without the intellectual snobbery or taste the pioneering spirit of Forth without the serious WTF? factor when rereading old code.

In interpreted languages small chunks of code can be developed independently of each other and tested immediately. This fast feedback cycle makes it much easier to explore a problem domain and figure out how to work with it effectively (or if we’re lucky elegantly). However interpreted languages have a reputation for being memory hungry and slow.

Efficient execution

This leads us to the second problem Go addresses: efficient execution. Systems programming is all about programs which other programs are going to rely on for key services so it’s highly desirable that languages addressing these problems produce programs which run in a timely fashion and with predictable memory overhead.

Compilation provides a huge boost to runtime performance compared to interpretation and the standard Go compiler features a range of optimisations which make it roughly competitive with C++ or Java. And because Go was developed with multi-core processors in mind it uses their resources effectively.

One area which raised concerns when Go was first released is its use of garbage collection rather than traditional programmer-controlled memory allocation. Garbage collection used to be a significant cost with 10ms set aside from every 50ms of program execution but in recent years there have been huge improvements by switching to a concurrent design and tailoring this to other features of Go’s design.

Ease of programming

The majority of systems languages were designed at a time when memory was limited and processors could generally only execute one instruction at a time (and that glacially slowly by modern standards). As a result these languages provide programmers with very primitive abstractions which no longer even vaguely resemble what’s really going on in most general purpose computers, and to make matters worse they often entail a lot of careful book-keeping just to track memory allocations and prevent leaks which will otherwise cause systems to become unstable.

Go was developed after these changes took place by a team with a long history of working on systems software and embraces programmer-friendly conveniences such as garbage collection and CSP concurrency which ease much of this burden. Its design also includes other features which aid ease of programming such as first-class functions, slices, hash maps and structural typing which can greatly simplify code analysis and reuse.

We’re going to have a lot of fun with all these features throughout this book.

And where did it come from?

Originally Go was developed in-house at Google, a company which thanks to its position in web search and online advertising is heavily tech driven. Google’s systems operate at ROFLscale - a scale which most of us can only dream of messing with - and its codebases measure multiple BLOC (that’s billion lines of code). The aspirations at the heart of Go’s design are essentially attempts to reign in the complexity of numerous codebases scattered throughout Google’s data-centres and written in C++, Python, and Java.

The initial development team comprised three industry veterans with a record of both innovation and successful commercial implementation: Ken Thompson, Rob Pike, and Robert Griesemer.

Ken Thompson is a legendary hacker, the original creator of UNIX whose B programming language was a strong influence on the design of hacker favourite C. Rob Pike meanwhile was deeply involved with the development of the UNIX, Plan 9 (along with Ken) and Inferno operating systems as well as the Dis virtual machine and a number of programming languages focused on concurrency. Robert Greisemer cut his teeth working on Java’s HotSpot VM and Google’s V8 JavaScript runtime, so between the three of them there’s a wealth of experience in systems software design and performance engineering.

At first glance Go is similar to another of Rob’s languages Limbo which he created for the Inferno operating system and it features the CSP approach to concurrency he and Ken had been playing with since their UNIX days. CSP (or Communicating Sequential Processes) was formally defined by British computer scientist Tony Hoare in the late 70s and was key to the technical success of the multi-processor InMos Transputer machines of the 1980s (though it’s programming language Occam was pretty nasty to work with).

However Go is a much smaller language than Limbo thanks to a key decision made at the start of the project: no suggested feature would be included in Go unless all three of the core team agreed it should be included. This makes for a highly conservative language design focused on must have features to achieve its stated design goals rather than nice to have features which might be at odds with them. This is quite a departure from the usual feature creep and special pleading seen in other languages.

Into the wild

When Go was first announced to the world on November 10th 2009 it was already a robust language but with a few rough edges like the use of make files for project builds, a weekly source release cycle which often saw package library APIs changing, and some fairly fiddly configuration via environment variables.

However key features like the language specification, structural typing, garbage collection, and goroutines were already fixed and a community of passionate users started to grow up around the language. Some of these came from the bifurcating Python scene in search of stability and static typing, others like me had need for a systems language but wanted to keep many of the free-wheeling conveniences we enjoyed in higher-level languages. Then there were the C/C++ and Java devs initially attracted by the reputations of Go’s designers who then stuck around for the fun they could have.

Looking back I think all three groups were pulling not only in different directions to each other, but also in different directions to the core team and it’s a testament to the flexibility of Go that a dedicated Rubyist like me can be as comfortable working with it as the most static-typing obsessed escapee from Python or generics-missing Java head.

The early buzz around Go wasn’t nearly as loud as TIOBE ratings suggested and the language would probably have remained a curiosity even with Google’s backing if it hadn’t been for the widespread adoption of Docker by the DevOps community.

Docker popularised containers, a standardised way of installing and running one or more lightweight virtual machines on a server. These virtual machines aren’t to be confused with the kind used by languages like Java or Erlang (and which form the meat of the software machines section of this book) but are better understood as an outgrowth of UNIX chroot jails based on the resource isolation features present in the Linux kernel. This allows one server to run multiple independent environments each with well-defined privileges and managed resources without requiring the overhead associated with more traditional virtualisation hypervisors like Xen or VMWare.

Google themselves are no strangers to containerisation and in 2015 released Kubernetes, an orchestration system for deploying and managing containers across large-scale clusters inspired by a previous in-house tool named Borg. Kubernetes is written in Go and in combination with Docker has helped simplify the once black art of microservice design. Not only are these powerful tools, their use of Go makes them great advertisements for the language.

Where next?

At the time of writing Go has been in the public eye for nine years. The release of version 1 in March 2012 came with the promise that there’d be no breaking language changes until a version 2 release and since version 1.1 development has followed a tight release schedule focused primarily on improving runtime performance and the standard library. Good software engineering practices aren’t particularly glamorous but the end results are impressive as the garbage collector and goroutine scheduler demonstrate.

It’s still far from clear when version 2 of Go will materialise or the extent of any changes to the language which will accompany it. One much-anticipated addition is some kind of support for generic types, the absence of which has been controversial since Go was first released. Generics are one of a number of different ways to enable compile-time metaprogramming and are popular in a number of languages such as C++ and Java, however they generally make the compilation process more complex and trade compile-time cost for runtime efficiency.

Go already allows metaprogramming at runtime via its reflection and unsafe APIs with minimal impact on compilation speeds but there is a runtime performance hit for any significant use of reflection which starts to make higher-level interpreted languages look competitive. And thanks to structural typing via interfaces it’s possible to write reasonably generic code based upon capability, though yet again there are runtime costs associated with this approach.

Personally I’d like Go version 2 to introduce runtime optimisations for both reflection and structural typing - maybe with some form of JIT compilation. This might well be possible for type switches by using call-site caching and making changes to the reflection API inspired by virtual machine designs.

There are also a number of active projects focused on richer compile-time metaprogramming using everything from code templating (the C++ approach) to Lisp-inspired macros. These all involve code generation and subsequent compilation and it would be interesting to see direct compile-time metaprogramming integrated into the language specification.

Then there’s enumeration via range statements which could be opened up to a wider selection of user-defined types. Conceptually this could be simple enough: add an interface to the standard library which yields successive elements and allow the compiler to spot this is implemented for a given type. But it’s hard to see how this approach wouldn’t get pretty ugly pretty fast.

A possible first step might be to add nodes and lists to the core types defined in the language specification as a analogous abstraction arrays & slices so that at least singly-linked and doubly-linked lists could be used in range statements.

Another area where Go could be improved is for mobile development. There’s quite a bit of work that’s gone on in this area over the past few years but Go still isn’t a first-class language on Android or iOS so it would be good to see something official in this direction.

Things are much further along with desktop applications such as LiteIDE which uses Qt5 bindings for its cross-platform GUI whilst TinyGo is bringing a decent subset of the language to minimalist embedded devices like the BBC micro:bit. Likewise GopherJS has pioneered Go in the web browser whilst Go version 1.11 now includes experimental support for WebAssembly. There are even bindings for the Vulkan graphics API.

So hopefully the version 2 release will make Go a credible language to use all the way from deep embedded systems right the way through to web browser front-ends. Who knows, perhaps we’ll even see some graphically intensive games being developed in Go.

Hello World

It’s a tradition in programming books to start with a canonical “Hello World” example and whilst I’ve never felt the usual presentation is particularly enlightening, I know we can spice things up a little to provide useful insights into how we write Go programs.

Let’s begin with the simplest Go program that will output text to the console.

Example 1.1.1
1 package main
2 
3 func main() {
4   println("hello world")
5 }

The first thing to note is that every Go source file belongs to a package, with the main package defining an executable program whilst all other packages represent libraries.

1 package main

For the main package to be executable it needs to include a main() function, which will be called following program initialisation.

3 func main() {

Notice that unlike C/C++ the main() function neither takes parameters nor has a return value. Whenever a program should interact with command-line parameters or return a value on termination these tasks are handled using functions in the standard package library. We’ll examine command-line parameters when developing Echo in the next chapter.

Finally let’s look at our payload.

4 println("hello world")

The println() function is one of a small set of builtin generic functions defined in the language specification and which in this case is usually used to assist debugging, whilst “hello world” is a value comprising an immutable string of characters in utf-8 format.

We can now run our program from the command-line (Terminal on MacOS X or Command Prompt on Windows) with the command

$ go run 01.go
hello world

Packages

Now we’re going to apply a technique which I plan to use throughout this book by taking this simple task and developing increasingly complex ways of expressing it in Go. This runs counter to how experienced programmers usually develop code but I feel this makes for a very effective way to introduce features of Go in rapid succession and have used it with some success during presentations and workshops.

There are a number of ways we can artificially complicate our hello world example and by the time we’ve finished I hope to have demonstrated all the features you can expect to see in the global scope of a Go package. Our first change is to remove the builtin println() function and replace it with something intended for production code.

Example 1.1.2
1 package main
2 import "fmt"
3 
4 func main() {
5   fmt.Println("hello world")
6 }

The structure of our program remains essentially the same, but we’ve introduced two new features.

2 import "fmt"

The import statement is a reference to the fmt package, one of many packages defined in Go’s standard runtime library. A package is a library which provides a group of related functions and data types we can use in our programs. In this case fmt provides functions and types associated with formatting text for printing and displaying it on a console or in the command shell.

5 fmt.Println("hello world")

One of the functions provided by fmt is Println() which takes one or more parameters and prints them to the console with a carriage return appended. Go assumes that any identifier starting with a capital letter is part of the public interface of a package whilst identifiers starting with any other letter or symbol are private to the package.

In production code we might choose to simplify matters a little by importing the fmt namespace into the namespace of the current source file, which requires we change our import statement.

2 import . "fmt"

And this consequently allows the explicit package reference to be removed from the Println() function call.

5 Println("hello world")

In this case we notice little gain however in later examples we’ll use this feature extensively to keep our code legible.

Example 1.1.3
1 package main
2 import . "fmt"
3 
4 func main() {
5   Println("hello world")
6 }

One aspect of imports that we’ve not yet looked at is Go’s builtin support for code hosted on a variety of popular social code-sharing sites such as GitHub and Google Code. Don’t worry, we’ll get to this in later chapters.

Constants

A significant proportion of Go codebases feature identifiers whose values will not change during the runtime execution of a program and our hello world example is no different, so we’re going to factor these out.

Example 1.1.4
1 package main
2 import . "fmt"
3 
4 const Hello = "hello"
5 const world = "world"
6 
7 func main() {
8   Println(Hello, world)
9 }

Here we’ve introduced two constants: Hello and world. Each identifier is assigned its value during compilation, and that value cannot be changed at runtime. As the identifier Hello starts with a capital letter the associated constant is visible to other packages - though this isn’t relevant in the context of a main package - whilst the identifier world starts with a lowercase letter and is only accessible within the main package.

We don’t need to specify the type of these constants as the Go compiler identifies them both as strings.

Another neat trick in Go’s armoury is multiple assignment so let’s see how this looks.

Example 1.1.5
1 package main
2 import . "fmt"
3 
4 const Hello, world = "hello", "world"
5 
6 func main() {
7   Println(Hello, world)
8 }

This is compact, but I personally find it too cluttered and prefer the more general form.

Example 1.1.6
 1 package main
 2 import . "fmt"
 3 
 4 const (
 5   Hello = "hello"
 6   world =  "world"
 7 )
 8 
 9 func main() {
10   Println(Hello, world)
11 }

Because the Println() function is variadic (i.e. can take a varible number of parameters) we can pass it both constants and it will print them on the same line, separate by whitespace. fmt also provides the Printf() function which gives precise control over how its parameters are displayed using a format specifier which will be familiar to seasoned C/C++ programmers.

10 Printf("%v %v\n", Hello, world)

fmt defines a number of % replacement terms which can be used to determine how a particular parameter will be displayed. Of these %v is the most generally used as it allows the formatting to be specified by the type of the parameter. We’ll discuss this in depth when we look at user-defined types, but in this case it will simply replace a %v with the corresponding string.

When parsing strings the Go compiler recognises a number of escape sequences which are available to mark tabs, new lines and specific unicode characters. In this case we use \n to mark a new line.

Example 1.1.7
 1 package main
 2 import . "fmt"
 3 
 4 const (
 5   Hello = "hello"
 6   world =  "world"
 7 )
 8 
 9 func main() {
10   Printf("%v %v\n", Hello, world)
11 }

Variables

Constants are useful for referring to values which shouldn’t change at runtime, however most of the time when we’re referencing values in an imperative language like Go we need the freedom to change these values. We associate values which will change with variables. What follows is a simple variation of our Hello World program which allows the value of world to be changed at runtime by creating a new value and assigning it to the world variable.

Example 1.1.8
 1 package main
 2 import . "fmt"
 3 
 4 const Hello = "hello"
 5 var world = "world"
 6 
 7 func main() {
 8   world += "!"
 9   Println(Hello, world)
10 }

There are two important changes here. Firstly we’ve introduced syntax for declaring a variable and assigning a value to it. Once more Go’s ability to infer type allows us assign a string value to the variable world without explicitly specifying the type.

5 var world = "world"

However if we wish to be more explicit we can be.

5 var world string = "world"

Having defined world as a variable in the global scope we can modify its value in main(), and in this case we choose to append an exclamation mark. Strings in Go are immutable values so following the assignment world will reference a new value.

8 world += "!"

To add some extra interest I’ve chosen to use an augmented assignment operator. These are a syntactic convenience popular in many languages which allow the value contained in a variable to be modified and the resulting value then assigned to the same variable.

I don’t intend to expend much effort discussing scope in Go. The point of this book is to experiment and learn by playing with code, referring to the comprehensive language specification available from Google when you need to know the technicalities of a given point. However to illustrate the difference between global and local scope we’ll modify this program further.

Example 1.1.9
 1 package main
 2 import . "fmt"
 3 
 4 const Hello = "hello"
 5 var world = "world"
 6 
 7 func main() {
 8   world := world + "!"
 9   Println(Hello, world)
10 }

Here we’ve introduced a new local variable world within main() which takes its value from an operation concatenating the value of the global world variable with an exclamation mark. Within main() any subsequent reference to world will always access the local version of the variable without affecting the global world variable. The is known as shadowing.

The := operator marks an assignment declaration in which the type of the expression is inferred from the type of the value being assigned. If we chose to declare the local variable separately from the assignment we’d have to give it a different name to avoid a compilation error.

Example 1.1.10
 1 package main
 2 import . "fmt"
 3 
 4 const Hello = "hello"
 5 var world = "world"
 6 
 7 func main() {
 8   var w string
 9   w = world + "!"
10   Println(Hello, w)
11 }

Another thing to note in this example is that when w is declared it’s also initialised to the zero value, which in the case of string happens to be ”“. This is a string containing no characters.

In fact all variables in Go are initialised to the zero value for their type when they’re declared and this eliminates an entire category of initialisation bugs which could otherwise be difficult to identify.

Functions

Having looked at how to reference values in Go and how to use the Println() function to display them, it’s only natural to wonder how we can implement our own functions. Obviously we’ve already implemented main() which hints at what’s involved, but main() is something of a special case as it exist to allow a Go program to execute and it neither requires any parameters nor produces any values to be used elsewhere in the program.

Example 1.1.11
 1 package main
 2 import . "fmt"
 3 
 4 const Hello = "hello"
 5 
 6 func main() {
 7   Println(Hello, world())
 8 }
 9 
10 func world() string {
11   return "world"
12 }

In this example we’ve introduced world(), a function which to the outside world has the same operational purpose as the variable of the same name that we used in the previous section.

The empty brackets () indicate that there are no parameters passed into the function when it’s called, whilst string tells us that a single value is returned and it’s of type string. Anywhere that a valid Go program would expect a string value we can instead place a call to world() and the value returned will satisfy the compiler. The use of return is required by the language specification whenever a function specifies return values, and in this case it tells the compiler that the value of world() is the string “world”.

Go is unusual in that its syntax allows a function to return more than one value and as such each function takes two sets of (), the first for parameters and the second for results. We could therefore write our function in long form as

10 func world() (string) {
11   return "world"
12 }

In this next example we use a somewhat richer function signature, passing the parameter name which is a string value into the function message(), and assigning the function’s return value to message which is a variable declared and available throughout the function.

Example 1.1.12
 1 package main
 2 import "fmt"
 3 
 4 func main() {
 5   fmt.Println(message("world"))
 6 }
 7 
 8 func message(name string) (message string) {
 9   message = fmt.Sprintf("hello %v", name)
10   return message
11 }

As with world() the message() function can be used anywhere that the Go compiler expects to find a string value. However where world() simply returned a predetermined value, message() performs a calculation using the Sprintf() function and returns its result.

Sprintf() is similar to Printf() which we met when discussing constants, only rather than create a string according to a format and displaying it in the terminal it instead returns this string as a value which we can assign to a variable or use as a parameter in another function call such as Println().

Because we’ve explicitly named the return value we don’t need to reference it in the return statement as each of the named return values is implied.

Example 1.1.13
 1 package main
 2 import . "fmt"
 3 
 4 func main() {
 5   Println(message("world"))
 6 }
 7 
 8 func message(name string) (message string) {
 9   message = Sprintf("hello %v", name)
10   return
11 }

If we compare the main() and message() functions, we notice that main() doesn’t have a return statement. Likewise if we define our own functions without return values we can omit the return statement though later we’ll meet examples where we’d still use a return statement to prematurely exit a function.

Example 1.1.14
 1 package main
 2 import . "fmt"
 3 
 4 func main() {
 5   greet("world")
 6 }
 7 
 8 func greet(name string) {
 9   Println("hello", name)
10 }

In the next example we’ll see what a function which uses multiple return values looks like.

Example 1.1.15
 1 package main
 2 import . "fmt"
 3 
 4 func main() {
 5   Println(message())
 6 }
 7 
 8 func message() (string, string) {
 9   return "hello", "world"
10 }

Because message() returns two values we can use it in any context where at least two parameters can be consumed. Println() happens to be a variadic function, which we’ll explain in a moment, and takes zero or more parameters so it happily consumes both of the values message() returns.

For our final example we’re going to implement our own variadic function.

Example 1.1.16
 1 package main
 2 import . "fmt"
 3 
 4 func main() {
 5   print("Hello", "world")
 6 }
 7 
 8 func print(v ...interface{}) {
 9   Println(v...)
10 }

We have three interesting things going on here which need explaining. Firstly I’ve introduced a new type, interface{}, which acts as a proxy for any other type in a Go program. We’ll discuss the details of this shortly but for now it’s enough to know that anywhere an interface{} is accepted we can provide a string.

In the function signature we use v …interface{} to declare a parameter v which takes any number of values. These are received by print() as a sequence of values and the subsequent call to Println(v…) uses this same sequence as this is the sequence expected by Println().

So why did we use …interface{} in defining our parameters instead of the more obvious …string? The Println() function is itself defined as Println(…interface{}) so to provide a sequence of values en masse we likewise need to use …interface{} in the type signature of our function. Otherwise we’d have to create a []interface{} (a slice of interface{} values, a concept we’ll cover in detail in a later chanpter) and copy each individual element into it before passing it into Println().

Encapsulation

In this chapter we’ll for the most part be using Go’s primitive types and types defined in various standard packages without any comment on their structure, however a key aspect of modern programming languages is the encapsulation of related data into structured types and Go supports this via the struct type. A struct describes an area of allocated memory which is subdivided into slots for holding named values, where each named value has its own type. A typical example of a struct in action would be

Example 1.1.17
 1 package main
 2 
 3 import "fmt"
 4 
 5 type Message struct {
 6   X string
 7   y *string
 8 }
 9 
10 func (v Message) Print() {
11   if v.y != nil {
12     fmt.Println(v.X, *v.y)
13   } else {
14     fmt.Println(v.X)
15   }
16 }
17 
18 func (v *Message) Store(x, y string) {
19   v.X = x
20   v.y = &y
21 }
22 
23 func main() {
24   m := &Message{}
25   m.Print()
26   m.Store("Hello", "world")
27   m.Print()
28 }
$ go run 17.go

Hello world

Here we’ve defined a struct Message which contains two values: X and y. Go uses a very simple rule for deciding if an identifier is visible outside of the package in which it’s defined which applies to both package-level constants and variables, and type names, methods and fields. If the identifier starts with a capital letter it’s visible outside the package otherwise its private to the package.

The Go language spec guarantees that all variables will be initialised to the zero value for their type. For a struct type this means that every field will be initialised to an appropriate zero value. Therefore when we declare a value of type Message the Go runtime will initialise all of its elements to their zero value (in this case a zero-length string and a nil pointer respectively), and likewise if we create a Message value using a literal

24   m := &Message{}

Having declared a struct type we can declare any number of method functions which will operate on this type. In this case we’ve introduced Print() which is called on a Message value to display it in the terminal, and Store() which is called on a pointer to a Message value to change its contents. The reason Store() applies to a pointer is that we want to be able to change the contents of the Message and have these changes persist. If we define the method to work directly on the value these changes won’t be propagated outside the method’s scope. To test this for yourself, make the following change to the program

Example 1.1.18
18 func (v Message) Store(x, y string) {

If you’re familiar with functional programming then the ability to use values immutably this way will doubtless spark all kinds of interesting ideas.

There’s another struct trick I want to show off before we move on and that’s type embedding using an anonymous field. Go’s design has upset quite a few people with an inheritance-based view of object orientation because it lacks inheritance, however thanks to type embedding we’re able to compose types which act as proxies to the methods provided by anonymous fields. As with most things, an example will make this much clearer

Example 1.1.19 Type Embedding
 1 package main
 2 
 3 import "fmt"
 4 
 5 type HelloWorld struct {}
 6 
 7 func (h HelloWorld) String() string {
 8   return "Hello world"
 9 }
10 
11 type Message struct {
12   HelloWorld
13 }
14 
15 func main() {
16   m := &Message{}
17   fmt.Println(m.HelloWorld.String())
18   fmt.Println(m.String())
19   fmt.Println(m)
20 }
$ go run 19.go
Hello world
Hello world
Hello world

Here we’re declaring a type HelloWorld which in this case is just an empty struct, but which in reality could be any declared type. HelloWorld defines a String() method which can be called on any HelloWorld value. We then declare a type Message which embeds the HelloWorld type by defining an anonymous field of the HelloWorld type. Wherever we encounter a value of type Message and wish to call String() on its embedded HelloWorld value we can do so by calling String() directly on the value, calling String() on the Message value, or in this case by allowing fmt.Println() to match it with the fmt.Stringer interface.

Any declared type can be embedded, so in our next example we’re going to base HelloWorld on the primitive bool boolean type to prove the point

Example 1.1.20 Type Embedding
 1 package main
 2 import "fmt"
 3 
 4 type HelloWorld bool
 5 
 6 func (h HelloWorld) String() (r string) {
 7   if h {
 8     r = "Hello world"
 9   }
10   return
11 }
12 
13 type Message struct {
14   HelloWorld
15 }
16 
17 func main() {
18   m := &Message{ HelloWorld: true }
19   fmt.Println(m)
20   m.HelloWorld = false
21   fmt.Println(m)
22   m.HelloWorld = true
23   fmt.Println(m)
24 }

In our final example we’ve declared the Hello type and embedded it in Message, then we’ve implemented a new String() method which allows a Message value more control over how it’s printed

Example 1.1.21 Type Embedding
 1 package main
 2 import "fmt"
 3 
 4 type Hello struct {}
 5 
 6 func (h Hello) String() string {
 7   return "Hello"
 8 }
 9 
10 type Message struct {
11   *Hello
12   World string
13 }
14 
15 func (v Message) String() (r string) {
16   if v.Hello == nil {
17     r = v.World
18   } else {
19     r = fmt.Sprintf("%v %v", v.Hello, v.World)
20   }
21   return
22 }
23 
24 func main() {
25   m := &Message{}
26   fmt.Println(m)
27   m.Hello = new(Hello)
28   fmt.Println(m)
29   m.World = "world"
30   fmt.Println(m)
31 }
$ go run 21.go

Hello 
Hello world

Generalisation

Encapsulation is of huge benefit when writing complex programs and it also enables one of the more powerful features of Go’s type system, the interface. An interface is similar to a struct in that it combines one or more elements but rather than defining a type in terms of the data items it contains, an interface defines it in terms of a set of method signatures which it must implement.

Once declared an interface can be used just like any other declared type, allowing functions and variables to operate with unknown types based solely on their required behaviour. Go’s type inference system will then recognise compliant values as instances of the interface, allowing us to write generalised code with little fuss.

In the next example we’re going to introduce a simple interface (by far the most common kind) which matches any type with a func String() string method signature.

Example 1.1.22
 1 package main
 2 import "fmt"
 3 
 4 type Stringer interface {
 5   String() string
 6 }
 7 
 8 type Hello struct {}
 9 
10 func (h Hello) String() string {
11   return "Hello"
12 }
13 
14 type World struct {}
15 
16 func (w *World) String() string {
17   return "world"
18 }
19 
20 type Message struct {
21   X Stringer
22   Y Stringer
23 }
24 
25 func (v Message) String() (r string) {
26   switch {
27   case v.X == nil && v.Y == nil:
28   case v.X == nil:
29     r = v.Y.String()
30   case v.Y == nil:
31     r = v.X.String()
32   default:
33     r = fmt.Sprintf("%v %v", v.X, v.Y)
34   }
35   return
36 }
37 
38 func main() {
39   m := &Message{}
40   fmt.Println(m)
41   m.X = new(Hello)
42   fmt.Println(m)
43   m.Y = new(World)
44   fmt.Println(m)
45   m.Y = m.X
46   fmt.Println(m)
47   m = &Message{ X: new(World), Y: new(Hello) }
48   fmt.Println(m)
49   m.X, m.Y = m.Y, m.X
50   fmt.Println(m)
51 }
$ go run 22.go

Hello
Hello world
Hello Hello
world Hello
Hello world

This interface is copied directly from fmt.Stringer, so we can simplify our code a little by using that interface instead

Example 1.1.23
17 type Message struct {
18   X fmt.Stringer
19   Y fmt.Stringer
20 }

As Go is strongly typed interface values contain both a pointer to the value contained in the interface, and the concrete type of the stored value. This allows us to perform type assertions to confirm that the value inside an interface matches a particular concrete type

Example 1.1.24
 1 package main
 2 import "fmt"
 3 
 4 type Hello struct {}
 5 
 6 func (h Hello) String() string {
 7   return "Hello"
 8 }
 9 
10 type World struct {}
11 
12 func (w *World) String() string {
13   return "world"
14 }
15 
16 type Message struct {
17   X fmt.Stringer
18   Y fmt.Stringer
19 }
20 
21 func (v Message) IsGreeting() (ok bool) {
22   if _, ok = v.X.(*Hello); !ok {
23     _, ok = v.Y.(*Hello)
24   }
25   return
26 }
27 
28 func main() {
29   m := &Message{}
30   fmt.Println(m.IsGreeting())
31   m.X = new(Hello)
32   fmt.Println(m.IsGreeting())
33   m.Y = new(World)
34   fmt.Println(m.IsGreeting())
35   m.Y = m.X
36   fmt.Println(m.IsGreeting())
37   m = &Message{ X: new(World), Y: new(Hello) }
38   fmt.Println(m.IsGreeting())
39   m.X, m.Y = m.Y, m.X
40   fmt.Println(m.IsGreeting())
41 }
go run 24.go
false
true
true
true
true
true

Here we’ve replaced Message’s String() method with IsGreeting(), a predicate which uses a pair of type assertions to tell us whether or not one of Message’s data fields contains a value of concrete type Hello.

So far in these examples we’ve been using pointers to Hello and World so the interface variables are storing pointers to pointers to these values (i.e. **Hello and **World) rather than pointers to the values themselves (i.e. *Hello and *World). In the case of World we have to do this to comply with the fmt.Stringer interface because String() is defined for *World and if we modify main to assign a World value to either field we’ll get a compile-time error

Example 1.1.25
29 func main() {
30   m := &Message{}
31   fmt.Println(m.IsGreeting())
32   m.X = Hello{}
33   fmt.Println(m.IsGreeting())
34   m.X = new(Hello)
35   fmt.Println(m.IsGreeting())
36   m.X = World{}
37 }
$ go run 25.go
# command-line-arguments
./25.go:36: cannot use World literal (type World) as type fmt.Stringer in assignment:
	World does not implement fmt.Stringer (String method has pointer receiver)

The final thing to mention about interfaces is that they support embedding of other interfaces. This allows us to compose a new, more restrictive interface based on one or more existing interfaces. Rather than demonstrate this with an example we’re going to look at code lifted directly from the standard io package which does this

67 type Reader interface {
68   Read(p []byte) (n int, err error)
69 }
78 type Writer interface {
79   Write(p []byte) (n int, err error)
80 }
106 type ReadWriter interface {
107   Reader
108   Writer
109 }

Here io is declaring three interfaces, the Reader and Writer which are independent of each other, and the ReadWriter which combines both. Any time we declare a variable, field or function parameter in terms of a ReaderWriter we know we can use both the Read() and Write() methods to manipulate it.

Startup

One of the less-discussed aspects of computer programs is the need to initialise many of them to a pre-determined state before they begin executing. Whilst this is probably the worst place to start discussing what to many people may appear to be advanced topics, one of my goals in this chapter is to cover all of the structural elements that we’ll meet when we examine more complex programs.

Every Go package may contain one or more init() functions specifying actions that should be taken during program initialisation. This is the one case I’m aware of where multiple declarations of the same identifier can occur without either resulting in a compilation error or the shadowing of a variable. In the following example we use the init() function to assign a value to our world variable

Example 1.1.26
 1 package main
 2 import . "fmt"
 3 
 4 const Hello = "hello"
 5 var world  string
 6 
 7 func init() {
 8   world = "world"
 9 }
10 
11 func main() {
12   Println(Hello, world)
13 }

However the init() function can contain any valid Go code, allowing us to place the whole of our program in init() and leaving main() as a stub to convince the compiler that this is indeed a valid Go program.

Example 1.1.27
 1 package main
 2 import . "fmt"
 3 
 4 const Hello = "hello"
 5 var world  string
 6 
 7 func init() {
 8   world = "world"
 9   Println(Hello, world)
10 }
11 
12 func main() {}

When there are multiple init() functions the order in which they’re executed is indeterminate so in general it’s best not to do this unless you can be certain the init() functions don’t interact in any way. The following happens to work as expected on my development computer but an implementation of Go could just as easily arrange it to run in reverse order or even leave deciding the order of execution until runtime.

Example 1.1.28
 1 package main
 2 import . "fmt"
 3 
 4 const Hello = "hello"
 5 var world  string
 6 
 7 func init() {
 8   Print(Hello, " ")
 9   world = "world"
10 }
11 
12 func init() {
13   Printf("%v\n", world)
14 }
15 
16 func main() {}

HTTP

So far our treatment of Hello World has followed the traditional route of printing a preset message to the console. Anyone would think we were living in the fuddy-duddy mainframe era of the 1970s instead of the shiny 21st Century, when web and mobile applications rule the world.

Turning Hello World into a web application is surprisingly simple, as the following example demonstrates.

Example 1.1.29
 1 package main
 2 import . "fmt"
 3 import "net/http"
 4 
 5 const MESSAGE = "hello world"
 6 const ADDRESS = ":1024"
 7 
 8 func main() {
 9   http.HandleFunc("/hello", Hello)
10   if e := http.ListenAndServe(ADDRESS, nil); e != nil {
11     Println(e)
12   }
13 }
14 
15 func Hello(w http.ResponseWriter, r *http.Request) {
16   w.Header().Set("Content-Type", "text/plain")
17   Fprintf(w, MESSAGE)
18 }
$ go run 29.go

Our web server is now listening on localhost port 1024 (usually the first non-privileged port on most Unix-like operating systems) and if we visit the url http://localhost:1024/hello with a web browser our server will return Hello World in the response body.

Image 1.29 http://localhost:1024/hello

The first thing to note is that the net/http package provides a fully-functional web server which requires very little configuration. All we have to do to get our content to the browser is define a handler, which in this case is a function to call whenever an http.Request is received, and then launch a server to listen on the desired address with http.ListenAndServe(). http.ListenAndServe returns an error if it’s unable to launch the server for some reason, which in this case we print to the console.

We’re going to import the net/http package into the current namespace and assume our code won’t encounter any runtime errors to make the simplicity even more apparent. If you run into any problems whilst trying the examples which follow, reinserting the if statement will allow you to figure out what’s going on.

Example 1.1.30
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const MESSAGE = "hello world"
 6 const ADDRESS = ":1024"
 7 
 8 func main() {
 9   HandleFunc("/hello", Hello)
10   ListenAndServe(ADDRESS, nil)
11 }
12 
13 func Hello(w ResponseWriter, r *Request) {
14   w.Header().Set("Content-Type", "text/plain")
15   Fprintf(w, MESSAGE)
16 }

HandleFunc() registers a URL in the web server as the trigger for a function, so when a web request targets the URL the associated function will be executed to generate the result. The specified handler function is passed both a ResponseWriter to send output to the web client and the Request which is being replied to. The ResponseWriter is a file handle so we can use the fmt.Fprint() family of file-writing functions to create the response body.

Finally we launch the server using ListenAndServe() which will block for as long as the server is active, returning an error if there is one to report.

In this example I’ve declared a function Hello and by referring to this in the call to HandleFunc() this becomes the function which is registered. However Go also allows us to define functions anonymously where we wish to use a function value, as demonstrated in the following variation on our theme.

Example 1.1.31
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const MESSAGE = "hello world"
 6 const ADDRESS = ":1024"
 7 
 8 func main() {
 9   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
10     w.Header().Set("Content-Type", "text/plain")
11     Fprintf(w, MESSAGE)
12   })
13   ListenAndServe(ADDRESS, nil)
14 }

Functions are first-class values in Go and here HandleFunc() is passed an anonymous function value which is created at runtime. This value is a closure so it can also access variables in the lexical scope in which it’s defined. We’ll treat closures in greater depth later in the book, but for now here’s an example which demonstrates their basic premise by defining a variable messages in main() and then accessing it from within the anonymous function.

Example 1.1.32
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const ADDRESS = ":1024"
 6 
 7 func main() {
 8   message := "hello world"
 9   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
10     w.Header().Set("Content-Type", "text/plain")
11     Fprintf(w, message)
12   })
13   ListenAndServe(ADDRESS, nil)
14 }

This is only a very brief taster of what’s possible using net/http so we’ll conclude by serving our hello world web application over an SSL connection.

Example 1.1.33
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const SECURE_ADDRESS = ":1025"
 6 
 7 func main() {
 8   message := "hello world"
 9   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
10     w.Header().Set("Content-Type", "text/plain")
11     Fprintf(w, message)
12   })
13   ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
14 }

Before we run this program we first need to generate a certificate and a public key, which we can do using crypto/tls/generate_cert.go in the standard package library.

$ go run $GOROOT/src/pkg/crypto/tls/generate_cert.go -ca=true -host="localhost"
2014/05/16 20:41:53 written cert.pem
2014/05/16 20:41:53 written key.pem
$ go run 33.go
Image 1.33 https://localhost:1025/hello

If you’re anything like me (and you have my sympathy if you are) then the next thought to idle through your mind will be a fairly obvious question: given that we can serve our content over both HTTP and HTTPS connections, how do we do both from the same program?

To answer this we have to know a little - but not a lot - about how to model concurrency in a Go program. The go keyword marks a goroutine which is a lightweight thread scheduled by the Go runtime. How this is implemented under the hood doesn’t matter, all we need to know is that when a goroutine is launched it takes a function call and creates a separate thread of execution for it. Here we’re going to launch a goroutine to run the HTTP server then run the HTTPS server in the main flow of execution.

Example 1.1.34
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const ADDRESS = ":1024"
 6 const SECURE_ADDRESS = ":1025"
 7 
 8 func main() {
 9   message := "hello world"
10   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
11     w.Header().Set("Content-Type", "text/plain")
12     Fprintf(w, message)
13   })
14 
15   go func() {
16     ListenAndServe(ADDRESS, nil)
17   }()
18 
19   ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
20 }

When I first wrote this code it actually used two goroutines, one for each server. Unfortunately no matter how busy any particular goroutine is, when the main() function returns our program will exit and our web servers will terminate. So I tried the primitive approach we all know and love from C

10 func main() {
11   message := "hello world"
12   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
13     w.Header().Set("Content-Type", "text/plain")
14     Fprintf(w, message)
15   })
16 
17   go func() {
18     ListenAndServe(ADDRESS, nil)
19   }()
20 
21   go func() {
22     ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
23   }()
24 
25   for {}
26 }

Here we’re using an infinite for loop to prevent program termination: it’s inelegant, but this is a small program and dirty hacks have their appeal. Whilst semantically correct this unfortunately doesn’t work either because of the way goroutines are scheduled: the infinite loop can potentially starve the thread scheduler and prevent the other goroutines from running.

$ go version
go version go1.3 darwin/amd64

In any event an infinite loop is a nasty, unnecessary hack as Go allows concurrent elements of a program to communicate with each other via channels, allowing us to rewrite our code as

Example 1.1.35
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const ADDRESS = ":1024"
 6 const SECURE_ADDRESS = ":1025"
 7 
 8 func main() {
 9   message := "hello world"
10   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
11     w.Header().Set("Content-Type", "text/plain")
12     Fprintf(w, message)
13   })
14 
15   done := make(chan bool)
16   go func() {
17     ListenAndServe(ADDRESS, nil)
18     done <- true
19   }()
20 
21   ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
22   <- done
23 }

For the next pair of examples we’re going to use two separate goroutines to run our HTTP and HTTPS servers, yet again coordinating program termination with a shared channel. In the first example we’ll launch both of the goroutines from the main() function, which is a fairly typical code pattern

Example 1.1.36
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const ADDRESS = ":1024"
 6 const SECURE_ADDRESS = ":1025"
 7 
 8 func main() {
 9   message := "hello world"
10   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
11     w.Header().Set("Content-Type", "text/plain")
12     Fprintf(w, message)
13   })
14 
15   done := make(chan bool)
16   go func() {
17     ListenAndServe(ADDRESS, nil)
18     done <- true
19   }()
20 
21   go func () {
22     ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
23     done <- true
24   }()
25   <- done
26   <- done
27 }

For our second deviation we’re going to launch a goroutine from main() which will run our HTTPS server and this will launch the second goroutine which manages our HTTP server

Example 1.1.37
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const ADDRESS = ":1024"
 6 const SECURE_ADDRESS = ":1025"
 7 
 8 func main() {
 9   message := "hello world"
10   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
11     w.Header().Set("Content-Type", "text/plain")
12     Fprintf(w, message)
13   })
14 
15   done := make(chan bool)
16   go func () {
17     go func() {
18       ListenAndServe(ADDRESS, nil)
19       done <- true
20     }()
21 
22     ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
23     done <- true
24   }()
25   <- done
26   <- done
27 }

There’s a certain amount of fragile repetition in this code as we have to remember to explicitly create a channel, and then to send and receive on it multiple times to coordinate execution. As Go provides first-order functions (i.e. allows us to refer to functions the same way we refer to data, assigning instances of them to variables and passing them around as parameters to other functions) we can refactor the server launch code as follows

Example 1.1.38
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const ADDRESS = ":1024"
 6 const SECURE_ADDRESS = ":1025"
 7 
 8 func main() {
 9   message := "hello world"
10   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
11     w.Header().Set("Content-Type", "text/plain")
12     Fprintf(w, message)
13   })
14 
15   Spawn(
16     func() { ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil) },
17     func() { ListenAndServe(ADDRESS, nil) },
18   )
19 }
20 
21 func Spawn(f ...func()) {
22   done := make(chan bool)
23 
24   for _, s := range f {
25     go func() {
26       s()
27       done <- true
28     }()
29   }
30 
31   for l := len(f); l > 0; l-- {
32     <- done
33   }
34 }

However this doesn’t work as expected, so let’s see if we can get any further insight

$ go vet 38.go
38.go:28: range variable s captured by func literal
exit status 1

Running go with the vet command runs a set of heuristics against our source code to check for common errors which wouldn’t be caught during compilation. In this case we’re being warned about this code

26 for _, s := range f {
27   go func() {
28     s()
29     done <- true
30   }()
31 }

Here we’re using a closure so it refers to the variable s in the for..range statement, and as the value of s changes on each successive iteration, so this is reflected in the call s().

To demonstrate this we’ll try a variant where we introduce a delay on each loop iteration much greater than the time taken to launch the goroutine.

Example 1.1.39
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 import "time"
 5 
 6 const ADDRESS = ":1024"
 7 const SECURE_ADDRESS = ":1025"
 8 
 9 func main() {
10   message := "hello world"
11   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
12     w.Header().Set("Content-Type", "text/plain")
13     Fprintf(w, message)
14   })
15 
16   Spawn(
17     func() { ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil) },
18     func() { ListenAndServe(ADDRESS, nil) },
19   )
20 }
21 
22 func Spawn(f ...func()) {
23   done := make(chan bool)
24 
25   for _, s := range f {
26     go func() {
27       s()
28       done <- true
29     }()
30     time.Sleep(time.Second)
31   }
32 
33   for l := len(f); l > 0; l-- {
34     <- done
35   }
36 }

When we run this we get the behaviour we expect with both HTTP and HTTPS servers running on their respective ports and responding to browser traffic. However this is hardly an elegant or practical solution and there’s a much better way of achieving the same effect.

Example 1.1.40format:go
26   for _, s := range f {
27     go func(server func()) {
28       server()
29       done <- true
30     }(s)
31   }

By accepting the parameter server to the goroutine’s closure we can pass in the value of s and capture it so that on successive iterations of the range our goroutines use the correct value.

Spawn() is an example of how powerful Go’s support for first-class functions can be, allowing us to run any arbitrary piece of code and wait for it to signal completion. It’s also a variadic function, taking as many or as few functions as desired and setting each of them up correctly.

If we now reach for the standard library we discover that another alternative is to use a sync.WaitGroup to keep track of how many active goroutines we have in our program and only terminate the program when they’ve all completed their work. Yet again this allows us to run both servers in separate goroutines and manage termination correctly.

Example 1.1.41
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 import "sync"
 5 
 6 const ADDRESS = ":1024"
 7 const SECURE_ADDRESS = ":1025"
 8 
 9 func main() {
10   message := "hello world"
11   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
12     w.Header().Set("Content-Type", "text/plain")
13     Fprintf(w, message)
14   })
15 
16   var servers sync.WaitGroup
17   servers.Add(1)
18   go func() {
19     defer servers.Done()
20     ListenAndServe(ADDRESS, nil)
21   }()
22 
23   servers.Add(1)
24   go func() {
25     defer servers.Done()
26     ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
27   }()
28   servers.Wait()
29 }

As there’s a certain amount of redundancy in this, let’s refactor a little by packaging server initiation into a new Launch() function. Launch() takes a parameter-less function and wraps this in a closure which will be launched as a goroutine in a separate thread of execution. Our sync.WaitGroup variable servers has been turned into a global variable to simplify the function signature of Launch(). When we call Launch() we’re freed from the need to manually increment servers prior to goroutine startup, and we use a defer statement to automatically call servers.Done() when the goroutine terminates even in the event that the goroutine crashes.

Example 1.1.42
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 import "sync"
 5 
 6 const ADDRESS = ":1024"
 7 const SECURE_ADDRESS = ":1025"
 8 
 9 var servers sync.WaitGroup
10 
11 func main() {
12   message := "hello world"
13   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
14     w.Header().Set("Content-Type", "text/plain")
15     Fprintf(w, message)
16   })
17 
18   Launch(func() {
19     ListenAndServe(ADDRESS, nil)
20   })
21 
22   Launch(func() {
23     ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
24   })
25   servers.Wait()
26 }
27 
28 func Launch(f func()) {
29   servers.Add(1)
30   go func() {
31     defer servers.Done()
32     f()
33   }()
34 }

The Environment

The main shells used with modern operating systems (Linux, OSX, FreeBSD, Windows, etc.) provide a persistent environment which can be queried by running programs, allowing a user to store configuration values in named variables. Go supports reading and writing these variables using the os package functions Getenv() and Setenv().

In our next example we’re going to query the environment for the variable SERVE_HTTP which we’ll assume contains the default address on which to serve unencrypted web content.

Example 1.1.43
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 import "os"
 5 import "sync"
 6 
 7 const SECURE_ADDRESS = ":1025"
 8 
 9 var address string
10 var servers sync.WaitGroup
11 
12 func init() {
13   if address = os.Getenv("SERVE_HTTP"); address == "" {
14     address = ":1024"
15   }
16 }
17 
18 func main() {
19   message := "hello world"
20   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
21     w.Header().Set("Content-Type", "text/plain")
22     Fprintf(w, message)
23   })
24 
25   Launch(func() {
26     ListenAndServe(address, nil)
27   })
28 
29   Launch(func() {
30     ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
31   })
32   servers.Wait()
33 }
34 
35 func Launch(f func()) {
36   servers.Add(1)
37   go func() {
38     defer servers.Done()
39     f()
40   }()
41 }

Here we’ve defined a global variable address which we set in init() to either the value provided in SERVE_HTTP or a default value “:1024”.

$ go run 43.go
Image 1.43a http://localhost:1024/hello
$ SERVE_HTTP=":3030" go run 43.go
Image 1.43b http://localhost:3030/hello

If we now extend this further to make the program fully configurable from the environment we arrive at

Example 1.1.44
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 import "os"
 5 import "sync"
 6 
 7 var (
 8   address string
 9   secure_address string
10   certificate string
11   key string
12 )
13 var servers sync.WaitGroup
14 
15 func init() {
16   if address = os.Getenv("SERVE_HTTP"); address == "" {
17     address = ":1024"
18   }
19 
20   if secure_address = os.Getenv("SERVE_HTTPS"); secure_address == "" {
21     secure_address = ":1025"
22   }
23 
24   if certificate = os.Getenv("SERVE_CERT"); certificate == "" {
25     certificate = "cert.pem"
26   }
27 
28   if key = os.Getenv("SERVE_KEY"); key == "" {
29     key = "key.pem"
30   }
31 }
32 
33 func main() {
34   message := "hello world"
35   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
36     w.Header().Set("Content-Type", "text/plain")
37     Fprintf(w, message)
38   })
39 
40   Launch(func() {
41     ListenAndServe(address, nil)
42   })
43 
44   Launch(func() {
45     ListenAndServeTLS(secure_address, certificate, key, nil)
46   })
47   servers.Wait()
48 }
49 
50 func Launch(f func()) {
51   servers.Add(1)
52   go func() {
53     defer servers.Done()
54     f()
55   }()
56 }
$ SERVE_HTTP=":3030" SERVE_HTTPS=":4040" go run 44.go

Handling Signals

If you’ve been running our example in the terminal and wondering how to terminate it without exiting the shell then you probably come from a GUI background and haven’t met control-C and its relatives (or rather you have, but most likely as cut’n’paste shortcuts).

Both Windows and Unix-style operating systems have the concept of a signal which can be sent from one process to another, and for historic reasons many of these can be manually entered using a control-key combination. This is a useful convenience but shutting down a production server this way can result in data loss or corruption. However that’s not the case with our Hello World server, so we have an excellent excuse to examine how to catch a termination signal and do something of our own choosing.

To listen for signals in Go we use the os/signal package in the standard library, which allows us to register a channel (an atomic queue for transferring messages between goroutines at runtime) on which notifications are to be received using the signal.Notify() function. Which signals will be made available depends largely on which operating system you’re working with and Go provides only two as standard across Windows and Unix: os.Interrupt and os.Kill. Of these os.Interrupt can be sent with control-C whilst os.Kill equates to SIGKILL on *nixen and is usually a non-maskable interrupt, meaning that it terminates execution and will never be received by Notify().

In the following example we’re initialising a signal handler at program startup. This consists of a goroutine containing an infinite loop and blocking on a channel of fixed size (in this case able to hold only one element at a time). The signal handler should be trapping the Interrupt and Kill signals. In both cases if we catch the signal we print a message to the console before exiting, however as previously mentioned the Kill signal (which can be sent from another shell session using the kill command) will never be received by our Go code.

Example 1.1.45
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 import "os"
 5 import "os/signal"
 6 import . "sync"
 7 
 8 const ADDRESS = ":1024"
 9 const SECURE_ADDRESS = ":1025"
10 
11 var servers WaitGroup
12 
13 func init() {
14   go SignalHandler(make(chan os.Signal, 1))
15 }
16 
17 func main() {
18   message := "hello world"
19   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
20     w.Header().Set("Content-Type", "text/plain")
21     Fprintf(w, message)
22   })
23 
24   Launch(func() {
25     ListenAndServe(ADDRESS, nil)
26   })
27 
28   Launch(func() {
29     ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
30   })
31   servers.Wait()
32 }
33 
34 func Launch(f func()) {
35   servers.Add(1)
36   go func() {
37     defer servers.Done()
38     f()
39   }()
40 }
41 
42 func SignalHandler(c chan os.Signal) {
43   signal.Notify(c, os.Interrupt)
44   for s := <- c; ; s = <- c {
45     switch s {
46     case os.Interrupt:
47       Println("^C received")
48       os.Exit(0)
49     case os.Kill:
50       Println("SIGKILL received")
51       os.Exit(1)
52     }
53   }
54 }

When we run this in the terminal on a Mac and press control-C we’ll see something like

$ go run 45.go
^C^C received

The key point of signals is that they allow our program to apply its own logic to events. In the following example we’re going to override the Interrupt signal sent by control-C so that the program continues execution. We’re then going to scan for other signals and use these to terminate the program. The syscall package defines a number of os.Signal values which can be detected by Notify() and of these I’ve chosen SIGABRT, SIGTERM and SIGQUIT as plausible termination signals. We’ll treat SIGABRT as an error condition and the other two as clean terminations.

Something else to note is that our signal handler is using a standard for loop statement to poll for input from the signal channel and then compare it to the cases of a switch statement. As the signal handler is designed to run for as long as Notify() is receiving signals we can simplify this a little

Example 1.1.46
45 func SignalHandler(c chan os.Signal) {
46   signal.Notify(c, os.Interrupt, syscall.SIGABRT, syscall.SIGTERM, syscall.SIGQUIT)
47   for s := <- c; ; s = <- c {
48     switch s {
49     case os.Interrupt:
50       Println("interrupt - continue running")
51     case syscall.SIGABRT:
52       Println("abnormal exit")
53       os.Exit(1)
54     case syscall.SIGTERM, syscall.SIGQUIT:
55       Println("clean shutdown")
56       os.Exit(0)
57     }
58   }
59 }
$ go build 46.go
$ ./46
^Cinterrupt - continue running
^Cinterrupt - continue running
^\clean shutdown

We can send a SIGABRT signal using the kill command in a subshell and force our program to terminate abnormally

$ go build 46.go
$ ./46
^Z
[1]+  Stopped                 go run 46.go
$ ps
  PID TTY           TIME CMD
41097 ttys016    0:00.48 -bash
58713 ttys016    0:00.10 go run 46.go
58716 ttys016    0:00.01 /var/folders/25/ybgksr451vxf78xk1svymm5c0000gn/T/go-build543807549/command-li\
ne-arguments/_obj/exe/46
57608 ttys017    0:00.08 -bash
$ kill -SIGABRT 58716
$ fg
go run 46.go
abnormal exit
exit status 1

So far we’ve looked at how our program receives signals, however it’s also possible to send signals. For now we’re going to focus on sending a SIGABRT signal from our program to itself when there’s an error launching one of the servers, in this case by setting ADDRESS and SECURE_ADDRESS to the same value.

Example 1.1.47
 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 import "os"
 5 import "os/signal"
 6 import . "sync"
 7 import "syscall"
 8 
 9 const ADDRESS = ":1024"
10 const SECURE_ADDRESS = ":1024"
11 
12 var servers WaitGroup
13 
14 func init() {
15   go SignalHandler(make(chan os.Signal, 1))
16 }
17 
18 func main() {
19   message := "hello world"
20   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
21     w.Header().Set("Content-Type", "text/plain")
22     Fprintf(w, message)
23   })
24 
25   Launch("HTTP", func() error {
26     return ListenAndServe(ADDRESS, nil)
27   })
28 
29   Launch("HTTPS", func() error {
30     return ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
31   })
32   servers.Wait()
33 }
34 
35 func Launch(name string, f func() error) {
36   servers.Add(1)
37   go func() {
38     defer servers.Done()
39     if e := f(); e != nil {
40       Println(name, "->", e)
41       syscall.Kill(syscall.Getpid(), syscall.SIGABRT)
42     }
43   }()
44 }
45 
46 func SignalHandler(c chan os.Signal) {
47   signal.Notify(c, os.Interrupt, syscall.SIGABRT, syscall.SIGTERM, syscall.SIGQUIT)
48   for s := <- c; ; s = <- c {
49     switch s {
50     case syscall.SIGABRT:
51       Println("abnormal exit")
52       os.Exit(1)
53     case os.Interrupt, syscall.SIGTERM, syscall.SIGQUIT:
54       Println("clean shutdown")
55       os.Exit(0)
56     }
57   }
58 }

We’ve modified our Launch() function to take a name which can be displayed as part of an error message, and its function parameter now has the signature func() error which specifies that it must return an error value, which is what’s returned by both ListenAndServe() and ListenAndServeTLS(). In the event the error (which is a predeclared interface) contains a value then we know an error condition’s occurred and can send a SIGABRT signal with syscall.Kill(). As Kill() is able to send signals to any running process we need to specify the ID of the current process, which we find using syscall.Getpid().

$ go run 47.go
2014/06/25 14:42:25 HTTPS -> listen tcp :1024: bind: address already in use
abnormal exit
exit status 1

TCP/IP

Printing text in a web browser is a cool trick, but what of real network programming? You know, the kind that bearded sandle-wearing *nix hackers go in for? It turns out this is surprisingly simple

Example 1.1.48 TCP/IP server
 1 package main
 2 import . "fmt"
 3 import "net"
 4 
 5 func main() {
 6   if listener, e := net.Listen("tcp", ":1024"); e == nil {
 7     for {
 8       if connection, e := listener.Accept(); e == nil {
 9         go func(c net.Conn) {
10           defer c.Close()
11           Fprintln(c, "hello world")
12         }(connection)
13       }
14     }
15   }
16 }

The net package revolves around server the Listener and client Connection types. A Listener is an interface which allows any type implementing its specified methods - Accept(), Close() and Addr() - to be used interchangeably and is a key tool in Go for generalising program design. Writing a server then becomes a simple process of

- Listen() on a specified protocol and port number
- Accept() incoming connections
- for each net.Conn, run a handler in a separate goroutine
- read from and write to the connection whilst performing work
- close each connection when it finishes its work

Here we start to see some of the power of interfaces as a net.Conn implements the Writer interface defined in the io package, and fmt.Fprintf() takes any type which integrates io.Writer as its target.

Moving away from HTTP means abandoning the browser for testing but both *nix and Windows have a handy command-line utility called telnet which allows us to connect directly to a TCP/IP server and interact with it. We’ll get into the interaction side of things later in the book, for now here’s an example run of our program.

$ go run 48.go &
[1] 17415
$ telnet localhost 1024
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
hello world
Connection closed by foreign host.

Telnet’s a useful tool, but it’d be nice if we could connect our own client to the server as this could then be built for any platform supported by Go. For stream-oriented protocols like TCP/IP we do this using the net.Dial() function to open a net.Conn connection to a server and we can then interact with this using the io.Reader and io.Writer interfaces. These interfaces are supported throughout the Go standard package library, allowing files and streaming connections to be used interchangeably.

Example 1.1.49 TCP/IP client
 1 package main
 2 import "bufio"
 3 import . "fmt"
 4 import "net"
 5 
 6 func main() {
 7   if connection, e := net.Dial("tcp", ":1024"); e == nil {
 8     defer connection.Close()
 9     if text, e := bufio.NewReader(connection).ReadString('\n'); e == nil {
10       Printf(text)
11     }
12   }
13 }

Because a net.Conn represents streams of data flowing between client and server we’ve introduced the bufio package to our client so that the data it’s receiving is buffered. This avoids our having to write our own code for buffering incoming data and is another example of the flexibility Go’s interfaces provide.

$ go run 48.go &
[1] 6102
$ go run 49.go
hello world
$ go run 49.go
hello world

Most books on network programming tend to stop at vanilla TCP/IP and leave figuring out how to establish a secure connection between client and server as an exercise for the reader. However we’re not likely to get another chance to look at this problem with the same lack of clutter that Hello World provides, and anyway we know how to generate a key and a certificate from our HTTPS adventure so we might as well reuse the knowledge. This time we’re going to need two sets of keys so let’s take care of that first

$ cp cert.pem server.cert.pem
$ cp key.pem server.key.pem
$ go run $GOROOT/src/pkg/crypto/tls/generate_cert.go -ca=true -host="localhost"
2014/05/16 20:41:53 written cert.pem
2014/05/16 20:41:53 written key.pem
$ cp cert.pem client.cert.pem
$ cp key.pem client.key.pem

Now we have our keys sorted, let’s take a look at what a TCP/IP server looks like in Go

Example 1.1.50 TCP/IP server with tls
 1 package main
 2 import "crypto/rand"
 3 import "crypto/tls"
 4 import . "fmt"
 5 
 6 func main() {
 7   if certificate, e := tls.LoadX509KeyPair("server.cert.pem", "server.key.pem"); e == nil {
 8     config := tls.Config{
 9       Certificates: []tls.Certificate{ certificate },
10       Rand: rand.Reader,
11     }
12 
13     if listener, e := tls.Listen("tcp", ":1025", &config); e == nil {
14       for {
15         if connection, e := listener.Accept(); e == nil {
16           go func(c *tls.Conn) {
17             defer c.Close()
18             Fprintln(c, "hello world")
19           }(connection.(*tls.Conn))
20         }
21       }
22     }
23   }
24 }

Importing crypto/tls provides us with an equivalent API to that defined in net and this means that as tls.Listen() fulfils the net.Listener interface our connections will be of type net.Conn. As a result if we want to pass the connection around inside our code we either have to import net so we can use net.Conn or perform a type assertion to use the connection as a *tls.Conn. We’ve made the latter choice here.

18   if connection, e := listener.Accept(); e == nil {
19     go func(c *tls.Conn) {
20       defer c.Close()
21       Fprintln(c, "hello world")
22     }(connection.(*tls.Conn))
23   }

For a server we import crypto/rand to access rand.Reader, a cryptographically secure pseudo-random number generator which we’ll be using as a source of randomness in the TLS connection. We then create a certificate using tls.LoadX509KeyPair() to load the server key pair and if this is successful then we set up a listener to accept incoming connections and write “Hello World” to a client.

As we’re using TLS we can’t test this version of Hello World using telnet so instead we need to write a client. Yet again this requires a keypair and where in our previous client we called net.Dial() we now use tls.Dial(), resulting in a very similar program.

Example 1.1.51 TCP/IP client with tls
 1 package main
 2 import "bufio"
 3 import "crypto/tls"
 4 import . "fmt"
 5 
 6 func main() {
 7   if certificate, e := tls.LoadX509KeyPair("client.cert.pem", "client.key.pem"); e == nil {
 8     config := tls.Config{
 9       Certificates: []tls.Certificate{ certificate },
10       InsecureSkipVerify: true,
11     }
12 
13     if connection, e := tls.Dial("tcp", ":1025", &config); e == nil {
14       defer connection.Close()
15       if text, e := bufio.NewReader(connection).ReadString('\n'); e == nil {
16         Printf(text)
17       }
18     }
19   }
20 }
$ go run 50.go &
[1] 6107
$ go run 51.go
hello world
$ go run 51.go
hello world

Looking back at our HTTP experiments, we were able to write a program which served over both HTTP and HTTPS connections. It’d be nice to do something similar with TCP/IP, if only to compare the two code-paths.

Example 1.1.52 TCP/IP dual-mode server
 1 package main
 2 import "crypto/rand"
 3 import "crypto/tls"
 4 import . "fmt"
 5 import "net"
 6 import "sync"
 7 
 8 var servers sync.WaitGroup
 9 
10 func main() {
11   if listener, e := net.Listen("tcp", ":1024"); e == nil {
12     Serve(listener)
13   }
14 
15   Serve(TLSListener("server.cert.pem", "server.key.pem", ":1025"))
16   servers.Wait()
17 }
18 
19 func TLSListener(cert, key, address string) (r net.Listener) {
20   if certificate, e := tls.LoadX509KeyPair(cert, key); e == nil {
21     config := tls.Config{
22       Certificates: []tls.Certificate{ certificate },
23       Rand: rand.Reader,
24     }
25     if listener, e := tls.Listen("tcp", address, &config); e == nil {
26       r = listener
27     }
28   }
29   return
30 }
31 
32 func Serve(listener net.Listener) {
33   if listener != nil {
34     Launch(func() {
35       for {
36         if connection, e := listener.Accept(); e == nil {
37           go func(c net.Conn) {
38             defer c.Close()
39             Fprintln(c, "hello world")
40           }(connection)
41         }
42       }
43     })
44   }
45 }
46 
47 func Launch(f func()) {
48   servers.Add(1)
49   go func() {
50     defer servers.Done()
51     f()
52   }()
53 }

We’ve reused Launch() from Example 1.33 to manage the lifecycle of our two server goroutines and introduced Serve() to phrase the server behaviour in terms of the net.Listener interface. We then move all the setup code for creating a tls.Listener into a separate function TLSListener() which returns a net.Listener value as tls.Listener complies with its interface, or a nil value if tls.Listen() returns an error.

If we now run this server we can connect to it with both of our client programs.

$ go run 52.go &
[1] 6278
$ go run 49.go
hello world
$ go run 51.go
hello world
$ go run 51.go
hello world
$ go run 49.go
hello world

UDP

Both TCP/IP and HTTP communications are connection-oriented and this involves a reasonable amount of handshaking and error-correction to assemble data packets in the correct order. For most applications this is exactly how we want to organise our network communications but sometimes the size of our messages is sufficiently small that we can fit them into individual packets, and when this is the case the UDP protocol is an ideal candidate.

As with our previous examples we still need both server and client applications.

Example 1.1.53 UDP server
 1 package main
 2 import . "fmt"
 3 import "net"
 4 
 5 var HELLO_WORLD = ([]byte)("Hello World\n")
 6 
 7 func main() {
 8   if address, e := net.ResolveUDPAddr("udp", ":1024"); e == nil {
 9     if server, e := net.ListenUDP("udp", address); e == nil {
10       for buffer := MakeBuffer(); ; buffer = MakeBuffer() {
11         if n, client, e := server.ReadFromUDP(buffer); e == nil {
12           go func(c *net.UDPAddr, packet []byte) {
13             if n, e := server.WriteToUDP(HELLO_WORLD, c); e == nil {
14               Printf("%v bytes written to: %v\n", n, c)
15             }
16           }(client, buffer[:n])
17         }
18       }
19     }
20   }
21 }
22 
23 func MakeBuffer() (r []byte) {
24   return make([]byte, 1024)
25 }

We have a somewhat more complex code pattern here than with TCP/IP to take account of the difference in underlying semantics: UDP is an unreliable transport dealing in individual packets (datagrams) which are independent of each other, therefore a server doesn’t maintain streams to any of its clients and these are responsible for any error-correction or packet ordering which may be necessary to coordinate successive signals. Because of these differences from TCP/IP we end up with the following generic workflow

- net.ResolveUDPAddr() to resolve the address
- net.ListenUDP() opens a UDP port and listens for traffic
- net.ReadFromUDP() copies incoming data into a buffer and provides the remote client’s address
- net.WriteToUDP() writes data back to the remote client’s address

For trivial uses of UDP we could probably forego the use of a separate goroutine to process each received packet, and indeed we may also have an application architecture where instead of processing the packet we’d hand it off to a message queue elsewhere. However many real-world examples such as serving DNS requests may introduce appreciable delays for processing and by using goroutines we ensure the server itself isn’t stalled.

Here our main overhead is the cost of buffer allocation as we use a different data buffer for each request. In a real-world example we’d very likely introduce a buffer pool which would expand and contract with demand, and reuse individual buffers once their associated request has completed. This is surprisingly simple to implement in Go and we’ll look at this in detail in a later chapter.

Our client has the same basic boilerplate as the server, only we use net.DialUDP() to set up a connection. We could use net.ReadFromUDP() and net.WriteToUDP() however as a net.UDPConn connection implements the io.ReadWriter interface we can use bufio.Reader to manage reading, and for writes the connection already knows the server address. As the server only knows about clients by receiving data from them we start our interaction with a UDPConn.Write() and then perform the buffered ReadString() to get a response.

Example 1.1.54 UDP client
 1 package main
 2 import "bufio"
 3 import . "fmt"
 4 import "net"
 5 
 6 var CRLF = ([]byte)("\n")
 7 
 8 func main() {
 9   if address, e := net.ResolveUDPAddr("udp", ":1024"); e == nil {
10     if server, e := net.DialUDP("udp", nil, address); e == nil {
11       defer server.Close()
12       if _, e = server.Write(CRLF); e == nil {
13         if text, e := bufio.NewReader(server).ReadString('\n'); e == nil {
14           Printf("%v", text)
15         }
16       }
17     }
18   }
19 }

Let’s give this a test run in the shell.

$ go run 53.go &
[2] 12777
$ go run 54.go
12 bytes written to: 127.0.0.1:58015
Hello World
$ go run 54.go
12 bytes written to: 127.0.0.1:50159
Hello World
$ go run 54.go
12 bytes written to: 127.0.0.1:51813
Hello World

Note how each time we run the client program it’s assigned a different network port by the operating system each time net.DialUDP is called.

We can make this apparent by performing multiple sequences of Write() and Read() operations

Example 1.1.55 UDP client
 1 package main
 2 import "bufio"
 3 import . "fmt"
 4 import "net"
 5 
 6 var CRLF = ([]byte)("\n")
 7 
 8 func main() {
 9   if address, e := net.ResolveUDPAddr("udp", ":1024"); e == nil {
10     if server, e := net.DialUDP("udp", nil, address); e == nil {
11       defer server.Close()
12       for i := 0; i < 3; i++ {
13         if _, e = server.Write(CRLF); e == nil {
14           if text, e := bufio.NewReader(server).ReadString('\n'); e == nil {
15             Printf("%v: %v", i, text)
16           }
17         }
18       }
19     }
20   }
21 }
$ go run 53.go &
[1] 12883
$ go run 55.go
12 bytes written to: 127.0.0.1:51732
0: Hello World
12 bytes written to: 127.0.0.1:51732
1: Hello World
12 bytes written to: 127.0.0.1:51732
2: Hello World
$ go run 55.go
12 bytes written to: 127.0.0.1:55504
0: Hello World
12 bytes written to: 127.0.0.1:55504
1: Hello World
12 bytes written to: 127.0.0.1:55504
2: Hello World

RSA obfuscated UDP

With all of our network examples to date we’ve included a secure transport option, but UDP doesn’t have a secured mode so we appear stuck with sending our message unencrypted for all the world to see. This is fine for a message such as Hello World which we’re happy for intervening network nodes to observe, but what if we want to send confidential data in our UDP packet?

In the following example we’re going to use our existing client RSA key-pair by sending the public key to our server which will then encrypt the message with this key and send it back to the client. The client already possesses the RSA private key so it’s a simple task to decrypt the message and display it. When we send the public key we could do so in a number of different formats: as an RSA pem file; as a raw binary buffer; or, serialised in some form. As both client and server are written in Go we’ll opt for the serialisation format provided in package gob. This is a pragmatic choice as if we were to send a pem file then that’d make it obvious that we’re using an encrypted format, and if we use a raw binary buffer we’d have to include a discussion of Go’s unsafe and reflection packages which are covered later in this book.

Example 1.1.56 RSA-enabled UDP server
 1 package main
 2 import "bytes"
 3 import "crypto/rand"
 4 import "crypto/rsa"
 5 import "crypto/sha1"
 6 import "encoding/gob"
 7 import . "fmt"
 8 import . "net"
 9 
10 var HELLO_WORLD = []byte("Hello World")
11 var RSA_LABEL = []byte("served")
12 
13 func main() {
14   Serve(":1025", func(connection *UDPConn, c *UDPAddr, packet *bytes.Buffer) (n int) {
15     var key rsa.PublicKey
16     if e := gob.NewDecoder(packet).Decode(&key); e == nil {
17       if response, e := rsa.EncryptOAEP(sha1.New(), rand.Reader, &key, HELLO_WORLD, RSA_LABEL); e == n\
18 il {
19         n, _ = connection.WriteToUDP(response, c)
20       }
21     }
22     return
23   })
24 }
25 
26 func Serve(address string, f func(*UDPConn, *UDPAddr, *bytes.Buffer) int) {
27   Launch(address, func(connection *UDPConn) {
28     for {
29       buffer := make([]byte, 1024)
30       if n, client, e := connection.ReadFromUDP(buffer); e == nil {
31         go func(c *UDPAddr, b []byte) {
32           if n := f(connection, c, bytes.NewBuffer(b)); n != 0 {
33             Println(n, "bytes written to", c)
34           }
35         }(client, buffer[:n])
36       }
37     }
38   })
39 }
40 
41 func Launch(address string, f func(*UDPConn)) {
42   if a, e := ResolveUDPAddr("udp", address); e == nil {
43     if server, e := ListenUDP("udp", a); e == nil {
44       f(server)
45     }
46   }
47 }

So, the first thing of note is that we’ve refactored connection management into Serve() to make the server code easier to follow, and then we’re passing a function literal into this with the tasks to be performed each time a client connects. For now this is a quick hack so we’re not launching Serve() in its own goroutine with all the extra boilerplate for sync.WaitGroup which we’ve seen in previous examples. However we are spawning a separate goroutine for each packet received so that the server doesn’t block, and as an added bonus each time data is written to a client the number of bytes transferred is logged.

For each connection we read the client’s message which we know should be a valid public key in gob format. To decode this we create a gob.Decoder with the message as its base, then Decode() this to get a valid rsa.PublicKey which we then use to encrypt our message with rsa.EncryptOAEP(). The main thing to note here is that RSA_LABEL is a parameter which must be set the same for both rsa.EncryptOAEP() and rsa.DecryptOAEP() for the message to be correctly read by the latter. There’s no reason why this couldn’t be configured on a per-connection basis.

Now, let’s take a look at our client application

Example 1.1.57 RSA-enabled UDP client
 1 package main
 2 import "bytes"
 3 import "crypto/rand"
 4 import "crypto/rsa"
 5 import "crypto/sha1"
 6 import "crypto/x509"
 7 import "encoding/gob"
 8 import "encoding/pem"
 9 import "io/ioutil"
10 import . "fmt"
11 import . "net"
12 
13 var RSA_LABEL = []byte("served")
14 
15 func main() {
16   Connect(":1025", func(server *UDPConn, private_key *rsa.PrivateKey) {
17     cipher_text := MakeBuffer()
18     if n, e := server.Read(cipher_text); e == nil {
19       if plain_text, e := rsa.DecryptOAEP(sha1.New(), rand.Reader, private_key, cipher_text[:n], RSA_L\
20 ABEL); e == nil {
21         Println((string)(plain_text))
22       }
23     }
24   })
25 }
26 
27 func Connect(address string, f func(*UDPConn, *rsa.PrivateKey)) {
28   LoadPrivateKey("client.key.pem", func(private_key *rsa.PrivateKey) {
29     if address, e := ResolveUDPAddr("udp", ":1025"); e == nil {
30       if server, e := DialUDP("udp", nil, address); e == nil {
31         defer server.Close()
32         SendKey(server, private_key.PublicKey, func() {
33           f(server, private_key)
34         })
35       }
36     }
37   })
38 }
39 
40 func LoadPrivateKey(file string, f func(*rsa.PrivateKey)) {
41   if file, e := ioutil.ReadFile(file); e == nil {
42     if block, _ := pem.Decode(file); block != nil {
43       if block.Type == "RSA PRIVATE KEY" {
44         if key, _ := x509.ParsePKCS1PrivateKey(block.Bytes); key != nil {
45           f(key)
46         }
47       }
48     }
49   }
50   return
51 }
52 
53 func SendKey(server *UDPConn, public_key rsa.PublicKey, f func()) {
54   var encoded_key bytes.Buffer
55   if e := gob.NewEncoder(&encoded_key).Encode(public_key); e == nil {
56     if _, e = server.Write(encoded_key.Bytes()); e == nil {
57       f()
58     }
59   }
60 }
61 
62 func MakeBuffer() (r []byte) {
63   return make([]byte, 1024)
64 }
$ go run 56.go &
[1] 66945
$ go run 57.go
256 bytes written to 127.0.0.1:51328
Hello World
$ go run 57.go
256 bytes written to 127.0.0.1:64834
Hello World
$ go run 57.go
256 bytes written to 127.0.0.1:50982
Hello World

The most obvious thing about this code is the heavy use of function literals, giving it a clean compositional feel. This is an aesthetic I picked up working with Ruby and which I always missed when dipping back into C or other low-level languages, so expect to variations on this style in later chapters.

Connect() is the client version of Serve(), abstracting away the details of contacting a UDP server, and the meat of our program’s interaction is a simple Read() of an encrypted message from the server which is then decrypted using rsa.DecryptOAEP() and displayed. Before our code initiates the connection though we need it to load an RSA key-pair so we can transmit our public key to the server. We do this in LoadPrivateKey() which uses ioutil.ReadFile() to load a PEM-encoded file into memory and ensure it contains a private key before invoking a function passed to it as a parameter. In this case the passed function sets up the connection, sends the public key to the server and then invokes the function passed to Connect() in SendKey().

To keep SendKey() as generic as possible it takes a parameterless function which is basically just a closure into the caller’s environment. In the case of Connect() the closure we pass to SendKey() binds to the server and private_key variables.

Error Handling

The examples in this chapter are for the most part designed to follow the happy path as our interest is in seeing some simple Go code that we can later build upon. The one obvious exception was when we explored signal handling and used the presence of an error as an excuse to send a SIGABRT to terminate the server. However error-handling is a large part of most real-world programming - especially in a system level language.

Go takes a typically pragmatic approach to error handling, the language specification defining the error type as a predeclared interface

type error interface {
	Error() string
}

In the following example we’re going to rewrite our encrypted UDP server from example 1.56 so that start-up errors cause the server to terminate and signal an error to the shell whilst client errors will log an appropriate message for later analysis using the log package. The default behaviour of the log package is to write its output to stderr so it integrates well with traditional *nix tools and infrastructure.

Whilst our program is still trivial in purpose, we now have all the basic conveniences for running a scalable server and integrating it with third-party monitoring and logging tools at deployment.

Example 1.1.58
 1 package main
 2 import "bytes"
 3 import "crypto/rand"
 4 import "crypto/rsa"
 5 import "crypto/sha1"
 6 import "encoding/gob"
 7 import "log"
 8 import . "net"
 9 
10 var HELLO_WORLD = []byte("Hello World")
11 var RSA_LABEL = []byte("served")
12 
13 func main() {
14   Serve(":1025", func(connection *UDPConn, c *UDPAddr, packet *bytes.Buffer) (n int) {
15     var e error
16     var key rsa.PublicKey
17     var response []byte
18 
19     if e = gob.NewDecoder(packet).Decode(&key); e != nil {
20       log.Println("unable to decode wrapper:", c)
21     }
22 
23     if response, e = rsa.EncryptOAEP(sha1.New(), rand.Reader, &key, HELLO_WORLD, RSA_LABEL); e != nil {
24       log.Println("unable to encrypt server response")
25     }
26 
27     if n, e = connection.WriteToUDP(response, c); e != nil {
28       log.Println("unable to write response to client:", c)
29     }
30     return
31   })
32 }
33 
34 func Serve(address string, f func(*UDPConn, *UDPAddr, *bytes.Buffer) int) {
35   Launch(address, func(connection *UDPConn) {
36     for {
37       buffer := make([]byte, 1024)
38       if n, client, e := connection.ReadFromUDP(buffer); e == nil {
39         go func(c *UDPAddr, b []byte) {
40           if n := f(connection, c, bytes.NewBuffer(b)); n != 0 {
41             log.Println(n, "bytes written to", c)
42           }
43         }(client, buffer[:n])
44       } else {
45         log.Println(address, e.Error())
46       }
47     }
48   })
49 }
50 
51 func Launch(address string, f func(*UDPConn)) {
52   var e error
53   var a *UDPAddr
54   var server *UDPConn
55 
56   if a, e = ResolveUDPAddr("udp", address); e != nil {
57     log.Fatalln("unable to resolve UDP address:", e.Error())
58   }
59 
60   if server, e = ListenUDP("udp", a); e != nil {
61     log.Fatalln("can't open socket for listening:", e.Error())
62   }
63 
64   f(server)
65 }
$ go run 58.go &
[1] 15397
$ go run 57.go
2016/07/13 09:13:11 256 bytes written to 127.0.0.1:65099
Hello World
$ go run 58.go
2016/07/13 09:13:51 can't open socket for listening: listen udp :1025: bind: address already in use
exit status 1

This is the only error condition that’s likely to occur in our shell-based test environment but if we forcibly corrupt the gob encoding of our client’s response key by modifying its SendKey function to drop the first byte we can emulate key corruption in transit

55 func SendKey(server *UDPConn, public_key rsa.PublicKey, f func()) {
56   var encoded_key bytes.Buffer
57   if e := gob.NewEncoder(&encoded_key).Encode(public_key); e == nil {
58     if _, e = server.Write(encoded_key.Bytes()[1:]); e == nil {
59       f()
60     }
61   }
62 }
$ go run 58.go &
[1] 16289
$ go run 57_corrupt_key.go
2016/07/13 09:24:44 unable to decode wrapper: 127.0.0.1:55085
2016/07/13 09:24:44 unable to encrypt server response

One of the cool things about interfaces is that they’re reference types, something we’ll routinely use to decide whether an error has occurred or not. If it has then the interface will contain an error value, and if not the interface itself will be a nil value. This leads to the common code pattern

if _, e := SomeCall(); e != nil {
  log.Println("some error", e)
  // this is our sad path
} else {
  // perform desired actions
}

Our sad path will generally either return it’s own error to the calling function or terminate the program with a call to log.Fatalln() or os.Exit(). Because Go functions allow for multiple return values, their use is accompanied by the convention that the last value returned will be of type error. This convention encourages us to handle errors where they occur rather than bubbling exceptions up the call stack, as would be the case in many languages. As there are rare occasions when exceptions might prove a useful idiom we’ll look at how we can achieve a similar outcome in the next section.

For now though we’re going to tidy this code up a little by using the if…{}…else if {}… construct which thanks to if’s ability to combine an assignment with a test leads to very succinct code.

Example 1.1.59
package main
import "bytes"
import "crypto/rand"
import "crypto/rsa"
import "crypto/sha1"
import "encoding/gob"
import "log"
import . "net"

var HELLO_WORLD = []byte("Hello World")
var RSA_LABEL = []byte("served")

func main() {
  Serve(":1025", func(connection *UDPConn, c *UDPAddr, packet *bytes.Buffer) (n int) {
    var key rsa.PublicKey
    var response []byte

    if e := gob.NewDecoder(packet).Decode(&key); e != nil {
      log.Println("unable to decode wrapper:", c)
    } else if response, e = rsa.EncryptOAEP(sha1.New(), rand.Reader, &key, HELLO_WORLD, RSA_LABEL); e \
!= nil {
      log.Println("unable to encrypt server response")
    } else if n, e = connection.WriteToUDP(response, c); e != nil {
      log.Println("unable to write response to client:", c)
    }
    return
  })
}

func Serve(address string, f func(*UDPConn, *UDPAddr, *bytes.Buffer) int) {
  Launch(address, func(connection *UDPConn) {
    for {
      buffer := make([]byte, 1024)
      if n, client, e := connection.ReadFromUDP(buffer); e == nil {
        go func(c *UDPAddr, b []byte) {
          if n := f(connection, c, bytes.NewBuffer(b)); n != 0 {
            log.Println(n, "bytes written to", c)
          }
        }(client, buffer[:n])
      } else {
        log.Println(address, e.Error())
      }
    }
  })
}

func Launch(address string, f func(*UDPConn)) {
  var connection *UDPConn

  if a, e := ResolveUDPAddr("udp", address); e != nil {
    log.Fatalln("unable to resolve UDP address:", e.Error())
  } else if connection, e = ListenUDP("udp", a); e != nil {
    log.Fatalln("can't open socket for listening:", e.Error())
  }
  f(connection)
}

Whilst I personally find this easier on the eye in many cases, it’s a less common idiom than successive if statements. It’s also important to remember Go’s scoping rules when using this construction as these allow each assignment to introduce new variables local to that if statement’s scope. As such some care should be taken in variable naming to avoid accidental shadowing.

Because error is defined as an interface rather than a concrete type we can declare our own types for error handling and then check for them to specialise error handling behaviour. In the next example we introduce the LaunchError type which complies with the predeclared error interface by implementing its own Error() method.

Example 1.1.60
 1 package main
 2 import "bytes"
 3 import "crypto/rand"
 4 import "crypto/rsa"
 5 import "crypto/sha1"
 6 import "encoding/gob"
 7 import "fmt"
 8 import "log"
 9 import . "net"
10 
11 var HELLO_WORLD = []byte("Hello World")
12 var RSA_LABEL = []byte("served")
13 
14 type LaunchError []interface{}
15 
16 func (l LaunchError) Error() (r string) {
17   if len(l) > 0 {
18     r = fmt.Sprintf(l[0].(string), l[1:]...)
19   }
20   return
21 }
22 
23 func NewLaunchError(format string, v ...interface{}) (l LaunchError) {
24   return LaunchError(append([]interface{}{ format }, v))
25 }
26 
27 func main() {
28   Serve(":1025", func(connection *UDPConn, c *UDPAddr, packet *bytes.Buffer) (n int) {
29     var key rsa.PublicKey
30     var response []byte
31 
32     if e := gob.NewDecoder(packet).Decode(&key); e != nil {
33       log.Println("unable to decode wrapper:", c)
34     } else if response, e = rsa.EncryptOAEP(sha1.New(), rand.Reader, &key, HELLO_WORLD, RSA_LABEL); e \
35 != nil {
36       log.Println("unable to encrypt server response")
37     } else if n, e = connection.WriteToUDP(response, c); e != nil {
38       log.Println("unable to write response to client:", c)
39     }
40     return
41   })
42 }
43 
44 func Serve(address string, f func(*UDPConn, *UDPAddr, *bytes.Buffer) int) {
45   e := Launch(address, func(connection *UDPConn) (e error) {
46     defer func() {
47       if x := recover(); x != nil {
48         e = LaunchError{ "serve failure %v", x }
49       }
50     }()
51     for {
52       buffer := make([]byte, 1024)
53       if n, client, e := connection.ReadFromUDP(buffer); e == nil {
54         go func(c *UDPAddr, b []byte) {
55           if n := f(connection, c, bytes.NewBuffer(b)); n != 0 {
56             log.Println(n, "bytes written to", c)
57           }
58         }(client, buffer[:n])
59       } else {
60         log.Println(address, e.Error())
61       }
62     }
63     return
64   })
65 
66   if e, ok:= e.(LaunchError); ok {
67     log.Fatalln(e.Error())
68   }
69 }
70 
71 func Launch(address string, f func(*UDPConn) error) error {
72   var connection *UDPConn
73 
74   if a, e := ResolveUDPAddr("udp", address); e != nil {
75     return NewLaunchError("unable to resolve UDP address:", e.Error())
76   } else if connection, e = ListenUDP("udp", a); e != nil {
77     return LaunchError{ "can't open socket for listening:", e.Error() }
78   }
79   return f(connection)
80 }

This is quite a complicated example, so let’s take a look at the various changes we’ve made in detail.

17 type LaunchError []interface{}
18 
19 func (l LaunchError) Error() (r string) {
20   if len(l) > 0 {
21     r = fmt.Sprintf(l[0].(string), l[1:]...)
22   }
23   return
24 }
25 
26 func NewLaunchError(format string, v ...interface{}) (l LaunchError) {
27   return LaunchError(append([]interface{}{ format }, v))
28 }

For simplicity we’ve made LaunchError a slice of interface{} values and declared an Error() method which uses the first element in the slice as a format string and successive elements as parameters for fmt.Sprintf(). The NewLaunchError() function is a typical example of a value constructor which we’d export from a package, and later in the code we show both construction this way and using a LaunchError{} literal.

73 func Launch(address string, f func(*UDPConn) error) error {
74   var connection *UDPConn
75 
76   if a, e := ResolveUDPAddr("udp", address); e != nil {
77     return NewLaunchError("unable to resolve UDP address:", e.Error())
78   } else if connection, e = ListenUDP("udp", a); e != nil {
79     return LaunchError{ "can't open socket for listening:", e.Error() }
80   }
81   return f(connection)
82 }

We’ve made another change to Launch() related to our new approach to error handling, which is to propogate these errors back to the caller via a return value - and just for completeness we’re allowing errors to bubble up from the function parameter f as well. This leads to changes in Serve as well.

46 func Serve(address string, f func(*UDPConn, *UDPAddr, *bytes.Buffer) int) {
47   e := Launch(address, func(connection *UDPConn) (e error) {
48     defer func() {
49       if x := recover(); x != nil {
50         e = LaunchError{ "serve failure %v", e }
51       }
52     }()

Here we set a value for e from the error returned by Launch() as well as intercepting any panic raised by the associated closure with defer and instead returning a LaunchError rather than crashing the program. As defer takes a closure the e referenced inside it is the same e as that declared by the function literal func(connection *UDPConn) (e error), a pattern encountered in many existing Go codebases. We then use the returned error value before exiting Serve().

68   if e, ok:= e.(LaunchError); ok {
69     log.Fatalln(e.Error())
70   }

It should come as no surprise that Go provides support for creating error values without our having to define error types, in the form of errors.New and fmt.Errorf. Using these we can remove the LaunchError type and rewrite our program.

Example 1.1.61
 1 package main
 2 import "bytes"
 3 import "crypto/rand"
 4 import "crypto/rsa"
 5 import "crypto/sha1"
 6 import "encoding/gob"
 7 import "errors"
 8 import "fmt"
 9 import "log"
10 import . "net"
11 
12 var HELLO_WORLD = []byte("Hello World")
13 var RSA_LABEL = []byte("served")
14 
15 func main() {
16   Serve(":1025", func(connection *UDPConn, c *UDPAddr, packet *bytes.Buffer) (n int) {
17     var key rsa.PublicKey
18     var response []byte
19 
20     if e := gob.NewDecoder(packet).Decode(&key); e != nil {
21       log.Println("unable to decode wrapper:", c)
22     } else if response, e = rsa.EncryptOAEP(sha1.New(), rand.Reader, &key, HELLO_WORLD, RSA_LABEL); e \
23 != nil {
24       log.Println("unable to encrypt server response")
25     } else if n, e = connection.WriteToUDP(response, c); e != nil {
26       log.Println("unable to write response to client:", c)
27     }
28     return
29   })
30 }
31 
32 func Serve(address string, f func(*UDPConn, *UDPAddr, *bytes.Buffer) int) {
33   e := Launch(address, func(connection *UDPConn) (e error) {
34     defer func() {
35       if x := recover(); x != nil {
36         e = fmt.Errorf("serve failure %v", x)
37       }
38     }()
39     for {
40       buffer := make([]byte, 1024)
41       if n, client, e := connection.ReadFromUDP(buffer); e == nil {
42         go func(c *UDPAddr, b []byte) {
43           if n := f(connection, c, bytes.NewBuffer(b)); n != 0 {
44             log.Println(n, "bytes written to", c)
45           }
46         }(client, buffer[:n])
47       } else {
48         log.Println(address, e.Error())
49       }
50     }
51     return
52   })
53 
54   if e != nil {
55     log.Fatalln(e.Error())
56   }
57 }
58 
59 func Launch(address string, f func(*UDPConn) error) error {
60   var connection *UDPConn
61 
62   if a, e := ResolveUDPAddr("udp", address); e != nil {
63     return fmt.Errorf("unable to resolve UDP address: %v", e)
64   } else if connection, e = ListenUDP("udp", a); e != nil {
65     return errors.New(fmt.Sprintf("can't open socket for listening: %v", e.Error()))
66   }
67   return f(connection)
68 }

Exceptions

If we consider these programs for a minute or two it becomes apparent that propagating our errors this way works very well when we wish to deal with an error immediately, which is usually the case. However there are occasions when an error will need to be propagated through several layers of function calls, and when this is the case there’s a lot of boilerplate involved in intermediate functions in the call stack.

We can do away with much of this by rolling our own lightweight equivalent of exceptions using defer and the panic() and recover() calls. In the next example we’ll do just this, introducing the Exception type which is an interface with error embedded within it. This means that any error value will also be useable as an Exception value.

Example 1.1.62
 1 package main
 2 import "bytes"
 3 import "crypto/rand"
 4 import "crypto/rsa"
 5 import "crypto/sha1"
 6 import "encoding/gob"
 7 import "fmt"
 8 import "log"
 9 import . "net"
10 import "os"
11 
12 var HELLO_WORLD = []byte("Hello World")
13 var RSA_LABEL = []byte("served")
14 
15 type Exception interface {
16   error
17 }
18 
19 func Raise(message string, parameters ...interface{}) {
20   panic(fmt.Errorf(message, parameters...))
21 }
22 
23 func Rescue(f func()) {
24   defer func() {
25     if e := recover(); e != nil {
26       if e, ok := e.(Exception); ok {
27         log.Println("Exception:", e.Error())
28         os.Exit(1)
29       } else {
30         panic(e)
31       }
32     }
33   }()
34 
35   f()
36 }
37 
38 func main() {
39   Serve(":1025", func(connection *UDPConn, c *UDPAddr, packet *bytes.Buffer) (n int) {
40     var key rsa.PublicKey
41     var response []byte
42 
43     Rescue(func() {
44       if e := gob.NewDecoder(packet).Decode(&key); e != nil {
45         Raise("unable to decode wrapper: %v", c)
46       } else if response, e = rsa.EncryptOAEP(sha1.New(), rand.Reader, &key, HELLO_WORLD, RSA_LABEL); \
47 e != nil {
48         Raise("unable to encrypt server response")
49       } else if n, e = connection.WriteToUDP(response, c); e != nil {
50         Raise("unable to write response to client: %v", c)
51       }
52     })
53     return
54   })
55 }
56 
57 func Serve(address string, f func(*UDPConn, *UDPAddr, *bytes.Buffer) int) {
58   Rescue(func() {
59     e := Launch(address, func(connection *UDPConn) (e error) {
60       for {
61         buffer := make([]byte, 1024)
62         if n, client, e := connection.ReadFromUDP(buffer); e == nil {
63           go func(c *UDPAddr, b []byte) {
64             if n := f(connection, c, bytes.NewBuffer(b)); n != 0 {
65               log.Println(n, "bytes written to", c)
66             }
67           }(client, buffer[:n])
68         } else {
69           log.Println(address, e.Error())
70         }
71       }
72       return
73     })
74 
75     if e != nil {
76       log.Fatalln(e.Error())
77     }
78   })
79 }
80 
81 func Launch(address string, f func(*UDPConn) error) error {
82   var connection *UDPConn
83 
84   if a, e := ResolveUDPAddr("udp", address); e != nil {
85     Raise("unable to resolve UDP address: %v", e)
86   } else if connection, e = ListenUDP("udp", a); e != nil {
87     Raise("can't open socket for listening: %v", e.Error())
88   }
89   return f(connection)
90 }

Our updated code looks surprisingly similar to that of Example 1.60 with a set of type declarations replacing LaunchError with Exception and NewLaunchError() with Raise(), which as its name suggests generates a panic to propagate the Exception value back up the calling stack. We’ve also introduced Rescue()

22 func Raise(message string, parameters ...interface{}) {
23   panic(fmt.Errorf(message, parameters...))
24 }

To intercept this panic we need a defer statement somewhere up the call stack which can handle it, otherwise it will bubble up through main() and the program will terminate with a stack trace. In our implementation of Rescue() we set up a deferred function which will check panic values with a type assertion and where these match the Exception interface the program will be terminated cleanly.

26 func Rescue(f func()) {
27   defer func() {
28     if e := recover(); e != nil {
29       if e, ok := e.(Exception); ok {
30         log.Println("Exception:", e.Error())
31         os.Exit(1)
32       } else {
33         panic(e)
34       }
35     }
36   }()
37 
38   f()
39 }
$ go run 62.go &
[1] 53769
$ go run 62.go
2016/07/13 17:39:33 Exception: can't open socket for listening: listen udp :1025: bind: address alread\
y in use
exit status 1
$ go run 57_corrupt_key.go 
2016/07/13 17:39:37 Exception: unable to decode wrapper: 127.0.0.1:49325
exit status 1
^Csignal: interrupt
[1]+  Exit 1                  go run 62.go

The semantics here are subtly different to our previous example and any error in the server will cause it to terminate. Instead we want launch errors to terminate the server whilst connection errors from talking with a particular client should log the error and return to listening for another connection. The easiest way to do this is to parameterise Rescue() so that it receives both the function to guard and a function with signature func(Exception) to respond according to the Exception value generated.

[Example 1.1.63a] Rescue() function definition
26 func Rescue(f func(), r func(Exception)) {
27   defer func() {
28     if e := recover(); e != nil {
29       if e, ok := e.(Exception); ok {
30         r(e)
31       } else {
32         panic(e)
33       }
34     }
35   }()
36 
37   f()
38 }
[Example 1.1.63b] Rescue() function in use
67       Rescue(
68         func() {
69           buffer := make([]byte, 1024)
70           if n, client, e := connection.ReadFromUDP(buffer); e == nil {
71             go func(c *UDPAddr, b []byte) {
72               if n := f(connection, c, bytes.NewBuffer(b)); n != 0 {
73                 log.Println(n, "bytes written to", c)
74               }
75             }(client, buffer[:n])
76           } else {
77             Raise("%v: %v", address, e.Error())
78           }
79         },
80         func(e Exception) {
81           log.Println(e.Error())
82         },
83       )
[Example 1.1.63c] Launch() rewritten to use Rescue()
67 func Launch(address string, f func(*UDPConn) error) {
68   var connection *UDPConn
69 
70   Rescue(
71     func() {
72       if a, e := ResolveUDPAddr("udp", address); e != nil {
73         Raise("unable to resolve UDP address: %v", e)
74       } else if connection, e = ListenUDP("udp", a); e != nil {
75         Raise("can't open socket for listening: %v", e)
76       } else if e = f(connection); e != nil {
77         Raise("connection error: %v", e)
78       }
79     },
80     func(e Exception) {
81       log.Println(e.Error())
82       os.Exit(1)
83     },
84   )
85 }

This now gives us a primitive domain specific language for exceptions using Rescue() blocks to guard behaviour and Raise() to trigger them. Unfortunately an Exception is essentially untyped in the context where it’s dealt with and this places all the burden for exception handling into a single closure. However generally languages which support exception handling allow exceptions to be differentiated by subtype, so let’s see what we can do to achieve a similar effect.

The obvious first step is to introduce a new type which implements the Exception interface and then use a type switch in an exception handler to specialise its behaviour

[Example 1.1.64a] defining an Exception type
26 type LaunchException error
27 
28 func RaiseLaunchException(message string, parameters ...interface{}) {
29   panic(LaunchException(fmt.Errorf(message, parameters...)))
30 }
[Example 1.1.64b] checking for a specific exception
 94 func Launch(address string, f func(*UDPConn) error) {
 95   var connection *UDPConn
 96 
 97   Rescue(
 98     func() {
 99       if a, e := ResolveUDPAddr("udp", address); e != nil {
100         RaiseLaunchException("unable to resolve UDP address: %v", e)
101       } else if connection, e = ListenUDP("udp", a); e != nil {
102         RaiseLaunchException("can't open socket for listening: %v", e)
103       } else if e = f(connection); e != nil {
104         Raise("connection error: %v", e)
105       }
106     },
107     func(e Exception) {
108       switch e := e.(type) {
109       case LaunchException:
110         log.Println("Launch Exception:", e.Error())
111       default:
112         log.Println(e.Error())
113       }
114       os.Exit(1)
115     },
116   )
117 }

The type switch allows us to select different courses of action depending on the concrete type of the value contained in the Exception, and we can also specify a default case to handle unknown values.

$ go run 64.go &
[1] 90620
$ go run 64.go
2016/07/16 00:03:17 Launch Exception can't open socket for listening: listen udp :1025: bind: address \
already in use
exit status 1
$ go run 57.go
2016/07/16 01:00:53 256 bytes written to 127.0.0.1:63574
Hello World
$ go run 57_corrupt_key.go 
2016/07/16 01:00:58 Exception: unable to decode wrapper: 127.0.0.1:54657
^Csignal: interrupt
$ go run 57.go
2016/07/16 01:02:09 256 bytes written to 127.0.0.1:52033
Hello World

The use of type switches in this manner is a common idiom in go and I quite like this solution as the only magic at work here is our abuse of the panic()/recover() mechanism. However I’d really prefer to split exception handling into several different functions, each keyed to a particular exception type. This is relatively easy to do but requires that we pop open go’s hood at runtime using type reflection.

Like many modern languages Go has an extensive reflection system which allows us to introspect any value at runtime - including the closures which our Rescue() function accepts - and then attempt to use that information to control the execution of our program.

[Example 1.1.65] checking for a specific exception
45 func Rescue(f func(), r ...interface{}) {
46   defer func() {
47     if e := recover(); e != nil {
48       if e, ok := e.(Exception); ok {
49         et := reflect.TypeOf(e)
50         for _, handler := range r {
51           if h := reflect.ValueOf(handler); h.Kind() == reflect.Func && h.Type().NumIn() == 1 {
52             switch hpt := h.Type().In(0); {
53             case et == hpt:
54               fallthrough
55             case hpt.Kind() == reflect.Interface && et.Implements(hpt):
56               h.Call([]reflect.Value{ reflect.ValueOf(e) })
57               return
58             }
59           }
60         }
61       }
62       panic(e)
63     }
64   }()
65 
66   f()
67 }

This seems like a pretty gnarly piece of code on first inspection so let’s rephrase it in English to get a better understanding for what’s it’s doing before looking at the details of the reflect API

47 given that we've recovered from a panic()
48   if we're handling any value whose type fulfils the Exception interface
49     determine the concrete type of the Exception value
50     step through the list of handlers specified in the Rescue() call
51       if the handler is a function and accepts exactly one parameter
52         find out what the type of its parameter is
53         if the parameter is the same type as our exception's concrete type
54         or the parameter is an interface which the exception implements
55           call the handler with the exception as its parameter
56           return from the defered function, continuing execution normally
57   in all other cases
58     propagate the value received by recover() up the unwinding call stack

This essentially boils down to finding out the runtime types of the values our function encounters and then making choices about how to proceed. To achieve this we’ve used two functions central to working with reflection. Firstly there’s reflect.TypeOf() which takes any value and returns a reflect.Type value, then there’s reflect.ValueOf() which also takes any concrete value and returns a reflect.Value.

Both Type and Value expose large APIs designed to operate on all runtime types and as a consequence many of these methods are prone to generating runtime panics if used incorrectly. Both types implement a Kind() method which can be used to provide basic safeguards

51           if h := reflect.ValueOf(handler); h.Kind() == reflect.Func && h.Type().NumIn() == 1 {
52             switch hpt := h.Type().In(0); {
53             case et == hpt:
54               fallthrough
55             case hpt.Kind() == reflect.Interface && et.Implements(hpt):
56               h.Call([]reflect.Value{ reflect.ValueOf(e) })
57               return
58             }
59           }

With type safety guarantees in place we can easily figure out whether or not we’re dealing with a recognisable exception handler (i.e. a function taking a parameter compatible with the Exception interface) and if we are the next question is how do we invoke this function? The reflect.Value type defines a method Call() which executes a function and takes as its parameter a []reflect.Value containing the actual parameters for the function, each wrapped as a reflect.Value

56               h.Call([]reflect.Value{ reflect.ValueOf(e) })

For completeness we’re going to refactor Rescue() by separating out the code for attempting an exception handling function call, making this easier to maintain

[Example 1.1.66] calling an exception handler via reflection
33 func attemptCall(e Exception, handler interface{}) (ok bool) {
34   if h := reflect.ValueOf(handler); h.Kind() == reflect.Func {
35     et := reflect.TypeOf(e)
36     if hpt := h.Type().In(0); et == hpt || et.Implements(hpt) {
37       h.Call([]reflect.Value{reflect.ValueOf(e)})
38       return true
39     }
40   }
41   return
42 }
43 
44 func Rescue(f func(), r ...interface{}) {
45   defer func() {
46     if e := recover(); e != nil {
47       if e, ok := e.(Exception); ok {
48         for _, h := range r {
49           if attemptCall(e, h) {
50             return
51           }
52         }
53       }
54       panic(e)
55     }
56   }()
57 
58   f()
59 }

Echo

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Arguments

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Flags

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Command-line Boilerplate and Standard I/O

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Conditional Flags

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Errors

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Going Loopy

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Adventures in Iteration

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Linear Sequences

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

The for {} construct

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

User Defined Slices

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Iterating Through Arrays

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Arrays and Slices Exposed

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Iteration and structured types

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

mappings

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Iteration and maps

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Software Machines

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Software Machines

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

array stacks

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

cactus stacks

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

hash maps

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

heaps

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

switch dispatchers

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

direct threaded dispatchers

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

indirect threaded dispatchers

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

assembler

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

tail calls

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

architectures

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

fun with types

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

timers

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Instruction Set

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

processor core

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

accumulator machine

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

stack machine

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

register machine

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

vector machine

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Software Machines

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

memory

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Functional Programming

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Metaprogramming and First-Class Functions

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Pure Functions, Expressions, and Recursion

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Partial Application and Currying

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Lazy Evaluation and Memoization

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Functions

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

The Machine View

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Adding Human Readability

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Procedures and Functions in Go

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

The Mathematical View

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Simple Factorials

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

First-Class and Higher-Order Functions

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Closures

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Currying

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Recursion

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

funcs() Which Call Themselves

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Mathematical Functions Which Call Themselves

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Error Handling the Go Way

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Changing Types

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Odds & Sods

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Maps and Hashes

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Go maps

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

A simple Map implementation

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Types

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Interfaces

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Interfaces, pt 1

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

package adder

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Pretty Pictures

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Basic Mandelbrot

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Phong Shading

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Communication by Sharing

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Concurrency

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

synchronous

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

asynchronous

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

map/reduce

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

map/reduce

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Errors, Exceptions & Flow Control

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Catch & Throw

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Stack Traces

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Exceptions

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Duck Typing, Reflection and Type Manipulation

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

package generalise

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

raw

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Beyond Go

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Interfacing with Dynamic Libraries

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

SQLite 3

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.

Ruby?

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/GoNotebook.