Preface

It was a cold, dark night at the tail-end of 1980 and I was attending an open evening for the Grammar School I’d later spend seven long, slow years attending when I first met a programmable computer, one of a series of steel-framed rack-mount monstrosities with trailing ribbon cables and large clunky floppy drives that would have looked equally at home in a Soviet space capsule or a Quatermass-era germ warfare lab. These were clustered in a long, narrow, beige room, its styrofoam tiles lit by anaemic fluorescent tubes and dominated by an ageing teletype machine, in the midst of one of those asbestos prefab 1960s science blocks so popular with the British educational establishment. The white heat of technology aged much faster than the aviation industry it was meant to replace.

This certainly wasn’t the first time I’d seen a computer. That honour probably belongs to Tomorrow’s World, the BBC’s flagship futurology show. My first clear memory though is from a couple of years earlier when I found a book on technology in my local library with photos of the LEO 1, describing how these electronic machines could ‘think’ and ‘solve problems’. This was a common theme of the 1970s with sentient computers in TV shows like Blake’s 7 and movies such as Futureworld and Logan’s Run.

Experience has made me all too painfully aware that it’s actually programmers who do the thinking and problem solving, not the machines, but to the yet unsullied mind of an eight year old proto-boffin the idea that a machine could ‘think’ was fascinating - and as inscrutable as the collection of disassembled transister radios and alarm clocks cluttering my bedroom draws. A few months later a family friend with an interest in technology gave me a current issue of Personal Computer World which made me realise that computers were something real people might one day play with at home, and not just the heroes of my favourite sci-fi shows. From then on anything I could find which mentioned computers was read cover to cover until the pages were dog-eared and yellowed.

However it was that night in 1980 which looms large in my memory. That night which set the course of my life.

The following year I passed my 11+ - the only time incidentally that I’ve ever aced an exam - and joined the first year intake. When two years later I received my first computer as a Christmas present I’d already taught myself rudimentary spaghetti-code programming in out-of-hours sessions on Research Machines 380Zs, typing in program listings from home micro magazines and hobbyist-oriented books in one of the plethora of semi-compatible BASIC dialects, then hacking them mostly into working shape.

When I was dependent on the school machines I was lucky if I’d get two hours of keyboard time in a week, and that competed with my growing interest in polydice and pen-and-paper RPGs. Having a computer of my own changed all that, the slim off-white Oric-1 plugged into an old disused black-and-white television in our spare room, and my relationship with code became much more… nuanced; programming emerging as the steady back-beat to my teenage life, something to do when not buried in homework, hanging out with friends or working on my Elite rating.

By the time I moved on to polytechnic I’d become a reasonably competent hacker, skilled in BASIC and latterly obsessed with FORTH. I’d even dabbled a little in Z80 assembler which back then was seen as a fairly natural thing for a teen into computers to mess with.

You’d probably expect me to have gone on to a Comp Sci degree with my love for coding but from what I could see that didn’t seem to be a big part of the curriculum, and anyway Comp Sci already had a reputation for social death. Really, who doesn’t want to be invited to parties?

Me apparently! Instead I spent the next four years blagging my way through an Applied Physics degree during daylight hours and sitting up half the night with hacker friends, writing stupid little programs in odd languages and hanging out on machines in distant places - much to the despair of my long-suffering tutors who thought I should be putting the same effort into numeric methods, FORTRAN and Modula-2. This was the pre-Web era when words like spam and worm were first being coined and most of the fun to be had online involved MUDs, telnet, X.25 pads, pdsoft archives, sneaker-net warez, and my good old friend JANET.

Of course I owned the obligatory mirror shades, a walkman, a clutch of goth and post-punk albums, an increasingly battered copy of Neuromancer, and a “super-slim” laptop with lead-acid battery. As a regular nethead I ran under a clutch of pseudonyms on alt.cyberpunk and alt.gothic, had an enviable arsenal of 1337 VMS hacks and various issues of 2600 and Mondo 2000. As I imagined a career building railguns for the Strategic Defence Initiative and autonomous robots with itchy little instinct-driven digital limbic systems I styled myself the heroine of Timbuk3’s ironic paean to misplaced scientific ambition The Future’s So Bright, I Gotta Wear Shades, a poster-girl surfer of The Third Wave. Kelly le Brock would obviously play me in the movie or maybe Terri Garber if it were made for television…

Don’t judge me or my generation too harshly: the late 80s were a strangely naive time as the looming shadows of the Cold War receded and our friends electric had yet to acquire the more sinister overtones of the mass surveillance society we live in today. It was still possible to dream of building artificial humans without immediately assuming their purpose would be violence and oppression, and Michael J Fox could rescue a multinational armed with nothing more than a winning smile, an MBA and an IBM PC. We had C dammit. C! And soon we’d have Java - at least until The Enterprise got their hands on it.

This book is my homage to those hacker years when PCs were still exciting just for being PCs and to the anarchic culture which throve in computer labs at certain campuses across the UK and US, the Trans-Atlantic distance already dissolving under the addictive social influence of USENET and MUDs. As such it’s not a professional book in any usual sense of the term, although I certainly hope the material within will be of interest to professionals and useful to them in solving real world problems using Go.

We won’t be concerning ourselves much with idiomatic style or community standards, both of which are already treated much better in other publications than I could hope to do. Nor will we spend much time on syntax save as an incidental.

However to reassure the more sober amongst you that this book does indeed have some intrinsic technical merit, I should probably mention that much of the material is drawn from a series of presentations and workshops I’ve delivered at well-respected software development conferences starting in 2009 and continuing to the present day as I’ve sought to share my passion for Go with a wider community.

The main advantage of reworking this material in book format is that I get to explore things at a more leisurely pace than with a live audience, and I hope in doing so to also recapture some of the anarchic can-do feel of those early books and magazine articles which first inspired me as a teenager. So what I’m really looking to create here is the kind of book about programming in Go that if my untutored younger self had found a copy, it would have kept her enthralled for months or even years. In some sense it’s therefore an attempt at a “write your own adventure” programming course - inspired perhaps by that marvellous book in Neal Stephenson’s The Diamond Age - which takes it for granted that even the casual reader can learn the full measure of Go without any other help.

I mention this because there are many clever, talented people in this world who’d really enjoy programming if only they thought they were capable. But somehow they’ve been convinced that programming is difficult. Well yes, it often is in the same way that writing a story or building with LEGO bricks or baking a cake can be difficult. But not for the bogus reasons propagated in media depictions of allegedly typical programmers!

You don’t need to be good at maths (I flunked calculus repeatedly), a straight-A student (my report card perennially read “could do better!”) or look anything like a stereotypical nerd (I’m an ex-goth turned yummy mummy). What matters is being interested in making things, and having the perseverance to keep going when things aren’t quite working how you imagine. Just like any other creative endeavour - whether that’s fixing up your house, icing a cake, repairing an engine, or writing a novel.

There will be some math and some theory along the way and I don’t apologise for this. Not only is programming a fun pastime in its own right it also provides a window on maths, science, engineering and social interactions. I can’t say in advance how deeply we’ll delve into any of these subjects and I reserve the right to follow wherever inspiration leads when it offers the chance to explore interesting programs.

So even if you’re looking at this book somewhat nervously after thumbing through the sample chapter, and worrying that perhaps you’re not the kind of person it’s aimed at, please do yourself the favour of setting aside all your preconceptions and at least playing with Go for a few days. It may feel tough at first but trust me, you’re most decidedly who I’m writing for, even if I’ve been too close to the machine for so many years that I haven’t much clue how to write for you. I guarantee that if you’re willing to invest some time in playing with my code and reading around the various topics I introduce that you’ll get at least as much from this book as those with CS degrees - indeed possible even as much as you’d get from studying a CS degree in the first place as they’re woefully light on practical application.

Why have I decided to focus this book on Go, and not one of the conventionally recognised beginner-friendly languages? That’s a good question. To start with I refute the proposition that Go is a difficult language. It’s one of the smallest languages I know of for general-purpose programming and definitely the smallest in the C-like family of systems languages. Despite that it features a number of powerful concepts like first-class functions, concurrency, and type inference which make coding a pleasure. So using Go to teach programming seems like a pretty reasonable choice.

But I’ve also made this decision for personal reasons.

For years I was best known for my avant garde talks on Ruby, a dynamic language which is both a great joy to work with and also commercially much in demand. If I were writing a conventional programming book to address an established audience then that’s where I’d be most comfortable claiming expertise, but frankly there are enough good books about Ruby already (and a pretty large catalogue of bad ones for that matter) so I’ve nothing to add to that genre.

And whilst I love Ruby to bits, that affection is tempered by the culture which has grown up in response to its commercial success: design patterns; test-driven development; SCRUM; Kanban; continuous deployment; best practices. These are all useful things to know if you’re working in a large team at a major enterprise with all that entails, but that’s a different passion to the one I feel. And frankly that’s not how I believe new programmers get the bug for coding, any more than mechanics get it from reading service manuals and measuring tolerances. Passion starts with play and play is the aim of this book. It’s a hacker’s book not an engineer’s book.

Go has been an immense source of untarnished joy for me since I first encountered it, fresh off a flight from a Polish tech conference in November 2009. At the time it was still a little ropey around the edges with makefiles and all kinds of odd limitations, but it was already the easiest way to write efficient concurrent programs in a C-like language and I admit I fell immediately in love.

It’s that love that’s sustained me through almost a decade of Go adventures and that’s led me to talk about it publicly at every opportunity, ligging my way onto the bill of some of the most highly regarded software development conferences in Europe and the United States, and ultimately leading me to write this book.

For this reason I’ve started with a large body of example code drawn from those conference sessions and very little text beyond this preface. In the coming months and years I’ll be adding additional code organically as the opportunity arises or in response to readers’ queries, and shaping a narrative to fit. There isn’t a publication schedule, and I do my own editing.

This Dear Reader is where you come in. I want this book to be interesting and quirky and packed with useful content so please let me know if there’s anything which doesn’t make sense to you, or offer any ideas you have for new directions. I can’t promise that every new idea will be used in quite the way you envisage but I’ll do my best to turn each into interesting code examples so we can all learn more about Go. And trust me, I’ll be learning just as much as you will because programming isn’t one of those disciplines where we ingest a fixed set of facts and then we’re done. This is the knowledge economy we’re messing with here which means millions of creative minds are busily rewriting everything all the time, sometimes for good reason but often just because it amuses them to do so.

Sadly there’ll be long gaps where very little gets added to the text because I’ve had to take on a commercial project to pay my bills with all that entails: like most people in their mid-40s I have a family to feed, a mortgage to pay, and a never-ending hardware habit to fund. I hope you’ll bear with me when these stalls happen. The longest to date was a little over two years and I felt a little guilty the entire time.

To make up for my somewhat peripatetic approach I intend this to be the only book I ever write on Go and that you need only purchase the electronic edition once to be covered for all time. This doesn’t include any print editions which I may agree to publish but that’s not an ambition at the time of writing.

For those on tight budgets or who fancy an ironman challenge, I’ve designed the free tutorial chapter Hello World so that it should be sufficient to learn Go without any outside assistance beyond the online documentation and some experimentation. There probably isn’t a better free resource on networking and cryptography in Go for the simple reason that there’s good money to be made teaching that stuff. If you’re coming from a C or Java background that should literally be all you need to bootstrap yourself into the language.

All source code will eventually be available online though I highly recommend manually typing it in as you work your way through the book. It’s easy to forget in an age of source control and GUI tools that coding is first and foremost a text-based process and there’s value in developing muscle memory whenever we get the chance. Code is also literature of a sort - a strange amalgam of poetry and contract negotiation like something from the dawn of writing - so there’s also great practical benefit in learning to read code. There are no short cuts to the insight that provides.

Now give me your hand and together let’s enjoy some amazing adventures in the land of Go.

Ellie

London, 2018

Introducing Go

In this section we’re going to explore Go through a series of increasingly complex programs which we can interact with either through the command line or across a network.

Getting Started

If you’re of an impatient nature (my teenage self certainly was) temper that enthusiasm for a second and read Google’s install docs to get your Go install up-and-running. You might also like to install LiteIDE which is an excellent cross-platform development environment for Go which also happens to be written in Go.

This book is laid out as a series of parts focused on different topics I’m interested in and the order you read it in is up to you with one clear exception: Hello World is designed as a quick tutorial introduction to Go so if you don’t know the language that’s the place to start. Even if you do know Go you’ll probably find other stuff in Hello World which will be useful as it covers network communications and cryptography.

Now Go have fun messing with my code.

So what is Go?

Well obviously it’s a programming language, which means an artificial language intended for humans to read and write but which can be usefully translated into the 1s and 0s understood by digital computers.

But is it a good programming language?

Opinion on the internet has been bitterly divided on this question (which is hardly unusual) since Go was released in November 2009 and the language has many detractors who either see it as deficient in some crucial feature without which efficient programs can’t possibly be productively developed, or else as a wholehearted nostalgia trip back into the 1980s by a team who’ve failed to move with the times.

So let’s look at the reality and see if these criticisms are justified.

Open source

Go’s developed as an open source project so anyone with an interest can read through its source code, make modifications for their own purposes, or get involved in its future development. However it’s also a carefully curated project and unlike many popular languages there’s a tightly defined language specification which is intended to be read and understood by anyone working with Go.

These are pragmatic choices. Not only do many eyes make light work of code quality, a clear and readable language specification makes it very clear what Go is and isn’t trying to achieve.

Efficient compilation

Work started on the language at Google in 2007 as a way to address three key problems in systems programming.

First there’s efficient compilation, the process of taking the source code for a program and turning it into an executable form with a minimum of fuss. As increasingly complex programs are developed in traditional systems languages like C and C++ they become dependent on equally complex build tools and compilation times can stretch into minutes or hours.

This isn’t a particularly new phenomenon. The systems I was working on in the 90s often had compile times in the order of 10 to 30 minutes so builds were a great excuse to make a cup of tea whilst waiting to see if things compiled successfully. Still there’s only so many cups of tea you can make in a day and maintain any semblance of productivity.

Sadly one of the main reasons for the popularity of interpreted languages like Ruby and Python (both of which have other much more compelling reasons to explore) is the interactive nature of developing with them. This basically allows devs to embrace the same smugness the Lisp crowd exude without the intellectual snobbery or taste the pioneering spirit of Forth without the serious WTF? factor when rereading old code.

In interpreted languages small chunks of code can be developed independently of each other and tested immediately. This fast feedback cycle makes it much easier to explore a problem domain and figure out how to work with it effectively (or if we’re lucky elegantly). However interpreted languages have a reputation for being memory hungry and slow.

Efficient execution

This leads us to the second problem Go addresses: efficient execution. Systems programming is all about programs which other programs are going to rely on for key services so it’s highly desirable that languages addressing these problems produce programs which run in a timely fashion and with predictable memory overhead.

Compilation provides a huge boost to runtime performance compared to interpretation and the standard Go compiler features a range of optimisations which make it roughly competitive with C++ or Java. And because Go was developed with multi-core processors in mind it uses their resources effectively.

One area which raised concerns when Go was first released is its use of garbage collection rather than traditional programmer-controlled memory allocation. Garbage collection used to be a significant cost with 10ms set aside from every 50ms of program execution but in recent years there have been huge improvements by switching to a concurrent design and tailoring this to other features of Go’s design.

Ease of programming

The majority of systems languages were designed at a time when memory was limited and processors could generally only execute one instruction at a time (and that glacially slowly by modern standards). As a result these languages provide programmers with very primitive abstractions which no longer even vaguely resemble what’s really going on in most general purpose computers, and to make matters worse they often entail a lot of careful book-keeping just to track memory allocations and prevent leaks which will otherwise cause systems to become unstable.

Go was developed after these changes took place by a team with a long history of working on systems software and embraces programmer-friendly conveniences such as garbage collection and CSP concurrency which ease much of this burden. Its design also includes other features which aid ease of programming such as first-class functions, slices, hash maps and structural typing which can greatly simplify code analysis and reuse.

We’re going to have a lot of fun with all these features throughout this book.

And where did it come from?

Originally Go was developed in-house at Google, a company which thanks to its position in web search and online advertising is heavily tech driven. Google’s systems operate at ROFLscale - a scale which most of us can only dream of messing with - and its codebases measure multiple BLOC (that’s billion lines of code). The aspirations at the heart of Go’s design are essentially attempts to reign in the complexity of numerous codebases scattered throughout Google’s data-centres and written in C++, Python, and Java.

The initial development team comprised three industry veterans with a record of both innovation and successful commercial implementation: Ken Thompson, Rob Pike, and Robert Griesemer.

Ken Thompson is a legendary hacker, the original creator of UNIX whose B programming language was a strong influence on the design of hacker favourite C. Rob Pike meanwhile was deeply involved with the development of the UNIX, Plan 9 (along with Ken) and Inferno operating systems as well as the Dis virtual machine and a number of programming languages focused on concurrency. Robert Greisemer cut his teeth working on Java’s HotSpot VM and Google’s V8 JavaScript runtime, so between the three of them there’s a wealth of experience in systems software design and performance engineering.

At first glance Go is similar to another of Rob’s languages Limbo which he created for the Inferno operating system and it features the CSP approach to concurrency he and Ken had been playing with since their UNIX days. CSP (or Communicating Sequential Processes) was formally defined by British computer scientist Tony Hoare in the late 70s and was key to the technical success of the multi-processor InMos Transputer machines of the 1980s (though it’s programming language Occam was pretty nasty to work with).

However Go is a much smaller language than Limbo thanks to a key decision made at the start of the project: no suggested feature would be included in Go unless all three of the core team agreed it should be included. This makes for a highly conservative language design focused on must have features to achieve its stated design goals rather than nice to have features which might be at odds with them. This is quite a departure from the usual feature creep and special pleading seen in other languages.

Into the wild

When Go was first announced to the world on November 10th 2009 it was already a robust language but with a few rough edges like the use of make files for project builds, a weekly source release cycle which often saw package library APIs changing, and some fairly fiddly configuration via environment variables.

However key features like the language specification, structural typing, garbage collection, and goroutines were already fixed and a community of passionate users started to grow up around the language. Some of these came from the bifurcating Python scene in search of stability and static typing, others like me had need for a systems language but wanted to keep many of the free-wheeling conveniences we enjoyed in higher-level languages. Then there were the C/C++ and Java devs initially attracted by the reputations of Go’s designers who then stuck around for the fun they could have.

Looking back I think all three groups were pulling not only in different directions to each other, but also in different directions to the core team and it’s a testament to the flexibility of Go that a dedicated Rubyist like me can be as comfortable working with it as the most static-typing obsessed escapee from Python or generics-missing Java head.

The early buzz around Go wasn’t nearly as loud as TIOBE ratings suggested and the language would probably have remained a curiosity even with Google’s backing if it hadn’t been for the widespread adoption of Docker by the DevOps community.

Docker popularised containers, a standardised way of installing and running one or more lightweight virtual machines on a server. These virtual machines aren’t to be confused with the kind used by languages like Java or Erlang (and which form the meat of the software machines section of this book) but are better understood as an outgrowth of UNIX chroot jails based on the resource isolation features present in the Linux kernel. This allows one server to run multiple independent environments each with well-defined privileges and managed resources without requiring the overhead associated with more traditional virtualisation hypervisors like Xen or VMWare.

Google themselves are no strangers to containerisation and in 2015 released Kubernetes, an orchestration system for deploying and managing containers across large-scale clusters inspired by a previous in-house tool named Borg. Kubernetes is written in Go and in combination with Docker has helped simplify the once black art of microservice design. Not only are these powerful tools, their use of Go makes them great advertisements for the language.

Where next?

At the time of writing Go has been in the public eye for nine years. The release of version 1 in March 2012 came with the promise that there’d be no breaking language changes until a version 2 release and since version 1.1 development has followed a tight release schedule focused primarily on improving runtime performance and the standard library. Good software engineering practices aren’t particularly glamorous but the end results are impressive as the garbage collector and goroutine scheduler demonstrate.

It’s still far from clear when version 2 of Go will materialise or the extent of any changes to the language which will accompany it. One much-anticipated addition is some kind of support for generic types, the absence of which has been controversial since Go was first released. Generics are one of a number of different ways to enable compile-time metaprogramming and are popular in a number of languages such as C++ and Java, however they generally make the compilation process more complex and trade compile-time cost for runtime efficiency.

Go already allows metaprogramming at runtime via its reflection and unsafe APIs with minimal impact on compilation speeds but there is a runtime performance hit for any significant use of reflection which starts to make higher-level interpreted languages look competitive. And thanks to structural typing via interfaces it’s possible to write reasonably generic code based upon capability, though yet again there are runtime costs associated with this approach.

Personally I’d like Go version 2 to introduce runtime optimisations for both reflection and structural typing - maybe with some form of JIT compilation. This might well be possible for type switches by using call-site caching and making changes to the reflection API inspired by virtual machine designs.

There are also a number of active projects focused on richer compile-time metaprogramming using everything from code templating (the C++ approach) to Lisp-inspired macros. These all involve code generation and subsequent compilation and it would be interesting to see direct compile-time metaprogramming integrated into the language specification.

Then there’s enumeration via range statements which could be opened up to a wider selection of user-defined types. Conceptually this could be simple enough: add an interface to the standard library which yields successive elements and allow the compiler to spot this is implemented for a given type. But it’s hard to see how this approach wouldn’t get pretty ugly pretty fast.

A possible first step might be to add nodes and lists to the core types defined in the language specification as a analogous abstraction arrays & slices so that at least singly-linked and doubly-linked lists could be used in range statements.

Another area where Go could be improved is for mobile development. There’s quite a bit of work that’s gone on in this area over the past few years but Go still isn’t a first-class language on Android or iOS so it would be good to see something official in this direction.

Things are much further along with desktop applications such as LiteIDE which uses Qt5 bindings for its cross-platform GUI whilst TinyGo is bringing a decent subset of the language to minimalist embedded devices like the BBC micro:bit. Likewise GopherJS has pioneered Go in the web browser whilst Go version 1.11 now includes experimental support for WebAssembly. There are even bindings for the Vulkan graphics API.

So hopefully the version 2 release will make Go a credible language to use all the way from deep embedded systems right the way through to web browser front-ends. Who knows, perhaps we’ll even see some graphically intensive games being developed in Go.

Hello World

It’s a tradition in programming books to start with a canonical “Hello World” example and whilst I’ve never felt the usual presentation is particularly enlightening, I know we can spice things up a little to provide useful insights into how we write Go programs.

Let’s begin with the simplest Go program that will output text to the console.

1 package main
2 
3 func main() {
4   println("hello world")
5 }

The first thing to note is that every Go source file belongs to a package, with the main package defining an executable program whilst all other packages represent libraries.

1 package main

For the main package to be executable it needs to include a main() function, which will be called following program initialisation.

3 func main() {

Notice that unlike C/C++ the main() function neither takes parameters nor has a return value. Whenever a program should interact with command-line parameters or return a value on termination these tasks are handled using functions in the standard package library. We’ll examine command-line parameters when developing Echo in the next chapter.

Finally let’s look at our payload.

4 println("hello world")

The println() function is one of a small set of builtin generic functions defined in the language specification and which in this case is usually used to assist debugging, whilst “hello world” is a value comprising an immutable string of characters in utf-8 format.

We can now run our program from the command-line (Terminal on MacOS X or Command Prompt on Windows) with the command

$ go run 01.go
hello world

Packages

Now we’re going to apply a technique which I plan to use throughout this book by taking this simple task and developing increasingly complex ways of expressing it in Go. This runs counter to how experienced programmers usually develop code but I feel this makes for a very effective way to introduce features of Go in rapid succession and have used it with some success during presentations and workshops.

There are a number of ways we can artificially complicate our hello world example and by the time we’ve finished I hope to have demonstrated all the features you can expect to see in the global scope of a Go package. Our first change is to remove the builtin println() function and replace it with something intended for production code.

1 package main
2 import "fmt"
3 
4 func main() {
5   fmt.Println("hello world")
6 }

The structure of our program remains essentially the same, but we’ve introduced two new features.

2 import "fmt"

The import statement is a reference to the fmt package, one of many packages defined in Go’s standard runtime library. A package is a library which provides a group of related functions and data types we can use in our programs. In this case fmt provides functions and types associated with formatting text for printing and displaying it on a console or in the command shell.

5 fmt.Println("hello world")

One of the functions provided by fmt is Println() which takes one or more parameters and prints them to the console with a carriage return appended. Go assumes that any identifier starting with a capital letter is part of the public interface of a package whilst identifiers starting with any other letter or symbol are private to the package.

In production code we might choose to simplify matters a little by importing the fmt namespace into the namespace of the current source file, which requires we change our import statement.

2 import . "fmt"

And this consequently allows the explicit package reference to be removed from the Println() function call.

5 Println("hello world")

In this case we notice little gain however in later examples we’ll use this feature extensively to keep our code legible.

1 package main
2 import . "fmt"
3 
4 func main() {
5   Println("hello world")
6 }

One aspect of imports that we’ve not yet looked at is Go’s builtin support for code hosted on a variety of popular social code-sharing sites such as GitHub and Google Code. Don’t worry, we’ll get to this in later chapters.

Constants

A significant proportion of Go codebases feature identifiers whose values will not change during the runtime execution of a program and our hello world example is no different, so we’re going to factor these out.

1 package main
2 import . "fmt"
3 
4 const Hello = "hello"
5 const world = "world"
6 
7 func main() {
8   Println(Hello, world)
9 }

Here we’ve introduced two constants: Hello and world. Each identifier is assigned its value during compilation, and that value cannot be changed at runtime. As the identifier Hello starts with a capital letter the associated constant is visible to other packages - though this isn’t relevant in the context of a main package - whilst the identifier world starts with a lowercase letter and is only accessible within the main package.

We don’t need to specify the type of these constants as the Go compiler identifies them both as strings.

Another neat trick in Go’s armoury is multiple assignment so let’s see how this looks.

1 package main
2 import . "fmt"
3 
4 const Hello, world = "hello", "world"
5 
6 func main() {
7   Println(Hello, world)
8 }

This is compact, but I personally find it too cluttered and prefer the more general form.

 1 package main
 2 import . "fmt"
 3 
 4 const (
 5   Hello = "hello"
 6   world =  "world"
 7 )
 8 
 9 func main() {
10   Println(Hello, world)
11 }

Because the Println() function is variadic (i.e. can take a varible number of parameters) we can pass it both constants and it will print them on the same line, separate by whitespace. fmt also provides the Printf() function which gives precise control over how its parameters are displayed using a format specifier which will be familiar to seasoned C/C++ programmers.

10 Printf("%v %v\n", Hello, world)

fmt defines a number of % replacement terms which can be used to determine how a particular parameter will be displayed. Of these %v is the most generally used as it allows the formatting to be specified by the type of the parameter. We’ll discuss this in depth when we look at user-defined types, but in this case it will simply replace a %v with the corresponding string.

When parsing strings the Go compiler recognises a number of escape sequences which are available to mark tabs, new lines and specific unicode characters. In this case we use \n to mark a new line.

 1 package main
 2 import . "fmt"
 3 
 4 const (
 5   Hello = "hello"
 6   world =  "world"
 7 )
 8 
 9 func main() {
10   Printf("%v %v\n", Hello, world)
11 }

Variables

Constants are useful for referring to values which shouldn’t change at runtime, however most of the time when we’re referencing values in an imperative language like Go we need the freedom to change these values. We associate values which will change with variables. What follows is a simple variation of our Hello World program which allows the value of world to be changed at runtime by creating a new value and assigning it to the world variable.

 1 package main
 2 import . "fmt"
 3 
 4 const Hello = "hello"
 5 var world = "world"
 6 
 7 func main() {
 8   world += "!"
 9   Println(Hello, world)
10 }

There are two important changes here. Firstly we’ve introduced syntax for declaring a variable and assigning a value to it. Once more Go’s ability to infer type allows us assign a string value to the variable world without explicitly specifying the type.

5 var world = "world"

However if we wish to be more explicit we can be.

5 var world string = "world"

Having defined world as a variable in the global scope we can modify its value in main(), and in this case we choose to append an exclamation mark. Strings in Go are immutable values so following the assignment world will reference a new value.

8 world += "!"

To add some extra interest I’ve chosen to use an augmented assignment operator. These are a syntactic convenience popular in many languages which allow the value contained in a variable to be modified and the resulting value then assigned to the same variable.

I don’t intend to expend much effort discussing scope in Go. The point of this book is to experiment and learn by playing with code, referring to the comprehensive language specification available from Google when you need to know the technicalities of a given point. However to illustrate the difference between global and local scope we’ll modify this program further.

 1 package main
 2 import . "fmt"
 3 
 4 const Hello = "hello"
 5 var world = "world"
 6 
 7 func main() {
 8   world := world + "!"
 9   Println(Hello, world)
10 }

Here we’ve introduced a new local variable world within main() which takes its value from an operation concatenating the value of the global world variable with an exclamation mark. Within main() any subsequent reference to world will always access the local version of the variable without affecting the global world variable. The is known as shadowing.

The := operator marks an assignment declaration in which the type of the expression is inferred from the type of the value being assigned. If we chose to declare the local variable separately from the assignment we’d have to give it a different name to avoid a compilation error.

 1 package main
 2 import . "fmt"
 3 
 4 const Hello = "hello"
 5 var world = "world"
 6 
 7 func main() {
 8   var w string
 9   w = world + "!"
10   Println(Hello, w)
11 }

Another thing to note in this example is that when w is declared it’s also initialised to the zero value, which in the case of string happens to be ”“. This is a string containing no characters.

In fact all variables in Go are initialised to the zero value for their type when they’re declared and this eliminates an entire category of initialisation bugs which could otherwise be difficult to identify.

Functions

Having looked at how to reference values in Go and how to use the Println() function to display them, it’s only natural to wonder how we can implement our own functions. Obviously we’ve already implemented main() which hints at what’s involved, but main() is something of a special case as it exist to allow a Go program to execute and it neither requires any parameters nor produces any values to be used elsewhere in the program.

 1 package main
 2 import . "fmt"
 3 
 4 const Hello = "hello"
 5 
 6 func main() {
 7   Println(Hello, world())
 8 }
 9 
10 func world() string {
11   return "world"
12 }

In this example we’ve introduced world(), a function which to the outside world has the same operational purpose as the variable of the same name that we used in the previous section.

The empty brackets () indicate that there are no parameters passed into the function when it’s called, whilst string tells us that a single value is returned and it’s of type string. Anywhere that a valid Go program would expect a string value we can instead place a call to world() and the value returned will satisfy the compiler. The use of return is required by the language specification whenever a function specifies return values, and in this case it tells the compiler that the value of world() is the string “world”.

Go is unusual in that its syntax allows a function to return more than one value and as such each function takes two sets of (), the first for parameters and the second for results. We could therefore write our function in long form as

10 func world() (string) {
11   return "world"
12 }

In this next example we use a somewhat richer function signature, passing the parameter name which is a string value into the function message(), and assigning the function’s return value to message which is a variable declared and available throughout the function.

 1 package main
 2 import "fmt"
 3 
 4 func main() {
 5   fmt.Println(message("world"))
 6 }
 7 
 8 func message(name string) (message string) {
 9   message = fmt.Sprintf("hello %v", name)
10   return message
11 }

As with world() the message() function can be used anywhere that the Go compiler expects to find a string value. However where world() simply returned a predetermined value, message() performs a calculation using the Sprintf() function and returns its result.

Sprintf() is similar to Printf() which we met when discussing constants, only rather than create a string according to a format and displaying it in the terminal it instead returns this string as a value which we can assign to a variable or use as a parameter in another function call such as Println().

Because we’ve explicitly named the return value we don’t need to reference it in the return statement as each of the named return values is implied.

 1 package main
 2 import . "fmt"
 3 
 4 func main() {
 5   Println(message("world"))
 6 }
 7 
 8 func message(name string) (message string) {
 9   message = Sprintf("hello %v", name)
10   return
11 }

If we compare the main() and message() functions, we notice that main() doesn’t have a return statement. Likewise if we define our own functions without return values we can omit the return statement though later we’ll meet examples where we’d still use a return statement to prematurely exit a function.

 1 package main
 2 import . "fmt"
 3 
 4 func main() {
 5   greet("world")
 6 }
 7 
 8 func greet(name string) {
 9   Println("hello", name)
10 }

In the next example we’ll see what a function which uses multiple return values looks like.

 1 package main
 2 import . "fmt"
 3 
 4 func main() {
 5   Println(message())
 6 }
 7 
 8 func message() (string, string) {
 9   return "hello", "world"
10 }

Because message() returns two values we can use it in any context where at least two parameters can be consumed. Println() happens to be a variadic function, which we’ll explain in a moment, and takes zero or more parameters so it happily consumes both of the values message() returns.

For our final example we’re going to implement our own variadic function.

 1 package main
 2 import . "fmt"
 3 
 4 func main() {
 5   print("Hello", "world")
 6 }
 7 
 8 func print(v ...interface{}) {
 9   Println(v...)
10 }

We have three interesting things going on here which need explaining. Firstly I’ve introduced a new type, interface is accepted we can provide a string.

In the function signature we use v …interface{} to declare a parameter v which takes any number of values. These are received by print() as a sequence of values and the subsequent call to Println(v…) uses this same sequence as this is the sequence expected by Println().

So why did we use …interface) so to provide a sequence of values en masse we likewise need to use …interface (a slice of interface{} values, a concept we’ll cover in detail in a later chanpter) and copy each individual element into it before passing it into Println().

Encapsulation

In this chapter we’ll for the most part be using Go’s primitive types and types defined in various standard packages without any comment on their structure, however a key aspect of modern programming languages is the encapsulation of related data into structured types and Go supports this via the struct type. A struct describes an area of allocated memory which is subdivided into slots for holding named values, where each named value has its own type. A typical example of a struct in action would be

 1 package main
 2 
 3 import "fmt"
 4 
 5 type Message struct {
 6   X string
 7   y *string
 8 }
 9 
10 func (v Message) Print() {
11   if v.y != nil {
12     fmt.Println(v.X, *v.y)
13   } else {
14     fmt.Println(v.X)
15   }
16 }
17 
18 func (v *Message) Store(x, y string) {
19   v.X = x
20   v.y = &y
21 }
22 
23 func main() {
24   m := &Message{}
25   m.Print()
26   m.Store("Hello", "world")
27   m.Print()
28 }

$ go run 17.go

Hello world

Here we’ve defined a struct Message which contains two values: X and y. Go uses a very simple rule for deciding if an identifier is visible outside of the package in which it’s defined which applies to both package-level constants and variables, and type names, methods and fields. If the identifier starts with a capital letter it’s visible outside the package otherwise its private to the package.

The Go language spec guarantees that all variables will be initialised to the zero value for their type. For a struct type this means that every field will be initialised to an appropriate zero value. Therefore when we declare a value of type Message the Go runtime will initialise all of its elements to their zero value (in this case a zero-length string and a nil pointer respectively), and likewise if we create a Message value using a literal

24   m := &Message{}

Having declared a struct type we can declare any number of method functions which will operate on this type. In this case we’ve introduced Print() which is called on a Message value to display it in the terminal, and Store() which is called on a pointer to a Message value to change its contents. The reason Store() applies to a pointer is that we want to be able to change the contents of the Message and have these changes persist. If we define the method to work directly on the value these changes won’t be propagated outside the method’s scope. To test this for yourself, make the following change to the program

18 func (v Message) Store(x, y string) {

If you’re familiar with functional programming then the ability to use values immutably this way will doubtless spark all kinds of interesting ideas.

There’s another struct trick I want to show off before we move on and that’s type embedding using an anonymous field. Go’s design has upset quite a few people with an inheritance-based view of object orientation because it lacks inheritance, however thanks to type embedding we’re able to compose types which act as proxies to the methods provided by anonymous fields. As with most things, an example will make this much clearer

 1 package main
 2 
 3 import "fmt"
 4 
 5 type HelloWorld struct {}
 6 
 7 func (h HelloWorld) String() string {
 8   return "Hello world"
 9 }
10 
11 type Message struct {
12   HelloWorld
13 }
14 
15 func main() {
16   m := &Message{}
17   fmt.Println(m.HelloWorld.String())
18   fmt.Println(m.String())
19   fmt.Println(m)
20 }

$ go run 19.go
Hello world
Hello world
Hello world

Here we’re declaring a type HelloWorld which in this case is just an empty struct, but which in reality could be any declared type. HelloWorld defines a String() method which can be called on any HelloWorld value. We then declare a type Message which embeds the HelloWorld type by defining an anonymous field of the HelloWorld type. Wherever we encounter a value of type Message and wish to call String() on its embedded HelloWorld value we can do so by calling String() directly on the value, calling String() on the Message value, or in this case by allowing fmt.Println() to match it with the fmt.Stringer interface.

Any declared type can be embedded, so in our next example we’re going to base HelloWorld on the primitive bool boolean type to prove the point

 1 package main
 2 import "fmt"
 3 
 4 type HelloWorld bool
 5 
 6 func (h HelloWorld) String() (r string) {
 7   if h {
 8     r = "Hello world"
 9   }
10   return
11 }
12 
13 type Message struct {
14   HelloWorld
15 }
16 
17 func main() {
18   m := &Message{ HelloWorld: true }
19   fmt.Println(m)
20   m.HelloWorld = false
21   fmt.Println(m)
22   m.HelloWorld = true
23   fmt.Println(m)
24 }

In our final example we’ve declared the Hello type and embedded it in Message, then we’ve implemented a new String() method which allows a Message value more control over how it’s printed

 1 package main
 2 import "fmt"
 3 
 4 type Hello struct {}
 5 
 6 func (h Hello) String() string {
 7   return "Hello"
 8 }
 9 
10 type Message struct {
11   *Hello
12   World string
13 }
14 
15 func (v Message) String() (r string) {
16   if v.Hello == nil {
17     r = v.World
18   } else {
19     r = fmt.Sprintf("%v %v", v.Hello, v.World)
20   }
21   return
22 }
23 
24 func main() {
25   m := &Message{}
26   fmt.Println(m)
27   m.Hello = new(Hello)
28   fmt.Println(m)
29   m.World = "world"
30   fmt.Println(m)
31 }

$ go run 21.go

Hello 
Hello world

Generalisation

Encapsulation is of huge benefit when writing complex programs and it also enables one of the more powerful features of Go’s type system, the interface. An interface is similar to a struct in that it combines one or more elements but rather than defining a type in terms of the data items it contains, an interface defines it in terms of a set of method signatures which it must implement.

Once declared an interface can be used just like any other declared type, allowing functions and variables to operate with unknown types based solely on their required behaviour. Go’s type inference system will then recognise compliant values as instances of the interface, allowing us to write generalised code with little fuss.

In the next example we’re going to introduce a simple interface (by far the most common kind) which matches any type with a func String() string method signature.

 1 package main
 2 import "fmt"
 3 
 4 type Stringer interface {
 5   String() string
 6 }
 7 
 8 type Hello struct {}
 9 
10 func (h Hello) String() string {
11   return "Hello"
12 }
13 
14 type World struct {}
15 
16 func (w *World) String() string {
17   return "world"
18 }
19 
20 type Message struct {
21   X Stringer
22   Y Stringer
23 }
24 
25 func (v Message) String() (r string) {
26   switch {
27   case v.X == nil && v.Y == nil:
28   case v.X == nil:
29     r = v.Y.String()
30   case v.Y == nil:
31     r = v.X.String()
32   default:
33     r = fmt.Sprintf("%v %v", v.X, v.Y)
34   }
35   return
36 }
37 
38 func main() {
39   m := &Message{}
40   fmt.Println(m)
41   m.X = new(Hello)
42   fmt.Println(m)
43   m.Y = new(World)
44   fmt.Println(m)
45   m.Y = m.X
46   fmt.Println(m)
47   m = &Message{ X: new(World), Y: new(Hello) }
48   fmt.Println(m)
49   m.X, m.Y = m.Y, m.X
50   fmt.Println(m)
51 }

$ go run 22.go

Hello
Hello world
Hello Hello
world Hello
Hello world

This interface is copied directly from fmt.Stringer, so we can simplify our code a little by using that interface instead

17 type Message struct {
18   X fmt.Stringer
19   Y fmt.Stringer
20 }

As Go is strongly typed interface values contain both a pointer to the value contained in the interface, and the concrete type of the stored value. This allows us to perform type assertions to confirm that the value inside an interface matches a particular concrete type

 1 package main
 2 import "fmt"
 3 
 4 type Hello struct {}
 5 
 6 func (h Hello) String() string {
 7   return "Hello"
 8 }
 9 
10 type World struct {}
11 
12 func (w *World) String() string {
13   return "world"
14 }
15 
16 type Message struct {
17   X fmt.Stringer
18   Y fmt.Stringer
19 }
20 
21 func (v Message) IsGreeting() (ok bool) {
22   if _, ok = v.X.(*Hello); !ok {
23     _, ok = v.Y.(*Hello)
24   }
25   return
26 }
27 
28 func main() {
29   m := &Message{}
30   fmt.Println(m.IsGreeting())
31   m.X = new(Hello)
32   fmt.Println(m.IsGreeting())
33   m.Y = new(World)
34   fmt.Println(m.IsGreeting())
35   m.Y = m.X
36   fmt.Println(m.IsGreeting())
37   m = &Message{ X: new(World), Y: new(Hello) }
38   fmt.Println(m.IsGreeting())
39   m.X, m.Y = m.Y, m.X
40   fmt.Println(m.IsGreeting())
41 }

go run 24.go
false
true
true
true
true
true

Here we’ve replaced Message’s String() method with IsGreeting(), a predicate which uses a pair of type assertions to tell us whether or not one of Message’s data fields contains a value of concrete type Hello.

So far in these examples we’ve been using pointers to Hello and World so the interface variables are storing pointers to pointers to these values (i.e. **Hello and **World) rather than pointers to the values themselves (i.e. *Hello and *World). In the case of World we have to do this to comply with the fmt.Stringer interface because String() is defined for *World and if we modify main to assign a World value to either field we’ll get a compile-time error

29 func main() {
30   m := &Message{}
31   fmt.Println(m.IsGreeting())
32   m.X = Hello{}
33   fmt.Println(m.IsGreeting())
34   m.X = new(Hello)
35   fmt.Println(m.IsGreeting())
36   m.X = World{}
37 }

$ go run 25.go
# command-line-arguments
./25.go:36: cannot use World literal (type World) as type fmt.Stringer in assignment:
	World does not implement fmt.Stringer (String method has pointer receiver)

The final thing to mention about interfaces is that they support embedding of other interfaces. This allows us to compose a new, more restrictive interface based on one or more existing interfaces. Rather than demonstrate this with an example we’re going to look at code lifted directly from the standard io package which does this

67 type Reader interface {
68   Read(p []byte) (n int, err error)
69 }

78 type Writer interface {
79   Write(p []byte) (n int, err error)
80 }

106 type ReadWriter interface {
107   Reader
108   Writer
109 }

Here io is declaring three interfaces, the Reader and Writer which are independent of each other, and the ReadWriter which combines both. Any time we declare a variable, field or function parameter in terms of a ReaderWriter we know we can use both the Read() and Write() methods to manipulate it.

Startup

One of the less-discussed aspects of computer programs is the need to initialise many of them to a pre-determined state before they begin executing. Whilst this is probably the worst place to start discussing what to many people may appear to be advanced topics, one of my goals in this chapter is to cover all of the structural elements that we’ll meet when we examine more complex programs.

Every Go package may contain one or more init() functions specifying actions that should be taken during program initialisation. This is the one case I’m aware of where multiple declarations of the same identifier can occur without either resulting in a compilation error or the shadowing of a variable. In the following example we use the init() function to assign a value to our world variable

 1 package main
 2 import . "fmt"
 3 
 4 const Hello = "hello"
 5 var world  string
 6 
 7 func init() {
 8   world = "world"
 9 }
10 
11 func main() {
12   Println(Hello, world)
13 }

However the init() function can contain any valid Go code, allowing us to place the whole of our program in init() and leaving main() as a stub to convince the compiler that this is indeed a valid Go program.

 1 package main
 2 import . "fmt"
 3 
 4 const Hello = "hello"
 5 var world  string
 6 
 7 func init() {
 8   world = "world"
 9   Println(Hello, world)
10 }
11 
12 func main() {}

When there are multiple init() functions the order in which they’re executed is indeterminate so in general it’s best not to do this unless you can be certain the init() functions don’t interact in any way. The following happens to work as expected on my development computer but an implementation of Go could just as easily arrange it to run in reverse order or even leave deciding the order of execution until runtime.

 1 package main
 2 import . "fmt"
 3 
 4 const Hello = "hello"
 5 var world  string
 6 
 7 func init() {
 8   Print(Hello, " ")
 9   world = "world"
10 }
11 
12 func init() {
13   Printf("%v\n", world)
14 }
15 
16 func main() {}

HTTP

So far our treatment of Hello World has followed the traditional route of printing a preset message to the console. Anyone would think we were living in the fuddy-duddy mainframe era of the 1970s instead of the shiny 21st Century, when web and mobile applications rule the world.

Turning Hello World into a web application is surprisingly simple, as the following example demonstrates.

 1 package main
 2 import . "fmt"
 3 import "net/http"
 4 
 5 const MESSAGE = "hello world"
 6 const ADDRESS = ":1024"
 7 
 8 func main() {
 9   http.HandleFunc("/hello", Hello)
10   if e := http.ListenAndServe(ADDRESS, nil); e != nil {
11     Println(e)
12   }
13 }
14 
15 func Hello(w http.ResponseWriter, r *http.Request) {
16   w.Header().Set("Content-Type", "text/plain")
17   Fprintf(w, MESSAGE)
18 }

$ go run 29.go

Our web server is now listening on localhost port 1024 (usually the first non-privileged port on most Unix-like operating systems) and if we visit the url http://localhost:1024/hello with a web browser our server will return Hello World in the response body.

The first thing to note is that the net/http package provides a fully-functional web server which requires very little configuration. All we have to do to get our content to the browser is define a handler, which in this case is a function to call whenever an http.Request is received, and then launch a server to listen on the desired address with http.ListenAndServe(). http.ListenAndServe returns an error if it’s unable to launch the server for some reason, which in this case we print to the console.

We’re going to import the net/http package into the current namespace and assume our code won’t encounter any runtime errors to make the simplicity even more apparent. If you run into any problems whilst trying the examples which follow, reinserting the if statement will allow you to figure out what’s going on.

 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const MESSAGE = "hello world"
 6 const ADDRESS = ":1024"
 7 
 8 func main() {
 9   HandleFunc("/hello", Hello)
10   ListenAndServe(ADDRESS, nil)
11 }
12 
13 func Hello(w ResponseWriter, r *Request) {
14   w.Header().Set("Content-Type", "text/plain")
15   Fprintf(w, MESSAGE)
16 }

HandleFunc() registers a URL in the web server as the trigger for a function, so when a web request targets the URL the associated function will be executed to generate the result. The specified handler function is passed both a ResponseWriter to send output to the web client and the Request which is being replied to. The ResponseWriter is a file handle so we can use the fmt.Fprint() family of file-writing functions to create the response body.

Finally we launch the server using ListenAndServe() which will block for as long as the server is active, returning an error if there is one to report.

In this example I’ve declared a function Hello and by referring to this in the call to HandleFunc() this becomes the function which is registered. However Go also allows us to define functions anonymously where we wish to use a function value, as demonstrated in the following variation on our theme.

 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const MESSAGE = "hello world"
 6 const ADDRESS = ":1024"
 7 
 8 func main() {
 9   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
10     w.Header().Set("Content-Type", "text/plain")
11     Fprintf(w, MESSAGE)
12   })
13   ListenAndServe(ADDRESS, nil)
14 }

Functions are first-class values in Go and here HandleFunc() is passed an anonymous function value which is created at runtime. This value is a closure so it can also access variables in the lexical scope in which it’s defined. We’ll treat closures in greater depth later in the book, but for now here’s an example which demonstrates their basic premise by defining a variable messages in main() and then accessing it from within the anonymous function.

 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const ADDRESS = ":1024"
 6 
 7 func main() {
 8   message := "hello world"
 9   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
10     w.Header().Set("Content-Type", "text/plain")
11     Fprintf(w, message)
12   })
13   ListenAndServe(ADDRESS, nil)
14 }

This is only a very brief taster of what’s possible using net/http so we’ll conclude by serving our hello world web application over an SSL connection.

 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const SECURE_ADDRESS = ":1025"
 6 
 7 func main() {
 8   message := "hello world"
 9   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
10     w.Header().Set("Content-Type", "text/plain")
11     Fprintf(w, message)
12   })
13   ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
14 }

Before we run this program we first need to generate a certificate and a public key, which we can do using crypto/tls/generate_cert.go in the standard package library.

$ go run $GOROOT/src/pkg/crypto/tls/generate_cert.go -ca=true -host="localhost"
2014/05/16 20:41:53 written cert.pem
2014/05/16 20:41:53 written key.pem
$ go run 33.go

If you’re anything like me (and you have my sympathy if you are) then the next thought to idle through your mind will be a fairly obvious question: given that we can serve our content over both HTTP and HTTPS connections, how do we do both from the same program?

To answer this we have to know a little - but not a lot - about how to model concurrency in a Go program. The go keyword marks a goroutine which is a lightweight thread scheduled by the Go runtime. How this is implemented under the hood doesn’t matter, all we need to know is that when a goroutine is launched it takes a function call and creates a separate thread of execution for it. Here we’re going to launch a goroutine to run the HTTP server then run the HTTPS server in the main flow of execution.

 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const ADDRESS = ":1024"
 6 const SECURE_ADDRESS = ":1025"
 7 
 8 func main() {
 9   message := "hello world"
10   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
11     w.Header().Set("Content-Type", "text/plain")
12     Fprintf(w, message)
13   })
14 
15   go func() {
16     ListenAndServe(ADDRESS, nil)
17   }()
18 
19   ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
20 }

When I first wrote this code it actually used two goroutines, one for each server. Unfortunately no matter how busy any particular goroutine is, when the main() function returns our program will exit and our web servers will terminate. So I tried the primitive approach we all know and love from C

10 func main() {
11   message := "hello world"
12   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
13     w.Header().Set("Content-Type", "text/plain")
14     Fprintf(w, message)
15   })
16 
17   go func() {
18     ListenAndServe(ADDRESS, nil)
19   }()
20 
21   go func() {
22     ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
23   }()
24 
25   for {}
26 }

Here we’re using an infinite for loop to prevent program termination: it’s inelegant, but this is a small program and dirty hacks have their appeal. Whilst semantically correct this unfortunately doesn’t work either because of the way goroutines are scheduled: the infinite loop can potentially starve the thread scheduler and prevent the other goroutines from running.

$ go version
go version go1.3 darwin/amd64

In any event an infinite loop is a nasty, unnecessary hack as Go allows concurrent elements of a program to communicate with each other via channels, allowing us to rewrite our code as

 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const ADDRESS = ":1024"
 6 const SECURE_ADDRESS = ":1025"
 7 
 8 func main() {
 9   message := "hello world"
10   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
11     w.Header().Set("Content-Type", "text/plain")
12     Fprintf(w, message)
13   })
14 
15   done := make(chan bool)
16   go func() {
17     ListenAndServe(ADDRESS, nil)
18     done <- true
19   }()
20 
21   ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
22   <- done
23 }

For the next pair of examples we’re going to use two separate goroutines to run our HTTP and HTTPS servers, yet again coordinating program termination with a shared channel. In the first example we’ll launch both of the goroutines from the main() function, which is a fairly typical code pattern

 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const ADDRESS = ":1024"
 6 const SECURE_ADDRESS = ":1025"
 7 
 8 func main() {
 9   message := "hello world"
10   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
11     w.Header().Set("Content-Type", "text/plain")
12     Fprintf(w, message)
13   })
14 
15   done := make(chan bool)
16   go func() {
17     ListenAndServe(ADDRESS, nil)
18     done <- true
19   }()
20 
21   go func () {
22     ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
23     done <- true
24   }()
25   <- done
26   <- done
27 }

For our second deviation we’re going to launch a goroutine from main() which will run our HTTPS server and this will launch the second goroutine which manages our HTTP server

 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 
 5 const ADDRESS = ":1024"
 6 const SECURE_ADDRESS = ":1025"
 7 
 8 func main() {
 9   message := "hello world"
10   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
11     w.Header().Set("Content-Type", "text/plain")
12     Fprintf(w, message)
13   })
14 
15   done := make(chan bool)
16   go func () {
17     go func() {
18       ListenAndServe(ADDRESS, nil)
19       done <- true
20     }()
21 
22     ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
23     done <- true
24   }()
25   <- done
26   <- done
27 }

There’s a certain amount of fragile repetition in this code as we have to remember to explicitly create a channel, and then to send and receive on it multiple times to coordinate execution. As Go provides first-order functions (i.e. allows us to refer to functions the same way we refer to data, assigning instances of them to variables and passing them around as parameters to other functions) we can refactor the server launch code as follows

However this doesn’t work as expected, so let’s see if we can get any further insight

$ go vet 38.go
38.go:28: range variable s captured by func literal
exit status 1

Running go with the vet command runs a set of heuristics against our source code to check for common errors which wouldn’t be caught during compilation. In this case we’re being warned about this code

26 for _, s := range f {
27   go func() {
28     s()
29     done <- true
30   }()
31 }

Here we’re using a closure so it refers to the variable s in the for..range statement, and as the value of s changes on each successive iteration, so this is reflected in the call s().

To demonstrate this we’ll try a variant where we introduce a delay on each loop iteration much greater than the time taken to launch the goroutine.

 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 import "time"
 5 
 6 const ADDRESS = ":1024"
 7 const SECURE_ADDRESS = ":1025"
 8 
 9 func main() {
10   message := "hello world"
11   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
12     w.Header().Set("Content-Type", "text/plain")
13     Fprintf(w, message)
14   })
15 
16   Spawn(
17     func() { ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil) },
18     func() { ListenAndServe(ADDRESS, nil) },
19   )
20 }
21 
22 func Spawn(f ...func()) {
23   done := make(chan bool)
24 
25   for _, s := range f {
26     go func() {
27       s()
28       done <- true
29     }()
30     time.Sleep(time.Second)
31   }
32 
33   for l := len(f); l > 0; l-- {
34     <- done
35   }
36 }

When we run this we get the behaviour we expect with both HTTP and HTTPS servers running on their respective ports and responding to browser traffic. However this is hardly an elegant or practical solution and there’s a much better way of achieving the same effect.

26   for _, s := range f {
27     go func(server func()) {
28       server()
29       done <- true
30     }(s)
31   }

By accepting the parameter server to the goroutine’s closure we can pass in the value of s and capture it so that on successive iterations of the range our goroutines use the correct value.

Spawn() is an example of how powerful Go’s support for first-class functions can be, allowing us to run any arbitrary piece of code and wait for it to signal completion. It’s also a variadic function, taking as many or as few functions as desired and setting each of them up correctly.

If we now reach for the standard library we discover that another alternative is to use a sync.WaitGroup to keep track of how many active goroutines we have in our program and only terminate the program when they’ve all completed their work. Yet again this allows us to run both servers in separate goroutines and manage termination correctly.

 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 import "sync"
 5 
 6 const ADDRESS = ":1024"
 7 const SECURE_ADDRESS = ":1025"
 8 
 9 func main() {
10   message := "hello world"
11   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
12     w.Header().Set("Content-Type", "text/plain")
13     Fprintf(w, message)
14   })
15 
16   var servers sync.WaitGroup
17   servers.Add(1)
18   go func() {
19     defer servers.Done()
20     ListenAndServe(ADDRESS, nil)
21   }()
22 
23   servers.Add(1)
24   go func() {
25     defer servers.Done()
26     ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
27   }()
28   servers.Wait()
29 }

As there’s a certain amount of redundancy in this, let’s refactor a little by packaging server initiation into a new Launch() function. Launch() takes a parameter-less function and wraps this in a closure which will be launched as a goroutine in a separate thread of execution. Our sync.WaitGroup variable servers has been turned into a global variable to simplify the function signature of Launch(). When we call Launch() we’re freed from the need to manually increment servers prior to goroutine startup, and we use a defer statement to automatically call servers.Done() when the goroutine terminates even in the event that the goroutine crashes.

 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 import "sync"
 5 
 6 const ADDRESS = ":1024"
 7 const SECURE_ADDRESS = ":1025"
 8 
 9 var servers sync.WaitGroup
10 
11 func main() {
12   message := "hello world"
13   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
14     w.Header().Set("Content-Type", "text/plain")
15     Fprintf(w, message)
16   })
17 
18   Launch(func() {
19     ListenAndServe(ADDRESS, nil)
20   })
21 
22   Launch(func() {
23     ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
24   })
25   servers.Wait()
26 }
27 
28 func Launch(f func()) {
29   servers.Add(1)
30   go func() {
31     defer servers.Done()
32     f()
33   }()
34 }

The Environment

The main shells used with modern operating systems (Linux, OSX, FreeBSD, Windows, etc.) provide a persistent environment which can be queried by running programs, allowing a user to store configuration values in named variables. Go supports reading and writing these variables using the os package functions Getenv() and Setenv().

In our next example we’re going to query the environment for the variable SERVE_HTTP which we’ll assume contains the default address on which to serve unencrypted web content.

 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 import "os"
 5 import "sync"
 6 
 7 const SECURE_ADDRESS = ":1025"
 8 
 9 var address string
10 var servers sync.WaitGroup
11 
12 func init() {
13   if address = os.Getenv("SERVE_HTTP"); address == "" {
14     address = ":1024"
15   }
16 }
17 
18 func main() {
19   message := "hello world"
20   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
21     w.Header().Set("Content-Type", "text/plain")
22     Fprintf(w, message)
23   })
24 
25   Launch(func() {
26     ListenAndServe(address, nil)
27   })
28 
29   Launch(func() {
30     ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
31   })
32   servers.Wait()
33 }
34 
35 func Launch(f func()) {
36   servers.Add(1)
37   go func() {
38     defer servers.Done()
39     f()
40   }()
41 }

Here we’ve defined a global variable address which we set in init() to either the value provided in SERVE_HTTP or a default value “:1024”.

$ go run 43.go

$ SERVE_HTTP=":3030" go run 43.go

If we now extend this further to make the program fully configurable from the environment we arrive at

$ SERVE_HTTP=":3030" SERVE_HTTPS=":4040" go run 44.go

Handling Signals

If you’ve been running our example in the terminal and wondering how to terminate it without exiting the shell then you probably come from a GUI background and haven’t met control-C and its relatives (or rather you have, but most likely as cut’n’paste shortcuts).

Both Windows and Unix-style operating systems have the concept of a signal which can be sent from one process to another, and for historic reasons many of these can be manually entered using a control-key combination. This is a useful convenience but shutting down a production server this way can result in data loss or corruption. However that’s not the case with our Hello World server, so we have an excellent excuse to examine how to catch a termination signal and do something of our own choosing.

To listen for signals in Go we use the os/signal package in the standard library, which allows us to register a channel (an atomic queue for transferring messages between goroutines at runtime) on which notifications are to be received using the signal.Notify() function. Which signals will be made available depends largely on which operating system you’re working with and Go provides only two as standard across Windows and Unix: os.Interrupt and os.Kill. Of these os.Interrupt can be sent with control-C whilst os.Kill equates to SIGKILL on *nixen and is usually a non-maskable interrupt, meaning that it terminates execution and will never be received by Notify().

In the following example we’re initialising a signal handler at program startup. This consists of a goroutine containing an infinite loop and blocking on a channel of fixed size (in this case able to hold only one element at a time). The signal handler should be trapping the Interrupt and Kill signals. In both cases if we catch the signal we print a message to the console before exiting, however as previously mentioned the Kill signal (which can be sent from another shell session using the kill command) will never be received by our Go code.

 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 import "os"
 5 import "os/signal"
 6 import . "sync"
 7 
 8 const ADDRESS = ":1024"
 9 const SECURE_ADDRESS = ":1025"
10 
11 var servers WaitGroup
12 
13 func init() {
14   go SignalHandler(make(chan os.Signal, 1))
15 }
16 
17 func main() {
18   message := "hello world"
19   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
20     w.Header().Set("Content-Type", "text/plain")
21     Fprintf(w, message)
22   })
23 
24   Launch(func() {
25     ListenAndServe(ADDRESS, nil)
26   })
27 
28   Launch(func() {
29     ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
30   })
31   servers.Wait()
32 }
33 
34 func Launch(f func()) {
35   servers.Add(1)
36   go func() {
37     defer servers.Done()
38     f()
39   }()
40 }
41 
42 func SignalHandler(c chan os.Signal) {
43   signal.Notify(c, os.Interrupt)
44   for s := <- c; ; s = <- c {
45     switch s {
46     case os.Interrupt:
47       Println("^C received")
48       os.Exit(0)
49     case os.Kill:
50       Println("SIGKILL received")
51       os.Exit(1)
52     }
53   }
54 }

When we run this in the terminal on a Mac and press control-C we’ll see something like

$ go run 45.go
^C^C received

The key point of signals is that they allow our program to apply its own logic to events. In the following example we’re going to override the Interrupt signal sent by control-C so that the program continues execution. We’re then going to scan for other signals and use these to terminate the program. The syscall package defines a number of os.Signal values which can be detected by Notify() and of these I’ve chosen SIGABRT, SIGTERM and SIGQUIT as plausible termination signals. We’ll treat SIGABRT as an error condition and the other two as clean terminations.

Something else to note is that our signal handler is using a standard for loop statement to poll for input from the signal channel and then compare it to the cases of a switch statement. As the signal handler is designed to run for as long as Notify() is receiving signals we can simplify this a little

45 func SignalHandler(c chan os.Signal) {
46   signal.Notify(c, os.Interrupt, syscall.SIGABRT, syscall.SIGTERM, syscall.SIGQUIT)
47   for s := <- c; ; s = <- c {
48     switch s {
49     case os.Interrupt:
50       Println("interrupt - continue running")
51     case syscall.SIGABRT:
52       Println("abnormal exit")
53       os.Exit(1)
54     case syscall.SIGTERM, syscall.SIGQUIT:
55       Println("clean shutdown")
56       os.Exit(0)
57     }
58   }
59 }

$ go build 46.go
$ ./46
^Cinterrupt - continue running
^Cinterrupt - continue running
^\clean shutdown

We can send a SIGABRT signal using the kill command in a subshell and force our program to terminate abnormally

$ go build 46.go
$ ./46
^Z
[1]+  Stopped                 go run 46.go
$ ps
  PID TTY           TIME CMD
41097 ttys016    0:00.48 -bash
58713 ttys016    0:00.10 go run 46.go
58716 ttys016    0:00.01 /var/folders/25/ybgksr451vxf78xk1svymm5c0000gn/T/go-build543807549/command-li\
ne-arguments/_obj/exe/46
57608 ttys017    0:00.08 -bash
$ kill -SIGABRT 58716
$ fg
go run 46.go
abnormal exit
exit status 1

So far we’ve looked at how our program receives signals, however it’s also possible to send signals. For now we’re going to focus on sending a SIGABRT signal from our program to itself when there’s an error launching one of the servers, in this case by setting ADDRESS and SECURE_ADDRESS to the same value.

 1 package main
 2 import . "fmt"
 3 import . "net/http"
 4 import "os"
 5 import "os/signal"
 6 import . "sync"
 7 import "syscall"
 8 
 9 const ADDRESS = ":1024"
10 const SECURE_ADDRESS = ":1024"
11 
12 var servers WaitGroup
13 
14 func init() {
15   go SignalHandler(make(chan os.Signal, 1))
16 }
17 
18 func main() {
19   message := "hello world"
20   HandleFunc("/hello", func(w ResponseWriter, r *Request) {
21     w.Header().Set("Content-Type", "text/plain")
22     Fprintf(w, message)
23   })
24 
25   Launch("HTTP", func() error {
26     return ListenAndServe(ADDRESS, nil)
27   })
28 
29   Launch("HTTPS", func() error {
30     return ListenAndServeTLS(SECURE_ADDRESS, "cert.pem", "key.pem", nil)
31   })
32   servers.Wait()
33 }
34 
35 func Launch(name string, f func() error) {
36   servers.Add(1)
37   go func() {
38     defer servers.Done()
39     if e := f(); e != nil {
40       Println(name, "->", e)
41       syscall.Kill(syscall.Getpid(), syscall.SIGABRT)
42     }
43   }()
44 }
45 
46 func SignalHandler(c chan os.Signal) {
47   signal.Notify(c, os.Interrupt, syscall.SIGABRT, syscall.SIGTERM, syscall.SIGQUIT)
48   for s := <- c; ; s = <- c {
49     switch s {
50     case syscall.SIGABRT:
51       Println("abnormal exit")
52       os.Exit(1)
53     case os.Interrupt, syscall.SIGTERM, syscall.SIGQUIT:
54       Println("clean shutdown")
55       os.Exit(0)
56     }
57   }
58 }

We’ve modified our Launch() function to take a name which can be displayed as part of an error message, and its function parameter now has the signature func() error which specifies that it must return an error value, which is what’s returned by both ListenAndServe() and ListenAndServeTLS(). In the event the error (which is a predeclared interface) contains a value then we know an error condition’s occurred and can send a SIGABRT signal with syscall.Kill(). As Kill() is able to send signals to any running process we need to specify the ID of the current process, which we find using syscall.Getpid().

$ go run 47.go
2014/06/25 14:42:25 HTTPS -> listen tcp :1024: bind: address already in use
abnormal exit
exit status 1

TCP/IP

Printing text in a web browser is a cool trick, but what of real network programming? You know, the kind that bearded sandle-wearing *nix hackers go in for? It turns out this is surprisingly simple

 1 package main
 2 import . "fmt"
 3 import "net"
 4 
 5 func main() {
 6   if listener, e := net.Listen("tcp", ":1024"); e == nil {
 7     for {
 8       if connection, e := listener.Accept(); e == nil {
 9         go func(c net.Conn) {
10           defer c.Close()
11           Fprintln(c, "hello world")
12         }(connection)
13       }
14     }
15   }
16 }

The net package revolves around server the Listener and client Connection types. A Listener is an interface which allows any type implementing its specified methods - Accept(), Close() and Addr() - to be used interchangeably and is a key tool in Go for generalising program design. Writing a server then becomes a simple process of

• Listen() on a specified protocol and port number
• Accept() incoming connections
• for each net.Conn, run a handler in a separate goroutine
• read from and write to the connection whilst performing work
• close each connection when it finishes its work

Here we start to see some of the power of interfaces as a net.Conn implements the Writer interface defined in the io package, and fmt.Fprintf() takes any type which integrates io.Writer as its target.

Moving away from HTTP means abandoning the browser for testing but both *nix and Windows have a handy command-line utility called telnet which allows us to connect directly to a TCP/IP server and interact with it. We’ll get into the interaction side of things later in the book, for now here’s an example run of our program.

$ go run 48.go &
[1] 17415
$ telnet localhost 1024
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
hello world
Connection closed by foreign host.

Telnet’s a useful tool, but it’d be nice if we could connect our own client to the server as this could then be built for any platform supported by Go. For stream-oriented protocols like TCP/IP we do this using the net.Dial() function to open a net.Conn connection to a server and we can then interact with this using the io.Reader and io.Writer interfaces. These interfaces are supported throughout the Go standard package library, allowing files and streaming connections to be used interchangeably.

 1 package main
 2 import "bufio"
 3 import . "fmt"
 4 import "net"
 5 
 6 func main() {
 7   if connection, e := net.Dial("tcp", ":1024"); e == nil {
 8     defer connection.Close()
 9     if text, e := bufio.NewReader(connection).ReadString('\n'); e == nil {
10       Printf(text)
11     }
12   }
13 }

Because a net.Conn represents streams of data flowing between client and server we’ve introduced the bufio package to our client so that the data it’s receiving is buffered. This avoids our having to write our own code for buffering incoming data and is another example of the flexibility Go’s interfaces provide.

$ go run 48.go &
[1] 6102
$ go run 49.go
hello world
$ go run 49.go
hello world

Most books on network programming tend to stop at vanilla TCP/IP and leave figuring out how to establish a secure connection between client and server as an exercise for the reader. However we’re not likely to get another chance to look at this problem with the same lack of clutter that Hello World provides, and anyway we know how to generate a key and a certificate from our HTTPS adventure so we might as well reuse the knowledge. This time we’re going to need two sets of keys so let’s take care of that first

$ cp cert.pem server.cert.pem
$ cp key.pem server.key.pem
$ go run $GOROOT/src/pkg/crypto/tls/generate_cert.go -ca=true -host="localhost"
2014/05/16 20:41:53 written cert.pem
2014/05/16 20:41:53 written key.pem
$ cp cert.pem client.cert.pem
$ cp key.pem client.key.pem

Now we have our keys sorted, let’s take a look at what a TCP/IP server looks like in Go

 1 package main
 2 import "crypto/rand"
 3 import "crypto/tls"
 4 import . "fmt"
 5 
 6 func main() {
 7   if certificate, e := tls.LoadX509KeyPair("server.cert.pem", "server.key.pem"); e == nil {
 8     config := tls.Config{
 9       Certificates: []tls.Certificate{ certificate },
10       Rand: rand.Reader,
11     }
12 
13     if listener, e := tls.Listen("tcp", ":1025", &config); e == nil {
14       for {
15         if connection, e := listener.Accept(); e == nil {
16           go func(c *tls.Conn) {
17             defer c.Close()
18             Fprintln(c, "hello world")
19           }(connection.(*tls.Conn))
20         }
21       }
22     }
23   }
24 }

Importing crypto/tls provides us with an equivalent API to that defined in net and this means that as tls.Listen() fulfils the net.Listener interface our connections will be of type net.Conn. As a result if we want to pass the connection around inside our code we either have to import net so we can use net.Conn or perform a type assertion to use the connection as a *tls.Conn. We’ve made the latter choice here.

18   if connection, e := listener.Accept(); e == nil {
19     go func(c *tls.Conn) {
20       defer c.Close()
21       Fprintln(c, "hello world")
22     }(connection.(*tls.Conn))
23   }

For a server we import crypto/rand to access rand.Reader, a cryptographically secure pseudo-random number generator which we’ll be using as a source of randomness in the TLS connection. We then create a certificate using tls.LoadX509KeyPair() to load the server key pair and if this is successful then we set up a listener to accept incoming connections and write “Hello World” to a client.

As we’re using TLS we can’t test this version of Hello World using telnet so instead we need to write a client. Yet again this requires a keypair and where in our previous client we called net.Dial() we now use tls.Dial(), resulting in a very similar program.

 1 package main
 2 import "bufio"
 3 import "crypto/tls"
 4 import . "fmt"
 5 
 6 func main() {
 7   if certificate, e := tls.LoadX509KeyPair("client.cert.pem", "client.key.pem"); e == nil {
 8     config := tls.Config{
 9       Certificates: []tls.Certificate{ certificate },
10       InsecureSkipVerify: true,
11     }
12 
13     if connection, e := tls.Dial("tcp", ":1025", &config); e == nil {
14       defer connection.Close()
15       if text, e := bufio.NewReader(connection).ReadString('\n'); e == nil {
16         Printf(text)
17       }
18     }
19   }
20 }

$ go run 50.go &
[1] 6107
$ go run 51.go
hello world
$ go run 51.go
hello world

Looking back at our HTTP experiments, we were able to write a program which served over both HTTP and HTTPS connections. It’d be nice to do something similar with TCP/IP, if only to compare the two code-paths.

 1 package main
 2 import "crypto/rand"
 3 import "crypto/tls"
 4 import . "fmt"
 5 import "net"
 6 import "sync"
 7 
 8 var servers sync.WaitGroup
 9 
10 func main() {
11   if listener, e := net.Listen("tcp", ":1024"); e == nil {
12     Serve(listener)
13   }
14 
15   Serve(TLSListener("server.cert.pem", "server.key.pem", ":1025"))
16   servers.Wait()
17 }
18 
19 func TLSListener(cert, key, address string) (r net.Listener) {
20   if certificate, e := tls.LoadX509KeyPair(cert, key); e == nil {
21     config := tls.Config{
22       Certificates: []tls.Certificate{ certificate },
23       Rand: rand.Reader,
24     }
25     if listener, e := tls.Listen("tcp", address, &config); e == nil {
26       r = listener
27     }
28   }
29   return
30 }
31 
32 func Serve(listener net.Listener) {
33   if listener != nil {
34     Launch(func() {
35       for {
36         if connection, e := listener.Accept(); e == nil {
37           go func(c net.Conn) {
38             defer c.Close()
39             Fprintln(c, "hello world")
40           }(connection)
41         }
42       }
43     })
44   }
45 }
46 
47 func Launch(f func()) {
48   servers.Add(1)
49   go func() {
50     defer servers.Done()
51     f()
52   }()
53 }

We’ve reused Launch() from Example 1.33 to manage the lifecycle of our two server goroutines and introduced Serve() to phrase the server behaviour in terms of the net.Listener interface. We then move all the setup code for creating a tls.Listener into a separate function TLSListener() which returns a net.Listener value as tls.Listener complies with its interface, or a nil value if tls.Listen() returns an error.

If we now run this server we can connect to it with both of our client programs.

$ go run 52.go &
[1] 6278
$ go run 49.go
hello world
$ go run 51.go
hello world
$ go run 51.go
hello world
$ go run 49.go
hello world

UDP

Both TCP/IP and HTTP communications are connection-oriented and this involves a reasonable amount of handshaking and error-correction to assemble data packets in the correct order. For most applications this is exactly how we want to organise our network communications but sometimes the size of our messages is sufficiently small that we can fit them into individual packets, and when this is the case the UDP protocol is an ideal candidate.

As with our previous examples we still need both server and client applications.

 1 package main
 2 import . "fmt"
 3 import "net"
 4 
 5 var HELLO_WORLD = ([]byte)("Hello World\n")
 6 
 7 func main() {
 8   if address, e := net.ResolveUDPAddr("udp", ":1024"); e == nil {
 9     if server, e := net.ListenUDP("udp", address); e == nil {
10       for buffer := MakeBuffer(); ; buffer = MakeBuffer() {
11         if n, client, e := server.ReadFromUDP(buffer); e == nil {
12           go func(c *net.UDPAddr, packet []byte) {
13             if n, e := server.WriteToUDP(HELLO_WORLD, c); e == nil {
14               Printf("%v bytes written to: %v\n", n, c)
15             }
16           }(client, buffer[:n])
17         }
18       }
19     }
20   }
21 }
22 
23 func MakeBuffer() (r []byte) {
24   return make([]byte, 1024)
25 }

We have a somewhat more complex code pattern here than with TCP/IP to take account of the difference in underlying semantics: UDP is an unreliable transport dealing in individual packets (datagrams) which are independent of each other, therefore a server doesn’t maintain streams to any of its clients and these are responsible for any error-correction or packet ordering which may be necessary to coordinate successive signals. Because of these differences from TCP/IP we end up with the following generic workflow

• net.ResolveUDPAddr() to resolve the address
• net.ListenUDP() opens a UDP port and listens for traffic
• net.ReadFromUDP() copies incoming data into a buffer and provides the remote client’s address
• net.WriteToUDP() writes data back to the remote client’s address

For trivial uses of UDP we could probably forego the use of a separate goroutine to process each received packet, and indeed we may also have an application architecture where instead of processing the packet we’d hand it off to a message queue elsewhere. However many real-world examples such as serving DNS requests may introduce appreciable delays for processing and by using goroutines we ensure the server itself isn’t stalled.

Here our main overhead is the cost of buffer allocation as we use a different data buffer for each request. In a real-world example we’d very likely introduce a buffer pool which would expand and contract with demand, and reuse individual buffers once their associated request has completed. This is surprisingly simple to implement in Go and we’ll look at this in detail in a later chapter.

Our client has the same basic boilerplate as the server, only we use net.DialUDP() to set up a connection. We could use net.ReadFromUDP() and net.WriteToUDP() however as a net.UDPConn connection implements the io.ReadWriter interface we can use bufio.Reader to manage reading, and for writes the connection already knows the server address. As the server only knows about clients by receiving data from them we start our interaction with a UDPConn.Write() and then perform the buffered ReadString() to get a response.

 1 package main
 2 import "bufio"
 3 import . "fmt"
 4 import "net"
 5 
 6 var CRLF = ([]byte)("\n")
 7 
 8 func main() {
 9   if address, e := net.ResolveUDPAddr("udp", ":1024"); e == nil {
10     if server, e := net.DialUDP("udp", nil, address); e == nil {
11       defer server.Close()
12       if _, e = server.Write(CRLF); e == nil {
13         if text, e := bufio.NewReader(server).ReadString('\n'); e == nil {
14           Printf("%v", text)
15         }
16       }
17     }
18   }
19 }

Let’s give this a test run in the shell.

$ go run 53.go &
[2] 12777
$ go run 54.go
12 bytes written to: 127.0.0.1:58015
Hello World
$ go run 54.go
12 bytes written to: 127.0.0.1:50159
Hello World
$ go run 54.go
12 bytes written to: 127.0.0.1:51813
Hello World

Note how each time we run the client program it’s assigned a different network port by the operating system each time net.DialUDP is called.

We can make this apparent by performing multiple sequences of Write() and Read() operations

 1 package main
 2 import "bufio"
 3 import . "fmt"
 4 import "net"
 5 
 6 var CRLF = ([]byte)("\n")
 7 
 8 func main() {
 9   if address, e := net.ResolveUDPAddr("udp", ":1024"); e == nil {
10     if server, e := net.DialUDP("udp", nil, address); e == nil {
11       defer server.Close()
12       for i := 0; i < 3; i++ {
13         if _, e = server.Write(CRLF); e == nil {
14           if text, e := bufio.NewReader(server).ReadString('\n'); e == nil {
15             Printf("%v: %v", i, text)
16           }
17         }
18       }
19     }
20   }
21 }

$ go run 53.go &
[1] 12883
$ go run 55.go
12 bytes written to: 127.0.0.1:51732
0: Hello World
12 bytes written to: 127.0.0.1:51732
1: Hello World
12 bytes written to: 127.0.0.1:51732
2: Hello World
$ go run 55.go
12 bytes written to: 127.0.0.1:55504
0: Hello World
12 bytes written to: 127.0.0.1:55504
1: Hello World
12 bytes written to: 127.0.0.1:55504
2: Hello World

RSA obfuscated UDP

With all of our network examples to date we’ve included a secure transport option, but UDP doesn’t have a secured mode so we appear stuck with sending our message unencrypted for all the world to see. This is fine for a message such as Hello World which we’re happy for intervening network nodes to observe, but what if we want to send confidential data in our UDP packet?

In the following example we’re going to use our existing client RSA key-pair by sending the public key to our server which will then encrypt the message with this key and send it back to the client. The client already possesses the RSA private key so it’s a simple task to decrypt the message and display it. When we send the public key we could do so in a number of different formats: as an RSA pem file; as a raw binary buffer; or, serialised in some form. As both client and server are written in Go we’ll opt for the serialisation format provided in package gob. This is a pragmatic choice as if we were to send a pem file then that’d make it obvious that we’re using an encrypted format, and if we use a raw binary buffer we’d have to include a discussion of Go’s unsafe and reflection packages which are covered later in this book.

 1 package main
 2 import "bytes"
 3 import "crypto/rand"
 4 import "crypto/rsa"
 5 import "crypto/sha1"
 6 import "encoding/gob"
 7 import . "fmt"
 8 import . "net"
 9 
10 var HELLO_WORLD = []byte("Hello World")
11 var RSA_LABEL = []byte("served")
12 
13 func main() {
14   Serve(":1025", func(connection *UDPConn, c *UDPAddr, packet *bytes.Buffer) (n int) {
15     var key rsa.PublicKey
16     if e := gob.NewDecoder(packet).Decode(&key); e == nil {
17       if response, e := rsa.EncryptOAEP(sha1.New(), rand.Reader, &key, HELLO_WORLD, RSA_LABEL); e == n\
18 il {
19         n, _ = connection.WriteToUDP(response, c)
20       }
21     }
22     return
23   })
24 }
25 
26 func Serve(address string, f func(*UDPConn, *UDPAddr, *bytes.Buffer) int) {
27   Launch(address, func(connection *UDPConn) {
28     for {
29       buffer := make([]byte, 1024)
30       if n, client, e := connection.ReadFromUDP(buffer); e == nil {
31         go func(c *UDPAddr, b []byte) {
32           if n := f(connection, c, bytes.NewBuffer(b)); n != 0 {
33             Println(n, "bytes written to", c)
34           }
35         }(client, buffer[:n])
36       }
37     }
38   })
39 }
40 
41 func Launch(address string, f func(*UDPConn)) {
42   if a, e := ResolveUDPAddr("udp", address); e == nil {
43     if server, e := ListenUDP("udp", a); e == nil {
44       f(server)
45     }
46   }
47 }

So, the first thing of note is that we’ve refactored connection management into Serve() to make the server code easier to follow, and then we’re passing a function literal into this with the tasks to be performed each time a client connects. For now this is a quick hack so we’re not launching Serve() in its own goroutine with all the extra boilerplate for sync.WaitGroup which we’ve seen in previous examples. However we are spawning a separate goroutine for each packet received so that the server doesn’t block, and as an added bonus each time data is written to a client the number of bytes transferred is logged.

For each connection we read the client’s message which we know should be a valid public key in gob format. To decode this we create a gob.Decoder with the message as its base, then Decode() this to get a valid rsa.PublicKey which we then use to encrypt our message with rsa.EncryptOAEP(). The main thing to note here is that RSA_LABEL is a parameter which must be set the same for both rsa.EncryptOAEP() and rsa.DecryptOAEP() for the message to be correctly read by the latter. There’s no reason why this couldn’t be configured on a per-connection basis.

Now, let’s take a look at our client application

 1 package main
 2 import "bytes"
 3 import "crypto/rand"
 4 import "crypto/rsa"
 5 import "crypto/sha1"
 6 import "crypto/x509"
 7 import "encoding/gob"
 8 import "encoding/pem"
 9 import "io/ioutil"
10 import . "fmt"
11 import . "net"
12 
13 var RSA_LABEL = []byte("served")
14 
15 func main() {
16   Connect(":1025", func(server *UDPConn, private_key *rsa.PrivateKey) {
17     cipher_text := MakeBuffer()
18     if n, e := server.Read(cipher_text); e == nil {
19       if plain_text, e := rsa.DecryptOAEP(sha1.New(), rand.Reader, private_key, cipher_text[:n], RSA_L\
20 ABEL); e == nil {
21         Println((string)(plain_text))
22       }
23     }
24   })
25 }
26 
27 func Connect(address string, f func(*UDPConn, *rsa.PrivateKey)) {
28   LoadPrivateKey("client.key.pem", func(private_key *rsa.PrivateKey) {
29     if address, e := ResolveUDPAddr("udp", ":1025"); e == nil {
30       if server, e := DialUDP("udp", nil, address); e == nil {
31         defer server.Close()
32         SendKey(server, private_key.PublicKey, func() {
33           f(server, private_key)
34         })
35       }
36     }
37   })
38 }
39 
40 func LoadPrivateKey(file string, f func(*rsa.PrivateKey)) {
41   if file, e := ioutil.ReadFile(file); e == nil {
42     if block, _ := pem.Decode(file); block != nil {
43       if block.Type == "RSA PRIVATE KEY" {
44         if key, _ := x509.ParsePKCS1PrivateKey(block.Bytes); key != nil {
45           f(key)
46         }
47       }
48     }
49   }
50   return
51 }
52 
53 func SendKey(server *UDPConn, public_key rsa.PublicKey, f func()) {
54   var encoded_key bytes.Buffer
55   if e := gob.NewEncoder(&encoded_key).Encode(public_key); e == nil {
56     if _, e = server.Write(encoded_key.Bytes()); e == nil {
57       f()
58     }
59   }
60 }
61 
62 func MakeBuffer() (r []byte) {
63   return make([]byte, 1024)
64 }

$ go run 56.go &
[1] 66945
$ go run 57.go
256 bytes written to 127.0.0.1:51328
Hello World
$ go run 57.go
256 bytes written to 127.0.0.1:64834
Hello World
$ go run 57.go
256 bytes written to 127.0.0.1:50982
Hello World

The most obvious thing about this code is the heavy use of function literals, giving it a clean compositional feel. This is an aesthetic I picked up working with Ruby and which I always missed when dipping back into C or other low-level languages, so expect to variations on this style in later chapters.

Connect() is the client version of Serve(), abstracting away the details of contacting a UDP server, and the meat of our program’s interaction is a simple Read() of an encrypted message from the server which is then decrypted using rsa.DecryptOAEP() and displayed. Before our code initiates the connection though we need it to load an RSA key-pair so we can transmit our public key to the server. We do this in LoadPrivateKey() which uses ioutil.ReadFile() to load a PEM-encoded file into memory and ensure it contains a private key before invoking a function passed to it as a parameter. In this case the passed function sets up the connection, sends the public key to the server and then invokes the function passed to Connect() in SendKey().

To keep SendKey() as generic as possible it takes a parameterless function which is basically just a closure into the caller’s environment. In the case of Connect() the closure we pass to SendKey() binds to the server and private_key variables.

Error Handling

The examples in this chapter are for the most part designed to follow the happy path as our interest is in seeing some simple Go code that we can later build upon. The one obvious exception was when we explored signal handling and used the presence of an error as an excuse to send a SIGABRT to terminate the server. However error-handling is a large part of most real-world programming - especially in a system level language.

Go takes a typically pragmatic approach to error handling, the language specification defining the error type as a predeclared interface

type error interface {
	Error() string
}

In the following example we’re going to rewrite our encrypted UDP server from example 1.56 so that start-up errors cause the server to terminate and signal an error to the shell whilst client errors will log an appropriate message for later analysis using the log package. The default behaviour of the log package is to write its output to stderr so it integrates well with traditional *nix tools and infrastructure.

Whilst our program is still trivial in purpose, we now have all the basic conveniences for running a scalable server and integrating it with third-party monitoring and logging tools at deployment.

 1 package main
 2 import "bytes"
 3 import "crypto/rand"
 4 import "crypto/rsa"
 5 import "crypto/sha1"
 6 import "encoding/gob"
 7 import "log"
 8 import . "net"
 9 
10 var HELLO_WORLD = []byte("Hello World")
11 var RSA_LABEL = []byte("served")
12 
13 func main() {
14   Serve(":1025", func(connection *UDPConn, c *UDPAddr, packet *bytes.Buffer) (n int) {
15     var e error
16     var key rsa.PublicKey
17     var response []byte
18 
19     if e = gob.NewDecoder(packet).Decode(&key); e != nil {
20       log.Println("unable to decode wrapper:", c)
21     }
22 
23     if response, e = rsa.EncryptOAEP(sha1.New(), rand.Reader, &key, HELLO_WORLD, RSA_LABEL); e != nil {
24       log.Println("unable to encrypt server response")
25     }
26 
27     if n, e = connection.WriteToUDP(response, c); e != nil {
28       log.Println("unable to write response to client:", c)
29     }
30     return
31   })
32 }
33 
34 func Serve(address string, f func(*UDPConn, *UDPAddr, *bytes.Buffer) int) {
35   Launch(address, func(connection *UDPConn) {
36     for {
37       buffer := make([]byte, 1024)
38       if n, client, e := connection.ReadFromUDP(buffer); e == nil {
39         go func(c *UDPAddr, b []byte) {
40           if n := f(connection, c, bytes.NewBuffer(b)); n != 0 {
41             log.Println(n, "bytes written to", c)
42           }
43         }(client, buffer[:n])
44       } else {
45         log.Println(address, e.Error())
46       }
47     }
48   })
49 }
50 
51 func Launch(address string, f func(*UDPConn)) {
52   var e error
53   var a *UDPAddr
54   var server *UDPConn
55 
56   if a, e = ResolveUDPAddr("udp", address); e != nil {
57     log.Fatalln("unable to resolve UDP address:", e.Error())
58   }
59 
60   if server, e = ListenUDP("udp", a); e != nil {
61     log.Fatalln("can't open socket for listening:", e.Error())
62   }
63 
64   f(server)
65 }