Tooling
In this chapter we will discuss some tooling you may find useful for writing and running production Go applications.
Godoc
Godoc uses the comments in your code to generate documentation. You can use it via the command line, or as an HTTP server where it will generate HTML pages.
To install, run:
go get golang.org/x/tools/cmd/godoc
You can run godoc on the standard library, or packages in your own GOPATH. To see the documentation for encoding/json for example, run:
$ godoc encoding/json
To see the documentation for a specific function, such as Marshal:
$ godoc encoding/json Marshal
To run the HTTP server locally:
$ godoc -http:6060
This will run a godoc HTTP server locally, where you can see the generated HTML documentation for packages in your GOPATH as well as the standard library.
There is also a hosted version of godoc at https://godoc.org. If you host your code on GitHub for example, godoc.org can generate the documentation for it. It’s a good idea to keep your comments clean and up to date in case others are checking your package’s page on godoc.org. Golint is useful for surfacing parts of your code that need comments.
More can be found at the official Go blog post “Godoc: documenting Go code”, but to summarize the main points:
“The convention is simple: to document a type, variable, constant, function, or even a package, write a regular comment directly preceding its declaration, with no intervening blank line.”
Package-level comments take the form of a comment directly above the package declaration, starting with “Package [name] …”, like so:
1 // Package sort provides primitives for sorting slices and user-defined
2 // collections.
3 package sort
but if your package comment is long, you can split it out into a separate doc.go file, which only contains the comment and a package clause. See net/http/doc.go for an example.
Go Guru
Go Guru is “a tool for answering questions about Go source code.”9 It ships with a command-line tool, but you can also integrate it into an editor. You can install it with:
go get -u golang.org/x/tools/cmd/guru
You can find a list of supported editor integrations on the Using Go Guru document linked in the godoc for guru. In this section we’ll show screenshots of guru in vim.
We’ll assume you have vim-go installed; if not please see the “Editor Integration” section of the “Installing Go” chapter in the beginning of this book.
Let’s take a look at one of the questions that guru answers for us: “what concrete types implement this interface?”
Navigate to the line that contains an interface, and type :GoImplements:
Then hit Enter. Your vim window should split and your cursor should be in the quickfix list on the bottom half, with a list of files containing concrete structs that implement the interface:
Hit Enter on any one of those and you’ll jump to the struct definition in the listed file. To get back to the list, do <Ctrl-W> j. Then you can scroll through it as before, or quit out of it as usual with :q.
Go ahead and try some of the other guru commands like :GoReferrers, :GoCallees, and :GoCallers. You can find more help on guru for vim-go at the vim-go-tutorial, and a list of guru queries and other help output with guru -help.
Race Detector
Go comes with a builtin mechanism for detecting race conditions. There are multiple ways to invoke it:
In this section we’ll write some code that contains a data race, and catch it with a test with the race detector enabled. A data race occurs when two goroutines try to access the same object in memory concurrently, one of which is trying to write to the object, and there is no lock in place to control access to the object.
1 package cat
2
3 import (
4 "log"
5 )
6
7 // Cat is a small, carnivorous mammal with excellent night vision
8 type Cat struct {
9 noise string
10 }
11
12 // SetNoise sets the noise that our cat makes
13 func (c *Cat) SetNoise(n string) {
14 c.noise = n
15 }
16
17 // Noise makes the cat make a noise
18 func (c *Cat) Noise() string {
19 return c.noise
20 }
21
22 func updateCat(c *Cat) {
23 go c.SetNoise("にゃん")
24
25 log.Println(c.Noise())
26 }
In the above code, we execute a goroutine that sets the Noise attribute of the Cat argument to "にゃん". That goroutine goes off and runs in the background and the flow of execution continues to where we attempt to log c.Noise. This causes a race condition as we might write the value in the goroutine at the same time as reading it in the log.Println call.
Without considering the race condition and reading the code from top to bottom, we expect updateCat to set the noise attribute for the passed-in "cat" to "にゃん". So let’s write a test that makes that assertion for us:
1 package cat
2
3 import (
4 "testing"
5 )
6
7 func TestUpdateCat(t *testing.T) {
8 c := Cat{noise: "meow"}
9 c = updateCat(c)
10 if got := c.Noise(); got != "にゃん" {
11 t.Fatalf("c.Noise() = %q, want %q", got, "にゃん")
12 }
13 }
When running this test normally with go test, we might get lucky and it will pass:
When we run the test with the race detector enabled, however, we’ll see something different:
That’s a lot of output, but let’s take a look at the first two blocks of text:
This tells us exactly where our data race is. Our package’s filename is cat.go, so if we narrow it down to the lines containing our file, we can see the write occurred here:
and the read here:
And indeed if we check our code, those are the lines where we attempt to set c.noise, as well as read c.noise.
So how do we fix this? We’re going to need a lock around our data structure. We’ll use a sync.Mutex to lock our Cat structure whenever we read or write to it:
1 package cat
2
3 import (
4 "log"
5 "sync"
6 )
7
8 // Cat is a small, carnivorous mammal with excellent night vision
9 type Cat struct {
10 mu sync.Mutex
11 noise string
12 }
13
14 // SetNoise sets the noise that our cat makes
15 func (c *Cat) SetNoise(n string) {
16 c.mu.Lock()
17 defer c.mu.Unlock()
18 c.noise = n
19 }
20
21 // Noise makes the cat make a noise
22 func (c *Cat) Noise() string {
23 c.mu.Lock()
24 defer c.mu.Unlock()
25
26 return c.noise
27 }
28
29 func updateCat(c *Cat) *Cat {
30 go c.SetNoise("にゃん")
31
32 log.Println(c.Noise())
33
34 return c
35 }
Now we run the test again:
and the race condition is fixed.
You can also build your application with the race detector enabled, and see potential data races while running the application, or during an integration test. As an example, let’s make an API that accepts user-contributed entries about the countries the user has visited, and a description of their trip:
1 package main
2
3 import (
4 "encoding/json"
5 "flag"
6 "fmt"
7 "log"
8 "net/http"
9 "strings"
10 "unicode"
11 )
12
13 var (
14 addr = flag.String("http", "127.0.0.1:8000", "HTTP listen address")
15 countries = map[string]string{
16 "england": "England",
17 "japan": "Japan",
18 "southafrica": "South Africa",
19 "unitedstates": "United States",
20 }
21 )
22
23 // Entry is a user-submitted entry summarizing their trip
24 // to a given country
25 type Entry struct {
26 Country string
27 Text string
28 }
29
30 // removeSpaces is a function for strings.Map that removes
31 // all spaces from a string
32 func removeSpaces(r rune) rune {
33 if unicode.IsSpace(r) {
34 return -1
35 }
36 return r
37 }
38
39 // normalizeCountry takes an Entry and normalizes its country name
40 func normalizeCountry(e *Entry) error {
41 co := strings.Map(removeSpaces, strings.ToLower(e.Country))
42 if _, ok := countries[co]; !ok {
43 return fmt.Errorf("invalid country %q", e.Country)
44 }
45
46 e.Country = countries[co]
47
48 return nil
49 }
50
51 // EntryPostHandler is the POST endpoint for entries
52 func EntryPostHandler(w http.ResponseWriter, req *http.Request) {
53 decoder := json.NewDecoder(req.Body)
54 defer req.Body.Close()
55 var entry Entry
56 err := decoder.Decode(&entry)
57 if err != nil {
58 log.Printf("ERROR: could not decode Entry: %s", err)
59 w.WriteHeader(http.StatusBadRequest)
60 w.Write([]byte(fmt.Sprintf("Could not decode Entry: %s\n", err)))
61 return
62 }
63
64 go func(e *Entry) {
65 normalizeCountry(e)
66 }(&entry)
67
68 log.Printf("INFO: received Entry %v", entry)
69 }
70
71 func main() {
72 http.HandleFunc("/entries", EntryPostHandler)
73
74 log.Printf("Running on %s ...", *addr)
75 log.Fatal(http.ListenAndServe(*addr, nil))
76 }
You can probably spot the race condition already - we’re trying to normalize the country name in a goroutine in the background, then immediately trying to log the entry. Running the server with go run normally won’t give us any errors:
and we can even POST an Entry:
we then see this on the server:
which is wrong - we’re supposed to be normalizing the country name to “England”. Let’s see what happens when we run the server with -race enabled:
That looks fine, but now let’s try to POST an Entry again:
and there is our data race. We could fix this race in a similar manner to the way we fixed the cat race earlier, but instead let’s try using a sync.WaitGroup:
1 package main
2
3 import (
4 "encoding/json"
5 "flag"
6 "fmt"
7 "log"
8 "net/http"
9 "strings"
10 "sync"
11 "unicode"
12 )
13
14 var (
15 addr = flag.String("http", "127.0.0.1:8000", "HTTP listen address")
16 countries = map[string]string{
17 "england": "England",
18 "japan": "Japan",
19 "southafrica": "South Africa",
20 "unitedstates": "United States",
21 }
22 )
23
24 // Entry is a user-submitted entry summarizing their trip
25 // to a given country
26 type Entry struct {
27 Country string
28 Text string
29 }
30
31 // removeSpaces is a function for strings.Map that removes
32 // all spaces from a string
33 func removeSpaces(r rune) rune {
34 if unicode.IsSpace(r) {
35 return -1
36 }
37 return r
38 }
39
40 // normalizeCountry takes an Entry and normalizes its country name
41 func normalizeCountry(e *Entry) error {
42 co := strings.Map(removeSpaces, strings.ToLower(e.Country))
43 if _, ok := countries[co]; !ok {
44 return fmt.Errorf("invalid country %q", e.Country)
45 }
46
47 e.Country = countries[co]
48
49 return nil
50 }
51
52 // EntryPostHandler is the POST endpoint for entries
53 func EntryPostHandler(w http.ResponseWriter, req *http.Request) {
54 decoder := json.NewDecoder(req.Body)
55 defer req.Body.Close()
56 var entry Entry
57 err := decoder.Decode(&entry)
58 if err != nil {
59 log.Printf("ERROR: could not decode Entry: %s", err)
60 w.WriteHeader(http.StatusBadRequest)
61 w.Write([]byte(fmt.Sprintf("Could not decode Entry: %s\n", err)))
62 return
63 }
64
65 var wg sync.WaitGroup
66 wg.Add(1)
67 go func(e *Entry) {
68 defer wg.Done()
69 normalizeCountry(e)
70 }(&entry)
71 wg.Wait()
72
73 log.Printf("INFO: received Entry %v", entry)
74 }
75
76 func main() {
77 http.HandleFunc("/entries", EntryPostHandler)
78
79 log.Printf("Running on %s ...", *addr)
80 log.Fatal(http.ListenAndServe(*addr, nil))
81 }
You can see that we now have a sync.WaitGroup, onto which we add a delta of 1 to the counter. Inside the goroutine we decrement the counter with defer wg.Done(), then we block until the counter is zero with wg.Wait(). Since we’re blocking until the goroutine finishes, there is no longer a data race:
and we see that our country is normalized now to “England”.
Go Report Card
Full disclosure: we are the authors of this free and open source tool.
Go Report Card is a web application that gives packages a grade based on how well they pass various linters and tools. It is a popular application in the open source community, with thousands of projects using the badge to indicate the code quality. You can try it on goreportcard.com if your source code is open source, or run the server locally to use the tool on your internal network or private repositories.
Staticcheck
Staticcheck is a static analysis tool for Go programs. It helps for simplifying code and catches some issues such as unused error values. It also shows some style suggestions such as suggesting that the first character of an error string should not be capitalized.10 Here is an example of running staticcheck on the Go Report Card source:
And here is another example of staticcheck being run on code that doesn’t set an error value properly:
1 package main
2
3 import (
4 "errors"
5 "fmt"
6 )
7
8 func badFunc() error {
9 return errors.New("an error has occurred")
10 }
11
12 func foo() error {
13 return nil
14 }
15
16 func main() {
17 var err error
18
19 if err := foo(); err == nil {
20 err = badFunc()
21 }
22
23 fmt.Println(err)
24 }
We might think that the err = badFunc() line would set the outside var err error to errors.New("an error has occurred"), but actually when we print the err in the main scope it prints <nil>:
Staticcheck will alert us that the err value inside the if statement is never used:
Go-fuzz
Go-fuzz is a randomized testing tool for Go packages. Fuzzing is a method of testing code by providing various types of input to a program and monitoring that program for crashes or memory leaks. Having an automated fuzzing process is a great way to get ahead of potential input bugs and allows us to write more robust programs.
To download go-fuzz, run:
go get -u github.com/dvyukov/go-fuzz/go-fuzz github.com/dvyukov/go-fuzz/go-fuzz-build
Now we must add a Fuzz function in the package we want to fuzz. Let’s use Nihongo.io, a Japanese dictionary written in Go, as an example:
1 package dictionary
2
3 import (
4 "compress/gzip"
5 "log"
6 "os"
7 )
8
9 func Fuzz(data []byte) int {
10 file, err := os.Open("../../data/edict2.json.gz")
11 if err != nil {
12 log.Fatal("Could not load edict2.json.gz: ", err)
13 }
14 defer file.Close()
15
16 reader, err := gzip.NewReader(file)
17 if err != nil {
18 log.Fatal("Could not create reader: ", err)
19 }
20
21 dict, err := Load(reader)
22 if err != nil {
23 log.Fatal("Could not load dictionary: ", err)
24 }
25
26 dict.Search(string(data), 10)
27
28 return 1
29 }
This file exists in the github.com/gojp/nihongo/lib/dictionary package. We then run go-fuzz-build and then go-fuzz (in our case with a high `-timeout flag since it takes some time to load the dictionary) to start the fuzzing:
and we see that fuzzer is running successfully.
According to the go-fuzz README:
The function must return 1 if the fuzzer should increase priority of the given input during subsequent fuzzing (for example, the input is lexically correct and was parsed successfully); -1 if the input must not be added to corpus even if gives new coverage; and 0 otherwise; other values are reserved for future use.
govalidate
govalidate is a useful command for validating your Go environment. It runs such checks as whether you have the latest Go version installed, whether your $PATH environment variable contains $GOPATH/bin, and others.
Pinning tools with Go modules
It’s good to ensure that new contributors can clone a repo and run all the necessary tools. If a project enforces in the build that staticcheck must pass, contributors should be able to run it locally. One way to do this is with a tools.go file.
Let’s assume we want contributors and our build to run staticcheck. We create a tools/ directory with a tools.go file:
1 // +build tools
2
3 package tools
4
5 import (
6 _ "honnef.co/go/tools/cmd/staticcheck"
7 )
Our Makefile can have:
and when we run make staticcheck, if we don’t already have staticcheck installed, go will install it based on the generated entry in go.mod.
Sometimes we want our projects to use specific versions of these tools. To do this, modify the tool’s version in go.mod.