In this article by Nic Jackson, author of the book Building Microservices with Go, we will examine the encoding/json package to see just how easy Go makes it for us to use JSON objects for our requests and responses.

(For more resources related to this topic, see here.)

Reading and writing JSON

Thanks to the encoding/json package, which is built into the standard library, encoding and decoding JSON to and from Go types is both fast and easy. It implements the simplistic Marshal and Unmarshal functions; however, if we need them, the package also provides Encoder and Decoder types, which allow us greater control when reading and writing streams of JSON data. In this section, we are going to examine both of these approaches, but first let's take a look at how simple it is to convert a standard Go struct into its corresponding JSON string.

Marshalling Go structs to JSON

To encode JSON data, the encoding/json package provides the Marshal function, which has the following signature:

func Marshal(v interface{}) ([]byte, error)

This function takes one parameter, which is of the interface type, so that's pretty much any object you can think of, since interface represents any type in Go. It returns a tuple of ([]byte, error). You will see this return style quite frequently in Go. Some languages implement a try...catch approach, which encourages an error to be thrown when an operation cannot be performed. Go suggests the (return type, error) pattern, where the error is nil when an operation succeeds.

In Go, unhanded errors are a bad thing, and while the language does implement the panic and recover functions, which resemble exception handling in other languages, the situations in which you should use them are quite different (The Go Programming Language, Donovan and Kernighan). In Go, panic causes normal execution to stop, and all deferred function calls in the Go routine are executed; the program will then crash with a log message. It is generally used for unexpected errors that indicate a bug in the code, and good, robust Go code will attempt to handle these runtime exceptions and return a detailed error object back to the calling function.

This pattern is exactly what is implemented with the Marshal function. In case Marshal cannot create a JSON-encoded byte array from the given object, which could be due to a runtime panic, then this is captured and an error object detailing the problem is returned to the caller.

Let's try this out, expanding on our existing example. Instead of simply printing a string from our handler, let's create a simple struct for the response and return that:

10 type helloWorldResponse struct {

11   Message string

12 }

In our handler, we will create an instance of this object, set the message, and then use the Marshal function to encode it to a string before returning.

Let's see what that will look like:

23 func helloWorldHandler(w http.ResponseWriter, r *http.Request) {

24   response := helloWorldResponse{Message: "HelloWorld"}

25   data, err := json.Marshal(response)

26   if err != nil {

27     panic("Ooops")

28   }

29

30   fmt.Fprint(w, string(data))

31 }

Now when we rerun our program and refresh our browser, we'll see the following output rendered in valid JSON:

{"Message":"Hello World"}

This is awesome, but the default behavior of Marshal is to take the literal name of the field and use that as the field in the JSON output. What if I prefer to use camel case and would rather see message—could we just rename the field in our struct message?

Unfortunately, we can't because in Go, lowercase properties are not exported. Marshal will ignore these and will not include them in the output.

All is not lost: the encoding/json package implements struct field attributes, which allow us to change the output for the property to anything we choose. The example code is as follows:

10 type helloWorldResponse struct {

11   Message string `json:"message"`

12 }

Using the struct field's tags, we can have greater control over how the output will look. In the preceding example, when we marshal this struct, the output from our server would be the following:

{"message":"Hello World"}

This is exactly what we want, but we can use field tags to control the output even further. We can convert object types and even ignore a field altogether if we need to:

struct helloWorldResponse {

// change the output field to be "message"

   Message   string `json:"message"`

   // do not output this field

   Author string `json:"-"`

   // do not output the field if the value is empty

   Date   string `json:",omitempty"`

   // convert output to a string and rename "id"

   Id   int   `json:"id, string"`

}

The channel, complex types, and functions cannot be encoded in JSON. Attempting to encode these types will result in an UnsupportedTypeError being returned by the Marshal function.

It also can't represent cyclic data structures, so if your stuct contains a circular reference, then Marshal will result in an infinite recursion, which is never a good thing for a web request.

If we want to export our JSON pretty formatted with indentation, we can use the MarshallIndent function, which allows you to pass an additional string parameter to specify what you would like the indent to be—two spaces, not a tab, right?

func MarshalIndent(v interface{}, prefix, indent string) ([]byte, error)

The astute reader might have noticed that we are decoding our struct into a byte array and then writing that to the response stream. This does not seem to be particularly efficient, and in fact, it is not. Go provides encoders and decoders, which can write directly to a stream. Since we already have a stream with the ResponseWriter interface, let's do just that.

Before we do so, I think we need to look at the ResponseWriter interface a little to see what is going on there.

ResponseWriter is an interface that defines three methods:

// Returns the map of headers which will be sent by the

// WriteHeader method.

Header()

// Writes the data to the connection. If WriteHeader has not

// already been called then Write will call

// WriteHeader(http.StatusOK).

Write([]byte) (int, error)

// Sends an HTTP response header with the status code.

WriteHeader(int)

If we have a ResponseWriter, how can we use this with fmt.Fprint(w io.Writer, a ...interface{})? This method requires a Writer interface as a parameter, and we have a ResponseWriter. If we look at the signature for Writer, we can see that it is the following:

Write(p []byte) (n int, err error)

Because the ResponseWriter interface implements this method, it also satisfies the Writer interface; therefore, any object that implements ResponseWriter can be passed to any function that expects Writer.

Amazing! Go rocks—but we don't have an answer to our question: is there any better way to send our data to the output stream without marshalling to a temporary string before we return it?

The encoding/json package has a function called NewEncoder. This returns an Encoder object, which can be used to write JSON straight to an open writer, and guess what—we have one of those:

func NewEncoder(w io.Writer) *Encoder

So instead of storing the output of Marshal into a byte array, we can write it straight to the HTTP response, as shown in the following code:

func helloWorldHandler(w http.ResponseWriter, r *http.Request) {

   response := HelloWorldResponse{Message: "HelloWorld"}

   encoder := json.NewEncoder(w)

   encoder.Encode(&response)

}

We will look at benchmarking in a later chapter, but to see why this is important, here's a simple benchmark to check the two methods against each other; have a look at the output:

go test -v -run="none" -bench=. -benchtime="5s" -benchmem

testing: warning: no tests to run

PASS

BenchmarkHelloHandlerVariable   10000000 1211 ns/op 248 B/op 5 allocs/op

BenchmarkHelloHandlerEncoder   10000000 662 ns/op 8 B/op   1 allocs/op

ok     github.com/nicholasjackson/building-microservices-in-go/chapter1/bench 20.650s

Using the Encoder rather than marshalling to a byte array is nearly 50% faster. We are dealing with nanoseconds here, so that time may seem irrelevant, but it isn't; this was two lines of code. If you have that level of inefficiency throughout the rest of your code, your application will run slower, you will need more hardware to satisfy the load, and that will cost you money. There is nothing clever in the differences between the two methods—all we have done is understood how the standard packages work and chosen the correct option for our requirements. That is not performance tuning, that is understanding the framework.

Unmarshalling JSON to Go structs

Now that we have learned how we can send JSON back to the client, what if we need to read input before returning the output? We could use URL parameters, and we will see what that is all about in the next chapter, but usually, you will need more complex data structures, which include the service to accept JSON as part of an HTTP POST request.

If we apply techniques similar to those we learned in the previous section (to write JSON), reading JSON is just as easy. To decode JSON into a stuct, the encoding/json package provides us with the Unmarshal function:

func Unmarshal(data []byte, v interface{}) error

The Unmarshal function works in the opposite way to Marshal: it allocates maps, slices, and pointers as required. Incoming object keys are matched using either the struct field name or its tag and will work with a case-insensitive match; however, an exact match is preferred. Like Marshal, Unmarshal will only set exported struct fields: those that start with an upper case letter.

We start by adding a new struct to represent the request, while Unmarshal can decode the JSON into an interface{} array, which would be of one of the following types:

map[string]interface{} // for JSON objects
[]interface{} // for JSON arrays

Which type it is depends on whether our JSON is an object or an array.

In my opinion, it is much clearer to the readers of our code if we explicitly state what we are expecting as a request. We can also save ourselves work by not having to manually cast the data when we come to use it.

Remember two things:

You do not write code for the compiler; you write code for humans to understand
You will spend more time reading code than you do writing it

We are going to do ourselves a favor by taking into account these two points and creating a simple struct to represent our request, which will look like this:

14 type helloWorldRequest struct {

15   Name string `json:"name"`

16 }

Again, we are going to use struct field tags because while we could let Unmarshal do case-insensitive matching so that {"name": "World} would correctly unmarshal into the struct the same as {"Name": "World"}, when we specify a tag, we are being explicit about the request form, and that is a good thing. In terms of speed and performance, it is also about 10% faster, and remember: performance matters.

To access the JSON sent with the request, we need to take a look at the http.Request object passed to our handler. The following listing does not show all the methods in the request, just the ones we are going to be immediately dealing with. For the full documentation, I recommend checking out the docs at https://godoc.org/net/http#Request.

type Requests struct {

…

// Method specifies the HTTP method (GET, POST, PUT, etc.).

Method string

// Header contains the request header fields received by the server. The type Header is a link to map[string] []string.

Header Header

// Body is the request's body.

Body io.ReadCloser

…

}

The JSON that has been sent with the request is accessible in the Body field. The Body field implements the io.ReadCloser interface as a stream and does not return []byte or string data. If we need the data contained in the body, we can simply read it into a byte array, like the following example:

30 body, err := ioutil.ReadAll(r.Body)

31 if err != nil {

32     http.Error(w, "Bad request", http.StatusBadRequest)

33     return

34 }

Here is something we'll need to remember: we are not calling Body.Close(); if we were making a call with a client, we would need to do this as it is not automatically closed; however, when used in a ServeHTTP handler, the server automatically closes the request stream.

To see how this all works inside our handler, we can look at the following handler:

28 func helloWorldHandler(w http.ResponseWriter, r *http.Request) {

29

30   body, err := ioutil.ReadAll(r.Body)

31   if err != nil {

32     http.Error(w, "Bad request", http.StatusBadRequest)

33             return

34   }

35

36   var request HelloWorldRequest

37   err = json.Unmarshal(body, &request)

38   if err != nil {

39     http.Error(w, "Bad request", http.StatusBadRequest)

40             return

41   }

42

43 response := HelloWorldResponse{Message: "Hello " + request.Name}

44

45   encoder := json.NewEncoder(w)

46   encoder.Encode(response)

47 }

Let's run this example and see how it works; to test it, we can simply use curl to send a request to the running server. If you feel more comfortable using a GUI tool, then Postman, which is available for the Google Chrome browser, will work just fine. Otherwise, feel free to use your preferred tool.

$ curl localhost:8080/helloworld -d '{"name":"Nic"}'

You should see the following response:

{"message":"Hello Nic"}

What do you think will happen if you do not include a body with your request?

$ curl localhost:8080/helloworld

If you guessed correctly that you would get an "HTTP status 400 Bad Request" error, then you win a prize. The following error replies to the request with the given message and status code:

func Error(w ResponseWriter, error string, code int)

Once we have sent this, we need to return, stopping further execution of the function as this does not close the ResponseWriter and return flow to the calling function automatically.

You might think you are done, but have a go and see whether you can improve the performance of the handler. Think about the things we were talking about when marshaling JSON.

Got it?

Well, if not, here is the answer: again, all we are doing is using Decoder, which is the opposite of the Encoder function we used when writing JSON, as shown in the following code example. This nets an instant 33% performance increase, and with less code, too.

27 func helloWorldHandler(w http.ResponseWriter, r *http.Request) {

28

29   var request HelloWorldRequest

30   decoder := json.NewDecoder(r.Body)

31

32   err := decoder.Decode(&request)

33   if err != nil {

34     http.Error(w, "Bad request", http.StatusBadRequest)

35             return

36   }

37

38   response := HelloWorldResponse{Message: "Hello " + request.Name}

39

40   encoder := json.NewEncoder(w)

41   encoder.Encode(response)

42 }

Now that you can see just how easy it is to encode and decode JSON with Go, I would recommend taking 5 minutes to spend some time digging through the documentation for the encoding/json package as there is a whole lot more than you can do with it:

https://golang.org/pkg/encoding/json/