Thanks to the encoding /json package, which is built into the standard library encoding and decoding JSON to and from Go types is both fast and easy. It implements the simplistic Marshal and Unmarshal functions; however, if we need them, the package also provides Encoder and Decoder types that allow us greater control when reading and writing streams of JSON data. In this section, we are going to examine both of these approaches, but first let's take a look at how simple it is to convert a standard Go struct into its corresponding JSON string.
Reading and writing JSON
Marshalling Go structs to JSON
To encode JSON data, the encoding/json package provides the Marshal function, which has the following signature:
func Marshal(v interface{}) ([]byte, error)
This function takes one parameter, which is of type interface, so pretty much any object you can think of since interface represents any type in Go. It returns a tuple of ([]byte, error), you will see this return style quite frequently in Go, some languages implement a try catch approach that encourages an error to be thrown when an operation cannot be performed, Go suggests the pattern (return type, error), where the error is nil when an operation succeeds.
In Go, unhandled errors are a bad thing, and whilst the language does implement Panic and Recover, which resemble exception handling in other languages, the situations where you should use these are quite different (see The Go Programming Language, Kernaghan). In Go, the panic function causes normal execution to stop and all deferred function calls in the Go routine are executed, the program will then crash with a log message. It is generally used for unexpected errors that indicate a bug in the code and good robust Go code will attempt to handle these runtime exceptions and return a detailed error object back to the calling function.
This pattern is exactly what is implemented with the Marshal function. In the instance that Marshal cannot create a JSON encoded byte array from the given object, which could be due to a runtime panic, then this is captured and an error object detailing the problem is returned to the caller.
Let's try this out, expanding on our existing example, instead of simply printing a string from our handler, let's create a simple struct for the response and return this instead.
Example 1.1 reading_writing_json_1/reading_writing_json_1.go
10 type helloWorldResponse struct {
11 Message string
12 }
In our handler, we will create an instance of this object, set the message, then use the Marshal function to encode it to a string before returning.
Let's see what that will look like:
23 func helloWorldHandler(w http.ResponseWriter, r *http.Request) {
24 response := helloWorldResponse{Message: "HelloWorld"}
25 data, err := json.Marshal(response)
26 if err != nil {
27 panic("Ooops")
28 }
29
30 fmt.Fprint(w, string(data))
31 }
Now, when we run our program again and refresh our browser, we see the following output rendered in valid JSON:
{"Message":"Hello World"}
This is awesome; however, the default behavior of Marshal is to take the literal name of the field and use this as the field in the JSON output. What if I prefer to use camel case and would rather see "message", could we just rename the field in the helloWorldResponse struct?
Unfortunately we can't, as in Go, lowercase properties are not exported, Marshal will ignore these and will not include them in the output.
All is not lost as the encoding/json package implements struct field attributes that allow us to change the output for the property to anything we choose.
Example 1.2 reading_writing_json_2/reading_writing_json_2.go
10 type helloWorldResponse struct {
11 Message string `json:"message"`
12 }
Using the struct field's tags, we can have greater control of how the output will look. In the preceding example, when we marshal this struct the output from our server would be:
{"message":"Hello World"}
This is exactly what we want, but we can use field tags to control the output even further. We can convert object types and even ignore a field altogether if we need to:
type helloWorldResponse struct {
// change the output field to be "message"
Message string `json:"message"`
// do not output this field
Author string `json:"-"`
// do not output the field if the value is empty
Date string `json:",omitempty"`
// convert output to a string and rename "id"
Id int `json:"id, string"`
}
Channel, complex types, and functions cannot be encoded in JSON; attempting to encode these types will result in an UnsupportedTypeError being returned by the Marshal function.
It also can't represent cyclic data structures; if your stuct contains a circular reference then Marshal will result in an infinite recursion, which is never a good thing for a web request.
If we want to export our JSON prettily formatted with indentation, we can use the MarshallIndent function, this allows you to pass an additional parameter of string to specify what you would like the indent to be. Two spaces right, not a tab?
func MarshalIndent(v interface{}, prefix, indent string) ([]byte, error)
The astute reader might have noticed that we are decoding our struct into a byte array and then writing that to the response stream, this does not seem to be particularly efficient and in fact it is not. Go provides Encoders and Decoders, which can write directly to a stream, since we already have a stream with the ResponseWriter then let's do just that.
Before we do, I think we need to look at the ResponseWriter a little to see what is going on there.
The ResponseWriter is an interface that defines three methods:
// Returns the map of headers which will be sent by the
// WriteHeader method.
Header()
// Writes the data to the connection. If WriteHeader has not
// already been called then Write will call
// WriteHeader(http.StatusOK).
Write([]byte) (int, error)
// Sends an HTTP response header with the status code.
WriteHeader(int)
If we have a ResponseWriter interface, how can we use this with fmt.Fprint(w io.Writer, a ...interface{})? This method requires a Writer interface as a parameter and we have a ResponseWriter interface. If we look at the signature for Writer we can see that it is:
Write(p []byte) (n int, err error)
Because the ResponseWriter interface implements this method, it also satisfies the interface Writer and therefore any object that implements ResponseWriter can be passed to any function that expects Writer.
Amazing, Go rocks, but we have not answered our question, Is there any better way to send our data to the output stream without marshalling to a temporary object before we return it?
The encoding/json package has a function called NewEncoder this returns us an Encoder object that can be used to write JSON straight to an open writer and guess what; we have one of those:
func NewEncoder(w io.Writer) *Encoder
So instead of storing the output of Marshal into a byte array, we can write it straight to the HTTP response.
Example 1.3 reading_writing_json_3/reading_writing_json_3.go:
func helloWorldHandler(w http.ResponseWriter, r *http.Request) {
response := HelloWorldResponse{Message: "HelloWorld"}
encoder := json.NewEncoder(w)
encoder.Encode(&response)
}
We will look at benchmarking in a later chapter, but to see why this is important we have created a simple benchmark to check the two methods against each other, have a look at the output.
Example 1.4 reading_writing_json_2/reading_writing_json_2.go:
$go test -v -run="none" -bench=. -benchtime="5s" -benchmem
BenchmarkHelloHandlerVariable-8 20000000 511 ns/op 248 B/op 5 allocs/op
BenchmarkHelloHandlerEncoder-8 20000000 328 ns/op 24 B/op 2 allocs/op
BenchmarkHelloHandlerEncoderReference-8 20000000 304 ns/op 8 B/op 1 allocs/op
PASS
ok github.com/building-microservices-with-go/chapter1/reading_writing_json_2 24.109s
Using Encoder rather than marshalling to a byte array is nearly 50% faster. We are dealing with nanoseconds here so that time may seem irrelevant, but it isn't; this was two lines of code. If you have that level of inefficiency throughout the rest of your code then your application will run slower, you will need more hardware to satisfy the load and that will cost you money. There is nothing clever in the differences between the two methods all we have done is understood how the standard packages work and chosen the correct option for our requirements, that is not performance tuning, that is understanding the framework.
Unmarshalling JSON to Go structs
Now we have learned how we can send JSON back to the client, what if we need to read input before returning the output? We could use URL parameters and we will see what that is all about in the next chapter, but commonly you will need more complex data structures that involve the service to accept JSON as part of an HTTP POST request.
Applying similar techniques that we learned in the previous section to write JSON, reading JSON is just as easy. To decode JSON into a stuct field the encoding/json package provides us with the Unmarshal function:
func Unmarshal(data []byte, v interface{}) error
The Unmarshal function works in the opposite way to Marshal; it allocates maps, slices, and pointers as required. Incoming object keys are matched using either the struct field name or its tag and will work with a case-insensitive match; however, an exact match is preferred. Like Marshal, Unmarshal will only set exported struct fields, those that start with an upper-case letter.
We start by adding a new struct field to represent the request, whilst Unmarshal can decode the JSON into an interface{}, which would be of map[string]interface{} // for JSON objects type or: []interface{} // for JSON arrays, depending if our JSON is an object or an array.
In my opinion it is much clearer to the readers of our code if we explicitly state what we are expecting as a request. We can also save ourselves work by not having to manually cast the data when we come to use it.
Remember two things:
- You do not write code for the compiler, you write code for humans to understand
- You will spend more time reading code than you do writing it
Taking these two points into account we create a simple struct to represent our request, which will look like this:
Example 1.5 reading_writing_json_4/reading_writing_json_4.go:
14 type helloWorldRequest struct {
15 Name string `json:"name"`
16 }
Again, we are going to use struct field tags as whilst we could let Unmarshal do case-insensitive matching so {"name": "World} would correctly unmarshal into the struct the same as {"Name": "World"}, when we specify a tag we are being explicit about the request form and that is a good thing. In terms of speed and performance it is also about 10% faster, and remember, performance matters.
To access the JSON sent with the request we need to take a look at the http.Request object passed to our handler. The following listing does not show all the methods on the request, just the ones we are going to be immediately dealing with, for full documentation I recommend checking out the documentation at https://godoc.org/net/http#Request:
type Requests struct {
...
// Method specifies the HTTP method (GET, POST, PUT, etc.).
Method string
// Header contains the request header fields received by the server. The type Header is a link to map[string] []string.
Header Header
// Body is the request's body.
Body io.ReadCloser
...
}
The JSON that has been sent with the request is accessible in the Body field. Body implements the interface io.ReadCloser as a stream and does not return a []byte or a string. If we need the data contained in the body, we can simply read it into a byte array, as shown in the following example:
30 body, err := ioutil.ReadAll(r.Body)
31 if err != nil {
32 http.Error(w, "Bad request", http.StatusBadRequest)
33 return
34 }
Here is something we'll need to remember. We are not calling Body.Close(), if we were making a call with a client we would need to do this as it is not automatically closed, however, when used in a ServeHTTP handler, the server automatically closes the request stream.
To see how this all works inside our handler, we can look at the following handler:
28 func helloWorldHandler(w http.ResponseWriter, r *http.Request) {
29
30 body, err := ioutil.ReadAll(r.Body)
31 if err != nil {
32 http.Error(w, "Bad request", http.StatusBadRequest)
33 return
34 }
35
36 var request helloWorldRequest
37 err = json.Unmarshal(body, &request)
38 if err != nil {
39 http.Error(w, "Bad request", http.StatusBadRequest)
40 return
41 }
42
43 response := helloWorldResponse{Message: "Hello " + request.Name}
44
45 encoder := json.NewEncoder(w)
46 encoder.Encode(response)
47 }
Let's run this example and see how it works. To test, we can simply use the curl command to send a request to the running server. If you feel more comfortable using a GUI tool than Postman (which is available for the Google Chrome browser), they will work just fine or feel free to use your preferred tool:
$ curl localhost:8080/helloworld -d '{"name":"Nic"}'
You should see the following response:
{"message":"Hello Nic"}
What do you think will happen if you do not include a body with your request?
$ curl localhost:8080/helloworld
If you guessed correctly, that you would get a HTTP status 400 Bad Request, then you win a prize:
func Error(w ResponseWriter, error string, code int)
Errors reply to the request with the given message and status code. Once we have sent this, we need to return stopping further execution of the function as this does not close the ResponseWriter interface and return flow to the calling function automatically.
Just before you think you are done, have a go and see if you can improve the performance of the handler. Think about the things we were talking about when marshaling JSON.
Got it?
Well if not here is the answer, again all we are doing is using the Decoder, which is the opposite of the Encoder that we used in writing JSON. It has an instant 33% performance increase and less code too.
Example 1.6 reading_writing_json_5/reading_writing_json_5.go:
27 func helloWorldHandler(w http.ResponseWriter, r *http.Request) {
28
29 var request HelloWorldRequest
30 decoder := json.NewDecoder(r.Body)
31
32 err := decoder.Decode(&request)
33 if err != nil {
34 http.Error(w, "Bad request", http.StatusBadRequest)
35 return
36 }
37
38 response := HelloWorldResponse{Message: "Hello " + request.Name}
39
40 encoder := json.NewEncoder(w)
41 encoder.Encode(response)
42 }
Now we can see just how easy it is to encode and decode JSON with Go, I would recommend taking five minutes now to spend some time digging through the documentation for the encoding/json package (https://golang.org/pkg/encoding/json/) as there is a whole lot more that you can do with this.
