First, a few words about modules and packages would be helpful. A package is a cohesive unit of data types, constants, variables, and functions. You build and test packages, not individual files or modules. When you build a package, the build system collects and also builds all dependent packages. If the package name is main, building it will result in an executable. You can run the main package without producing a binary (more specifically, the Go build system first builds the package, produces the binary in a temporary location, and runs it). To use another package, you import it. Modules help with organizing multiple packages and the resolution of package references within a project. A module is simply a collection of packages. If you import a package into your program, the module containing that package will be added to go.mod, and a checksum of the contents of that module will be added to go.sum. Modules also help you to manage versions of your programs.

All files of a package are stored under a single directory on the filesystem. Every package has a name declared using the package directive, shared by all source files in it. The package name usually matches the directory name containing the files, but this is not necessarily so. For example, the main package is not usually under a directory named main/. The directory of the package determines the package’s “import path.” You import another package into your current package using the import <importPath> statement. Once you import a package, you use the names declared in that package using its package name (which is not necessarily the directory name).

A module name points to the location where the module contents are stored in a version control system on the Internet. At the time of writing, this is not a hard-and-fast requirement, so you can actually create module names that do not follow this convention. This should be avoided to prevent potential future incompatibilities with the build system. Your module names should be part of the import paths for the packages of those modules. In particular, module names whose first component (the part before the first /) does not have . are reserved for the standard library.

These concepts are illustrated in Figure 1.1.

Figure 1.1 – Modules and packages

1. The module name declared in go.mod is the repository path where the module can be found.
2. The import path in main.go defines where the imported package can be found. The Go build system will locate the package using this import path, and then it will locate the module containing the package by scanning the parent directories of the package path. Once the module is found, it will be downloaded to the module cache.
3. The package name defined in the imported module is the package name you use to access the symbols of that package. This can be different from the last component of the import path. In our example, the package name is example, but the import path for this package is github.com/bserdar/go-recipes-module.
4. The Example function is located in the example package.
5. The example package also imports another package contained in the same module. The build system will identify this package to be part of the same module and resolve the references, using the downloaded version of the module.

You will need a recent version of Go on your computer to build and run the examples in this chapter. The examples in this book were tested using Go version 1.22. The code from this chapter can be found at https://github.com/PacktPublishing/Go-Recipes-for-Developers/tree/main/src/chp1.

When you start working on a new project, the first thing to do is to create a module for it. A module is how Go manages dependencies.

How to do it...

1. Create a directory to store a new module.
2. Under that directory, use go mod init <moduleName> to create the new module. The go.mod file marks the root directory of a module. Any package under this directory will be a part of this module unless that directory also has a go.mod file. Although such nested modules are supported by the build system, there is not much to be gained from them.
3. To import a package in the same module, use moduleName/packagePath. When moduleName is the same as the location of the module on the internet, there are no ambiguities about what you are referring to.
4. For the packages under a module, the root of the module is the closest parent directory containing a go.mod file. All references to other packages within a module will be looked up in the directory tree under the module root.
5. Start by creating a directory to store the project files. Your current directory can be anywhere on the filesystem. I have seen people use a common directory to store their work, such as $HOME/projects (or \user\myUser\projects in Windows). You may choose to use a directory structure that looks like the module name, such as $HOME/github.com/mycompany/mymodule (or \user\myUser\github.com\mycompany\mymodule in Windows). Depending on your operating system, you may find a more suitable location.

Warning

Do not work under the src/ directory of your Go installation. That is the source code for the Go standard library.

Tip

You should not have an environment variable, GOPATH; if you have to keep it, do not work under it. This variable was used by an older mode of operation (Go version <1.13) that is now deprecated in favor of the Go module system.

Throughout this chapter, we will be using a simple program that displays a form in a web browser and stores the entered information in a database.

After creating the module directory, use go mod init. The following commands will create a webform directory under projects and initialize a Go module there:

$ cd projects
$ mkdir webform
$ go mod init github.com/examplecompany/webform

This will create a go.mod file in this directory that looks like this:

module github.com/PacktPublishing/Go-Recipes-for-Developers/chapter1/webform
go 1.21.0

Use a name that describes where your module can be found. Always use a URL structure such as the <host>.<domain>/location/to/module format (e.g., github.com/bserdar/jsonom). In particular, the first component of the module name should have a dot (.) (the Go build system checks this).

So, even though you can name the module something such as webform or mywork/webform, do not do so. However, you can use something such as workspace.local/webform. When in doubt, use the code repository name.

Once you have a new module, it is time to decide how you are going to organize the source files.

How to do it...

There are several established conventions, depending on the project:

• Use a standard layout, such as https://github.com/golang-standards/project-layout.
• A library with a narrow focus can put all the exported names at the module root, with implementation details optionally stored under internal packages. A module that produces a single executable with relatively few or no reusable components can also use the flat directory structure.

For a project like ours that produces an executable, the structure laid out in https://github.com/golang-standards/project-layout fits. So, let’s follow that template:

webform/
  go.mod
  cmd/
    webform/
      main.go
  web/
    static/
  pkg/
    ...
  internal/
    ...
  build/
    ci/
    package/
  configs/

Here, the cmd/webform directory will contain the main package. As you can see, this is one instance where the package name does not match the directory it is in. The Go build system will create executables using the directory name, so when you build the main package under cmd/webform, you get an executable named webform. If you have multiple executables built within a single module, you can accommodate them by creating a separate main package under a directory matching the program name, under the cmd/ directory.

The pkg/ directory will contain the exported packages of the program. These are packages that can be imported and reused in other projects.

If you have packages that are not usable outside this project, you should put them under the internal/ directory. The Go build system recognizes this directory and does not allow you to import packages under internal/ from other packages that are outside the directory containing the internal/ directory. With this setup, all the packages of our webform program will have access to the packages under internal/, but it will be inaccessible to packages importing this module.

The web/ directory will contain any web-related assets. In this example, we will have a web/static directory containing static web pages. You can also add web/templates to store server-side templates if you have any.

The build/package directory should have packaging scripts and configuration for cloud, container, packaging systems (dep, rpm, pkg, etc.).

The build/ci directory should have continuous integration tool scripts and configurations. If the continuous integration tool you are using requires its files to be in a certain directory other than this, you can create symbolic links, or simply put those files where the tool needs them instead of /build/ci.

The configs/ directory should contain the configuration file templates and default configurations.

You can also see projects that have the main package under the module root, eliminating the cmd/ directory. This is a common layout when the module has only one executable:

webform/
  go.mod
  go.sum
  main.go
  internal/
    ...
  pkg/
    ...

Then there are modules without any main package. These are usually libraries that you can import into your projects. For example, https://github.com/google/uuid contains the popular UUID implementation using a flat directory structure.

Now that you have a module and a source tree with some Go files, you can build or run your program.

How to do it...

• Use go build to build the current package
• Use go build ./path/to/package to build the package in the given directory
• Use go build <moduleName> to build a module
• Use go run to run the current main package
• Use go run ./path/to/main/package to build and run the main package in the given directory
• Use go run <moduleName/mainpkg> to build and run the module’s main under the given directory

Let’s write the main function that starts an HTTP server. The following snippet is cmd/webform/main.go:

package main
import (
    "net/http"
)
func main() {
    server := http.Server{
        Addr:    ":8181",
        Handler: http.FileServer(http.Dir("web/static")),
    }
    server.ListenAndServe()
}

Currently, main only imports the standard library’s net/http package. It starts a server that serves the files under the web/static directory. Note that for this to work, you have to run the program from the module root:

$ go run ./cmd/webform

Always run the main package; avoid go run main.go. This will run main.go, excluding any other files in the main package. It will fail if you have other .go files that contain helper functions in the main package.

If you run this program from another directory, it will fail to find the web/static directory; because it is a relative path, it is resolved relative to the current directory.

When you run a program via go run, the program executable is placed in a temporary directory. To build the executable, use the following:

$ go build ./cmd/webform

This will create a binary in the current directory. The name of the binary will be determined by the last segment of the main package – in this case, webform. To build a binary with a different name, use the following:

$ go build -o wform ./cmd/webform

This will build a binary called wform.

Most projects will depend on third-party libraries that must be imported into them. The Go module system manages these dependencies.

How to do it...

1. Find the import path of the package you need to use in your project.
2. Add the necessary imports to the source files you use in the external package.
3. Use the go get or go mod tidy command to add the module to go.mod and go.sum. If the module was not downloaded before, this step will also download the module.

Tip

You can use https://pkg.go.dev to discover packages. It is also the place to publish documentation for the Go projects you publish.

Let’s add a database to our program from the previous section so that we can store the data submitted by the web form. For this exercise, we will use the SQLite database.

Change the cmd/webform/main.go file to import the database package and add the necessary database initialization code:

package main
import (
    "net/http"
    "database/sql"
    _ "modernc.org/sqlite"
    "github.com/PacktPublishing/Go-Recipes-for-Developers/src/chp1/
    webform/pkg/commentdb"
)
func main() {
    db, err := sql.Open("sqlite", "webform.db")
    if err != nil {
        panic(err)
    }
    commentdb.InitDB(db)
    server := http.Server{
        Addr:    ":8181",
        Handler: http.FileServer(http.Dir("web/static")),
    }
    server.ListenAndServe()
}

The _ "modernc.org/sqlite" line imports the SQLite driver into the project. The underscore is the blank identifier, meaning that the sqlite package is not directly used by this file and is only included for its side effects. Without the blank identifier, the compiler would complain that the import was not used. In this case, the modernc.org/sqlite package is a database driver, and when you import it, its init() functions will register the required driver with the standard library.

The next declaration imports the commentdb package from our module. Note that the complete module name is used to import the package. The build system will recognize the prefix of this import declaration as the current module name, and it will translate it to a local filesystem reference, which, in this case, is webform/pkg/commentdb.

On the db, err := sql.Open("sqlite", "webform.db") line, we use the database/sql package function, Open, to start a SQLite database instance. sqlite names the database driver, which was registered by the imported _ "modernc.org/sqlite".

The commentdb.InitDB(db) statement will call a function from the commentdb package .

Now, let’s see what commentdb.InitDB looks like. This is the webform/pkg/commentdb/initdb.go file:

package commentdb
import (
    "context"
    "database/sql"
)
const createStmt=`create table if not exists comments (
email TEXT,
comment TEXT)`
func InitDB(conn *sql.DB) {
    _, err := conn.ExecContext(context.Background(), createStmt)
    if err != nil {
        panic(err)
    }
}

As you can see, this function creates the database tables if they have not been created yet.

Note the capitalization of InitDB. If the first letter of a symbol name declared in a package is a capital letter, that symbol is accessible from other packages (i.e., it is exported). If not, the symbol can only be used within the package it is declared (i.e., it is not exported). The createStmt constant is not exported and will be invisible to other packages.

Let’s build the program:

$ go build ./cmd/webform
  cmd/webform/main.go:7:2: no required module provides package modernc.org/sqlite; to add it:
      go get modernc.org/sqlite

You can run go get modernc.org/sqlite to add a module to your project. Alternatively, you can run the following:

$ go get

That will get all the missing modules. Alternatively, you can run the following:

$ go mod tidy

go mod tidy will download all missing packages, update go.mod and go.sum with updated dependencies, and remove references to any unused modules. go get will only download missing modules.

Sometimes, you need a specific version of a third-party package because of API incompatibilities or a particular behavior you depend on.

How to do it...

• To get a specific version of a package, specify the version label:

  $ go get modernc.org/sqlite@v1.26.0

• To get the latest release of a specific major version of a package, use this:

  $ go get gopkg.in/yaml.v3

  Alternatively, use this:

  $ go get github.com/ory/dockertest/v3

• To import the latest available version, use this:

  $ go get modernc.org/sqlite

• You can also specify a different branch. The following will get a module from the devel branch, if there is one:

  $ go get modernc.org/sqlite@devel

• Alternatively, you can get a specific commit:

  $ go get modernc.org/sqlite@a8c3eea199bc8fdc39391d5d261eaa3577566050

As you can see, you can get a specific revision of a module using the @revision convention:

$ go get modernc.org/sqlite@v1.26.0

The revision part of the URL is evaluated by the version control system, which, in this case, is git, so any valid git revision syntax can be used.

Tip:

You can find which revision control systems are supported by checking out the src/cmd/go/alldocs.go file under your Go installation.

That also means you can use branches:

$ go get modernc.org/sqlite@master

Tip

The https://gopkg.in service translates version numbers to URLs compatible with the Go build system. Refer to the instructions on that website on how to use it.

The module cache is a directory where the Go build system stores downloaded module files. This section describes how to work with the module cache.

How to do it...

The module cache is, by default, under $GOPATH/pkg/mod, which is $HOME/go/pkg/mod when GOPATH is not set:

• By default, the Go build system creates read-only files under the module cache to prevent accidental modifications.
• To verify that the module cache is not modified and reflects the original versions of modules, use this:

  go mod verify

• To clean up the module cache, use this:

  go clean -modcache

The authoritative source for information about the module cache is the Go Modules Reference (https://go.dev/ref/mod)

Not every piece of code is reusable. Having a smaller API surface makes it easier for others to adapt and use your code. So, you should not export APIs that are specific to your program.

How to do it...

Create internal packages to hide implementation details from other packages. Anything under an internal package can only be imported from the packages under the package containing that internal package – that is, anything under myproject/internal can only be imported from the packages under myproject.

In our example, we placed the database access code into a package where it can be accessed by other programs. However, it does not make sense to expose the HTTP routes to others, as they are specific to this program. So, we will put them under the webform/internal package.

This is the internal/routes/routes.go file:

package routes
import (
    "database/sql"
    "github.com/gorilla/mux"
    "net/http"
)
func Build(router *mux.Router, conn *sql.DB) {
    router.Path("/form").
        Methods("GET").HandlerFunc(func(w http.ResponseWriter, r 
        *http.Request) {
        http.ServeFile(w, r, "web/static/form.html")
    })
    router.Path("/form").
        Methods("POST").HandlerFunc(func(w http.ResponseWriter, r 
        *http.Request) {
        handlePost(conn, w, r)
    })
}
func handlePost(conn *sql.DB, w http.ResponseWriter, r *http.Request) {
    email := r.PostFormValue("email")
    comment := r.PostFormValue("comment")
    _, err := conn.ExecContext(r.Context(), "insert into comments 
    (email,comment) values (?,?)",
    email, comment)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    http.Redirect(w, r, "/form", http.StatusFound)
}

Then, we change the main.go file to use the internal package:

package main
import (
    "database/sql"
    "net/http"
    "github.com/gorilla/mux"
    _ "modernc.org/sqlite"
    "github.com/PacktPublishing/Go-Recipes-for-Developers/src/chp1/
    webform/internal/routes"
    "github.com/PacktPublishing/Go-Recipes-for-Developers/src/chp1/
    webform/pkg/commentdb"
)
func main() {
    db, err := sql.Open("sqlite", "webform.db")
    if err != nil {
        panic(err)
    }
    commentdb.InitDB(db)
    r := mux.NewRouter()
    routes.Build(r, db)
    server := http.Server{
        Addr:    ":8181",
        Handler: r,
    }
    server.ListenAndServe()
}

Sometimes, you will work on multiple modules, or you download a module from a repository, make some changes to it, and then want to use the changed version instead of the version available on the repository.

How to do it...

Use the replace directive in go.mod to point to the local directory containing a module.

Let’s return to our example – suppose you want to make some changes to the sqlite package:

1. Clone it:

   $ ls
     webform
   $ git clone git@gitlab.com:cznic/sqlite.git
   $ ls
     sqlite
     webform

2. Modify the go.mod file under your project to point to the local copy of the module. go.mod becomes the following:

   module github.com/PacktPublishing/Go-Recipes-for-Developers/chapter1/webform
   go 1.22.1
   replace modernc.org/sqlite => ../sqlite
   require (
       github.com/gorilla/mux v1.8.1
       modernc.org/sqlite v1.27.0
   )
   ...

3. You can now make changes in the sqlite module on your system, and those changes will be built into your application.

Sometimes you need to work with multiple interdependent modules. A convenient way to do this is by defining a workspace. A workspace is simply a set of modules. If one of the modules within a workspace refers to a package in another module in the same workspace, it is resolved locally instead of that module being downloaded over the network.

How to do it...

1. To create a workspace, you have to have a parent directory containing all your work modules:

   $ cd ~/projects
   $ mkdir ws
   $ cd ws

2. Then, start a workspace using this:

   $ go work init

   This will create a go.work file in this directory.

3. Place the module you are working on into this directory.

   Let’s demonstrate this using our example. Let’s say we have the following directory structure:

   $HOME/
     projects/
       ws/
          go.work
          webform
          sqlite

   Now, we want to add the two modules, webform and sqlite, to the workspace. To do that, use this:

   $ go work use ./webform
   $ go work use ./sqlite

   These commands will add the two modules to your workspace. Any sqlite reference from the webform module will now be resolved to use the local copy of the module.

Go tooling uses the semantic versioning system. This means that the version numbers are of the X.Y.z form, broken down as follows:

• X is incremented for major releases that are not necessarily backward compatible.
• Y is incremented for minor releases that are incremental but backward-compatible
• z is incremented for backward-compatible patches

You can learn more about semantic versioning at https://semver.org.

How to do it...

• To publish a patch or minor version, tag the branch containing your changes with the new version number:

  $ git tag v1.0.0
  $ git push origin v1.0.0

• If you want to publish a new release that has an incompatible API with the previous releases, you should increment the major versions of that module. To release a new major version of your module, use a new branch:

  $ git checkout -b v2

  Then, change your module name in go.mod to end with /v2, and update all references in the source tree to use the /v2 version of the module.

For example, let’s say you released the first version of the webform module, v1.0.0. Then, you decided you would like to add new API endpoints. This would not be a breaking change, so you simply increment the minor version number – v1.1.0. But then it turns out some of the APIs you added were causing problems, so you removed them. Now, that is a breaking change, so you should publish v2.0.0 with it. How can you do that?

The answer is, you use a new branch in the version control system. Create the v2 branch:

$ git checkout -b v2

Then, change go.mod to reflect the new version:

module github.com/PacktPublishing/Go-Recipes-for-Developers/chapter1/webform/v2
go 1.22.1
require (
  ...
)

If there are multiple packages in the module, you have to update the source tree so that any references to packages within that module also use the v2 version.

Commit and push the new branch:

$ git add go.mod
$ git commit -m "New version"
$ git push origin v2

To use the new version, you now have to import the v2 version of the packages:

import "github.com/PacktPublishing/Go-Recipes-for-Developers/chapter1/webform/v2/pkg/commentdb"

This chapter focused on the concepts and mechanics of setting up and managing Go projects. It is by no means an exhaustive reference, but the recipes presented here should give you the basics of using the Go build system effectively.

The definitive guide for Go modules is the Go Modules Reference (https://go.dev/ref/mod).

Check out the Managing dependencies link (https://go.dev/doc/modules/managing-dependencies) for a detailed discussion on dependency management.

In the next chapter, we will start working with textual data.