Adopting a collaborative development process using GitHub
While you may be working on your own, it is good practice to adopt a development process that allows others to collaborate and one that ensures that the code is always ready to be deployed to production. We will achieve both aims by using a remote repository and Continuous Integration (CI).
A remote repository acts as a backup for all your code and makes it much easier to set up CI (testing, linting, and so on). We’ll use GitHub as I find it to have all the features needed, although other platforms, such as GitLab, are also valid and commonly used in the industry.
Rather than creating the repository through GitHub’s UI, we’ll use Terraform as set up earlier. To do so, we’ll first need a personal access token from GitHub, as explained at https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token. The token will need the repo
, workflow
, and delete_repo
scopes. This token is a secret and hence best placed in infrastructure/secrets.auto.tfvars and encrypted as described earlier in the Managing secrets section. The code should be placed into infrastructure/secrets.auto.tfvars as follows (replace abc1234
with your token):
github_token = "abc1234"
Terraform itself does not know how to interact with GitHub, which means that we need to install the GitHub provider to do so. This is done by adding the following code to infrastructure/main.tf:
terraform { required_providers { github = { source = "integrations/github" version = "~> 4.0" } } required_version = ">=1.0" }
With the provider present, we can describe the repository we would like to exist by adding the following code to infrastructure/github.tf:
variable "github_token" { sensitive = true } provider "github" { token = var.github_token } resource "github_repository" "tozo" { name = "tozo" visibility = "private" }
Finally, to actually create the repository, we need to initialize and apply Terraform as follows:
terraform init terraform apply
We should now set up git
so that it knows about the remote repository. To do this, we’ll need the correct path, which will depend on your GitHub account name and the name of your project. As my GitHub account name is pgjones and this project is called tozo, the path is pgjones/tozo, making the following command:
git remote add origin git@github.com:pgjones/tozo.git
To have our local branch track the remote origin
main
branch, run the following command:
git push --set-upstream origin main
To push our local changes on our main
branch to the remote feature
branch, run the following command:
git push origin main:feature
To pull the remote main
branch to update our local branch, run the following command:
git pull origin main
Most in this industry operate a development workflow based on merge (pull) requests, which we’ll also adopt. This workflow consists of the following steps:
- Develop a feature locally consisting of as few commits as makes sense (small changes).
- Push the feature to a remote
feature
branch. - Open a merge request based on that branch.
- Review the merge request, merging it to the
main
branch only if CI passes. - Pull the latest
main
branch and repeat.
With the repository created, we can now look at adding CI.
Adding continuous integration
GitHub provides a CI system called Actions that has a free tier, which we’ll use. To start, we need to create the following folder structure:
tozo └── .github └── workflows
Now we can configure a workflow that runs jobs on every change to the main
branch and every merge request by adding the following code to .github/workflows/ci.yml:
name: CI on: push: branches: [ main ] pull_request: branches: [ main ] workflow_dispatch: jobs:
This allows us to add jobs for the infrastructure, backend, and frontend.
Adding CI for the infrastructure code
We previously set up the commands to format and lint the infrastructure code as follows:
terraform fmt --check=true terraform validate
To have these run as part of CI, we need to add the following job to the .github/workflows/ci.yml file within the jobs
section:
infrastructure: runs-on: ubuntu-latest steps: - name: Install Terraform run: | sudo apt-get update && sudo apt-get install -y gnupg software-properties-common curl curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add - sudo apt-add-repository "deb [arch=amd64] https:// apt.releases.hashicorp.com $(lsb_release -cs) main" sudo apt-get update && sudo apt-get install terraform - uses: actions/checkout@v3 - name: Initialise Terraform run: terraform init - name: Check the formatting run: terraform fmt --check=true --recursive - name: Validate the code run: terraform validate
We can now add a job for the backend code.
Adding CI for the backend code
We previously set up the commands to format, lint, and test the backend code as follows:
pdm run format pdm run lint pdm run test
To have these run as part of CI, we will need to have a database service running as well, as the tests run against the database. Fortunately, GitHub supports PostgreSQL database services by running a PostgreSQL database alongside the CI job. We can make use of this database service and run the commands by adding the following job to the jobs
section in .github/workflows/ci.yml:
backend: runs-on: ubuntu-latest container: python:3.10.1-slim-bullseye services: postgres: image: postgres env: POSTGRES_DB: tozo_test POSTGRES_USER: tozo POSTGRES_PASSWORD: tozo POSTGRES_HOST_AUTH_METHOD: "trust" options: >- --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 defaults: run: working-directory: backend env: TOZO_QUART_DB_DATABASE_URL: "postgresql://tozo:tozo@ postgres:5432/tozo_test" steps: - uses: actions/checkout@v3 - name: Install system dependencies run: apt-get update && apt-get install -y postgresql postgresql-contrib - name: Initialise dependencies run: | pip install pdm pdm install - name: Linting run: pdm run lint - name: Testing run: pdm run test
We can now add a job for the frontend code.
Adding CI for the frontend code
We previously set up the commands to format, lint, test, and build the frontend code as follows:
npm run format npm run lint npm run test npm run build
We can make use of the service and run the commands by adding the following job to the jobs
section of .github/workflows/ci.yml:
frontend: runs-on: ubuntu-latest defaults: run: working-directory: frontend steps: - name: Use Node.js uses: actions/setup-node@v2 with: node-version: '18' - uses: actions/checkout@v3 - name: Initialise dependencies run: npm ci --cache .npm --prefer-offline - name: Check formatting run: npm run format - name: Linting run: npm run lint - name: Testing run: npm run test - name: Build run: npm run build
We now have everything we need in place to start developing our app. The folder structure at this stage is as follows:
tozo ├── .github │ └── workflows ├── backend │ ├── src │ │ └── backend │ └── tests ├── frontend │ ├── public │ └── src └── infrastructure
We now have all of our checks running on every change to the main
branch and for every pull request. This should ensure that our code remains at a high quality and alert us to any issues that may otherwise be missed.