DevOping - 2

DevOping - 2: Unit Testing and GitHub Actions

This is not intended to be a guide. I’m sure that everything explained here can be done in a better/easier/more efficient way. Here, I will explain the whole learning process and the solutions I found that could solve my problems.

In the last post, I introduced the project and explained how I configured the database and implemented some CRUD functions. I will continue the explanation by detailing how I created some unit tests and utilized them using GitHub Actions workflows.

Unit Testing

When we develop code, it is important to also develop tests to ensure that everything works as expected. A unit test is a test that validates that a function works as expected. To design unit tests, you have to consider different inputs and the corresponding outputs that the function should return.

If someone modifies the function—for example, by adding new functionality or extra code—and the outputs are not as expected, this indicates that the function has been broken and no longer functions correctly. Unit tests are incredibly helpful for detecting bugs at an early stage.

A test is also a function that prepares the input, calls the function being tested, and checks the output. There are libraries that help with all this. In my case, I used GoLang “testing” package.

Let’s do an example. This function tests the GetRestaurantbyID() function.

func TestGetRestaurant(t *testing.T) {
	//there is a sample restaurant in the database, this tests expects to retrieve it.
	restaurant, err := GetRestaurantByID(1)

	expectedRestaurant := models.Restaurant{
		ID:          1,
		Name:        "Sample Restaurant",
		Type:        "Mexican",
		Street_no:   123,
		Street_name: "Sample Street",
		City:        "Sample City",
		Postal_code: "12345",
		Country:     "Sample Country",
		Punctuation: 4,
		NumReviews:  10,
	}
	// Check if the function behaves as expected
	if err != nil {
		t.Errorf("GetRestaurantByID failed: %v", err)
	}

	if *restaurant != expectedRestaurant {
		t.Errorf("Expected %v, got %v", expectedRestaurant, *restaurant)
	}

}

My database contains a predefined restaurant with fields that I made up. Since I know that this restaurant exists in the DB, I can expect to obtain it when using the GetRestaurantByID() function with the restaurant ID, which I already know is 1.

You can see that I define a restaurant with the fields that I expect to obtain and compare it with the restaurant returned by the function being tested. If they are not the same, I raise a testing error.

All these testing functions were defined in a database_test.go file I created. In GoLang, if you execute the go test command, it will search for all the files that end with the suffix “_test.go” and execute all the testing functions there.

If all the tests are successful, we will see a PASS at the end of all tests; otherwise, we get a FAIL.

I wrote some tests to test some CRUD functions, but I didn’t spend the time to write exhaustive tests for all the functions.

Now that I had some tests, my goal was to automate them in a way where whenever a new push is made to the main branch of the code repository, the tests get executed.

Since I was using GitHub as my code repository, I used GitHub Actions to automate this.

GitHub Actions

GitHub Actions allow us to automate workflows whenever specific events (like pushes, merges, etc.) occur in our code repository. These workflows are defined using YAML language.

To create a workflow, in your repository’s root folder, create a new workflows folder. In this workflows folder, you can create a new .yml file with the name of the workflow.

The syntax is straightforward, and you can import other actions (as if they were libraries) and execute them. In this action, I just want to execute the go test command.

This was my initial main.yml workflow (SPOILER: It didn’t work).

#This code is not correct!!!
name: Go Test with MySQL

on:
  push:
    branches:
      - master

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Set up Go
      uses: actions/setup-go@v5
      with:
        go-version: '1.21'

    - name: Run tests
      id: tests
      working-directory: ./backend/database
      run: go test  

Let’s divide this workflow into chunks:

Here we define the name of the workflow using the name keyword. Afterwards, we specify when we want this workflow to be executed using the on keyword. This workflow will be executed every time a push is made to the master/main branch.

name: Go Test with MySQL

on:
  push:
    branches:
      - master

Afterwards, using the jobs keyword, we define different units of flow, each one running in a specific execution environment. In this case, we only have one job named “test” that will be executed in a Ubuntu environment. Inside this job, we can create different sequential tasks using the steps keyword.

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Set up Go
      uses: actions/setup-go@v5
      with:
        go-version: '1.21'
    - name: Run tests
      id: tests
      working-directory: ./backend/database
      run: go test  

The first task uses an already defined action (actions/checkout@v4) to check out all the code from the repository in this dedicated environment. Then, since we will be using GoLang, we make use of another predefined action (actions/setup-go@v5) to install it. Finally, we navigate to the directory where the database_tests.go file exists using the working-directory clause. Finally, using the run keyword, we can specify the commands that we want to execute.

Facing my first problem

The workflow that I provided before didn’t work. Well, it got executed, but the tests failed because the connection with the database wasn’t possible. I was facing three problems:

• I don’t have a MySQL database. Also, in my previous post, I explained how I installed a MySQL database on my computer and created users, tables, etc. In this environment, there isn’t a database with the tables defined.
• Even if I had a MySQL database, it would be empty. Some tests expect already existing data in the database, so I need to create this testing data before executing the tests.
• My database connection was using environment variables. In the previous post, I explained the code used to connect to the database. I use environmental variables to specify the port, IP, password, etc. Since this workflow is running in a different environment, the variables don’t exist.

I will explain how I solved all this.

mySQL in GitHub Actions instances

I spent a lot of time trying to find the proper solution to this. I was attempting to add a MySQL database using other actions from the marketplace. However, nothing seemed to solve the issue. After reading and trying things for several hours, I came across this post on Stack Overflow: https://stackoverflow.com/questions/72294279/how-to-connect-to-mysql-databas-using-github-actions

Here, it is mentioned that the instances that run the workflows already have MySQL installed but disabled by default. Hence, instead of installing MySQL, we just need to start the service. This is the task in the workflow file:

    - name: Start mySQL
      run: | 
        sudo systemctl start mysql.service
        sleep 10

Initialize the database

Now that we have a working database, we need to create all the tables and some database entries in order to execute the tests as expected. To do this, I created an init.sql file that contains SQL queries to create all the tables and necessary entries:

CREATE SCHEMA IF NOT EXISTS `restaurant_app` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci ;
USE `restaurant_app` ;

-- -----------------------------------------------------
-- Table `restaurant_app`.`restaurants`
-- -----------------------------------------------------
CREATE TABLE IF NOT EXISTS `restaurant_app`.`restaurants` (
  `id` INT NOT NULL AUTO_INCREMENT,
  `name` VARCHAR(255) NOT NULL,
  `type` VARCHAR(255) NULL DEFAULT NULL,
  `street_no` INT NULL DEFAULT NULL,
  `street_name` VARCHAR(255) NULL DEFAULT NULL,
  `city` VARCHAR(255) NULL DEFAULT NULL,
  `postal_code` VARCHAR(10) NULL DEFAULT NULL,
  `country` VARCHAR(255) NULL DEFAULT NULL,
  `punctuation` INT NULL DEFAULT NULL,
  `num_reviews` INT NULL DEFAULT NULL,
  PRIMARY KEY (`id`))
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8mb4
COLLATE = utf8mb4_0900_ai_ci;

//Create all tables and insert values using sql syntax

This is just the query that creates the schema and one table; you can imagine the rest.

Now that we have this script, we need to execute it.

When configuring my localhost database, I created a specific user with a specific password, but for this temporary environment, we can use the default root username/password. The task that makes use of the init.sql file and executes it to initialize the database looks like this:

    - name: Execute SQL script
      working-directory: ./backend/database
      run: |
        sudo mysql --user=root --password=root < ./init.sql

Environment variables

My code connects to the database using environmental variables to specify the IP, port, username, password, and database. I need to create these variables in the Workflow environment. You can specify them at the beginning of your file:

env: 
  DB_USERNAME: root
  DB_PASSWORD: root
  DB_DATABASE: restaurant_app
  DB_PORT: 3306
  DB_HOSTNAME: localhost

We use the default username and password as mentioned before, and the hostname is still localhost since the database is in the same VM where the workflow gets executed.

With all three modifications, the final main.yml file works like this:

name: Go Test with MySQL

env: 
  DB_USERNAME: root
  DB_PASSWORD: root
  DB_DATABASE: restaurant_app
  DB_PORT: 3306
  DB_HOSTNAME: localhost
on:
  push:
    branches:
      - master

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Set up Go
      uses: actions/setup-go@v5
      with:
        go-version: '1.21'
    - name: Start mySQL
      run: | 
        sudo systemctl start mysql.service
        sleep 10
    - name: Execute SQL script
      working-directory: ./backend/database
      run: |
        sudo mysql --user=root --password=root < ./init.sql

    - name: Run tests
      id: tests
      working-directory: ./backend/database
      run: go test  

GitHub Actions in action

In your GitHub repository, you can go to the “Actions” menu and you will see all the executions of your workflows:

If the workflow fails, you can see what tasks and error occurred. For example, on this push my workflow failed because the GetRestaurantByID function failed.

In a further post, we will create more workflows to do more cool things like building Docker images and deploying them into AWS EC2 instances.

At this point, we have a database with some CRUD functions and a workflow that sets up a dedicated environment to execute unit tests. In the next post, we will develop endpoints to interact with the database.