Writing tests and benchmarks

When we start developing, tests take a backseat more often than not. There are several reasons why this might be necessary at the time, but the inability to set up a testing framework and surroundings is not one of them. Unlike many languages, Rust supports testing right out of the box. This recipe covers how to use these tools.

Although we mostly talk about unit testing here, that is, tests on a function/struct level, the tools remain the same for integration tests.

Getting ready

Again, this recipe is best worked on in its own project space. Use cargo new testing --lib to create the project. Inside the project directory, create another folder and call it tests.

Additionally, the benchmarks feature is still only available on the nightly branch of Rust. It is required to install the nightly build of Rust: rustup install nightly.

How to do it...

Follow these steps to learn more about creating a test suite for your Rust projects:

1. Once created, a library project already contains a very simple test (probably to encourage you to write more). The cfg(test) and test attributes tell cargo (the test runner) how to deal with the module:

#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
        assert_eq!(2 + 2, 4);
    }
}

2. Before we add further tests, let's add a subject that needs testing. In this case, let's use something interesting: a singly linked list from our other book (Hands-On Data Structures and Algorithms with Rust) made generic. It consists of three parts. First is a node type:

#[derive(Clone)]
struct Node<T> where T: Sized + Clone {
    value: T,
    next: Link<T>,
}

impl<T> Node<T> where T: Sized + Clone {
    fn new(value: T) -> Rc<RefCell<Node<T>>> {
        Rc::new(RefCell::new(Node {
            value: value,
            next: None,
        }))
    }
}

Second, we have a Link type to make writing easier:

type Link<T> = Option<Rc<RefCell<Node<T>>>>;

The last type is the list complete with functions to add and remove nodes. First, we have the type definition:

#[derive(Clone)]
pub struct List<T> where T: Sized + Clone {
    head: Link<T>,
    tail: Link<T>,
    pub length: usize,
}

In the impl block, we can then specify the operations for the type:

impl<T> List<T> where T: Sized + Clone {
    pub fn new_empty() -> List<T> {
        List { head: None, tail: None, length: 0 }
    }

    pub fn append(&mut self, value: T) {
        let new = Node::new(value);
        match self.tail.take() {
            Some(old) => old.borrow_mut().next = Some(new.clone()), 
            None => self.head = Some(new.clone())
        }; 
        self.length += 1;
        self.tail = Some(new);
    }

    pub fn pop(&mut self) -> Option<T> {
        self.head.take().map(|head| {
            if let Some(next) = head.borrow_mut().next.take() {
                self.head = Some(next);
            } else {
                self.tail.take();
            }
            self.length -= 1;
            Rc::try_unwrap(head)
                .ok()
                .expect("Something is terribly wrong")
                .into_inner()
                .value
        })
    }
}

3. With the list ready to be tested, let's add some tests for each function, starting with a benchmark:

#[cfg(test)]
mod tests {
    use super::*;
    extern crate test;
    use test::Bencher;

    #[bench]
    fn bench_list_append(b: &mut Bencher) {
        let mut list = List::new_empty();
        b.iter(|| {
            list.append(10);
        });
    }

Add some more tests for basic list functionality inside the test module:

    #[test]
    fn test_list_new_empty() {
        let mut list: List<i32> = List::new_empty();
        assert_eq!(list.length, 0);
        assert_eq!(list.pop(), None);
    } 

    #[test]
    fn test_list_append() {
        let mut list = List::new_empty();
        list.append(1);
        list.append(1);
        list.append(1);
        list.append(1);
        list.append(1);
        assert_eq!(list.length, 5);
    }


    #[test]
    fn test_list_pop() {
        let mut list = List::new_empty();
        list.append(1);
        list.append(1);
        list.append(1);
        list.append(1);
        list.append(1);
        assert_eq!(list.length, 5);
        assert_eq!(list.pop(), Some(1));
        assert_eq!(list.pop(), Some(1));
        assert_eq!(list.pop(), Some(1));
        assert_eq!(list.pop(), Some(1));
        assert_eq!(list.pop(), Some(1));
        assert_eq!(list.length, 0);
        assert_eq!(list.pop(), None);
    }
}

4. It's also a good idea to have an integration test that tests the library from end to end. For that, Rust offers a special folder in the project called tests, which can house additional tests that treat the library as a black box. Create and open the tests/list_integration.rs file to add a test that inserts 10,000 items into our list:

use testing::List;

#[test]
fn test_list_insert_10k_items() {
    let mut list = List::new_empty();
    for _ in 0..10_000 {
        list.append(100);
    }
    assert_eq!(list.length, 10_000);
}

5. Great, now each function has one test. Try it out by running cargo +nightly test in the testing/ root directory. The result should look like this:

$ cargo test
   Compiling testing v0.1.0 (Rust-Cookbook/Chapter01/testing)
    Finished dev [unoptimized + debuginfo] target(s) in 0.93s
     Running target/debug/deps/testing-a0355a7fb781369f

running 4 tests
test tests::test_list_new_empty ... ok
test tests::test_list_pop ... ok
test tests::test_list_append ... ok
test tests::bench_list_append ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/list_integration-77544dc154f309b3

running 1 test
test test_list_insert_10k_items ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests testing

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

6. To run the benchmark, issue cargo +nightly bench:

cargo +nightly bench
   Compiling testing v0.1.0 (Rust-Cookbook/Chapter01/testing)
    Finished release [optimized] target(s) in 0.81s
     Running target/release/deps/testing-246b46f1969c54dd

running 4 tests
test tests::test_list_append ... ignored
test tests::test_list_new_empty ... ignored
test tests::test_list_pop ... ignored
test tests::bench_list_append ... bench: 78 ns/iter (+/- 238)

test result: ok. 0 passed; 0 failed; 3 ignored; 1 measured; 0 filtered out

Now, let's go behind the scenes to understand the code better.

How it works...

Testing frameworks are a third-party library in many programming languages although well-tested code should be the default! By providing a (tiny) testing framework along with a test runner and even a small benchmarking framework (only on nightly as of this writing), the barrier for testing your Rust code is significantly lower. Although there are still some missing features (for example, mocking), the community is working on providing many of these things via external crates.

After setting everything up in step 1, step 2 creates a singly linked list as the test subject. A singly linked list is a series of the same node types, connected with some sort of pointer. In this recipe, we decided to use the interior mutability pattern, which allows for borrowing mutably at runtime to modify the node it points to. The attached operations (append() and pop()) make use of this pattern. Step 3 then creates the tests that we can use to verify that our code does what we think it should. These tests cover the basic workings of the list: create an empty list, append a few items, and remove them again using pop.

Tests can be failed using a variety of assert! macros. They cover equals (assert_eq!), not equals (assert_ne!), Boolean conditions (assert!), and non-release mode compilation only (debug_assert!). With these available and attributes such as #[should_panic], there is no case that cannot be covered. Additionally, this great Rust book offers an interesting read as well: https://doc.rust-lang.org/book/ch11-01-writing-tests.html.

Step 4 adds a special integration test in a separate file. This restricts programmers to think like the user of the crate, without access to internal modules and functions that can be available in the nested tests module. As a simple test, we insert 10,000 items into the list to see whether it can handle the volume.

The +nightly parameter instructs cargo to use the nightly toolchain for this command.

Only in step 5 are we ready to run the benchmarks using cargo +nightly test, but tests are not automatically benchmarked. On top of that, benchmarks (cargo +nightly bench) compile the code using --release flags, thereby adding several optimizations that could lead to different outcomes from cargo +nightly test (including a headache for debugging those).

Step 6 shows the output of the benchmarking harness with nanosecond precision for each loop execution (and the standard deviation). Whenever doing any kind of performance optimization, have a benchmark ready to show that it actually worked!

Other nice things that the Rust documentation tool adds to testing are doctests. These are snippets that are compiled and executed as well as rendered as documentation. We were so delighted, we gave it its own recipe! So, let's move on to the next recipe.