Building the tokenizer
The tokenizer is the module in our system design that reads one or more characters from an arithmetic expression and translates it into a token. In other words, input is a set of characters and output is a set of tokens. In case you are wondering, examples of tokens are Add, Subtract, and Num(2.0).
We have to first create a data structure for two things:
- To store the input arithmetic expression from the user
- To represent the output tokens
In the following section, we will delve into how to determine the right data structures for the tokenizer module.
Tokenizer data structure
To store the input arithmetic expression, we can choose among the following data types:
- String slice
- String
We will choose the &str type, as we do not need to own the value or dynamically increase the size of the expression. This is because the user will provide the arithmetic expression once, and then the expression won't change for...
