It is required for you to install the SQLite runtime library when you're running the SQLite code. However, it is also useful (although not required) to install a SQLite interactive manager. You can download the precompiled binaries of SQLite tools from https://www.sqlite.org/download.html. However, version 3.11 or higher would be ideal.

Please note that if you're using Debian-derived Linux distribution, the libsqlite3-dev package should be installed.

It is also required for you to install and run the PostgreSQL Database Management System (DBMS) when you're running the PostgreSQL code. As with SQLite, it is useful but not required to install a PostgreSQL interactive manager. You can download the precompiled binary of PostgreSQL DBMS from https://www.postgresql.org/download/. However, version 7.4 or higher would be acceptable.

Installing and running the Redis server is necessary when you're running the Redis code. You can download it from https://redis.io/download.

The complete source code for this chapter can be found in the Chapter02 folder of the repository at https://github.com/PacktPublishing/Creative-Projects-for-Rust-Programmers. In this folder, there is a sub-folder for every project, plus a folder named data, which contains the data that we'll use as input for the projects.

In this chapter, we'll look at how to build a program that loads a JSON file and an XML file into three databases: a SQLite database, a PostgreSQL database, and a Redis key-value store. To avoid hardwiring the names and positions of the files and the database credentials into the program, we are going to load them from a TOML configuration file.

The final project is named transformer, but we'll explain this through several preliminary small projects:

• toml_dynamic and toml_static: These read a TOML file in two different ways.
• json_dynamic and json_static: These read a JSON file in two different ways.
• xml_example: This reads an XML file.
• sqlite_example: This creates two tables in a SQLite database, inserts records into them, and queries them.
• postgresql_example: This creates two tables in a PostgreSQL database, inserts records into them, and queries them.
• redis_example: This adds some data to a key-value store and queries it.

One simple and maintainable way to store information in a filesystem is to use a text file. This is also very efficient for data spanning no more than 100 KB. However, there are several competing standards for storing information in text files, such as INI, CSV, JSON, XML, YAML, and others.

The one used by Cargo is TOML. This is a really powerful format that is used by many Rust developers to store the configuration data of their apps. It is designed to be written by hand, using a text editor, but it can also be written by an application very easily.

The toml_dynamic and toml_static projects (using the toml crate) load data from a TOML file. Reading a TOML file is useful when configuring a software application, and this is what we'll do. We will use the data/config.toml file, which contains all of the parameters for the projects of this chapter.

You can also create or modify a TOML file by using code, but we are not going to do that. Being able to modify a TOML file can be useful in some scenarios, such as to save user preferences.

It is important to consider that when a TOML file is changed by a program, it undergoes dramatic restructuring:

• It acquires specific formatting, which you may dislike.
• It loses all of its comments.
• Its items are sorted alphabetically.

So, if you want to use the TOML format both for manually edited parameters and for program-saved data, you would be better off using two distinct files:

• One edited only by humans
• One edited primarily by your software, but occasionally also by humans

This chapter describes two projects in which a TOML file is read using different techniques. These techniques are to be used in two different cases:

• In a situation where we are not sure which fields are contained in the file, and so we want to explore it. In this case, we use the toml_dynamic program.
• In another situation where, in our program, we describe exactly which fields should be contained in the file and we don't accept a different format. In this case, we use the toml_static program.

Using toml_dynamic

The purpose of this section is to read the config.toml file, located in the data folder, when we want to explore the content of that file. The first three lines of this file are as follows:

[input]
xml_file = "../data/sales.xml"
json_file = "../data/sales.json"

After these lines, the file contains other sections. Among them is the [postgresql] section, which contains the following line:

database = "Rust2018"

To run this project, enter the toml_dynamic folder and type in cargo run ../data/config.toml. A long output should be printed. It will begin with the following lines:

Original: Table(
    {
        "input": Table(
            {
                "json_file": String(
                    "../data/sales.json",
                ),
                "xml_file": String(
                    "../data/sales.xml",
                ),
            },
        ),

Notice that this is just a verbose representation of the first three lines of the config.toml file. This output proceeds with emitting a similar representation for the rest of the file. After having printed the whole data structure representing the file that is read, the following line is added to the output:

 [Postgresql].Database: Rust2018

This is the result of a specific query on the data structure loaded when the file is read.

Let's look at the code of the toml_dynamic program:

1. Declare a variable that will contain a description of the whole file. This variable is initialized in the next three statements:

let config_const_values =

2. We add the pathname of the file from the first argument in the command line to config_path. Then, we load the contents of this file into the config_text string and we parse this string into a toml::Value structure. This is a recursive structure because it can have a Value property among its fields:

{
    let config_path = std::env::args().nth(1).unwrap();
    let config_text = 
     std::fs::read_to_string(&config_path).unwrap();
    config_text.parse::<toml::Value>().unwrap()
};

3. This structure is then printed using the debug structured formatting (:#?), and a value is retrieved from it:

println!("Original: {:#?}", config_const_values);
println!("[Postgresql].Database: {}",
    config_const_values.get("postgresql").unwrap()
    .get("database").unwrap()
    .as_str().unwrap());

Notice that to get the value of the "database" item contained the "postgresql" section, a lot of code is required. The get function needs to look for a string, which may fail. That is the price of uncertainty.

Using toml_static

On the other hand, if we are quite sure of the organization of our TOML file, we should use another technique shown in the project, toml_static.

To run it, open the toml_static folder and type in cargo run ../data/config.toml. The program will only print the following line:

[postgresql].database: Rust2018

This project uses two additional crates:

• serde: This enables the use of the basic serialization/deserialization operations.
• serde_derive: This provides a powerful additional feature known as the custom-derive feature, which allows you to serialize/deserialize using a struct.

serde is the standard serialization/deserialization library. Serialization is the process of converting data structures of the program into a string (or a stream). Deserialization is the reverse process; it is the process of converting a string (or a stream) into some data structures of the program.

To read a TOML file, we need to use deserialization.

In the code, first, a struct is defined for any section contained in the data/config.toml file. That file contains the Input, Redis, Sqlite, and Postgresql sections, and so we declare as many Rust structs as the sections of the file we want to read; then, the Config struct is defined to represent the whole file, having these sections as members.

For example, this is the structure for the Input section:

#[allow(unused)]
#[derive(Deserialize)]
struct Input {
    xml_file: String,
    json_file: String,
}

Notice that the preceding declaration is preceded by two attributes.

The allow(unused) attribute is used to prevent the compiler from warning us about unused fields in the following structure. It is convenient for us to avoid these noisy warnings. The derive(Deserialize) attribute is used to activate the automatic deserialization initiated by serde for the following structure.

After these declarations, it is possible to write the following line of code:

toml::from_str(&config_text).unwrap()

This invokes the from_str function, which parses the text of the file into a struct. The type of that struct is not specified in this expression, but its value is assigned to the variable declared in the first line of the main function:

 let config_const_values: Config =

So, its type is Config.

Any discrepancies between the file's contents and the struct type will be considered an error in this operation. So, if this operation is successful, any other operation on the structure cannot fail.

While the previous program (toml_dynamic) had a kind of dynamic typing, such as that of Python or JavaScript, this program has a kind of static typing, similar to Rust or C++.

The advantage of static typing appears in the last statement, where the same behavior as the long statement of the previous project is obtained by simply writing config_const_values.postgresql.database.

For storing data that is more complex than that which is stored in a configuration file, JSON format is more appropriate. This format is quite popular, particularly among those who use the JavaScript language.

We are going to read and parse the data/sales.json file. This file contains a single anonymous object, which contains two arrays—"products" and "sales".

The "products" array contains two objects, each one having three fields:

  "products": [
    {
      "id": 591,
      "category": "fruit",
      "name": "orange"
    },
    {
      "id": 190,
      "category": "furniture",
      "name": "chair"
    }
  ],

The "sales" array contains three objects, each one containing five fields:

"sales": [
    {
      "id": "2020-7110",
      "product_id": 190,
      "date": 1234527890,
      "quantity": 2.0,
      "unit": "u."
    },
    {
      "id": "2020-2871",
      "product_id": 591,
      "date": 1234567590,
      "quantity": 2.14,
      "unit": "Kg"
    },
    {
      "id": "2020-2583",
      "product_id": 190,
      "date": 1234563890,
      "quantity": 4.0,
      "unit": "u."
    }
  ]

The information in the arrays is about some products to sell and some sale transactions associated with those products. Notice that the second field of each sale ("product_id") is a reference to a product, and so it should be processed after the corresponding product object has been created.

We will see a pair of programs with the same behavior. They read the JSON file, increment the quantity of the second sale object by 1.5, and then save the whole updated structure into another JSON file.

Similarly to the TOML format case, there can also be a dynamic parsing technique used for JSON files, where the existence and type of any data field is checked by the application code, and a static parsing technique, where it uses the deserialization library to check the existence and type of any field.

So, we have two projects: json_dynamic and json_static. To run each of them, open its folder and type in cargo run ../data/sales.json ../data/sales2.json. The program will not print anything, but it will read the first file specified in the command line and create the second file that is specified.

The created file is similar to the read file, but with the following differences:

• The fields of the file created by json_dynamic are sorted in alphabetical order, while the fields of the file created by json_static are sorted in the same order as in the Rust data structure.
• The quantity of the second sale is incremented from 2.14 to 3.64.
• The final empty line is removed in both created files.

Now, we can see the implementations of the two techniques of serialization and deserialization.

The json_dynamic project

Let's look at the source code of the project:

1. This project gets the pathnames of two files from the command line—the existing JSON file ("input_path") to read into a memory structure and a JSON file to create ("output_path") by saving the loaded structure, after having modified it a bit.
2. Then, the input file is loaded into the string named sales_and_products_text and the generic serde_json::from_str::<Value> function is used to parse the string into a dynamically typed structure representing the JSON file. This structure is stored in the sales_and_products local variable.

Imagine that we want to change the quantity sold by the second sale transaction, incrementing it by 1.5 kilograms:

1. First, we must get to this value using the following expression:

sales_and_products["sales"][1]["quantity"]

2. This retrieves the "sales" sub-object of the general object. It is an array containing three objects.

3. Then, this expression gets the second item (starting from zero ([1])) of this array. This is an object representing a single sale transaction.
4. After this, it gets the "quantity" sub-object of the sale transaction object.
5. The value we have reached has a dynamic type that we think should be serde_json::Value::Number, and so we make a pattern matching with this type, specifying the if let Value::Number(n) clause.
6. If all is good, the matching succeeds and we get a variable named n—containing a number, or something that can be converted into a Rust floating-point number by using the as_f64 function. Lastly, we can increment the Rust number and then create a JSON number from it using the from_f64 function. We can then assign this object to the JSON structure using the same expression we used to get it:

sales_and_products["sales"][1]["quantity"]
    = Value::Number(Number::from_f64(
        n.as_f64().unwrap() + 1.5).unwrap());

7. The last statement of the program saves the JSON structure to a file. Here, the serde_json::to_string_pretty function is used. As the name suggests, this function adds formatting whitespace (blanks and new lines) to make the resulting JSON file more human-readable. There is also the serde_json::to_string function, which creates a more compact version of the same information. It is much harder for people to read, but it is somewhat quicker to process for a computer:

std::fs::write(
    output_path,
    serde_json::to_string_pretty(&sales_and_products).unwrap(),
).unwrap();

The json_static project

If, for our program, we are sure that we know the structure of the JSON file, a statically typed technique can and should be used instead. It is shown in the json_static project. The situation here is similar to that of the projects processing the TOML file.

The source code of the static version first declares three structs—one for every object type contained in the JSON file we are going to process. Each struct is preceded by the following attribute:

#[derive(Deserialize, Serialize, Debug)]

The JSON string is parsed using the serde_json::from_str::<SalesAndProducts> function. Then, the code to increment the quantity of sold oranges becomes quite simple:

sales_and_products.sales[1].quantity += 1.5

The rest of the is unchanged.

Another very text format is XML. Unfortunately, there is no stable serialization/deserialization library to manage XML format. However, this is not necessarily a shortcoming. In actual fact, XML format is often used to store large datasets; so large, in fact, that it would be inefficient to load them all before we start converting the data into an internal format. In these cases, it may be more efficient to scan the file or incoming stream and process it as long as it is read.

The xml_example project is a rather convoluted program that scans the XML file specified on the command line and, in a procedural fashion, loads information from the file into a Rust data structure. It is meant to read the ../data/sales.xml file. This file has a structure corresponding to the JSON file we sought in the previous section. The following lines show an excerpt of that file:

<?xml version="1.0" encoding="utf-8"?>
<sales-and-products>
    <product>
        <id>862</id>
    </product>
    <sale>
        <id>2020-3987</id>
    </sale>
</sales-and-products>

All XML files have a header in the first line and then one root element; in this case, the root element it is named sales-and-products. This element contains two kinds of elements—product and sale. Both kinds of elements have specific sub-elements, which are the fields of the corresponding data. In this example, only the id fields are shown.

To run the project, open its folder and type in cargo run ../data/sales.xml. Some lines will be printed on the console. The first four of them should be as follows:

Got product.id: 862.
Got product.category: fruit.
Got product.name: cherry.
  Exit product: Product { id: 862, category: "fruit", name: "cherry" }

These describe the contents of the specified XML file. In particular, the program found a product with ID 862, then it detected that it is a fruit, then that it is a cherry, and then, when the whole product had been read, the whole struct representing the product was printed. A similar output will appear for sales.

The parsing is performed using only the xml-rs crate. This crate enables a mechanism of parsing, shown in the following code excerpt:

let file = std::fs::File::open(pathname).unwrap();
let file = std::io::BufReader::new(file);
let parser = EventReader::new(file);
for event in parser {
    match &location_item {
        LocationItem::Other => ...
        LocationItem::InProduct => ...
        LocationItem::InSale => ...
    }
}

An object of the EventReader type scans the buffered file and it generates an event whenever a step is performed in the parsing. The application code handles these kinds of events according to their needs.

The word event is used by this crate, but the word transition would probably be a better description of the data extracted by the parser.

A complex language is hard to parse, but for languages as simple as our data, the situation during the parsing can be modeled by a state machine. To that purpose, three enum variables are declared in the source code: location_item, with the LocationItem type; location_product, with the LocationProduct type; and location_sale, with the LocationSale type.

The first one indicates the current position of the parsing in general. We can be inside a product (InProduct), inside a sale (InSale), or outside of both (Other). If we are inside a product, the LocationProduct enum indicates the current position of parsing inside the current product. This can be within any of the allowed fields or outside of all of them. Similar states happen for sales.

The iteration encounters several kinds of events. The main ones are the following:

• XmlEvent::StartElement: Signals that an XML element is beginning. It is decorated by the name of the beginning element and the possible attributes of that element.
• XmlEvent::EndElement: Signals that an XML element is ending. It is decorated by the name of the ending element.
• XmlEvent::Characters: Signals that the textual contents of an element is available. It is decorated by that available text.

The program declares a mutable product struct, with the Product type, and a mutable sale struct, with the Sale type. They are initialized with default values. Whenever there are some characters available, they are stored in the corresponding field of the current struct.

For example, consider a situation where the value of location_item is LocationItem::InProduct and the value of location_product is LocationProduct::InCategory—that is, we are in a category of a product. In this situation, there can be the name of the category or the end of the category. To get the name of the category, the code contains this pattern of a match statement:

Ok(XmlEvent::Characters(characters)) => {
    product.category = characters.clone();
    println!("Got product.category: {}.", characters);
}

In this statement, the characters variable gets the name of the category and a clone of it is assigned to the product.category field. Then, the name is printed to the console.