When it was first introduced in 2009, MongoDB took the database world by storm, and since that time it has rapidly gained in popularity. According to the 2019 StackOverflow developer survey (https://insights.stackoverflow.com/survey/2019#technology-_-databases), MongoDB is ranked fifth, with 26% of professional developers and 25.5% of all respondents saying they use MongoDB. DB-Engines (https://db-engines.com/en/ranking) also ranks MongoDB as the fifth most widely used database, using an algorithm that takes into account the frequency of search, DBA Stack Exchange and StackOverflow references, and the frequency with which MongoDB appears in job postings. What is of even more interest is that the trend graph generated by DB-Engines shows that the score (and therefore ranking) of MongoDB has grown by 200% since 2013. You can refer to https://db-engines.com/en/ranking_trend for more details.  In 2013, MongoDB was not even in the...

One of the most important distinctions between MongoDB and the traditional relational database management systems (RDBMS) is that instead of tables, rows, and columns, the basis for storage in MongoDB is a document. In a certain sense, you can think of the traditional RDBMS system as two dimensional, whereas MongoDB is three dimensional. Documents are typically modeled using JSON formatting and then inserted into the database where they are converted to a binary format for storage (more on that in later chapters!).

Related to the document basis for storage is the fact that MongoDB documents have no fixed schema. The main benefit of this is vastly reduced overhead. Database restructuring is a piece of cake, and doesn't cause the massive problems, website crashes, and security breaches seen in applications reliant upon a traditional RDBMS database restructuring.

The really great news for developers is that most modern programming applications are based...

Another feature that causes MongoDB to stand out from other database technologies is its ability to ensure high availability through a process known as replication. A server running MongoDB can have copies of its databases duplicated across two more servers. These copies are known as replica sets. Replica sets are organized through an election process whereby the members of the replica vote on which server becomes the primary. Other servers are then assigned the role of secondary. 

This arrangement not only ensures that the database is continuously available, but that it can also be used by application code by way of read preferences. A read preference tells the replica set which servers in the replica set are preferred. If the read preferences are set less restrictively, then the first server in the set to respond might be able to satisfy the request, thereby implementing a form of parallel process that has the potential to greatly enhance performance. This...

One more feature, among many, is MongoDB's ability to handle a massive amount of data. This is accomplished by splitting up a sizeable collection across multiple servers, creating a sharded cluster. In the process of splitting the collection, a shard key is chosen from among the fields present with the collection's documents. The shard key is then used by the sharded cluster balancer to determine the appropriate distribution of documents. Application program code is then able, by its knowledge of the value of the shard key, to direct queries to specific members of the sharded cluster, achieving potentially enormous performance gains. This setup is illustrated in the following diagram:

This topic is covered in extensive detail in Chapter 15, Deploying a Sharded Cluster. In the next section of this chapter, we have a look at the major differences between MongoDB 3 and MongoDB 4.x.

What's new and different in the MongoDB 4.x release can be broken down into two main categories: new features and internal enhancements. Let's look at the most significant new features first.

The most significant new features introduced in MongoDB 4.x include the following:

• Multidocument ACID transaction support
• Nonblocking secondary replica reads
• In-progress index build interruption 

In the database world, a transaction is a block of database operations that should be treated as if the entire block of commands was just a single command. An example would be where your application is performing end-of-the-month payroll processing. In order to maintain the integrity of the database, and your accounting files, you would need this set of operations to be safeguarded in the event of a failure. ACID is an acronym that stands for atomicity, consistency, isolation, and durability. It represents a set of principles that the database needs to follow in order to safeguard a block of database updates. For more information on ACID, you can refer to https://en.wikipedia.org/wiki/ACID.

In MongoDB 3, a write operation on a single document, even a document containing other embedded documents, was considered atomic. In MongoDB 4.x, multiple documents can be included in a single atomic transaction. Although invoking this support negatively...

MongoDB developers have often included read concerns (as mentioned previously) in their operations in order to shift the burden of response from the primary server in a replica set to its secondaries instead. This frees up the primary to process write operations. Traditionally, MongoDB, prior to version 4, blocked such secondary reads while an update from the primary was in progress. The block ensured that any data read from the secondary would appear exactly the same as data read from the primary.

The downside to this approach, however, was that while the block was in place, the secondary read operation had to wait, which in turn negatively impacted read performance. Likewise, if a read operation was requested prior to a write, it would hold up update operations between the primary and secondary, negatively impacting write performance.

Because of internal changes introduced in MongoDB 4.x to support multidocument transactions, storage engine timestamps and...

A big problem with versions of MongoDB prior to 4.4 is that the following commands error out if an index build (https://docs.mongodb.com/master/core/index-creation/#index-builds-on-populated-collections) operation is in progress:

• db.dropDatabase()
• db.collection.drop()
• db.collection.dropIndexes()

In MongoDB 4.4, when this happens, an attempt to force the in-progress index build operation is made. If successful, the index build is halted, and the drop*() operation continues without error. In the case of a drop*() performed on a replica set, the abort attempt is made on the primary. Once the primary commits to the abort, it then synchronizes to the secondaries.

There are a number of other new features that do not represent a massive paradigm shift, but are extremely useful nonetheless. These include improvements to the aggregation pipeline, field-level encryption, password() prompt, and wildcard indexes. Let's first have a look at aggregation pipeline improvements.

Another major new feature we discuss here involves the introduction of $convert, a new aggregation pipeline operator. This new operator allows the developer to change the data type of a document field while being processed in the pipeline. Target data types include double, string, Boolean, date, integer, long, and decimal. In addition, you can convert a field in the pipeline to the data type objectId, which is critically useful when you need direct access to the autogenerated unique identification field _id. For more information on the aggregation pipeline operator and $convert, go to https://docs.mongodb.com/master/core/aggregation-pipeline/#aggregation-pipeline.

The official programming language drivers for MongoDB 4.2 now support client-side field-level encryption. The implications for security improvements are enormous. This enhancement means that your applications can now provide end-to-end encryption for transmitted data down to the field level. So you could have a transmission of data from your application to MongoDB that includes, for example, an encrypted national identification number mixed in with otherwise plain-text data.

In many cases, it is highly undesirable to include a hard-coded password in a Mongo script. Starting with MongoDB 4.2, in place of a hard-coded password value, you can substitute a built-in JavaScript function passwordPrompt(). You can refer to https://docs.mongodb.com/master/reference/method/passwordPrompt/#passwordPrompt for more details on the function. This causes the Mongo shell to pause and wait for manual user input before proceeding. The password that is entered is then used as the password value.

Starting with MongoDB 4.2, support has been added for wildcard indexes (https://docs.mongodb.com/master/core/index-wildcard/#wildcard-indexes). This feature is useful for situations where the index is either not yet available or is unknown. For situations where the field is known and well established, it's best to create a normal index. There are cases, however, where you have a subset of documents that contain a particular field otherwise lacking in other documents in the collection. You might also be in a situation where a field is added later, and where the DBA has not yet had a chance to create an index on the new field. In these cases, adding a wildcard index allows MongoDB to perform a query more efficiently.

Starting with MongoDB 4.2, support for the Extended JSON v2 (https://docs.mongodb.com/master/reference/mongodb-extended-json/#mongodb-extended-json-v2) specification has been enabled for the following utilities:

• bsondump
• mongodump
• mongoexport
• mongoimport

Starting with MongoDB 4.2, there are now five verbosity log levels, each revealing increasing amounts of information. In MongoDB 4.0.6, you can now set a threshold on the maximum time it should take for data to replicate between members of a replica set. It's now possible to get the information from the MongoDB log file if that time is exceeded.

A further enhancement to diagnostics capabilities includes additional fields that are added to the output of the db.serverStatus() command that can be issued from a Mongo shell.

MongoDB 4.4 adds the ability to perform a hedged read (https://docs.mongodb.com/master/core/sharded-cluster-query-router/#hedged-reads) on a sharded cluster. By setting a hedged read preference option, applications are able to direct read requests to servers in replica sets other than the primary. The advantage of this approach is that the application simply takes the first result response, improving performance. The potential cost, of course, is that if a secondary responds, the data might be slightly out of date.