In the previous chapter, we learned about the concepts of Natural Language Processing (NLP) and text analytics. We also looked at various pre-processing steps in brief. In this chapter, we will learn how to deal with text data whose formats are mostly unstructured. Unstructured data cannot be represented in a tabular format. Therefore, it is essential to convert it into numeric features because most machine learning algorithms are capable of dealing only with numbers. More emphasis will be put on steps such as tokenization, stemming, lemmatization, and stop-word removal. You will also learn about two popular methods for feature extraction: bag of words and Term Frequency-Inverse Document Frequency, as well as various methods for creating new features from existing features. Finally, you will become familiar with how text data can be visualized.

To deal with data effectively, we need to understand the various forms in which it exists. Let's first understand the types of data that exist. There are two main ways to categorize data, by structure and by content, as explained in the upcoming sections.

Categorizing Data Based on Structure

On the basis of structure, data can be divided into three categories, namely structured, semi-structured, and unstructured, as shown in the following diagram:

Figure 2.1: Categorization based on content

These three categories are explained in detail here:

• Structured Data: This is the most organized form of data. It is represented in tabular formats such as Excel files and Comma-Separated Value (CSV) files. The following figure shows what structured data usually looks like:

Figure 2.2: Structured data

• Semi-Structured Data: This type of data is not presented in a tabular structure, but it can be represented...

Most of the time, text data cannot be used as it is. This is because the presence of various unknown symbols or links makes it dirty or unfit for use. Data cleaning is the art of extracting meaningful portions from data by eliminating unnecessary details. Consider the sentence, He tweeted, 'Live coverage of General Elections available at this.tv/show/ge2019. _/\_ Please tune in :) '.

Various symbols, such as "_/\_" and ":)," are present in the sentence. They do not contribute much to its meaning. We need to remove such unwanted details. This is done not only to focus more on the actual content but also to reduce computations. To achieve this, methods such as tokenization and stemming are used. We will learn about them one by one in the upcoming sections.

Tokenization

Tokenization and word tokenizers were briefly described in Chapter 1, Introduction to Natural Language Processing. Tokenization is the process of splitting sentences...

Let's understand feature extraction with real-life examples. Features represent the characteristics of a person or a thing. These characteristics may or may not uniquely represent a person or a thing. For instance, the general characteristics that a person possesses, such as the number of ears, hands, and legs, are generally not enough to identify that person uniquely. But characteristics such as fingerprints and DNA sequences can be used to recognize that person distinctly. Similarly, in feature extraction, we try to extract attributes from texts that represent those texts uniquely. These attributes are called features. Machine learning algorithms take only numeric features as input. So, it is of utmost importance to represent texts as numeric features. When dealing with texts, we extract both general and specific features. Sometimes, individual words constituting texts do not affect some features directly, such as the language of the text and the...

Feature engineering is a method for extracting new features from existing features. These new features are extracted as they tend to effectively explain variability in data. One application of feature engineering could be to calculate how similar different pieces of text are. There are various ways of calculating the similarity between two texts. The most popular methods are cosine similarity and Jaccard similarity. Let's learn about each of them:

• Cosine similarity: The cosine similarity between two texts is the cosine of the angle between their vector representations. BoW and TF-IDF matrices can be regarded as vector representations of texts.
• Jaccard similarity: This is the ratio of the number of terms common between two text documents to the total number of unique terms present in those texts.

  Let's understand this with the help of an example. Suppose there are two texts:

  Text 1: I like detective Byomkesh Bakshi.

  Text 2: Byomkesh Bakshi is not...

In this chapter, you have learned about various types of data and ways to deal with unstructured text data. Text data is usually untidy and needs to be cleaned and pre-processed. Pre-processing steps mainly consist of tokenization, stemming, lemmatization, and stop-word removal. After pre-processing, features are extracted from texts using various methods, such as BoW and TF-IDF. This step converts unstructured text data into structured numeric data. New features are created from existing features using a technique called feature engineering. In the last part of the chapter, we explored various ways of visualizing text data, such as word clouds.

In the next chapter, you will learn how to develop machine learning models to classify texts using the features you have learned to extract in this chapter. Moreover, different sampling techniques and model evaluation parameters will be introduced.