Understanding natural language processing

In the last few years, branches of artificial intelligence (AI) have created a lot of buzz, and those branches are data science, data analytics, predictive analysis, NLP, and so on.

As mentioned in the Preface of this book, we are focusing on Python and natural language processing. Let me ask you some questions--Do you really know what natural language is? What is natural language processing? What are the other branches involved in building expert systems using various concepts of natural language processing? How can we build intelligent systems using the concept of NLP?

Let's begin our roller coaster ride of understanding NLP.

What is natural language?

• As a human being, we express our thoughts or feelings via a language
• Whatever you speak, read, write, or listen to is mostly in the form of natural language, so it is commonly expressed as natural language
• For example:
  • The content of this book is a source of natural language
  • Whatever you speak, listen, and write in your daily life is also in the form of natural language
  • Movie dialogues are also a source of natural language
  • Your WhatsApp conversations are also considered a form of natural language

What is natural language processing?

• Now you have an understanding of what natural language is. NLP is a sub-branch of AI. Let's consider an example and understand the concept of NLP. Let's say you want to build a machine that interacts with humans in the form of natural language. This kind of an intelligent system needs computational technologies and computational linguistics to build it, and the system processes natural language like humans.
• You can relate the aforementioned concept of NLP to the existing NLP products from the world's top tech companies, such as Google Assistant from Google, Siri speech assistance from Apple, and so on.
• Now you will able to understand the definitions of NLP, which are as follows:
  • Natural language processing is the ability of computational technologies and/or computational linguistics to process human natural language
  • Natural language processing is a field of computer science, artificial intelligence, and computational linguistics concerned with the interactions between computers and human (natural) languages
  • Natural language processing can be defined as the automatic (or semi-automatic) processing of human natural language

What are the other branches involved in building expert systems using, various concepts of NLP? Figure 1.1 is the best way to know how many other branches are involved when you are building an expert system using NLP concepts:

Figure 1.1: NLP concepts

Figures 1.2 and 1.3 convey all the subtopics that are included in every branch given in Figure 1.1:

Figure 1.2: Sub-branches of NLP concepts

Figure 1.3 depicts the rest of the sub-branches:

Figure 1.3: Sub-branches of NLP concepts

How can we build an intelligent system using concepts of NLP? Figure 1.4 is the basic model, which indicates how an expert system can be built for NLP applications. The development life cycle is defined in the following figure:

Figure 1.4: Development life cycle

Let's see some of the details of the development life cycle of NLP-related problems:

1. If you are solving an NLP problem, you first need to understand the problem statement.
2. Once you understand your problem statement, think about what kind of data or corpus you need to solve the problem. So, data collection is the basic activity toward solving the problem.
3. After you have collected a sufficient amount of data, you can start analyzing your data. What is the quality and quantity of our corpus? According to the quality of the data and your problem statement, you need to do preprocessing.
4. Once you are done with preprocessing, you need to start with the process of feature engineering. Feature engineering is the most important aspect of NLP and data science related applications. We will be covering feature engineering related aspects in much more detail in Chapter 5, Feature Engineering and NLP Algorithms and Chapter 6, Advance Feature Engineering and NLP Algorithms.
5. Having decided on and extracted features from the raw preprocessed data, you are to decide which computational technique is useful to solve your problem statement, for example, do you want to apply machine learning techniques or rule-based techniques?.
6. Now, depending on what techniques you are going to use, you should ready the feature files that you are going to provide as an input to your decided algorithm.
7. Run your logic, then generate the output.
8. Test and evaluate your system's output.
9. Tune the parameters for optimization, and continue till you get satisfactory results.

We will be covering a lot of information very quickly in this chapter, so if you see something that doesn't immediately make sense, please do not feel lost and bear with me. We will explore all the details and examples from the next chapter onward, and that will definitely help you connect the dots.