Feature Engineering

Feature engineering is a method for extracting new features from existing features. These new features are extracted as they tend to effectively explain variability in data. One application of feature engineering could be to calculate how similar different pieces of text are. There are various ways of calculating the similarity between two texts. The most popular methods are cosine similarity and Jaccard similarity. Let's learn about each of them:

• Cosine similarity: The cosine similarity between two texts is the cosine of the angle between their vector representations. BoW and TF-IDF matrices can be regarded as vector representations of texts.
• Jaccard similarity: This is the ratio of the number of terms common between two text documents to the total number of unique terms present in those texts.

  Let's understand this with the help of an example. Suppose there are two texts:

  Text 1: I like detective Byomkesh Bakshi.

  Text 2: Byomkesh Bakshi is not...