Calculating sentiment analysis of text

Sentiment analysis is the ability to derive tone and feeling behind a word or series of words. This section will utilize techniques in python to calculate a sentiment analysis score from the 100 transactions in our dataset.

Getting ready

This section will require using functions and data types within PySpark. Additionally, we well importing the TextBlob library for sentiment analysis. In order to use SQL and data type functions within PySpark, the following must be imported:

from pyspark.sql.types import FloatType 

Additionally, in order to use TextBlob, the following library must be imported:

from textblob import TextBlob

How to do it...

The following section walks through the steps to apply sentiment score to the dataset.

1. Create a sentiment score function, sentiment_score, using the following script:

from textblob import TextBlob
def sentiment_score(chat):
    return TextBlob(chat).sentiment.polarity

2. Apply sentiment_score to each conversation response in the...