How to perform sentiment analysis using Python [Tutorial]

Sentiment analysis is one of the most popular applications of NLP. Sentiment analysis refers to the process of determining whether a given piece of text is positive or negative. In some variations, we consider "neutral" as a third option. This technique is commonly used to discover how people feel about a particular topic. This is used to analyze the sentiments of users in various forms, such as marketing campaigns, social media, e-commerce customers, and so on.

In this article, we will perform sentiment analysis using Python. This extract is taken from Python Machine Learning Cookbook by Prateek Joshi. This book contains 100 recipes that teach you how to perform various machine learning tasks in the real world.

How to Perform Sentiment Analysis in Python

Step 1: Create a new Python file, and import the following packages:

import nltk.classify.util
from nltk.classify import NaiveBayesClassifier
from nltk.corpus import movie_reviews

Step 2: Define a function to extract features:

def extract_features(word_list):
return dict([(word, True) for word in word_list])

Step 3: We need training data for this, so we will use movie reviews in NLTK:

if __name__=='__main__':
   # Load positive and negative reviews  
   positive_fileids = movie_reviews.fileids('pos')
   negative_fileids = movie_reviews.fileids('neg')

Step 4: Let's separate these into positive and negative reviews:

features_positive = [(extract_features(movie_reviews.words(fileids=[f])), 
           'Positive') for f in positive_fileids]
   features_negative = [(extract_features(movie_reviews.words(fileids=[f])), 
           'Negative') for f in negative_fileids]

Step 5: Divide the data into training and testing datasets:

# Split the data into train and test (80/20)
   threshold_factor = 0.8
   threshold_positive = int(threshold_factor * len(features_positive))
   threshold_negative = int(threshold_factor * len(features_negative))

Step 6: Extract the features:

features_train = features_positive[:threshold_positive] + features_negative[:threshold_negative]
   features_test = features_positive[threshold_positive:] + features_negative[threshold_negative:]  
   print "\nNumber of training datapoints:", len(features_train)
   print "Number of test datapoints:", len(features_test)

Step 7: We will use a Naive Bayes classifier. Define the object and train it:

# Train a Naive Bayes classifier
   classifier = NaiveBayesClassifier.train(features_train)
   print "\nAccuracy of the classifier:", nltk.classify.util.accuracy(classifier, features_test)

Step 8: The classifier object contains the most informative words that it obtained during analysis. These words basically have a strong say in what's classified as a positive or a negative review. Let's print them out:

print "\nTop 10 most informative words:"
   for item in classifier.most_informative_features()[:10]:
       print item[0]

Step 9: Create a couple of random input sentences:

# Sample input reviews
   input_reviews = [
       "It is an amazing movie", 
       "This is a dull movie. I would never recommend it to anyone.",
       "The cinematography is pretty great in this movie", 
       "The direction was terrible and the story was all over the place" 
   ]

Step 10: Run the classifier on those input sentences and obtain the predictions:

print "\nPredictions:"
   for review in input_reviews:
       print "\nReview:", review
       probdist = classifier.prob_classify(extract_features(review.split()))
       pred_sentiment = probdist.max()

Step 11: Print the output:

print "Predicted sentiment:", pred_sentiment 
       print "Probability:", round(probdist.prob(pred_sentiment), 2)

If you run this code, you will see three main things printed on the Terminal. The first is the accuracy, as shown in the following image:



The next is a list of most informative words:



The last is the list of predictions, which are based on the input sentences:

How does the Code work?

We use NLTK's Naive Bayes classifier for our task here. In the feature extractor function, we basically extract all the unique words. However, the NLTK classifier needs the data to be arranged in the form of a dictionary. Hence, we arranged it in such a way that the NLTK classifier object can ingest it.

Once we divide the data into training and testing datasets, we train the classifier to categorize the sentences into positive and negative. If you look at the top informative words, you can see that we have words such as "outstanding" to indicate positive reviews and words such as "insulting" to indicate negative reviews. This is interesting information because it tells us what words are being used to indicate strong reactions.

Thus we learn how to perform Sentiment Analysis in Python. For more interesting machine learning recipes read our book, Python Machine Learning Cookbook.

Understanding Sentiment Analysis and other key NLP concepts.

Twitter Sentiment Analysis.

Sentiment Analysis of the 2017 US elections on Twitter.