Twitter offers a series of APIs to provide programmatic access to Twitter data, including reading tweets, accessing user profiles, and posting content on behalf of a user.

In order to set up our project to access Twitter data, there are two preliminary steps, as follows:

The registration step will take a few minutes. Assuming that we are already logged in to our Twitter account, all we need to do is point our browser to the Application Management page at http://apps.twitter.com and create the new app.

Once the app is registered, under the Keys and Access Tokens tab, we can find the information we need to authenticate our application. The Consumer Key and Consumer Secret (also called API Key and API Secret, respectively) are a setting of your application. The Access Token and Access Token Secret are instead a setting for your user account. Your application can potentially ask for access to several users through their...

In order to interact with the Twitter APIs, we need a Python client that implements the different calls to the API itself. There are several options as we can see from the official documentation (https://dev.twitter.com/overview/api/twitter-libraries). None of them are officially maintained by Twitter and they are backed by the open source community. While there are several options to choose from, some of them almost equivalent, so we will choose to use Tweepy here as it offers a wider support for different features and is actively maintained.

The library can be installed via pip:

$ pip install tweepy==3.3.0 

The first part of the interaction with the...

This section is all about analyzing entities in tweets. We're going to perform some frequency analysis using the data collected in the previous section. Slicing and dicing this data will allow users to produce some interesting statistics that can be used to get some insights on the data and answer some questions.

Analyzing entities such as hashtags is interesting as these annotations are an explicit way for the author to label the topic of the tweet.

We start with the analysis of the tweets by Packt Publishing. As Packt Publishing supports and promotes open source software, we are interested in finding what kind of technologies are mentioned often by Packt Publishing.

The following script extracts the hashtags from a user timeline, producing a list of the most common ones:

# Chap02-03/twitter_hashtag_frequency.py 
import sys 
from collections import Counter 
import json 
 
def get_hashtags(tweet): 
  entities = tweet.get('entities', {}) 
  hashtags...

The previous section analyzed the entity field of a tweet. This provides useful knowledge on the tweet, because these entities are explicitly curated by the author of the tweet. This section will focus on unstructured data instead, that is, the raw text of the tweet. We'll discuss aspects of text analytics such as text preprocessing and normalization and we'll perform some statistical analysis on the tweets. Before digging the details, we'll introduce some terminology.

Tokenization is one of the important steps in the preprocessing phase. Given a stream of text (such as a tweet status), tokenization is the process of breaking this text down into individual units called tokens. In the simplest form, these units are words, but we could also work on a more complex tokenization that deals with phrases, symbols, and so on.

Tokenization sounds like a trivial task, and it's been widely studied by the natural language processing community. Chapter...