Identifying trend and seasonality
Trend, seasonality, and cyclic variations are the most important characteristics of time-series. A trend is the presence of a long-term increase or decrease in the sequence. Seasonality is a variation that occurs at specific regular intervals of less than a year. Seasonality can occur on different time spans such as daily, weekly, monthly, or yearly. Finally, cyclic variations are rises and falls that are not of a fixed frequency.
An important characteristic of time-series is stationarity. This refers to a property of time-series not to change distribution over time, or in other words, that the process that produces the time-series doesn't change with time. Time-Series that don't change over time are called stationary (or stationary processes). Many models or measures assume stationarity and might not work properly if the data is not stationary. Therefore, with these algorithms, the time-series should be decomposed first into the main signal, and then the seasonal and trend components. In this decomposition, we would subtract the trend and seasonal components from the original time-series.
In this section, we'll first go through an example of how to estimate trend and seasonality using curve fitting. Then, we'll look at other tools that can help discover trends, seasonality, and cyclic variations. These include statistics such as autocorrelation and the augmented Dickey–Fuller test, and visualizations such as the autocorrelation plot (also: lag plot) and the periodogram.
Let's start with a hopefully clear example of how seasonality and trend can be estimated in just a few lines of Python. For this, we'll come back to the GISS Surface Temperature Analysis dataset released by NASA. We'll load the dataset, and we'll do curve fitting, which comes straight out of the box in NumPy.
We'll download the dataset from Datahub (https://datahub.io/core/global-temp) or you can find it from the book's GitHub repository (in the chapter2 folder).
Then, we can load it up and pivot it:
temperatures = pd.read_csv('/Users/ben/Downloads/monthly_csv.csv')
temperatures['Date'] = pd.to_datetime(temperatures['Date'])
temperatures = temperatures.pivot('Date', 'Source', 'Mean')
Now we can use NumPy's polyfit functionality. It fits a polynomial of the form:
In this formula, k is the degree of the polynomial and b is the coefficients we are trying to find.
It is just a function in NumPy to fit the coefficients. We can use the same function to fit seasonal variation and trend. Since trend can dominate over seasonality, before estimating seasonality, we remove the trend:
from numpy import polyfit
def fit(X, y, degree=3):
coef = polyfit(X, y, degree)
trendpoly = np.poly1d(coef)
return trendpoly(X)
def get_season(s, yearly_periods=4, degree=3):
X = [i%(365/4) for i in range(0, len(s))]
seasonal = fit(X, s.values, degree)
return pd.Series(data=seasonal, index=s.index)
def get_trend(s, degree=3):
X = list(range(len(s)))
trend = fit(X, s.values, degree)
return pd.Series(data=trend, index=s.index)
Let's plot seasonality and trend on top of our global temperature increases!
import seaborn as sns
plt.figure(figsize=(12, 6))
temperatures['trend'] = get_trend(temperatures['GCAG'])
temperatures['season'] = get_season(temperatures['GCAG'] - temperatures['trend'])
sns.lineplot(data=temperatures[['GCAG', 'season', 'trend']])
plt.ylabel('Temperature change');
This is the graph that we get:
Figure 2.9: Temperature change from the late 19th century to today
This was to show that you can use plug-in functionality in NumPy for curve fitting in order to find both trend and seasonality. If you want to experiment further, you can play with the degree of the polynomial or with the seasonality component to see if you can get a better fit, or find another seasonality component. We could have used functionality from other libraries such as seasonal.seasonal_decompose() in statsmodels, or Facebook's Prophet, which decomposes using Fourier coefficients for the seasonal components.
Now that we've seen how to estimate seasonality and trend, let's move on to other statistics and visualizations. Continuing with the pollution dataset, and picking up the EEG dataset we saw in Chapter 1, we'll show practically in Python how to get these statistics and plots, and how to identify trend and seasonality.
Autocorrelation is the correlation of a signal with a lagged version of itself. The autocorrelation plot draws the autocorrelation as a function of lag. The autocorrelation plot can help find repeating patterns, and is often used in signal processing. The autocorrelation can help spot a periodic signal. Let's plot the autocorrelation of the pollution data:
pollution = pollution.pivot("Year", "City", "SPM")
pd.plotting.autocorrelation_plot(pollution['London'])
Here's the plot that we get:
Figure 2.10: Autocorrelaton plot of pollution in London
We can see high autocorrelations with a lag of only a few years. There is a negative autocorrelation at around 100 years, after which point the autocorrelation stays around 0.
The plot of SPM clearly shows that air pollution is not a stationary process, since the autocorrelation is not flat. You can also compare the run of pollution that shows there's a trend, and therefore the mean also changes – another indication that the series is not stationary.
We can also test this statistically. A test for stationarity is the augmented Dickey–Fuller test:
from statsmodels.tsa import stattools
stattools.adfuller(pollution['London'])
(-0.33721640804242853,
0.9200654843183897,
13,
303,
{'1%': -3.4521175397304784,
'5%': -2.8711265007266666,
'10%': -2.571877823851692},
1684.6992663493872)
The second return value is the p-value that gives the significance or the probability of obtaining test results at least as extreme as the observation given the null hypothesis. With p-values below 5% or 0.05 we would typically reject the null hypothesis, and we could assume that our time-series is stationary. In our case, we can't assume that the series is stationary.
We saw the graph of electroencephalography (EEG) signals in Chapter 1, Introduction to Time-Series with Python, and we mentioned that EEG signals show brain waves at several frequency ranges.
We can visualize this nicely. Let's go through it step by step in Python. We first need to do a few imports:
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib.dates import DateFormatter
import seaborn as sns
from sklearn.datasets import fetch_openml
OpenML is a project that provides benchmark datasets and a website for comparison of machine learning algorithms. The scikit-learn library provides an interface to OpenML that allows us to fetch data from OpenML. The whole measurement spans 117 seconds. So we need to set this up correctly as an index in pandas:
eeg = fetch_openml(data_id=1471, as_frame=True)
increment = 117 / len(eeg['data'])
import numpy as np
index = np.linspace(
start=0,
stop=increment*len(eeg['data']),
num=len(eeg['data'])
)
ts_index = pd.to_datetime(index, unit='s')
v1 = pd.Series(name='V1', data=eeg['data']['V1'].values, index=ts_index)
We can slice our dataset directly. Please note that the DatetimeIndex is anchored in 1970, but we can ignore this safely here:
slicing = (v1.index >= '1970-01-01 00:00:08') & (v1.index <='1970-01-01 00:01:10.000000000')
v1[slicing]
Here's the slice:
1970-01-01 00:00:08.006208692 4289.74
1970-01-01 00:00:08.014019627 4284.10
1970-01-01 00:00:08.021830563 4280.00
1970-01-01 00:00:08.029641498 4289.74
1970-01-01 00:00:08.037452433 4298.46
...
1970-01-01 00:01:09.962547567 4289.74
1970-01-01 00:01:09.970358502 4283.08
1970-01-01 00:01:09.978169437 4284.62
1970-01-01 00:01:09.985980373 4289.23
1970-01-01 00:01:09.993791308 4290.77
Name: V1, Length: 7937, dtype: float64
This slicing avoids an artifact, a strong spike, occurring at around 1:20.
The graph we saw in Chapter 1, we can plot as follows:
date_formatter = DateFormatter("%S")
ax = v1[slicing].plot(figsize=(12, 6))
ax.xaxis.set_major_formatter(date_formatter)
plt.ylabel('voltage')
Here's the graph again:
Figure 2.11: Voltage over time in an EEG signal
This is the plot of the EEG signal over time.
We can also resample the data to look at the series more coarsely, with less resolution, for example like this:
plt.subplot(311)
ax1 = v1[slicing].resample('1s').mean().plot(figsize=(12, 6))
ax1.xaxis.set_major_formatter(date_formatter)
plt.subplot(312)
ax1 = v1[slicing].resample('2s').mean().plot(figsize=(12, 6))
ax1.xaxis.set_major_formatter(date_formatter)
plt.subplot(313)
ax2 = v1[slicing].resample('5s').mean().plot(figsize=(12, 6))
ax2.xaxis.set_major_formatter(date_formatter)
plt.xlabel('seconds');
This is the graph with three subplots we get from resampling to frequencies of 1 second, 2 seconds, and 5 seconds, respectively:
Figure 2.12: Resampled EEG signals
Each of the resampled signals in the plot could be more or less useful for analysis depending on the application. For high-frequency analysis, we shouldn't resample at all, while if we are trying to remove as much noise as possible, we should resample to a more coarse time resolution.
We can look at cyclic activity on a plot of spectral density. We can do this by applying a Fourier transform. Here, we go with the Welch method, which averages over time before applying the discrete Fourier transform:
from scipy import signal
fs = len(eeg['data']) // 117
f, Pxx_den = signal.welch(
v1[slicing].values,
fs,
nperseg=2048,
scaling='spectrum'
)
plt.semilogy(f, Pxx_den)
plt.xlabel('frequency [Hz]')
plt.ylabel('PSD [V**2/Hz]')
The spectral density plot, the periodogram, looks like this:
Figure 2.13: Periodogram of the EEG signals
The information in this plot is like the autocorrelation plot that we drew for pollution, however, it gives us information about how prominent certain frequencies are. In this case we see that low frequencies are particularly powerful. In other words, the signal shows a slow oscillation.
This brings the chapter to an end. Let's summarize what we've covered.
