In the early 1970s, if you wanted to purchase a stock, you would need to engage a broker, who would charge you a fixed commission of nearly 1%. If you wanted to purchase an airline ticket, you would need to contact a travel agent, who would earn a commission of around 7%. And if you wanted to sell a home, you would contact a real estate agent, who would earn a commission of 6%. In 2018, you can do the first two essentially for free. The last one remains as it was in the 1970s.

Why is this the case and, more importantly, what does any of this have to do with machine learning? The reality is, it all comes down to data, and who has access to that data.

You might assume that you could easily access troves of real estate listing data quite easily through APIs or by web scraping real estate websites. You would be wrong. Well, wrong if you intend to follow...

Let's begin by inspecting the data points for each of our columns. We want to look for odd and outlier values in our data. We will start by looking at the bedroom and bathroom columns:

1. In the following code, we look at the unique values for bedrooms:

df['beds'].unique() 

The preceding code results in the following output:

2. Now, let's look at bathrooms. We do that in the following code:

df['baths'].unique() 

The preceding code results in the following output:

3. Based on the output from the two preceding queries, we see that we need to correct some items that have a leading underscore. Let's do that now:

df['beds'] = df['beds'].map(lambda x: x[1:] if x.startswith('_') else x) 
df['baths'] = df['baths'].map(lambda x: x[1:] if x.startswith('_') else...

When dealing with geographic data, as we are here, it is immensely valuable to be able to plot that information. One way of doing that is with something called a choropleth map. A choropleth is essentially a geographic heat map. We are going to build a choropleth to create a heat map of average rental price by ZIP code.

The first thing we will need to do this is the ZIP code. Unfortunately for us, our dataset does not contain ZIP code information. We do, however, have the address for the properties. With a little help from the Google Maps API, we can retrieve this information.

Currently, the Google Maps API is a paid API. The rates are reasonable, 1,000 calls for $5, but they also give you a credit of $200 each month (at the time of writing). They also allow you to sign up for a free trial before they will start billing you, and they won't bill unless...

Since this data is based on ZIP codes, the best way to visualize it is with a choropleth. If you're unfamiliar with a choropleth, it's simply a visualization that represents the data according to a color spectrum. Let's create one now using a Python mapping library called folium at https://github.com/python-visualization/folium. If you don't have folium installed, again, it can be done with pip install on the command line.

Now we'll go ahead and create our visualization:

import folium 
 
m = folium.Map(location=[40.748817, -73.985428], zoom_start=13) 
 
m.choropleth( 
    geo_data=open('nyc.json').read(), 
    data=zdf_mean, 
    columns=['zip', 'avg_rent'], 
    key_on='feature.properties.postalCode', 
    fill_color='YlOrRd', fill_opacity=0.7, line_opacity=0.2, 
    ) 
 
m 

There...

Let's begin modeling by using our dataset. We're going to examine the effect that the ZIP code and the number of bedrooms have on the rental price. We'll use two packages here: the first, statsmodels, we introduced in Chapter 1, The Python Machine Learning Ecosystem, but the second, patsy, https://patsy.readthedocs.org/en/latest/index.html, is a package that makes working with statsmodels easier. Patsy allows you to use R-style formulas when running a regression. Let's do that now:

import patsy 
import statsmodels.api as sm 
 
 
f = 'rent ~ zip + beds' 
y, X = patsy.dmatrices(f, zdf, return_type='dataframe') 
 
results = sm.OLS(y, X).fit() 
results.summary() 

The preceding code generates the following output:

Note that the preceding output is truncated.

With those few lines of code, we have just run our first machine...

At this point, we have only examined the relationship between the ZIP code, bedrooms, and rental price. And while our model had some explanatory benefit, we had a minimal dataset and far too few features to adequately examine the complex world of real estate valuation.

Fortunately, however, if we were to add more data and features to the model, we could use the exact same framework to expand our analysis.

Some possible future extensions to explore would be utilizing data for restaurants and bars available from APIs such as Foursquare or Yelp, or walkability and transportation-proximity measures from providers such as Walk Score.

There are a number of ways to extend the model, and I suggest if you do pursue working on a project such as this that you explore a variety of measures. More data is released every day and, with it, models can only improve.

In this chapter, we learned how to acquire data on real estate listings, how to utilize the functionality of pandas to manipulate and sanitize that data, how to inspect the data visually with choropleths, and finally, how to build and use regression modeling to price out an apartment.

At this point, we have just touched the surface of machine learning. In the chapters that follow, we'll go further into how to evaluate the quality of our model, and we'll also learn how to turn them into full-scale solutions.