ArcGIS Blueprints

Chapter 1. Extracting Real-Time Wildfire Data from ArcGIS Server with the ArcGIS REST API

The ArcGIS platform, which contains a number of different products, including ArcGIS Desktop, ArcGIS Pro, ArcGIS for Server, and ArcGIS Online, provides a robust environment to perform geographic analysis and mapping. The content produced by this platform can be integrated using the ArcGIS REST API and a programming language such as Python. Many of the applications we'll build in this book use the ArcGIS REST API as the bridge to exchange information between software products.

We're going to start by developing a simple ArcGIS Desktop custom script tool in ArcToolbox that connects to an ArcGIS Server map service to retrieve real-time wildfire information. The wildfire information will be retrieved from a USGS map service that provides real-time wildfire data. For this chapter and all other chapters in this book, the reader is expected to have intermediate-level experience of Python and ArcPy. Ideally, you should be running version 10.3 or 10.2 of ArcGIS Desktop. Previous versions of ArcGIS Desktop have some significant differences that may cause problems in the development of some applications in the book.

We'll use the ArcGIS REST API and the Python requests module to connect to the map service and request the data. The response from the map service will contain data that will be written to a feature class stored in a local geodatabase using the ArcPy data access module.

This will all be accomplished with a custom script tool attached to an ArcGIS Python Toolbox. ArcGIS Python toolboxes are relatively new; they were first introduced in version 10.1 of ArcGIS Desktop. They provide a Python-centric method to create custom toolboxes and tools. The older method to create toolboxes in ArcGIS Desktop, while still relevant, requires a combination of Python and a wizard-based approach to create tools.

In this chapter, we will cover the following topics:

ArcGIS Desktop Python's toolboxes
The ArcGIS Server map and feature services
The Python requests module
The Python JSON module
The ArcGIS REST API
The ArcPy data access module that is arcpy.da

A general overview of the Python libraries for ArcGIS is provided in the appendix of this book. It is recommended that you read this chapter before continuing with the appendix and other chapters.

Creating the ArcGIS Desktop Python Toolbox

There are two ways to create toolboxes in ArcGIS: script tools in custom toolboxes and script tools in Python toolboxes. Python toolboxes encapsulate everything in one place: parameters, validation code, and source code. This is not the case with custom toolboxes, which are created using a wizard and a separate script that processes the business logic.

A Python Toolbox functions like any other toolbox in ArcToolbox, but it is created entirely in Python and has a file extension of .pyt. It is created programmatically as a class named Toolbox. In this section, you will learn how to create a Python Toolbox and add a tool. You'll only create the basic structure of the toolbox and tool that will ultimately connect to an ArcGIS Server map service containing the wildfire data. In a later section, you'll complete the functionality of the tool by adding code that connects to the map service, downloads the current data, and inserts it into a feature class. Take a look at the following steps:

Open ArcCatalog: You can create a Python Toolbox in a folder by right-clicking on the Folder and navigating to New | Python Toolbox. In ArcCatalog, there is a folder called Toolboxes, and inside it, there is a My Toolboxes folder, as shown in the following screenshot. Right-click on this folder and navigate to New | Python Toolbox.
The name of the toolbox is controlled by the filename. Name the toolbox InsertWildfires.pyt.
The Python Toolbox file (.pyt) can be edited in any text or code editor. By default, the code will open in Notepad. However, you will want to use a more advanced Python development environment, such as PyScripter, IDLE, and so on. You can change this by setting the default editor for your script by navigating to Geoprocessing | Geoprocessing Options and going to the Editor section. In the following screenshot, you'll notice that I have set my editor to PyScripter, which is my preferred environment. You may want to change this to IDLE or whichever development environment you are currently using.
For example, to find the path to the executable for the IDLE development environment, you can navigate to Start | All Programs | ArcGIS | Python 2.7 | IDLE. Right-click on IDLE and select Properties to display the properties window. Inside the Target textbox, you should see a path to the executable, as shown in the following screenshot. You will want to copy and paste only the actual path starting with C:\Python27 and not the quotes that surround the path.
Copy and paste the path into the Editor and Debugger sections inside the Geoprocessing Options dialog box.
Right-click on InsertWildfires.pyt and select Edit. This will open the development environment you defined earlier, as shown in the following screenshot. Your environment will vary depending upon the editor that you have defined:
Remember that you will not be changing the name of the class, which is Toolbox. However, you will have to rename the Tool class to reflect the name of the tool you want to create. Each tool will have various methods, including __init__(), which is the constructor for the tool along with getParameterInfo(), isLicensed(), updateParameters(), updateMessages(), and execute(). You can use the __init__() method to set initialization properties, such as the tool's label and description. Find the class named Tool in your code, and change the name of this tool to USGSDownload, and set the label and description properties:
```
class USGSDownload(object):
  def __init__(self):
    """Define the tool (tool name is the name of the class)."""
    self.label = "USGS Download"
    self.description = "Download from USGS ArcGIS Server instance"
    self.canRunInBackground = False
```
Tip
Downloading the example code.
You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you
You can use the Tool class as a template for other tools that you'd like to add to the toolbox by copying and pasting the class and its methods. We're not going to do that in this chapter, but I wanted you to be aware of this.

You will need to add each tool to the tools property (the Python list) in the Toolbox class. Add the USGSDownload tool, as shown in the following code snippet:

  def __init__(self):
    """Define the toolbox (the name of the toolbox is the name of the .pyt file."""
    self.label = "Toolbox"
    self.alias = ""
    
    #List of tool classes associated with this toolbox
    self.tools = [USGSDownload]

Save your code.
When you close the code editor, your toolbox should be automatically refreshed. You can also manually refresh a toolbox by right-clicking on the InsertWildfires.pyt and selecting Refresh. If a syntax error occurs in your code, the toolbox icon will change, as shown in the following screenshot. Note the red X next to the toolbox. Your tool may not visible inside the toolbox either. If you've coded everything correctly, you will not see this icon, as shown in the following screenshot:
To see the error, right-click on InsertWildfires.pyt and select Check Syntax.
Assuming that you don't have any syntax errors, you should see the following screenshot of the Toolbox/Tool structure:

Working with tool parameters

Almost all tools have parameters. The parameter values will be set by the end user with the Tool dialog box, or they will be hardcoded within the script. When the tool is executed, the parameter values are sent to your tool's source code. Your tool reads these values and proceeds with its work. You use the getParameterInfo() method to define the parameters for your tool. Individual parameter objects are created as part of this process. If necessary, open InsertWildfires.pyt in your code editor and find the getParameterInfo() function in the USGSDownload class. Add the following parameters, and then we'll discuss what the code is doing:

def getParameterInfo(self):

        """Define parameter definitions"""
        # First parameter
        param0 = arcpy.Parameter(displayName="ArcGIS Server Wildfire URL",
                        name="url",
                        datatype="GPString",
                        parameterType="Required",
                        direction="Input")
        param0.value = "http://wildfire.cr.usgs.gov/arcgis/rest/services/geomac_dyn/MapServer/0/query"

        # Second parameter
        param1 = arcpy.Parameter(displayName="Output Feature Class",
                        name="out_fc",
                        datatype="DEFeatureClass",
                        parameterType="Required",
                        direction="Input")

Each parameter is created using arcpy.Parameter and is passed a number of arguments that define the object. For the first Parameter object (param0), we are going to capture a URL to an ArcGIS Server map service containing real-time wildfire data. We give it a display name ArcGIS Server Wildfire URL, which will be displayed on the dialog box for the tool. A name for the parameter, a datatype, a parameter type, and a direction. In the case of the first parameter (param0), we also assign an initial value, which is the URL to an existing map service containing the wildfire data. For the second parameter, we're going to define an output feature class where the wildfire data that is read from the map service will be written. An empty feature class to store the data has already been created for you.

Next we'll add both the parameters to a Python list called params and return, to the list to the calling function. Add the following code:

def getParameterInfo(self):

        """Define parameter definitions"""
        # First parameter
        param0 = arcpy.Parameter(displayName="ArcGIS Server Wildfire URL",
                        name="url",
                        datatype="GPString",
                        parameterType="Required",
                        direction="Input")
        param0.value = "http://wildfire.cr.usgs.gov/arcgis/rest/services/geomac_dyn/MapServer/0/query"

        # Second parameter
        param1 = arcpy.Parameter(displayName="Output Feature Class",
                        name="out_fc",
                        datatype="DEFeatureClass",
                        parameterType="Required",
                        direction="Input")

        params = [param0, param1]
        return params

Tool execution

The main work of a tool is done inside the execute() method. This is where the geoprocessing of your tool takes place. The execute() method, as shown in the following code snippet, can accept a number of arguments, including the tools self, parameters, and messages:

def execute(self, parameters, messages):
        """The source code of the tool."""
        return

To access the parameter values that are passed into the tool, you can use the valueAsText() method. The following steps will guide you through how to add and execute the execute() method:

Find the execute() function in the USGSDownload class and add the following code snippet to access the parameter values that will be passed into your tool. Remember from a previous step that the first parameter will contain a URL to a map service containing the wildfire data and the second parameter will be the output feature class where the data will be written:
```
def execute(self, parameters, messages):
        inFeatures = parameters[0].valueAsText
        outFeatureClass = parameters[1].valueAsText
```
At this point, you have created a Python Toolbox, added a tool, defined the parameters for the tool, and created variables that will hold the parameter values that the end user has defined. Ultimately, this tool will use the URL that is passed to the tool to connect to an ArcGIS Server map service, download the current wildfire data, create a feature class, and write the wildfire data to the feature class. However, we're going to save the geoprocessing tasks for later. For now, I just want you to print out the values of the parameters that have been entered so that we know that the structure of the tool is working correctly. Print this information using the following code:
```
def execute(self, parameters, messages):
        inFeatures = parameters[0].valueAsText
        arcpy.AddMessage(inFeatures)
        outFeatureClass = parameters[1].valueAsText
        arcpy.AddMessage(outFeatureClass)
```
Execute the tool by double-clicking on USGS Download from the InsertWildfires.pyt toolbox that you've created. You should see the following dialog box:
Leave the default URL parameter as it is and select an output feature class. You'll want to click on the Browse button and then navigate to the C:\ArcGIS_Blueprint_Python\Data\WildfireData folder and select the WildlandFires geodatabase. Inside, there is an empty feature class called CurrentFires. Select this feature class.
Now click on OK. The progress dialog box will contain the parameter values that you passed in. Your output feature class will probably be different than mine, as shown in the following screenshot:

We'll complete the actual geoprocessing of this tool in the next section.

Populating the feature class

In the previous section, you learned how to create a Python Toolbox and add tools. You created a new toolbox called InsertWildfires and added a tool called USGS Download. However, in that exercise, you didn't complete the geoprocessing operations that connect to an ArcGIS Server map service, query the service for current wildfires, and populate the feature class from the data pulled from the map service query. You'll complete these steps in the following section.

Installing pip and the requests module

This section of the application uses the Python requests module. If you don't already have this module installed on your computer, you will need to do this at this time using pip.

The pip is a package manager that serves as a repository and installation manager for Python modules. It makes finding and installing Python modules much easier. There are several steps that you'll need to follow in order to install pip and the requests module. Instructions to install pip and the requests module are provided in the first few steps:

Open the Environment Variables dialog box in Windows. The easiest way to display this dialog box is to go to Start and then type Environment Variables in the search box. This should display an Edit environment variables for your account entry. Select this item.
If you don't see a variable called PATH, click on the New button to create one. If you already have a PATH variable, you can just add to the existing content. Select the PATH variable, and click on Edit, and then set the value to C:\Python27\ArcGIS10.3; C:\Python27\ArcGIS10.3\Scripts. The first path will provide a reference to the location of the Python executable and the second will reference the location of pip when it is installed. This makes it possible to run Python and pip from the Command Prompt in Windows.
Click on OK and then click on OK again to save the changes.
Next, we'll install pip if you haven't already done so in the past. You need to install pip before you can install requests. In a web browser, go to https://pip.pypa.io/en/latest/installing.html and scroll down to the install pip section. Right-click on get-pip.py and select Save Link As or something similar. This will vary depending upon the browser you are using. Save it to your C:\ArcGIS_Blueprint_Python folder.
Open the Command Prompt in Windows, type python C:\ArcGIS_Blueprint_Python\get-pip.py, and press Enter on your keyboard. This will install pip.
In the Command Prompt, type pip install requests and press Enter on your keyboard. This will install the requests module.
Close the Command Prompt.

Requesting data from ArcGIS Server

In the following steps, we will learn how to request data from ArcGIS Server:

Open ArcCatalog and navigate to the location where you've created your Python Toolbox, it would look like following screenshot:
Right-click on InsertWildfires.pyt and select Edit to display the code for the toolbox.

First, we'll clean up a little by removing the AddMessage() functions. Clean up your execute() method so that it appears as follows:

def execute(self, parameters, messages):
        inFeatures = parameters[0].valueAsText
        outFeatureClass = parameters[1].valueAsText

Next, add the code that connects to the wildfire map service, to perform a query. In this step, you will also define the QueryString parameters that will be passed into the query of the map service. First, import the requests and json modules:

import arcpy
import requests, json


class Toolbox(object):
    def __init__(self):
        """Define the toolbox (the name of the toolbox is the name of the
        .pyt file)."""
        self.label = "Toolbox"
        self.alias = ""

        # List of tool classes associated with this toolbox
        self.tools = [USGSDownload]

Then, create the agisurl and json_payload variables that will hold the QueryString parameters. Note that, in this case, we have defined a WHERE clause so that only wildfires where the acres are greater than 5 will be returned. The inFeatures variable holds the ArcGIS Server Wildfire URL:

    def execute(self, parameters, messages):
        inFeatures = parameters[0].valueAsText
        outFeatureClass = parameters[1].valueAsText

        agisurl = inFeatures
        json_payload = { 'where': 'acres > 5', 'f': 'pjson',
        'outFields': 'latitude,longitude,incidentname,acres' }

Submit the request to the ArcGIS Server instance; the response should be stored in a variable called r. Print a message to the dialog box indicating the response as:

def execute(self, parameters, messages):
        inFeatures = parameters[0].valueAsText
        outFeatureClass = parameters[1].valueAsText

        agisurl = inFeatures
        json_payload = { 'where': 'acres > 5', 'f': 'pjson', 'outFields': 'latitude,longitude,incidentname,acres' }

        r = requests.get(agisurl, params=json_payload)
        arcpy.AddMessage("The response: " + r.text)

Test the code to make sure that we're on the right track. Save the file and refresh InsertWildfires in ArcCatalog. Execute the tool and leave the default URL. If everything is working as expected, you should see a JSON object output to the progress dialog box. Your output will probably vary from the following screenshot:

Return to the execute() method and convert the JSON object to a Python dictionary using the json.loads() method:

def execute(self, parameters, messages):
        inFeatures = parameters[0].valueAsText
        outFeatureClass = parameters[1].valueAsText

        agisurl = inFeatures
        json_payload = { 'where': 'acres > 5', 'f': 'pjson', 'outFields': 'latitude,longitude,incidentname,acres' }

        r = requests.get(inFeatures, params=json_payload)
        arcpy.AddMessage("The response: " + r.text)

        decoded = json.loads(r.text)

Inserting data in a feature class with the ArcPy data access module

The following steps will guide you, to insert data in a feature class with the help of the ArcPy data access module:

Now, we'll use the ArcPy data access module, that is Arcpy.da, to create an InsertCursor object by passing the output feature class defined in the tool dialog box along with the fields that will be populated:

    def execute(self, parameters, messages):
        inFeatures = parameters[0].valueAsText
        outFeatureClass = parameters[1].valueAsText

        agisurl = inFeatures
        json_payload = { 'where': 'acres > 5', 'f': 'pjson', 'outFields': 'latitude,longitude,fire_name,acres' }

        r = requests.get(inFeatures, params=json_payload)
        arcpy.AddMessage("The response: " + r.text)

        decoded = json.loads(r.text)
        cur = arcpy.da.InsertCursor(outFeatureClass, ("SHAPE@XY", "NAME", "ACRES"))

Create a For loop that you can see in the following code, and then we'll discuss what this section of code accomplishes:

def execute(self, parameters, messages):
        inFeatures = parameters[0].valueAsText
        outFeatureClass = parameters[1].valueAsText

        agisurl = inFeatures
        json_payload = { 'where': 'acres > 5', 'f': 'pjson', 'outFields': 'latitude,longitude,fire_name,acres' }

        r = requests.get(inFeatures, params=json_payload)
        arcpy.AddMessage("The response: " + r.text)

        decoded = json.loads(r.text)
        cur = arcpy.da.InsertCursor(outFeatureClass, ("SHAPE@XY", "NAME", "ACRES"))
        cntr = 1
        for rslt in decoded['features']:
            fireName = rslt['attributes']['incidentname']
            latitude = rslt['attributes']['latitude']
            longitude = rslt['attributes']['longitude']
            acres = rslt['attributes']['acres']
            cur.insertRow([(longitude,latitude),fireName, acres])
            arcpy.AddMessage("Record number: " + str(cntr) + " written to feature class")
            cntr = cntr + 1
        del cur

The first line simply creates a counter that will be used to display the progress information in the Progress Dialog box. We then start a For loop that loops through each of the features (wildfires) that have been returned. The decoded variable is a Python dictionary. Inside the For loop, we retrieve the wildfire name, latitude, longitude, and acres from the attributes dictionary. Finally, we call the insertRow() method to insert a new row into the feature class along with the wildfire name and acres as attributes. The progress information is written to the Progress Dialog box and the counter is updated.

Save the file and refresh your Python Toolbox.
Double-click on the USGS Download tool.
Leave the default URL and select the CurrentFires feature class in the WildlandFires geodatabase. The CurrentFires feature class is empty and has fields for NAMES and ACRES:
Click on OK to execute the tool. The number of features written to the feature class will vary depending upon the current wildfire activity. Most of the time, there is at least a little activity, but it is possible that there wouldn't be any wildfires in the U.S. as shown in the following screenshot:
View the feature class in ArcMap. To view the feature class in the following screenshot, I've plotted the points along with a Basemap topography layer. Your data will almost certainly be different than mine as we are pulling real-time data:

Amazon Customer Mar 11, 2017

A great resource that focuses on creating add-ins and Python toolboxes in ArcMap. A great bonus are a set of freely available Python packages that can be used to extend ArcMap so that you can interact with the ArcGIS REST API, among others. For intermediate and advanced arcpy users.

Amazon Verified review

ArcGIS Blueprints: Explore the robust features of Python to create real-world ArcGIS applications through exciting, hands-on projects

What do you get with eBook?

ArcGIS Blueprints

Chapter 1. Extracting Real-Time Wildfire Data from ArcGIS Server with the ArcGIS REST API

Design

Creating the ArcGIS Desktop Python Toolbox

Tip

Working with tool parameters

Tool execution

Populating the feature class

Installing pip and the requests module

Requesting data from ArcGIS Server

Inserting data in a feature class with the ArcPy data access module

Summary

Page 1 of 5

Key benefits

Description

Who is this book for?

What you will learn

Product Details

What do you get with eBook?

Product Details

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

People who bought this also bought

About the author

FAQs

ArcGIS Blueprints: Explore the robust features of Python to create real-world ArcGIS applications through exciting, hands-on projects

What do you get with eBook?

Contact Details

Billing Address

Tip

Key benefits

Description

Who is this book for?

What you will learn

Product Details

What do you get with eBook?

Contact Details

Billing Address

Product Details

Packt Subscriptions

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

People who bought this also bought

About the author

FAQs