In this tutorial, we will build a sample internet of things application using Google Cloud IoT.
We will start off by implementing the end-to-end solution, where we take the data from the DHT11 sensor and post it to the Google IoT Core state topic.
This article is an excerpt from the book, Enterprise Internet of Things Handbook, written by Arvind Ravulavaru.
End-to-end communication
To get started with Google IoT Core, we need to have a Google account. If you do not have a Google account, you can create one by navigating to this URL: https://accounts.google.com/SignUp?hl=en.
Once you have created your account, you can login and navigate to Google Cloud Console: https://console.cloud.google.com.
Setting up a project
The first thing we are going to do is create a project. If you have already worked with Google Cloud Platform and have at least one project, you will be taken to the first project in the list or you will be taken to the Getting started page.
As of the time of writing this book, Google Cloud Platform has a free trial for 12 months with $300 if the offer is still available when you are reading this chapter, I would highly recommend signing up:
Once you have signed up, let's get started by creating a new project. From the top menu bar, select the Select a Project dropdown and click on the plus icon to create a new project. You can fill in the details as illustrated in the following screenshot:
Click on the Create button. Once the project is created, navigate to the Project and you should land on the Home page.
Enabling APIs
Following are the steps to be followed for enabling APIs:
From the menu on the left-hand side, select APIs & Services | Library as shown in the following screenshot:
On the following screen, search for pubsub and select the Pub/Sub API from the results and we should land on a page similar to the following:
Click on the ENABLE button and we should now be able to use these APIs in our project.
Next, we need to enable the real-time API; search for realtime and we should find something similar to the following:
Click on the ENABLE & button.
Enabling device registry and devices
The following steps should be used for enabling device registry and devices:
From the left-hand side menu, select IoT Core and we should land on the IoT Core home page:
Instead of the previous screen, if you see a screen to enable APIs, please enable the required APIs from here.
Click on the & Create device registry button. On the Create device registry screen, fill the details as shown in the following table:
Field
Value
Registry ID
Pi3-DHT11-Nodes
Cloud region
us-central1
Protocol
MQTT
HTTP
Default telemetry topic
device-events
Default state topic
dht11
After completing all the details, our form should look like the following:
We will add the required certificates later on.
Click on the Create button and a new device registry will be created.
From the Pi3-DHT11-Nodes registry page, click on the Add device button and set the Device ID as Pi3-DHT11-Node or any other suitable name.
Leave everything as the defaults and make sure the Device communication is set to Allowed and create a new device.
On the device page, we should see a warning as highlighted in the following screenshot:
Now, we are going to add a new public key. To generate a public/private key pair, we need to have OpenSSL command line available. You can download and set up OpenSSL from here: https://www.openssl.org/source/.
Use the following command to generate a certificate pair at the default location on your machine:
openssl req -x509 -newkey rsa:2048 -keyout rsa_private.pem -nodes -out rsa_cert.pem -subj "/CN=unused"
If everything goes well, you should see an output as shown here:
Do not share these certificates anywhere; anyone with these certificates can connect to Google IoT Core as a device and start publishing data.
Now, once the certificates are created, we will attach them to the device we have created in IoT Core.
Head back to the device page of the Google IoT Core service and under Authentication click on Add public key. On the following screen, fill it in as illustrated:
The public key value is the contents of rsa_cert.pem that we generated earlier. Click on the ADD button.
Now that the public key has been successfully added, we can connect to the cloud using the private key.
Setting up Raspberry Pi 3 with DHT11 node
Now that we have our device set up in Google IoT Core, we are going to complete the remaining operation on Raspberry Pi 3 to send data.
Pre-requisites
The requirements for setting up Raspberry Pi 3 on a DHT11 node are:
One Raspberry Pi 3: https://www.amazon.com/Raspberry-Pi-Desktop-Starter-White/dp/B01CI58722
One breadboard: https://www.amazon.com/Solderless-Breadboard-Circuit-Circboard-Prototyping/dp/B01DDI54II/
One DHT11 sensor: https://www.amazon.com/HiLetgo-Temperature-Humidity-Arduino-Raspberry/dp/B01DKC2GQ0
Three male-to-female jumper cables: https://www.amazon.com/RGBZONE-120pcs-Multicolored-Dupont-Breadboard/dp/B01M1IEUAF/
If you are new to the world of Raspberry Pi GPIO's interfacing, take a look at this Raspberry Pi GPIO Tutorial: The Basics Explained on YouTube: https://www.youtube.com/watch?v=6PuK9fh3aL8.
The following steps are to be used for the setup process:
Connect the DHT11 sensor to Raspberry Pi 3 as shown in the following diagram:
Next, power up Raspberry Pi 3 and log in to it. On the desktop, create a new folder named Google-IoT-Device. Open a new Terminal and cd into this folder.
Setting up Node.js
Refer to the following steps to install Node.js:
Open a new Terminal and run the following commands:
$ sudo apt update
$ sudo apt full-upgrade
This will upgrade all the packages that need upgrades. Next, we will install the latest version of Node.js. We will be using the Node 7.x version:
$ curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash -
$ sudo apt install nodejs
This will take a moment to install, and once your installation is done, you should be able to run the following commands to see the version of Node.js and npm:
$ node -v
$ npm -v
Developing the Node.js device app
Now, we will set up the app and write the required code:
From the Terminal, once you are inside the Google-IoT-Device folder, run the following command:
$ npm init -y
Next, we will install jsonwebtoken (https://www.npmjs.com/package/jsonwebtoken) and mqtt (https://www.npmjs.com/package/mqtt) from npm. Execute the following command:
$ npm install jsonwebtoken mqtt--save
Next, we will install rpi-dht-sensor (https://www.npmjs.com/package/rpi-dht-sensor) from npm. This module will help in reading the DHT11 temperature and humidity values:
$ npm install rpi-dht-sensor --save
Your final package.json file should look similar to the following code snippet:
{
"name": "Google-IoT-Device",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo "Error: no test specified" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC",
"dependencies": {
"jsonwebtoken": "^8.1.1",
"mqtt": "^2.15.3",
"rpi-dht-sensor": "^0.1.1"
}
}
Now that we have the required dependencies installed, let's continue. Create a new file named index.js at the root of the Google-IoT-Device folder. Next, create a folder named certs at the root of the Google-IoT-Device folder and move the two certificates we created using OpenSSL there.
Your final folder structure should look something like this:
Open index.js in any text editor and update it as shown here:
var fs = require('fs');
var jwt = require('jsonwebtoken');
var mqtt = require('mqtt');
var rpiDhtSensor = require('rpi-dht-sensor');
var dht = new rpiDhtSensor.DHT11(2); // `2` => GPIO2
var projectId = 'pi-iot-project';
var cloudRegion = 'us-central1';
var registryId = 'Pi3-DHT11-Nodes';
var deviceId = 'Pi3-DHT11-Node';
var mqttHost = 'mqtt.googleapis.com';
var mqttPort = 8883;
var privateKeyFile = '../certs/rsa_private.pem';
var algorithm = 'RS256';
var messageType = 'state'; // or event
var mqttClientId = 'projects/' + projectId + '/locations/' + cloudRegion + '/registries/' + registryId + '/devices/' + deviceId;
var mqttTopic = '/devices/' + deviceId + '/' + messageType;
var connectionArgs = {
host: mqttHost,
port: mqttPort,
clientId: mqttClientId,
username: 'unused',
password: createJwt(projectId, privateKeyFile, algorithm),
protocol: 'mqtts',
secureProtocol: 'TLSv1_2_method'
};
console.log('connecting...');
var client = mqtt.connect(connectionArgs);
// Subscribe to the /devices/{device-id}/config topic to receive config updates.
client.subscribe('/devices/' + deviceId + '/config');
client.on('connect', function(success) {
if (success) {
console.log('Client connected...');
sendData();
} else {
console.log('Client not connected...');
}
});
client.on('close', function() {
console.log('close');
});
client.on('error', function(err) {
console.log('error', err);
});
client.on('message', function(topic, message, packet) {
console.log(topic, 'message received: ', Buffer.from(message, 'base64').toString('ascii'));
});
function createJwt(projectId, privateKeyFile, algorithm) {
var token = {
'iat': parseInt(Date.now() / 1000),
'exp': parseInt(Date.now() / 1000) + 86400 * 60, // 1 day
'aud': projectId
};
var privateKey = fs.readFileSync(privateKeyFile);
return jwt.sign(token, privateKey, {
algorithm: algorithm
});
}
function fetchData() {
var readout = dht.read();
var temp = readout.temperature.toFixed(2);
var humd = readout.humidity.toFixed(2);
return {
'temp': temp,
'humd': humd,
'time': new Date().toISOString().slice(0, 19).replace('T', ' ') // https://stackoverflow.com/a/11150727/1015046
};
}
function sendData() {
var payload = fetchData();
payload = JSON.stringify(payload);
console.log(mqttTopic, ': Publishing message:', payload);
client.publish(mqttTopic, payload, { qos: 1 });
console.log('Transmitting in 30 seconds');
setTimeout(sendData, 30000);
}
In the previous code, we first define the projectId, cloudRegion, registryId, and deviceId based on what we have created. Next, we build the connectionArgs object, using which we are going to connect to Google IoT Core using MQTT-SN. Do note that the password property is a JSON Web Token (JWT), based on the projectId and privateKeyFile algorithm.
The token that is created by this function is valid only for one day. After one day, the cloud will refuse connection to this device if the same token is used.
The username value is the Common Name (CN) of the certificate we have created, which is unused.
Using mqtt.connect(), we are going to connect to the Google IoT Core. And we are subscribing to the device config topic, which can be used to send device configurations when connected.
Once the connection is successful, we callsendData() every 30 seconds to send data to the state topic.
Save the previous file and run the following command:
$ sudo node index.js
And we should see something like this:
As you can see from the previous Terminal logs, the device first gets connected then starts transmitting the temperature and humidity along with time. We are sending time as well, so we can save it in the BigQuery table and then build a time series chart quite easily.
Now, if we head back to the Device page of Google IoT Core and navigate to the Configuration & state history tab, we should see the data that we are sending to the state topic here:
Now that the device is sending data, let's actually read the data from another client.
Reading the data from the device
For this, you can either use the same Raspberry Pi 3 or another computer. I am going to use MacBook as a client that is interested in the data sent by the Thing.
Setting up credentials
Before we start reading data from Google IoT Core, we have to set up our computer (for example, MacBook) as a trusted device, so our computer can request data. Let's perform the following steps to set the credentials:
To do this, we need to create a new Service account key. From the left-hand-side menu of the Google Cloud Console, select APIs & Services | Credentials. Then click on the Create credentials dropdown and select Service account key as shown in the following screenshot:
Now, fill in the details as shown in the following screenshot:
We have given access to the entire project for this client and as an Owner. Do not select these settings if this is a production application.
Click on Create and you will be asked to download and save the file.
Do not share this file; this file is as good as giving someone owner-level permissions to all assets of this project.
Once the file is downloaded somewhere safe, create an environment variable with the name GOOGLE_APPLICATION_CREDENTIALS and point it to the path of the downloaded file.
You can refer to Getting Started with Authentication at https://cloud.google.com/docs/authentication/getting-started if you are facing any difficulties.
Setting up subscriptions
The data from the device is being sent to Google IoT Core using the state topic. If you recall, we have named that topic dht11. Now, we are going to create a subscription for this topic:
From the menu on the left side, select Pub/Sub | Topics. Now, click on New subscription for the dht11 topic, as shown in the following screenshot:
Create a new subscription by setting up the options selected in this screenshot:
We are going to use the subscription named dht11-data to get the data from the state topic.
Setting up the client
Now that we have provided the required credentials as well as subscribed to a Pub/Sub topic, we will set up the Pub/Sub client. Follow these steps:
Create a folder named test_client inside the test_client directory. Now, run the following command:
$ npm init -y
Next, install the @google-cloud/pubsub (https://www.npmjs.com/package/@google-cloud/pubsub) module with the help of the following command:
$ npm install @google-cloud/pubsub --save
Create a file inside the test_client folder named index.js and update it as shown in this code snippet:
var PubSub = require('@google-cloud/pubsub');
var projectId = 'pi-iot-project';
var stateSubscriber = 'dht11-data'
// Instantiates a client
var pubsub = new PubSub({
projectId: projectId,
});
var subscription = pubsub.subscription('projects/' + projectId + '/subscriptions/' + stateSubscriber);
var messageHandler = function(message) {
console.log('Message Begin >>>>>>>>');
console.log('message.connectionId', message.connectionId);
console.log('message.attributes', message.attributes);
console.log('message.data', Buffer.from(message.data, 'base64').toString('ascii'));
console.log('Message End >>>>>>>>>>');
// "Ack" (acknowledge receipt of) the message
message.ack();
};
// Listen for new messages
subscription.on('message', messageHandler);
Update the projectId and stateSubscriber in the previous code. Now, save the file and run the following command:
$ node index.js
We should see the following output in the console:
This way, any client that is interested in the data of this device can use this approach to get the latest data.
With this, we conclude the section on posting data to Google IoT Core and fetching the data. In the next section, we are going to work on building a dashboard.
Building a dashboard
Now that we have seen how a client can read the data from our device on demand, we will move on to building a dashboard, where we display data in real time.
For this, we are going to use Google Cloud Functions, Google BigQuery, and Google Data Studio.
Google Cloud Functions
Cloud Functions are solution for serverless services. Cloud Functions is a lightweight solution for creating standalone and single-purpose functions that respond to cloud events.
You can read more about Google Cloud Functions at https://cloud.google.com/functions/.
Google BigQuery
Google BigQuery is an enterprise data warehouse that solves this problem by enabling super-fast SQL queries using the processing power of Google's infrastructure.
You can read more about Google BigQuery at https://cloud.google.com/bigquery/.
Google Data Studio
Google Data Studio helps to build dashboards and reports using various data connectors, such as BigQuery or Google Analytics.
You can read more about Google Data Studio at https://cloud.google.com/data-studio/.
As of April 2018, these three services are still in beta.
As we have already seen in the Architecture section, once the data is published on the state topic, we are going to create a cloud function that will get triggered by the data event on the Pub/Sub client. And inside our cloud function, we are going to get a copy of the published data and then insert it into the BigQuery dataset.
Once the data is inserted, we are going to use Google Data Studio to create a new report by linking the BigQuery dataset to the input.
So, let's get started.
Setting up BigQuery
The first thing we are going to do is set up BigQuery:
From the side menu of the Google Cloud Platform Console, our project page, click on the BigQuery URL and we should be taken to the Google BigQuery home page. Select Create new dataset, as shown in the following screenshot:
Create a new dataset with the values illustrated in the following screenshot:
Once the dataset is created, click on the plus sign next to the dataset and create an empty table. We are going to name the table dht11_data and we are going have three fields in it, as shown here:
Click on the Create Table button to create the table.
Now that we have our table ready, we will write a cloud function to insert the incoming data from Pub/Sub into this table.
Setting up Google Cloud Function
Now, we are going to set up a cloud function that will be triggered by the incoming data:
From the Google Cloud Console's left-hand-side menu, select Cloud Functions under Compute. Once you land on the Google Cloud Functions homepage, you will be asked to enable the cloud functions API. Click on Enable API:
Once the API is enabled, we will be on the Create function page. Fill in the form as shown here:
The Trigger is set to Cloud Pub/Sub topic and we have selected dht11 as the Topic.
Under the Source code section; make sure you are in the index.js tab and update it as shown here:
var BigQuery = require('@google-cloud/bigquery');
var projectId = 'pi-iot-project';
var bigquery = new BigQuery({
projectId: projectId,
});
var datasetName = 'pi3_dht11_dataset';
var tableName = 'dht11_data';
exports.pubsubToBQ = function(event, callback) {
var msg = event.data;
var data = JSON.parse(Buffer.from(msg.data, 'base64').toString());
// console.log(data);
bigquery
.dataset(datasetName)
.table(tableName)
.insert(data)
.then(function() {
console.log('Inserted rows');
callback(); // task done
})
.catch(function(err) {
if (err && err.name === 'PartialFailureError') {
if (err.errors && err.errors.length > 0) {
console.log('Insert errors:');
err.errors.forEach(function(err) {
console.error(err);
});
}
} else {
console.error('ERROR:', err);
}
callback(); // task done
});
};
In the previous code, we were using the BigQuery Node.js module to insert data into our BigQuery table. Update projectId, datasetName, and tableName as applicable in the code.
Next, click on the package.json tab and update it as shown:
{
"name": "cloud_function",
"version": "0.0.1",
"dependencies": {
"@google-cloud/bigquery": "^1.0.0"
}
}
Finally, for the Function to execute field, enter pubsubToBQ. pubsubToBQ is the name of the function that has our logic and this function will be called when the data event occurs.
Click on the Create button and our function should be deployed in a minute.
Running the device
Now that the entire setup is done, we will start pumping data into BigQuery:
Head back to Raspberry Pi 3 which was sending the DHT11 temperature and humidity data, and run the application. We should see the data being published to the state topic:
Now, if we head back to the Cloud Functions page, we should see the requests coming into the cloud function:
You can click on VIEW LOGS to view the logs of each function execution:
Now, head over to our table in BigQuery and click on the RUN QUERY button; run the query as shown in the following screenshot:
Now, all the data that was generated by the DHT11 sensor is timestamped and stored in BigQuery.
You can use the Save to Google Sheets button to save this data to Google Sheets and analyze the data there or plot graphs, as shown here:
Or we can go one step ahead and use the Google Data Studio to do the same.
Google Data Studio reports
Now that the data is ready in BigQuery, we are going to set up Google Data Studio and then connect both of them, so we can access the data from BigQuery in Google Data Studio:
Navigate to https://datastudio.google.com and log in with your Google account.
Once you are on the Home page of Google Data Studio, click on the Blank report template. Make sure you read and agree to the terms and conditions before proceeding.
Name the report PI3 DHT11 Sensor Data. Using the Create new data source button, we will create a new data source.
Click on Create new data source and we should land on a page where we need to create a new Data Source. From the list of Connectors, select BigQuery; you will be asked to authorize Data Studio to interface with BigQuery, as shown in the following screenshot:
Once we authorized, we will be shown our projects and related datasets and tables:
Select the dht11_data table and click on Connect. This fetches the metadata of the table as shown here:
Set the Aggregation for the temp and humd fields to Max and set the Type for time as Date & Time. Pick Minute (mm) from the sub-list.
Click on Add to report and you will be asked to authorize Google Data Studio to read data from the table.
Once the data source has been successfully linked, we will create a new time series chart.
From the menu, select Insert | Time Series link. Update the data configuration of the chart as shown in the following screenshot:
You can play with the styles as per your preference and we should see something similar to the following screenshot:
This report can then be shared with any user. With this, we have seen the basic features and implementation process needed to work with Google Cloud IoT Core as well other features of the platform.
If you found this post useful, do check out the book, Enterprise Internet of Things Handbook, to build state of the art IoT applications best-fit for Enterprises.
Cognitive IoT: How Artificial Intelligence is remoulding Industrial and Consumer IoT
Five Most Surprising Applications of IoT
How IoT is going to change tech teams
Read more
Germany
Slovakia
Canada
Brazil
Singapore
Hungary
Philippines
Mexico
Thailand
Ukraine
Luxembourg
Estonia
Lithuania
Norway
Chile
United States
Great Britain
India
Spain
South Korea
Ecuador
Colombia
Taiwan
Switzerland
Indonesia
Cyprus
Denmark
Finland
Poland
Malta
Czechia
New Zealand
Austria
Turkey
France
Sweden
Italy
Egypt
Belgium
Portugal
Slovenia
Ireland
Romania
Greece
Argentina
Malaysia
South Africa
Netherlands
Bulgaria
Latvia
Australia
Japan
Russia
