Decoupling applications using Pub/Sub

Pub/Sub is a messaging service that allows us to decouple applications that produce and receive events, creating a much more robust system due to the high availability and consistency that the service provides.

Before looking in detail at the different functionalities of Pub/Sub, it is important to understand the fundamental concepts of this technology, since we will mention them frequently in this section. These are presented here:

• Topic: Corresponds to a resource created within Pub/Sub where publishers will send their generated messages
• Subscription: Corresponds to a resource that represents the flow of messages from a specific topic to a particular subscriber
• Message: Corresponds to the information and attributes that are sent to a topic through a publisher and received by the subscribers to that topic

Pub/Sub types

Among the types of service that Pub/Sub offers us are the following:

• Pub/Sub: Multi-zone-type Pub/Sub version, with automatic provisioning, unlimited storage per topic, message retention of up to 7 days, and global endpoint exposures
• Pub/Sub Lite: Pub/Sub version of a zonal type that allows reduced costs, provision of resources before being used, maximum storage of 10 terabytes (TB) per topic, unlimited message retention, and exposure of regional endpoints

Pub/Sub

If we select the Pub/Sub type, our service will scale automatically, which is ideal for scenarios where there is not a constant flow of messages and where we have to be prepared for message peaks. In addition, being a global resource, both topics and subscriptions to these topics can occur in different Google Cloud projects.

Pub/Sub Lite

If we select the Pub/Sub Lite type, our service will have a limited capacity for resources established (number of partitions and storage). This is ideal for scenarios where the flow of messages is predictable and known in advance. If you want to increase the capacity of your service, it is possible to do so manually. In addition, as it is a zonal-type resource, both the topics and the subscriptions must be in the same project and zone.

Relationship options

Within the relationship options between publishers and subscribers when using Pub/Sub, we can find the following scenarios:

• Fan in: Multiple publishers send messages to multiple topics that are consumed by a single subscriber. This is also known as a many-to-one relationship.
• Fan out: A publisher posts a message on a topic that is consumed by multiple subscribers. This is also known as a one-to-many relationship.
• Many-to-many: Multiple publishers send messages to multiple topics that are consumed by multiple subscribers.

You can see a diagram of these Pub/Sub patterns here:

Figure 8.16 – Pub/Sub patterns

Publishing a message

To post a message within a Pub/Sub topic, the first thing we need to do is create a topic within Pub/Sub. For that, we use the following command from the Google Cloud SDK:

gcloud pubsub topics create TOPIC_ID

Here, TOPIC_ID is the ID with which a topic will be created.

Once a topic is created, we must assign the necessary permissions to the service account that we will use in our project to publish a message to the topic. For this, we can assign the roles/pubsub.publisher role.

Once we have the role that allows us to publish our message, we can proceed to code implementation. In this particular example, we will use Node.js, so we will install the Pub/Sub library using the following command:

npm install @google-cloud/pubsub --save

If you need to use another programming language, you can go to the official documentation for examples in the various programming languages supported by the Pub/Sub library, at https://cloud.google.com/pubsub/docs/publisher.

Then, to publish a message on the topic, we implement the following code:

const topicName = 'TOPIC_NAME';
const data = JSON.stringify({field: 'value'});
const {PubSub} = require('@google-cloud/pubsub');
const pubSubClient = new PubSub();
async function publishMessage() {
  
  const dataBuffer = Buffer.from(data);
  try {
    
    const messageId = await pubSubClient.topic(topicName).publish(dataBuffer);
    
  } catch (error) {
    //TODO: Handle error.
  }
}
publishMessage(); 

TOPIC_NAME is the name of the topic in which you want to publish a message (projects/PROJECT_ID/topics/TOPIC_ID).

In this way, we can publish our first message in a Pub/Sub topic.

Consuming a message using pull and push subscriptions

Within the different types of subscriptions available within Pub/Sub, we find the following:

• Push: In this type of subscription, Pub/Sub sends a request with a message to the subscribing application every time a message reaches a topic.
• Pull: In this type of subscription, the subscribing application is responsible for generating a request in order to obtain a message from a topic.

For scenarios where multiple topics within Pub/Sub must be processed by a single webhook, where there are webhooks implemented in Google App Engine (GAE) or Cloud Functions, or where it is not possible to configure dependencies in the webhooks such as credentials or client libraries, it is recommended to use push subscriptions.

To create a push subscription, use the following command from the Google Cloud SDK:

gcloud pubsub subscriptions create SUBSCRIPTION_ID --topic=TOPIC_ID –push-endpoint=PUSH_ENDPOINT

Here, the following applies:

• SUBSCRIPTION_ID is the ID with which a subscription will be created.
• TOPIC_ID is the ID of the topic on which a subscription will be created.
• PUSH_ENDPOINT is the Uniform Resource Locator (URL) of the webhook where a message received by a topic will be received.

In this way, we can configure a subscription that calls the indicated endpoint every time a message arrives on the topic.

In situations where there is a high message-flow rate (greater than one message per second), where the efficiency of message processing is critical, or where it is not possible to expose endpoints for webhooks with Secure Sockets Layer (SSL) certificates, it is recommended that pull subscriptions are used.

In order to create a pull-type subscription, use the following command from the Google Cloud SDK:

gcloud pubsub subscriptions create SUBSCRIPTION_ID --topic=TOPIC_ID

Here, the following applies:

• SUBSCRIPTION_ID is the ID with which a subscription will be created.
• TOPIC_ID is the ID of the topic on which a subscription will be created.

Unlike push-type subscriptions, where Pub/Sub is responsible for sending messages to our application through the indicated endpoint, in pull-type subscriptions we must constantly pull messages from a topic.

For this, we need to have a service account with the necessary permissions to obtain these messages, to which we assign the role of roles/pubsub.subscriber.

Once we have this role enabled, allowing us to read our messages, we can proceed to code implementation. In this example, we will use Node.js, so we will install the Pub/Sub library using the following command:

npm install @google-cloud/pubsub --save

If you need to use another programming language, go to the official documentation where there are examples in the programming languages supported by the Pub/Sub library, at https://cloud.google.com/pubsub/docs/pull.

Then, to read a message from a topic to which we are subscribed, we implement the following code:

const subscriptionName = 'SUBSCRIPTION_NAME';
const timeout = TIMEOUT_IN_SECONDS;
const {PubSub} = require('@google-cloud/pubsub');
const pubSubClient = new PubSub();
function listenForMessages() {
  const subscription = pubSubClient.subscription(subscriptionName);
  const messageHandler = message => {
    //TODO: get message information from message.data
    message.ack();
  };
  subscription.on('message', messageHandler);
  setTimeout(() => {
    subscription.removeListener('message', messageHandler);
  }, timeout * 1000);
}
listenForMessages();

Here, the following applies:

• SUBSCRIPTION_NAME is the name of a subscription associated with a topic to consume the message.
• TIMEOUT_IN_SECONDS is the time in seconds for which a subscription listening will pull for new messages.

In this way, we can create a subscription that allows us to programmatically obtain messages from our application.