Practical XMPP

Chapter 1. An Introduction to XMPP and Installing Our First Server

Picture this scene: It's 1999 and you have just received your brand new computer, which sports a super-fast 333 MHz processor, a massive 256 MB of RAM, and a copy of the latest operating system, Windows 98. Not only that, but it also has a 33.6 kbps modem. After setting a wallpaper of your favorite mountain range and adjusting the screensaver to have a cool floating 3D effect, you set about connecting yourself to the information superhighway.

Soon, colleagues and friends find out that you have got yourself connected and start inviting you to connect. Jennifer in the office is using ICQ, so you also install ICQ and start chatting. Meanwhile, you've received an e-mail from your friend Marty, who asks you to install AOL Instant Messenger. Not wanting to be left out, you sign up and connect there too. Several weeks later, you've also installed Yahoo! Instant Messenger and MSN Messenger.

Now, each time your modem completes making the bing-bong electronic noises, you instinctively fire up several instant messaging applications to ensure you don't miss any messages from friends, family, and colleagues. Worse still, if you want to have a group chat with Linda and Marvin, that's not possible since both of your friends have decided to use different instant messaging programs. You begin to think to yourself that there has to be a better way.

Enter XMPP, a federated secure set of standards that allow clients and servers to talk to each other in a common language.

In this chapter, we'll cover the following topics:

What is XMPP?
Uses of XMPP
XMPP and the Web
Installing Node.js
Installing, configuring, and testing our XMPP server
Creating a test account and connecting

What is XMPP?

Work on a new instant messaging system designed to solve the aforementioned problems began in 1998 with the development of the Extensible Messaging and Presence Protocol, or XMPP, and the first XMPP server was made available by January 1999.

Over the next 6 years, the Jabber software and standards were defined and eventually approved by the Internet Engineering Task Force (IETF). Jabber became known as XMPP during that process. The process of becoming part of the IETF meant that the protocol underwent a huge amount of scrutiny, laying the foundations for the maturity and security in the protocol that we are seeing today.

Additional standards were created later that extended the platform so that it could handle things such as voice and video calls, Multi-User Chat (MUC), publish-subscribe systems, avatars, vCards, and feature discovery. These standards are looked after by a body called the XMPP Standards Foundation, or XSF, which handles the publishing, editing, finalizing, and obsoleting of the standards.

The important thing to realize about XMPP is that rather than being a set of software, it is a set of standards that define how clients and servers should interact, much like HTTP is a standard rather than a specific piece of software like Apache. What this means is that there are many implementations of servers and clients, all interactively speaking a common language from proprietary solutions to open source offerings forming a large and rich ecosystem.

The XMPP logo

XMPP provides many advantages outside of its plethora of servers and clients, which includes the following:

Decentralized: Unlike big messaging silos such as MSN messenger or ICQ, XMPP is built from the ground up so that servers are able to intercommunicate, mimicking the format of the early Internet. With no single point of failure, a friend's server going down does not prevent you from communicating with other friends or colleagues.
Secure: The XMPP community works hard to ensure that the standards and implementations are highly secure, by adopting things such as mandatory server-to-server encryption and secure authentication mechanisms, and by becoming involved with (and implementing where appropriate) new secure mechanisms such as DNSSEC. This focus on security has led to XMPP being deployed in environments requiring the highest levels of security, from actively deployed armed forces through to financial institutions.
Scalable: With servers able to handle hundreds of thousands of connections at any time and due to XMPP's push architecture (versus HTTP's pull for example), hugely scaled systems not previously possible have been created.
Real-time: Sending messages in real time enables new ways of developing capabilities that were previously difficult to support, such as remote collaboration on a project and talking to people all over the world without the lag or complexity of older communication methods.
Multi-device: Somewhat ahead of its time, XMPP was built with multiple devices baked into the protocol. These are exposed as resources on your Jabber ID (JID), which means you can be connected to the same account from a desktop computer, a mobile phone, and a tablet at the same time and have messages routed appropriately.
Extensible: XMPP is an extensible platform, meaning new functionality can be added to the protocol without interfering with existing features. This means that new ways of working and new standards come into existence (such as WebRTC), which can be integrated into XMPP with ease. These extensions are called XMPP Extension Protocols, or XEP for short, and a full list (including obsolete or abandoned extensions) can be seen on the XMPP website at http://xmpp.org/extensions.
Mature: With hundreds of server and client implementations developed over 15 years, the standard can be said to be heavily battle-tested, and many deployments of XMPP systems demonstrate that the standards embodied by the protocol are scalable for today's Internet and the Internet of the future.

Lastly, another major advantage of using or developing on the XMPP platform is the amazing community that rallies around the protocol. Within the XSF and the community at large, there is a huge amount of experience, help, knowledge, and enthusiasm. The community has very active mailing lists, and chat rooms, as well as scores of developers writing blogs and articles and regularly speaking at events all over the world. This welcoming community means that it is easy to propose ideas, get feedback, and improve deployments or implementations with the knowledge that you will be supported by a great group of people.

Uses of XMPP

XMPP is used to solve many real-world problems and has made its way into many large services, often without users realizing. Some large examples of deployments or XMPP-based services include:

Jabber.org
WhatsApp
Google Cloud Messaging (GCM)
Facebook chat
Google Talk (discontinued)
HipChat

Additionally, many services also allow users to connect via an XMPP gateway, meaning that users can connect with their preferred clients rather than with those supplied by a company, such as Slack and Skype.

XMPP is deployed in many industries where its uses go beyond simple chat applications to include diverse purposes such as the following:

Building management
Gaming
System control
Log watching
Cyber-incident report management
Peer-to-peer session negotiation (such as with WebRTC)
Voice-over-IP (VoIP)
Identity services

XMPP and the Web

With the continuing rise of the real-time web and WebRTC applications, XMPP is becoming more relevant on the web than ever before. Its push architecture means that lightweight real-time applications can be built without continuously polling the server, making for more scalable web applications. Its mature standards mean that developers are quickly able to build real-world applications without having to design their own proprietary standards and benefit from the thought and problem solving that has gone into XMPP.

With all its advantages, one area that WebRTC hasn't solved (nor was it designed to solve) is the issue of session negotiation. Not only was a standard called Jingle available in XMPP before WebRTC arrived in the browser, but also it turns out that it's a great way of transferring session data to form peer-to-peer sessions. This means that users are now able to use WebRTC clients of their choice without having to visit and reply upon a meet me type URL or a proprietary signaling method.

Installing Node.js and library dependencies

Node.js was released in 2009, and since its release, it has quickly become a nearly indispensable tool for server side development using JavaScript. Node.js allows developers to create fast and scalable applications (especially when asynchronous IO libraries are used) in a single language throughout an application's stack, with JavaScript running on both the back-end and front-end.

While Node.js is not required for working with XMPP, all examples in this book will utilize libraries written in Node.js, some of which can also run on the browser.

Using Ubuntu 16.04 as a base, we're now going to install Node.js and some core dependencies for the libraries we'll be using.

First we'll install libicu-dev and libexpat-dev. These libraries allow us to perform case transformations on international characters and process XML, respectively:

$ sudo apt-get install libicu libexpat1-dev

Next, we'll work on installing Node.js itself. My preferred method of achieving this is to use nvm by Tim Caswell, which is available from this GitHub repository: https://github.com/creationix/nvm . Please check out the readme for the current installation instructions, but at the time of writing this book, the procedure is as follows:

$ curl https://raw.githubusercontent.com/creationix/nvm/v0.32.0
    /install.sh | bash
$ source ~/.bashrc
$ nvm install 6
$ node -v

This will install the latest version of the Node.js 6 release for you. The advantage of using nvm over the operating system packages is that it is easy to switch between different versions of Node.js with a simple command.

Installing our XMPP server

There are a large number of XMPP servers in the wild, from proprietary to open source, with communities of varying sizes. The XSF maintains a list of servers on its website at http://xmpp.org/xmpp-software/servers/ . However, this list does not take into account whether the projects are inactive or not; nor does it indicate the state of their implementations. Four open source servers worth having a look at which include:

Openfire: A hugely popular Java-based server with a large community supporting a large number of plugins and that is actively developed (for more information can be found at: http://www.igniterealtime.org/projects/openfire/)
Tigase: A popular Java-based server with active development and a great community, for more information can be found at http://www.tigase.net/
MongooseIM: An Erlang-based server forked from a previous XMPP server implementation and actively developed by Erlang Solutions, which can be found at https://github.com/esl/MongooseIM
Prosody: A fast and resource-light Lua-based server with a great core development team and an active community, which can be found at http://prosody.im

In this book, we're going to make use of Prosody since it is very easy to install, run, and configure. Before deploying to production, we recommend that you investigate the advantages and disadvantages of each system to see what works best for your particular requirements and environment. As XMPP is a standard, you'll be able to swap out servers without making any code changes, a great benefit of working with this setup.

Installing the server

Installing Prosody on an Ubuntu system is very easy since it is included in the main package repositories. In order to install Prosody, run the following command at the command line:

$ sudo apt-get install prosody

Prosody can then be started and stopped using the following standard methods:

$ sudo service prosody start
$ sudo service prosody stop

If you are not using Ubuntu, then Prosody may be available via your respective package manager. The best installation instructions for Prosody can be found at http://prosody.im/doc/install.

Prosody also produces Docker images of their releases, which are available from the official Docker hub: https://registry.hub.docker.com/u/prosody/prosody/ .

Configuring the server

By default, Prosody will store its configuration file at /etc/prosody/prosody.cfg.lua. The configuration file itself is rather simple and is written in the Lua language, which is the same language in which the Prosody server is written.

Shown here is the configuration file we are going to use throughout this book. Following the configuration example is an explanation of each part of these settings. As always, for the most up-to-date and detailed information, please visit the Prosody website (http://prosody.im):

modules_enabled = { 
        "roster"; 
        "saslauth"; 
        "tls"; 
        "dialback"; 
        "disco"; 
        "version"; 
        "uptime"; 
        "time"; 
        "ping", 
        "register"; 
        "posix"; 
        "bosh"; 
}; 
allow_registration = true; 
daemonize = true; 
consider_bosh_secure = true; 
cross_domain_bosh = true; 
pidfile = "/var/run/prosody/prosody.pid"; 
c2s_require_encryption = false 
authentication = "internal_plain" 
 
log = { 
        debug = "/var/log/prosody/prosody.log"; 
        error = "/var/log/prosody/prosody.err"; 
        { levels = { "error" }; to = "syslog";  }; 
} 
 
VirtualHost "localhost" 
    enabled = true 
    ssl = { 
        key = "/etc/prosody/certs/example.com.key"; 
        certificate = "/etc/prosody/certs/example.com.crt"; 
    } 
 
VirtualHost "anon.localhost" 
   authentication = "anonymous" 
 
Component "component.localhost" 
    component_secret = "mysecretcomponentpassword"

Now, let's discuss each of the sections of the configuration file:

modules_enabled: describes the modules to load when Prosody is started. At its core, Prosody is quite small, but many of the XMPP features are implemented within modules. If you ask members of the Prosody team whether the server supports a new feature, the response will generally be "There's a module for that." A full list of modules and configuration can be found at http://prosody.im/doc/modules .

The modules we are loading here are mostly the defaults which are set by Prosody on installation:

Module	Description
`roster`	`roster` is like an address book for XMPP.
`saslauth`	Simple Authentication and Security Layer (SASL) is a framework for authentication. It separates authentication mechanisms from application protocols, allowing any authentication mechanism supported by SASL to be used with any application protocol that uses SASL. It is within the SASL framework that we see the more secure authentication mechanisms.
`tls`	Transport Layer Security provides encrypted communications for XMPP data transfer.
`dialback`	Provides server identity verification using DNS when attempting to talk to remote servers.
`disco`	Short for discovery, this allows the server to advertise what features it supports to other servers and clients.
`version`	Replies to software version requests.
`uptime`	Reports on how long the server has been active.
`time`	Reports back on the time according to the server.
`ping`	Sets up the server to reply to ping requests.
`register`	Allows clients to create new user accounts on sign-up.
`Posix`	Required for `daemonizing` and `syslog` logging.
`Bosh`	Bidirectional-streams Over Synchronous HTTP (BOSH) is a long polling setup that allows two way XMPP communication over HTTP. It is a predecessor of WebSockets but still very useful in situations where connectivity isn't perfect.

allow_registration=true;: Allow users to create an account from a client.
daemonize = true;: Run Prosody as a daemon. If running in Docker, set this to false.
consider_bosh_secure = true;: Generally, rather than talking to Prosody's BOSH endpoint directly, we proxy it via a web server (such as Nginx) and use that to handle SSL. Therefore it's safe to consider BOSH requests as secure.
cross_domain_bosh = true;: This adds COR headers to BOSH responses to allow requests to come from any domain.
pidfile = "/var/run/prosody/prosody.pid";: This is the location of the Process ID (pid) file for Prosody.
c2s_require_encryption = false;: In production, we'd have this set to true for security, but for development, it's safe to set this to false. If set to true, then we'd be insisting on encrypted connections.
authentication = "internal_plain";: The authentication mechanism used for clients. Here, we're saying to store the password as plain text, but in production, we'd store passwords as encrypted strings. This then also affects the usable authentication mechanisms.
The log entry: Sets the log locations and the level at which we perform logging. Our setup pushes even the most detailed entries to the log files.
Next, we define a set of virtual hosts, much like we would with a web server. Here, we have defined two virtual hosts for different reasons:
- VirtualHost "localhost": This is our main virtual host and the one on which our users will exist. A user's JID will appear as test@localhost, for example.
- VirtualHost "anon.localhost": Here we define what is known as an anonymous domain. This allows users to connect without an XMPP account. Generally, it is a best practice to prevent anonymous accounts from communicating with other servers in order to prevent spam.
Lastly, we define a component connection (more about these later). We define our component to run on a subdomain. When a component attempts to connect to a server, the component sends a shared secret (i.e. a password known in configuration files for both the component and the server) as an authentication mechanism. Generally, components are run on the same server, so this method of authentication is considered appropriate.

Then we save this configuration file to /etc/prosody/prosody.cfg.lua and restart the server:

$ sudo service prosody restart

Testing our setup

Next, we're going to test our Prosody setup. First, we will test whether Prosody is listening to the client-to-server (C2S) and server-to-server (S2S) ports. By default, with XMPP, the C2S port is 5222 and the S2S port is 5269. We can perform a quick and easy test using telnet. If your server is running as expected, you will see an output similar to the following screenshot:

Testing Prosody connections using telnet

Next, we will also verify whether we have set up our BOSH endpoint correctly by visiting the endpoint in a browser, by visiting http://localhost:5280/http-bind/, port number 5280 being the default BOSH port, we should see the following screenshot:

Testing Prosody's BOSH endpoint from a browser

Creating a test account

Prosody has a great command-line utility, prosodyctl, which allows us to create and update user accounts as well as command the server. Here, we're going to use it to create our first XMPP account on the server. Type the following command in the command line:

$ sudo prosodyctl adduser test@localhost

You will then be prompted to enter a password. Let's use the Password password for now. Once this is complete, we have our XMPP account. We'll test this with a client in a moment.

Tip

If you ever forget your user passwords, it's simply a case of running the preceding command but replacing adduser with passwd. You will then be prompted for a new password.

Installing an XMPP client

Lastly, we'll install a standard desktop XMPP client, which will allow us to interact with our code. A list of XMPP clients is held on the XMPP website (http://xmpp.org/xmpp-software/clients/). Find one suitable for your platform and follow the installation instructions.

In this book, we're going to use Empathy (https://live.gnome.org/Empathy) since it comes installed as standard in the Ubuntu desktop:

First, load the Empathy application and you will be presented with the main window.
From here, select Accounts from the Empathy menu in the title bar.
Choose to add an account and select the Jabber option.

Setting up an XMPP account in Empathy on Ubuntu
Next, in the Jabber ID box, enter test@localhost; and for the Password, enter password.
Once your details are entered, pop open the Advanced box and uncheck Encryption required, since we won't need this while developing.
After clicking on Done, your client should be connected to your local XMPP server; it's not very exciting so far.

To doubly verify that you are connected, you can check the Prosody log file and you should see entries similar to the following:

$ sudo tail /var/log/prosody/prosody.log
Feb 09 20:37:43 c2s2250e30  info  Client connected
Feb 09 20:37:43 c2s2250e30  info  Authenticated as test2@localhost

Note

In addition to the main log file, there is also an error log file located at /var/log/prosody/prosody.err (by default). Should something be happening that you aren't expecting, this is also a good location to check.