The history of APIs
By now, you already know that there are different types of APIs that you can use depending on what you’re trying to achieve. It’s important to know how APIs were created and which events were the key contributors to their evolution. Learning how we got to where we are now is the first step to understanding how to build successful products on top of APIs.
To understand how APIs were invented, let’s go back in time to circa 1950. At that time, computing was just getting started, and the first known mention of a software module was made by Herman Goldstine and John von Neumann. The authors were referring to local APIs in the form of modules or, as they called it, “subroutines.” In their Report on the Mathematical and Logical Aspects of an Electronic Computing Instrument from 1947, they explain what a subroutine is. To the authors, a subroutine is a special kind of routine—a set of program instructions that can be reused. They describe that a subroutine has the purpose of being substituted inside any existing routine. It’s clear that the authors were focusing on reusability, as you’ve read before in this chapter. In fact, from then on, APIs have been used as a way to reduce the amount of programming required to build an application.
However, the work of Goldstine and von Neumann was purely theoretical and couldn’t be put into practice at that time. It was only when the Electronic Delay Storage Automatic Calculator, or EDSAC, was created that software modules were actually used. In 1951, Maurice Wilkes, the creator of EDSAC, was inspired by the work of Goldstine and von Neumann. Wilkes introduced the use of modules as a way to reuse functionality and make it easier to write programs from scratch. Wilkes, jointly with Wheeler and Gill, describes how subroutines could be used in their book The Preparation of Programs for an Electronic Digital Computer. In this, they explain that with a library of subroutines, it should be simple to program sophisticated calculations. You’d only have to write a master routine that, in turn, calls the various subroutines where the different calculations are executed.
Still, the first time the term “application program interface” (notice that the word used isn’t “programming”) appeared was probably in the Data Structures and Techniques for Remote Computer Graphics paper published in 1968. In this, its authors, Ira Walter Cotton and Frank S. Greatorex, notice the importance of the reusability of code in the context of hardware replacement. The paper mentions that even if you change the hardware, a “consistent application program interface could be maintained.”
Since then, the concept of reusability through the isolation of code to create APIs has been evolving. While initially, the focus was on APIs that could be used locally within an operating system, at a later stage, remote APIs were explored, most notably web APIs. In 2000, Roy Fielding published an intriguing Ph.D. dissertation titled Architectural Styles and the Design of Network-based Software Architecture. In it, Fielding analyzes the differences between local APIs—which are based on libraries or modules—and remote APIs. The author calls local APIs library-based and remote APIs network-based. While local APIs have the goal of offering entry points to code that can be reused, remote APIs aim to enable application interactions. According to Fielding, the only restriction that remote APIs have is the ability to read and write to the network where they operate. Let’s continue our exploration through the history of APIs. Keep reading to understand how one of the most popular operating systems is in the origins of web APIs.
Unix
Web APIs originated with the Unix operating system and its way of letting different applications—or processes—communicate with each other. In reality, Unix is not a single operating system. It’s a group of operating systems with the same root: AT&T Unix. The first version was created in the 1970s by Ken Thompson and Dennis Ritchie at the Bell Labs research center. AT&T decided to license Unix to other companies after the first version was released to the public in 1973. The licensing of Unix made it—and all its variants—one of the most used types of operating systems of all time. One of those variants, the Sun Microsystems Solaris operating system, has contributed the most to the history of web APIs.
From its inception, Unix has been recognized for its modular structure, where individual processes are created with simplicity in mind and aimed at seamless collaboration. This approach, referred to as the Unix philosophy, has become one of the primary reasons for its immense success and a crucial factor in the evolution of web APIs. Brian Kernighan, one of the developers of the Unix operating system, described interoperability during a demo performed in 1982:
Inter-process communication, or IPC, is a system integrated into Unix that enables the transfer of messages between various processes. IPC is a collection of APIs that allows developers to coordinate the execution of concurrent processes. The IPC framework offers multiple forms of communication, including pipes, message queues, semaphores, shared memory, and sockets, to accommodate the needs of diverse applications. However, it’s worth noting that, except for sockets, all other communication methods are confined to processes running on the same server, limiting their scope and functionality. It’s precisely with sockets that network APIs gained traction and different use cases emerged.
Network APIs
Sun Microsystems leveraged the functionality of network sockets to introduce a method of communicating with remote processes, known as RPC. The concept of RPC was first introduced in the 1980s as part of Sun’s Network File System (NFS) project and adhered to the calling conventions used in Unix and the C programming language. It rapidly gained popularity due to its ability to enable any running application to send a request over a network to another application, which would then respond with the result of the requested operation. The messages and responses are encoded using the External Data Representation format, or XDR, which provides a standard format understood by both the producer and consumer. The RPC protocol offers the capability to deliver messages with XDR payloads through either the User Datagram Protocol (UDP) or the Transmission Control Protocol (TCP), thereby providing compatibility with different network types. While UDP is a protocol more oriented toward performance, TCP offers more reliability in situations where the quality of the network is questionable.
The journey from the first implementations of RPC to its official publication as an Internet Engineering Task Force (IETF) Request for Comments, or RFC, took about a decade. The RPC protocol was first published as RFC 1831 in 1995 and underwent various transformations through subsequent versions until it reached its latest form in 2009, as described in RFC 5531. That year, Sun Microsystems changed the RPC license to the widely used three-clause BSD, which made it available for free to anyone. Today, most variations of the Unix operating system provide some form of native RPC support. At the same time, Microsoft Windows also offers official support for the protocol through its Services for UNIX (SFU) software package. Other operating systems offer RPC compatibility with various implementations for programming languages such as C, C++, Java, and .NET.
Despite RPC’s widespread popularity and reputation as a lean and straightforward protocol to implement and use, there may be better choices for heterogeneous network environments. One of the primary issues with RPC has to do with the passing of parameters and the interpretation of data between clients and servers that are written in different programming languages. Although RPC relies on the XDR format to describe payloads, the definition of data types can vary between operating systems and programming languages, resulting in the misinterpretation of information. This has led to the rise of other protocols that provide an abstraction layer for messages and parameters.
The concept of service-oriented architecture, or SOA, emerged as the prevailing standard for facilitating collaboration among applications operating in heterogeneous environments. Around the same period, various internet-based public services were gaining popularity, with the World Wide Web (WWW) particularly attracting the attention of a wider audience outside the academic community.
The web
In 1989, Tim Berners-Lee, an English scientist, created the WWW. Since then, it has become the primary means of accessing information and communicating online. During its early years, the web comprised simple interconnected blocks of information, known as web pages, which could be accessed to view information. These web pages were manually updated by webmasters, who were responsible for maintaining the content. With the rise of commercial web initiatives, various services were developed to allow individuals to upload and share personal information such as photos, blogs, and other forms of multimedia. This led to the creation of desktop applications that enabled users to interact with online services more efficiently. Initially, these applications were used to download information, but eventually, they allowed users to upload content to the web.
The way desktop content-creation applications communicated with the newly launched web services gave rise to what we now refer to as web APIs. One such early example was Flickr, a widely used photo-sharing service that allowed developers to interact with it via a web API. The API enabled developers to perform a range of tasks such as uploading, downloading, listing, and searching photos from a single user account or across the entire service. Business applications such as Salesforce also benefited from what web APIs had to offer. In fact, Salesforce was probably the first modern web API to be launched and used.
On the more closed side of the software industry, other protocols started to emerge, with the goal of simplifying the life of developers and integration designers. One such protocol gained significant popularity because of its natural integration with existing Microsoft tools. It was the Simple Object Access Protocol, or, in short, SOAP. Microsoft promoted SOAP, and it became the number one way to integrate its different products. SOAP became popular outside of the Microsoft world with support from several programming languages and operating systems. For some time, SOAP was seen as the successor of RPC and the way to connect all kinds of business applications. You can see a simplified illustration of a SOAP request here:
Figure 1.1 – A simplified illustration of a SOAP request
Around the same time, another protocol was being developed to utilize as many features of HTTP as possible and better meet the needs of web services. This resulted in the creation of the Representational State Transfer Protocol, commonly known as REST. Its creator was Roy Fielding, who you already know from before in this chapter. In his Ph.D. dissertation, Fielding not only described how remote APIs should operate but also invented REST:
REST is a hybrid style derived from several of the network-based architectural styles (...) and combined with additional constraints that define a uniform connector interface.
The network-based architectural styles that Fielding refers to include cache and client-server, two things that are familiar in web interactions. Compared to SOAP, REST is much easier to understand and process and a natural winner on the open web because it doesn’t need as many rules to operate as SOAP does. You can see a simple illustration of this protocol here:
Figure 1.2 – A simplified illustration of a REST request where data is sent from a client to a server to create (POST), replace (PUT), or modify (PATCH) a resource
However, because it’s more open, it doesn’t offer the level of control and security that SOAP offers. While that might not be an issue with non-critical applications, it is a must for business-related APIs. Because of that, another protocol was created. This time, the goal was to increase control over what was transmitted so that the information could be validated in a reproducible way.
Google’s Remote Procedure Call, or, in short, gRPC, was born in 2015. It started to be used by almost all of Google’s open web APIs. gRPC works on top of HTTP/2, the second version of HTTP, offering, among other things, low latency and the ability to stream data between servers and clients. However, the big advantage of gRPC over other protocols is that it uses a strict interface description language. Protocol Buffers, or Protobuf, is the format used by gRPC to describe the interface and messages shared between clients and servers. Unlike other languages, Protobuf is binary and offers high security and performance. However, Protobuf is oriented toward providing a way to remotely execute code that is available on the API server.
At around the same time, the need for openly querying large amounts of data started to grow. Facebook, a popular social network, pioneered the use of graph databases. It was clear that REST wasn’t the best way to access data that was always changing, and gRPC wasn’t the answer either. So, GraphQL, a data query and manipulation language, was created. Compared to other API architectures, its main difference is that it allows clients to define the shape of the data that they want to query—that, too, at runtime, in a dynamic way. Even though it doesn’t sound like much, this was a big deal because of two factors. On the one hand, it allowed bandwidth-conscious clients—such as mobile phones—to retrieve just the data they needed, saving precious bandwidth. On the other hand, it opened the available data graph to clients, allowing them to openly query all existing data through the API.
