The journey of a click through Rails abstraction layers

The primary goal of any web application is to serve web requests, where web implies communicating over the internet and request refers to data that must be processed and acknowledged by a server.

A simple task such as clicking on a link and opening a web page in a browser, which we perform hundreds of times every day, consists of dozens of steps, from resolving the IP address of a target service to displaying the response to the user.

In the modern world, every request passes through multiple intermediate servers (proxies, load balancers, content delivery networks (CDNs), and so on). For this chapter, the following simplified diagram will be enough to visualize the journey of a click in the context of a Rails app.

Figure 1.1 – A simplified diagram of a journey of the click (the Rails version)

The Rails part of this journey starts in a so-called web server – for example, Puma (https://github.com/puma/puma). It takes care of handling connections, transforming HTTP requests into a Ruby-friendly format, calling our Rails application, and sending the result back over the HTTP connection.

Communication models

Web applications can use other communication models, and not only the request-response one. Streaming and asynchronous (for example, WebSocket) models are not rare guests in modern Rails applications, especially after the addition of Hotwire (https://hotwired.dev/) to the stack. However, they usually play a secondary role, and most applications are still designed with the request-response model in mind. That’s why we only consider this model in this book.

Next, we will take a deeper look at the right part of the diagram in Figure 1.1. Getting to know the basics of request processing in Rails will help us to think in abstraction layers when designing our application. But first, we need to explain why layered architecture makes sense to web applications at all.

From web requests to abstraction layers

The life cycle of a web application consists of the bootstrap phase (configuration and initialization) and the serving phase. The bootstrap phase includes loading the application code, and initializing and configuring the framework components – that is, everything we need to do before accepting the first web request – before we enter the serving phase.

In the serving phase, the application acts as an executor, performing many independent units of work – handling web requests. Independent here means that every request is self-contained, and the way we process it (from a code point of view) doesn’t depend on previous or concurrent requests. This means that requests do not share a lot of state. In Ruby terms, when processing a request, we create many disposable objects, whose lifetimes are bound by the request’s lifetime.

How does this affect our application design? Since requests are independent, the serving phase could be seen as a conveyor-belt assembly line – we put request data (raw material) on the belt, pass it through multiple workstations, and get the response box at the end.

A natural reflection of this idea in application design would be the extraction of abstraction layers (workstations) and chaining them together to build a processing line. This process could also be called layering. Just like how assembly lines increase production efficiency in real life, architecture patterns improve software quality. In this book, we will discuss the layered architecture pattern, which is generic enough to fit many applications, especially Ruby on Rails ones.

What are the properties of a good abstraction layer? We will try to find the answer to this question throughout the book using examples; however, we can list some basic properties right away:

• An abstraction should have a single responsibility. However, the responsibilities themselves can be broad but should not overlap (thus, following the separation of concerns principle).
• Layers should be loosely coupled and have no circular or reverse dependencies. If we draw the request processing flow from top to bottom, the inter-layer connectors should never go up, and we should try to minimize the number of connections between layers. A physical assembly line is an example of perfect layering – every workstation (layer) has, at most, one workstation above and, at most, one below.
• Abstractions should not leak their internals. The main idea of extracting an abstraction is to separate an interface from the implementation. Extracting a common interface can be a challenging task by itself, but it always pays off in the long term.
• It should be possible to test abstractions in isolation. This item is usually a result of all the preceding, but it makes sense to pay attention to it explicitly, since thinking about testability can help us to come up with a better interface.

From a developer’s perspective, a good abstraction layer provides a clear interface to solve a common problem and is easy to refactor, debug, and test. A clear interface can be translated as one with the least possible conceptual overhead or just one that is simple.

Designing simple abstractions is a difficult task; that’s why you may hear that introducing abstractions makes working with the code base more complicated. The goal of this book is to teach you how to avoid this pitfall and learn how to design good abstractions.

How many abstraction layers are nice to have? The short answer is, it depends.

Let’s continue our assembly line analogy. The number of workstations grows as the assembly process becomes more sophisticated. We can also split existing stages into multiple new ones to make the process more efficient, and to assemble faster. Similarly, the number of abstraction layers increases with the evolution of a project’s business logic and the code base growth.

In real life, the efficiency metric is speed; in software development, it is also speed – the speed of shipping new features. This metric depends on many factors, many of which are not related to how we write our code. From the code perspective, the main factor is maintainability – how easy it is to add new features and introduce changes to the existing ones (including fixing bugs).

Applying software design patterns and extracting abstraction layers are the two main tools to keep maintainability high. Does it mean the more abstractions we have the more maintainable our code is?

Surely not. No one builds a car assembly line consisting of thousands of workstations by the number of individual nuts and screws, right? So, should we software engineers avoid introducing new abstractions just for the sake of introducing new abstractions? Of course not!

Overengineering is not a made-up problem; it does exist. Adding a new abstraction should be evaluated. We will learn some techniques when we start discussing particular abstraction layers later in this book. Now, let’s move on to Rails and see what the framework offers us out of the box in terms of abstraction layers.

A basic Rails application comes with just three abstractions – controllers, models, and views. (You are invited to decide whether they fit our definition of good or not by yourself.) Such a small number allows us to start building things faster and focus on a product, instead of spending time to please the framework (as it would be if had a dozen different layers). This is the Rails way.

In this book, we will learn how to extend the Rails way – how to gradually introduce new abstraction layers without losing the focus on product development. First, we need to learn more about the Rails way itself. Let’s take a look at some of the components that make up this approach with regard to web requests.

Rack

The component responsible for HTTP-to-Ruby (and vice versa) translation is called Rack (https://github.com/rack/rack). More precisely, it’s an interface describing two fundamental abstractions – request and response.

Rack is the contract between a web server (for example, Puma or Unicorn) and a Ruby application. It can be described using the following source:

request_env = { "HTTP_HOST" => "www.example.com", …}
response = application.call(request_env)
status, headers, body_iterator = *response

Let’s examine each line of the preceding code:

• The first one defines an HTTP request represented as a Hash. This Hash is called the request environment and contains HTTP headers and Rack-specific fields (such as rack.input to access the request body). This API and naming convention came from the old days of CGI web servers, which passed request data via environment variables.

Common Gateway Interface

Common Gateway Interface (CGI) is the first attempt to standardize the communication interface between web servers and applications. A CGI-compliant program must read request headers from env variables and the request body from STDIN and write the response to STDOUT. A CGI web server runs a new instance of the program for every request – an unaffordable luxury for today’s Rails applications. The FastCGI (https://fastcgi-archives.github.io/) protocol was developed to resolve this situation.

• The second line calls a Rack-compatible application, which is anything that responds to #call. That’s the only required method.
• The final line describes the structure of the return value. It is an array, consisting of three elements – a status code (integer), HTTP response headers (Hash), and an enumerable body (that is, anything that responds to #each and yields string values). Why is body not just a string? Using enumerables allows us to implement streaming responses, which could help us reduce memory allocation.

The simplest possible Rack application is just a Lambda returning a predefined response tuple. You can run it using the rackup command like this (note that the rackup gem must be installed):

$ rackup -s webrick --builder 'run ->(env) { [200, {}, ["Hello, Rack!"]] }'
[2022-07-25 11:15:44] INFO  WEBrick 1.7.0
[2022-07-25 11:15:44] INFO  WEBrick::HTTPServer#start: pid=85016 port=9292

Try to open a browser at http://localhost:9292 – you will see "Hello, Rack!" on a blank screen.

Rails on Rack

Where is the Rack’s #call method in a Rails application? Look at the config.ru file at the root of your Rails project. It’s a Rack configuration file, which describes how to run a Rack-compatible application (.ru stands for rack-up). You will see something like this:

require_relative "config/environment"
run Rails.application

Rails.application is a singleton instance of the Rails application, its web entry-point.

Now that we know where the Rails part of the click journey begins, let’s try to learn more about it.

The best way to see the amount of work a Rails app does while performing a unit of work is to trace all Ruby method calls during a single request-response cycle. For that, we can use the trace_location gem.

What a gem – trace_location

The trace_location (https://github.com/yhirano55/trace_location) gem is a curious developer’s little helper. Its main purpose is to learn what’s happening behind the scenes of simple APIs provided by libraries and frameworks. You will be surprised how complex the internals of the things you take for granted (say, user.save in Active Record) can be.

Designing simple APIs that solve complex problems shows true mastery of software development. Under the hood, this gem uses Ruby’s TracePoint API (https://rubyapi.org/3.2/o/tracepoint) – a powerful runtime introspection tool.

The fastest way to emulate web request handling is to open a Rails console (rails c) and run the following snippet:

request =
  Rack::MockRequest.env_for('http://localhost:3000')
TraceLocation.trace(format: :log) do
  Rails.application.call(request)
end

Look at the generated log file. Even for a new Rails application, the output would contain thousands of lines – serving a GET request in Rails is not a trivial task.

So, the number of Ruby methods invoked during an HTTP request is huge. What about the number of created Ruby objects? We can measure it using the built-in Ruby tools. In a Rails console, type the following:

was_alloc = GC.stat[:total_allocated_objects]
Rails.application.call(request)
new_alloc = GC.stat[:total_allocated_objects]
puts "Total allocations: #{new_alloc – was_alloc}"

For an action rendering nothing (head :ok), I get about 3,000 objects when running the preceding snippet. We can think of this number as a lower bound for Rails applications.

What do these numbers mean for us? The goal of this book is to demonstrate how we can leverage abstraction layers to keep our code base in a healthy state. At the same time, we shouldn’t forget about potential performance implications. Adding an abstraction layer results in adding more method calls and object allocations, but this overhead is negligible compared to what we already have. In Rails, abstractions do not make code slower (humans do).

Let’s run our tracer again and only include #call methods this time:

TraceLocation.trace(format: :log, methods: [:call]) do
  Rails.application.call(request)
end

This time, we only have a few hundred lines logged:

[Tracing events] C: Call, R: Return
C /usr/local/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/engine.rb:528 [Rails::Engine#call]
  C /usr/local/lib/ruby/gems/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:130 [ActionDispatch::HostAuthorization#call]
    C /usr/local/lib/ruby/gems/3.1.0/gems/rack-2.2.4/lib/rack/sendfile.rb:109 [Rack::Sendfile#call]
      C /usr/local/lib/ruby/gems/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/static.rb:22 [ActionDispatch::Static#call]
         // more lines here
      R /usr/local/lib/ruby/gems/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/static.rb:24 [ActionDispatch::Static#call]
    R /usr/local/lib/ruby/gems/3.1.0/gems/rack-2.2.4/lib/rack/sendfile.rb:140 [Rack::Sendfile#call]
  R /usr/local/lib/ruby/gems/3.1.0/gems/actionpack-7.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:131 [ActionDispatch::HostAuthorization#call]
R /usr/local/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/engine.rb:531 [Rails::Engine#call]

Each method is put twice in the log – first, when we enter it, and the second time when we return from it. Note that the #call methods are nested into each other; this is another important feature of Rack in action — middleware.

Pattern – middleware

Middleware is a component that wraps a core unit (function) execution and can inspect and modify input and output data without changing its interface. Middleware is usually chained, so each one invokes the next one, and only the last one in the chain executes the core logic. The chaining aims to keep middleware small and single-purpose. A typical use case for middleware is adding logging, instrumentation, or authentication (which short-circuits the chain execution). The pattern is popular in the Ruby community, and aside from Rack, it is used by Sidekiq, Faraday, AnyCable, and so on. In the non-Ruby world, the most popular example would be Express.js.

The following diagram shows how a middleware stack wraps the core functionality by intercepting inputs and enhancing outputs:

Figure 1.2 – The middleware pattern diagram

Rack is a modular framework, which allows you to extend basic request-handling functionality by injecting middleware. Middleware intercepts HTTP requests to perform some additional, usually utilitarian, logic – enhancing a Rack env object, adding additional response headers (for example, X-Runtime or CORS-related), logging the request execution, performing security checks, and so on.

Rails includes more than 20 middlewares by default. You can see the middleware stack by running the bin/rails middleware command:

$ bin/rails middleware
use ActionDispatch::HostAuthorization
use Rack::Sendfile
use ActionDispatch::Static
use ActionDispatch::Executor
use ActionDispatch::ServerTiming
use Rack::Runtime
... more ...
use Rack::Head
use Rack::ConditionalGet
use Rack::ETag
use Rack::TempfileReaper
run MyProject::Application.routes

Rails gives you full control of the middleware chain – you can add, remove, or rearrange middleware. The middleware stack can be called the HTTP pre-/post-processing layer. It should treat the application as a black box and know nothing about its business logic. A Rack middleware stack should not enhance the application web interface but act as a mediator between the outer world and the Rails application.

Rails routing

At the end of the preceding middleware list, there is a run command with Application.routes passed. The routes object (an instance of the ActionDispatch::Routing::RouteSet class) is a Rack application that uses the routes.rb file to match the request to a particular resolver – a controller-action pair or yet another Rack application.

Here is an example of the routing config with both controllers and applications:

Rails.application.routes.draw do
  # Define a resource backed by PostsController
  resources :posts, only: %i[show]
  # redirect() generates a Redirect Rack app instance
  get "docs/:article",
      to: redirect("/wiki/%{article}")
  # You can pass a lambda, too
  get "/_health", to: -> _env {
    [200, { "content-type" => "text/html" }, ["I'm alive"]]
  }
  # Proxy all matching requests to a Rack app
  mount ActionCable.server, at: "/cable"
end

This is the routing layer of a Rails application. All the preceding Rack resolvers can be implemented as Rack middleware; why did we put them into the routing layer? Redirects are a part of the application functionality, as well as WebSockets (Action Cable).

However, the health check endpoint can be seen as a property of a Rack app, and if it doesn’t use the application’s internal state to generate a response (as in our example), it can be moved to the middleware layer.

Similar to choosing between Rack and routes, we can have a routes versus controllers debate. With regards to the preceding example, we can ask, why not use controllers for redirects?

C for controller

Controllers comprise the next layer through which a web request passes. This is the first abstraction layer on our way. A controller is a concept that generalizes and standardizes the way we process inbound requests. In theory, controllers can be used not only for HTTP requests but also for any kind of request (since the controller is just a code abstraction).

In practice, however, that’s very unlikely – implementation-wise, controllers are highly coupled with HTTP/Rack. There is even an API to turn a controller’s action into a Rack app:

Rails.application.routes.draw do
  # The same as: resources :posts, only: %i[index]
  get "/posts", to: PostsController.action(:index)
end

MVC

Model–view–controller is one of the oldest architectural patterns, which was developed in the 1970s for GUI applications development. The pattern implies that a system consists of three components – Model, View, and Controller. Controller handles user actions and operates on Model; Model, in turn, updates View, which results in a UI update for the user. Although Rails is usually called an MVC framework, its data flow differs from the original one – Controller is responsible for updating View, and View can easily access and even modify Model.

The controller layer’s responsibility is to translate web requests into business actions or operations and trigger UI updates. This is an example of single responsibility, which consists of many smaller responsibilities – an actor (a user or an API client) authentication and authorization, requesting parameters validation, and so on. The same is true for every inbound abstraction layer, such as Action Cable channels or Action Mailbox mailboxes.

Coming back to the routing example and the redirects question, we can now answer it – since there is no business action behind the redirection logic, putting it into a controller is an abstraction misuse.

We will talk about controllers in detail in the following chapters.

Now, we have an idea of a click’s journey through a Rails application. However, not everything in Rails happens within a request-response cycle; our click has likely triggered some actions to be executed in the background.