Describing network architecture and its components

The term network architecture was introduced in the early 2000s, mimicking roles in the construction industry, where architects design and civil engineers build. Different companies use the term differently, but in this book, network architecture will be used to refer to the design of the network and its functions.

For a good network architecture, it is desirable to have a document describing in detail the first three layers of the network, from the physical layer to the routing layer. With this documentation, it is easy for the engineers to understand the physical connections, the Ethernet domains, and the routing protocols used.

Diagrams

A network diagram is mostly like a map, where the cities are the nodes and the roads are the links that connect them. For a network engineer, diagrams are crucial to describe how nodes are connected, and they also can group and demarcate important areas. A good diagram is easy to interpret and follow how data flows.

There are up to three types of diagrams; they can be integrated on the same page and graph, or they can be separated onto different pages. The main diagrams are one to show the physical connections, which can include the technology involved in the data link layer, and the switching and routing diagrams.

In Figure 1.6, we can see an example of a network diagram:

Figure 1.6 – Example of a network diagram

Figure 1.7 shows examples of network diagram symbols:

Figure 1.7 – Network diagram symbols

Network node names

A network node is a device that is essentially used to interconnect and serve as a transport of the data in the network. It can be either a hub, a switch, or a router. To help network engineers identify the node function, names are used to describe their main function. Here are some of them:

• Transit router: These are routers that have interfaces with other service providers. These links are normally used as a service to access other networks, therefore they have a cost because they are normally connected to other big carriers.
• Peer router: These routers have interfaces with other networks in a peer configuration, meaning none of the parts pay to use it. In these links, only the traffic between the peer companies is exchanged, and the traffic destinated to outside networks are not allowed. Accessing external networks would be the case when using transit routers.
• Core router: These are nodes that are in the center of the network. They normally handle a large amount of traffic and have high-speed interfaces. Their throughput capacity is the highest in the network, but they have fewer interfaces as they concentrate the traffic of the network.
• Distribution router: These are nodes that normally connect to the core and aggregation routers. They normally interconnect different locations of the network. They don’t have many interfaces and their throughput capacity is high, but not as high as the core router.
• Aggregation router: These routers normally aggregate the traffic from the access routers. They are normally located in the same area or location as the access routers, and they have fewer interfaces compared to the access routers.
• Access router: Some architects add a node that connects all last-mile networks or CPE nodes. These routers are located closer to the customer and have more interfaces than any other router.
• Top of the rack (TOR): TOR refers to nodes that can be either a switch or a router, depending on the architecture. They are responsible for connecting the servers in the rack to the rest of the network.
• Clos rack: A Clos network, as described before, is a technique to add connectivity to multiple servers using small devices. A Clos rack is seen by the rest of the network as a single unique block, and in terms of architecture, it acts as a single node, normally used as a single router with a large number of interfaces.
• CPE: CPE is the node that is installed at the customer’s location. It normally has one interface connecting to the last-mile network and one local interface that can be an Ethernet or a wireless Ethernet. These devices can also implement NAT, firewall and, in some cases, they have multiple local interfaces, which can act as a switch and a router. These nodes are cheap and small with very low throughput capacity compared to the other nodes.

The last-mile network

This term is used to describe the architecture used to connect the customer to the network. Normally, this term is only used for ISPs, but some corporations also use it to interconnect their branches.

The last-mile network has a range of coverage and normally doesn’t cross the 1 km mark but depends on the type of technology used. Here are some of the most common last-mile networks:

• Cable TV: There are several technologies used here to provide data communication using the cable TV that the customer has installed. The most used one is DOCSIS, which in 2017 was upgraded to version 4. This solution uses a single cable that is shared to several premises.
• Digital subscriber line (DSL): DSL uses the old telephone line to pass data communication. For that, there are lots of standards, and the most common ones are VDSL and ADSL. The DSL solutions don’t share the same media as cable TV does, and there is one cable for each customer.
• Fiber to the premises (FTTP): FTTP is when an optical cable arrives at a customer’s premises. Like cable TV, the most common implementation is a single fiber that crosses several different customers in a sharable manner. The most common technology is a passive optical network (PON) or, more specifically, the Gigabit Ethernet PON (GPON) (or G.984).

Important note

Further details on GPON networks can be found in the paper GPON in Telecommunication Network – November 2010 – Paper from the International Congress on Ultra Modern Telecommunications and Control Systems (ICUMT) conference, 2010.

• Wi-Fi: Normally, this technology is used privately inside a company or a home, but some ISPs use the Wireless Ethernet standards (IEEE 802.11 family) to provide the last mile to customers using omnidirectional antennas. This particular use is different depending on each country and it depends on the government’s legislation. They are normally advertised as Ethernet hotspots (https://en.wikipedia.org/wiki/Hotspot_(Wi-Fi)).
• Satellite: For data communication using satellites, there are two methods: one using geostationary satellites and the other using constellation satellites. The difference between them is the latency, as geostationary orbits very far from earth. The constellation method has low latency but has handover challenges as the satellites keep moving, normally having very low data throughput. The most famous technology using geostationary is VSAT. Internet using VSAT adds around 250 ms every time it has to travel from earth to the satellite, therefore it is a 500 ms round trip. But the dark ages of high latency might be over as SpaceX has announced they have finally solved the handover problem using the constellation method. This new service is called Starlink and has promised to have high capacity, low latency, and high availability using low orbit satellites.

Important note

A good discussion on the Starlink network can be found in the paper Starlink Analysis – July 15, 2021 – Research group ROADMAP-5G at the Carinthia University of Applied Sciences.

• Power line communication (PLC) or HomePlug: PLC, or broadband over power lines (BoPL), uses the power cables to communicate data. This is achieved by modulating high frequencies on the wire. Most transformers won’t be able to pass through the information as they act as a low-frequency cut filter, so it has to be contained within a house or between posts without a transformer. The most common technologies here are the HomePlug AV2 and IEEE 1901-2010 (https://ieeexplore.ieee.org/document/5678772).
• Mobile: Definitely the most popular network is the mobile last mile. Today, they use 5G technology, but other old networks are still in use, such as 4G (LTE), 3G, and GPRS.

Important note

More information on mobile technologies can be found at Evolution of Mobile Communication Technology towards 5G Networks and Challenges by A. Agarwal, K. Agarwal, S. Agarwal, and G. Misra – American Journal of Electrical and Electronic Engineering, 2019, Vol. 7, No. 2, pp. 34-37.

The physical architecture

The physical architecture is sometimes not necessarily the description of the cables or the fibers that will connect the devices but the infrastructure used by the network as a physical layer defined in the TCP/IP stack. This means we can reuse other foreign networks as a physical layer even though they have their own protocol stacks. Here are some of the possible physical technologies used in the architecture:

• Dark fiber: When connecting nodes, the term dark fiber means the nodes that are connected will be using a fiber that does not contain a repeater or underlying infrastructure. In the case of a connection between two nodes using dark fiber, if one node loses power, the other will not receive any light from the fiber. In this scenario, a fiber cut is perceived in both ends immediately, and interfaces go down instantaneously with a fiber cut. Only the packets in the output interface queue are discarded when a failure occurs.
• Synchronous Transport Module (STM): STM was initially created to multiplex digital phone lines, but later started to be used for data communication. The most common one was STM-1, which was 155 Mbps. Routers used to have an interface that could encapsulates STM frames toward an STM network. The STM network would just switch the frames from one end to the other. A cut in the fiber using this technology might not be perceived quickly enough, causing a huge amount of packet loss. As we will describe later, bidirectional forwarding detection (BFD) needs to be used here to avoid drastic problems.
• Dense wavelength-division multiplexing (DWDM): DWDM is an evolution of STM. The DWDM network is a switch network that also has a frame and time and wave division for each of the packets of data carried, similar to STM but enhanced. Similarly, BFD is necessary because a cut in the fiber here would not be perceived quickly enough, causing a huge amount of packet loss.
• Back to back: As explained before, the term back to back is normally used to designate the nodes that are connected directly without any other physical layer in between, such as repeaters.
• Network tunnels: Network tunnels are points of the network that are used to encapsulate the traffic and travel in a different network. Tunnels can be either Layer 2 or Layer 3 and are implemented to abstract the network that is being carried. In some network architectures, they are meant to reach a distant part of the network using a foreign infrastructure.
• VPN tunnels: These are like network tunnels. VPN tunnels normally add encryption.

The routing architecture

It’s important to define how the traffic will flow in the network. For that, we need to have a proper design in terms of routing distribution. This is necessary so failure remediation, redundant paths, load balancing, routing policies, and traffic agreements can be implemented. The architecture would have to include an internal routing protocol and an external routing protocol if connected outside. Here is a summary:

• Interior gateway protocol (IGP): IGP is a routing protocol that runs in a delimited area or location, normally internally within the same organization, as the name says. In the IGP domain, routers exchange path information by announcing and receiving topology updates. The most common IGPs use link state information to build the routing path topology. If an interface goes down, the update has to be propagated to the entire IGP domain. Isolated areas are used to avoid having to update a too-large topology and cause instability. Historically, the popular IGPs were RIP and EIGRP, but today, only Open Shortest Path First (OSPF) and Intermediate System-to-Intermediate System (IS-IS) are used.
• Exterior gateway protocol (EGP): EGP is a routing protocol used to exchange routing information between organizations. It normally does not contain link state information, only the path distance. The most common EGP protocol is Border Gateway Protocol (BGP).
• IS-IS: IS-IS is an IGP protocol designed by ISO, registered as ISO 10589. It is a link state protocol based on the shortest path algorithm called Dijkstra’s algorithm. It’s the second most used IGP.
• OSPF: OSPF is an IGP protocol designed by IETF, registered originally in 1989 by RFC1131 and updated a few times later. Version 3 is the last version described in RFC5340. OSPF also uses Dijkstra’s algorithm to calculate paths and is the most popular and used IGP. OSPF uses areas to scale and improve stability during routing database updates.
• BGP: BGP is a unique protocol used to exchange routing information between organizations. It was first introduced in 1989 in RFC1105. It is also one of the protocols with more updates and extensions on the IETF and can be used for different purposes, such as internal BGP (iBGP), Multiprotocol BGP(MP-BGP) defined in RFC4760, MPLS (MP-BGP), and recently, BGPsec, defined in 2017 in RFC8205. BGP is a path vector-based protocol, also known as a distance vector protocol, and it does not use link information like OSPF.
• Autonomous system number (ASN): Like the IP range, ASN is a unique number that is associated with an organization when starting using BGP to exchange routing tables. It is controlled by the five regional internet registries: ARIN in North America, LACNIC in Latin America, APNIC in Asia-Pacific, RIPE in Europe, and AFRINIC in Africa. When routing tables are exchanged using BGP, the ASN is carried on the path. For instance, Amazon.com uses ASN 16509 (https://whois.arin.net/rest/asn/AS16509).

Let’s explore how a network works in terms of its state.

Types of failure

In computer networks, a major problem is the instability caused by failures in routing tables, links, or nodes. If a node goes numb, for example, the CPU freezes, the other nodes have to detect it quickly so they can divert the traffic through a different path. But how can a failure be detected to reroute quickly enough? Let’s explore the types of failures first:

• Link failure: A link failure is when a connection between two nodes stops receiving or sending data because there is an interruption on the path. The failure can be caused by a physical problem, such as a fiber cut, environmental conditions, such as heavy rain, or because of middleware equipment failure. Nodes normally detect whether a link is down by the lack of signal on the input, but in some cases, such as when using repeaters or underlying networks (such as DWDM), the signal is present on the input but data can’t be delivered. So, it requires a higher-level protocol to monitor and detect the communication breakdown instead of the interface input signal alone; otherwise, data will be discarded continuously until a node decides to reroute the traffic, which can take several seconds in some cases.
• Node failure: A node can fail in several different ways; the most common ones are power loss and OS freeze. A software glitch can cause a router to freeze for minutes or even hours, causing packet loss or not, depending on where the freeze occurs, in either the forwarding plane or the control plane. Detecting this failure quickly is a bit harder because all interface signals are still present, and the forwarding plane might be still working.
• Flapping: Interface flapping is when the interface keeps going down for short periods without being detected. Flapping causes data loss without detection and normally is hard to be discovered without specific equipment to measure the medium connected normally on both ends. The term flapping also is used when a route keeps appearing and disappearing on the routing table, called route flapping.

Failure detection techniques

Here are some techniques to detect failure:

• Signal off: Interfaces have a very simple way of detecting failure, by the absence of the main signal or light. In the case of fiber, if the intensity of the light received is too low, it would consider the interface down. Note that this detection is made on the input interface.
• Protocol keep alive and hello packets: Some routing protocols have keep alive (or hello) messages to check whether their neighbors are still alive. In OSPF, the default period for hello packets is 10 seconds for LAN interfaces, and 30 seconds for P2P connections. BGP has a default of 30 seconds. For today’s network speed, 30 seconds is a lot of data lost. A 10 Gbps interface would discard a total of 37 GB if fully loaded. In today’s protocol implementation, the period of sending these messages can’t be shorter than a few seconds, which is still a long period of data lost.
• Link BFD: In 2010, IETF published RFC5880, which describes the BFD protocol, which was intended to allow routers to detect failure on their interfaces in the order of microseconds. The BFD message supports a minimum of 1 ms interval. BFD is normally implemented on the interface hardware, which allows it to respond without interrupting the main CPU.
• The BFD routing protocol: Link BFD is normally enabled in all interfaces of the network to detect failures quickly, but it would not help in the case of OS router freeze or control plane failure. To avoid packet loss in these cases, all major protocols have the BFD capability, including OSPF, IS-IS, and BGP. Although the BFD protocol message supports microsecond intervals, the implementation using routing protocols is normally in the order of milliseconds and limited to the number of points. The reason is that these messages need to be handled by the main CPU, and too many might cause performance degradation.
• Route flapping detection: The routing protocol can detect persistent route flapping and suppress it for a period. This is useful to avoid recalculating paths when a route is not actually stable. When suppression is in place, normally, the default route is taken.

Control plane and forwarding plane

It is very important to understand the difference between a forwarding plane and a control plane, especially if you are working on network automation. Let’s explore them in this section.

The forwarding plane, or data plane, is an abstract concept where some processes, equipment, and hardware are used to forward traffic through the network. In other words, the forwarding plane defines all entities in the network responsible for receiving data, transporting it, and delivering it.

The control plane is an abstract concept designated to all entities in the network responsible for constructing the data path, removing it, or updating it.

A forwarding plane works when data is carried from one input point, A, to another output point, B, but does not need to have a control plane working. The control plane would only work if a path does not exist from A to B. The control plane also works in case of a failure because the original path might be interrupted and needs to be constructed again.

So, why is this important in network automation? Because the control plane has to update forwarding paths if there is a problem with the forwarding plane, which can cause packet drop, jitter, and delays. A stable network does not require any path updates and consequently minimum work for the control plane. Network automation needs to avoid any particular automation that might cause the control plane to update the network.

Graceful restart

Usually, when a router restarts, all the routing peers detect that the session went down and then came up. This down/up transition results in the control plane working to recompute all the route paths, generating thousands of updates in the entire network and, consequently, causing a churn to the forwarding plane. This recomputation can also cause routing flaps, which may create transient forwarding black holes and transient forwarding loops. These transient problems also consume a lot of resources on the control plane of the routers affected.

Therefore, a graceful restart was created to avoid such drastic changes if a restart is required.

The idea is we could restart all control plane processes in one router without affecting the forwarding plane and the control plane of the other neighbor routers. In practice, a graceful restart is a method to restart the routing processes without affecting the forwarding plane.

In 2003, IETF published RFC3623 to define the implementation of the graceful restart for OSPF. Today, the main control plane protocols have some sort of graceful restart, including BGP, IS-IS, MPLS, RSVP, and LDP.

When building network automation, this kind of method is preferred to update the software.

In this section, we’ve reviewed network architecture and its components. We got more details on routing and physical architecture components. We also learned how important control and data plane separation is, along with the failure types. It is important to know these network terminologies to help with network automation. Next, we’re going to review network management and its components.