Cisco's Application Centric Infrastructure (ACI) is a big evolutionary step in data center networking, not because it adds programmability to the network--this has been a rising trend over the last few years--but because of the increased compatibility between vendors. This is where the real benefits are. 

We can see the start of this evolutionary step with Cisco's FlexPod (an amalgam of Cisco UCS, VMWare hypervisors, and NetApp storage). Here we see properly validated designs that span more than one vendor. This in itself was a big step; after all, it makes sense for a vendor to try and encourage the end user to purchase their equipment instead of their competitors'. This is done for two reasons: compatibility between devices and the vendor's financial success.

So, what of networks where one vendor can supply all of the equipment, from the networking to the storage and compute elements? It is actually quite rare to find an environment comprising one single vendor in the real world; most networks (and I am including virtualization platforms and storage within this term) have equipment from more than one vendor, because when you are looking for the best performance, you go with the big names (VMWare for virtualization, NetApp for storage, and so on) because they have longevity in the industry and the knowledge and support options that are required. The network becomes heterogeneous, because it needs to be in order to fulfill user, application, and business demands.

The downside to this is that we lose some degree of compatibility. There are industry-standard protocols that provide some level of compatibility back, such as SNMP (Simple Network Management Protocol), Syslog, and LLDP (Link Layer Discovery Protocol), that can facilitate alerting, logging, and communication between devices, but ACI takes this all one step further, taking the heterogeneous data center network and making it, well, homogenous. Through ACI, the data center can be configured rapidly as the application demands, and this includes physical and virtual network elements from multiple vendors. All of this can be performed through one GUI.

Before we dive in, let’s take a few moments to understand what ACI is all about, dispelling some of the myths along the way.

Myth: ACI is too expensive

ACI is not cheap to purchase; it is engineered for the data center, so it commands data center prices. Even the most basic of starter kits has a list price of $250,000. While a quarter of a million dollars is enough to get you started in the world of ACI, it is probably out of reach of most people. Even trying to sell ACI, as a "this could revolutionize our business" proposal, within most companies would be difficult. Despite the fact that most companies do not pay list price, ACI represents a huge risk, and for a number of reasons.

ACI is in its infancy, so adoption will be slow. The companies that have the easily available financial resources to dive into it are, most likely, the same kind of businesses that are not typically early adopters. Established companies that have the cash have more accountability to stakeholders, shareholders, and the public, so they are less likely to rush into investing six-figure sums than the eager startup company, to whom $250,000 represents a massive proportion of their available funds.

Nevertheless, as ACI becomes more prevalent, its adoption rate will increase, despite the cost (which can always be negotiated).

Myth: SDN (and ACI) will replace the engineer

The idea of software-defined networking (SDN) has caused quite a stir in the networking industry as engineers question whether having a programmable network will mean that the developer slowly takes their place. So, we have some degree of fear when it comes to ACI, yet SDN and ACI only represent a small portion of the market. As the infrastructure scales up and out, SDN makes more sense. In smaller deployments, the costs outweigh the benefits, yet SDN (and ACI) will never replace the network engineer. The developer does not speak the language of networks in the same way, that a traditional network engineer does not talk in development code. The two will remain separate entities in their little silos--ACI offers a bridge between the two, but both roles remain safe.

So as much as ACI is expensive, data center-specific, and occasionally perceived as a threat to the traditional network engineer, why should you look at it favorably?

This is SDN, the Cisco way

ACI allows the network administrator and application developers to work closer together. Applications change; networks change. Both have life cycles of varying length, and ACI allows these life cycles to coexist with each other and complement each other. Both teams can work together to achieve a common goal.

ACI reduces the complexity of the network with respect to deployment, management, and monitoring, and does this through a common policy framework. Applications can be deployed rapidly, and the administrative overhead on the network is significantly reduced. It is, therefore, application-centric and can facilitate services at layer 4 to 7 to enhance the application life cycle.

Through ACI, we can automate and program the network. We have a singular platform with which to provision the network. We can bring in, with ease, services such as virtualization (VMWare and Hyper-V), firewalls, load balancers, and a whole range of infrastructure that would previously have meant many hours being spent configuring and reconfiguring as the demands of the application changed.

This automation is performed through policies. Policies are centrally configured on APICs (Application Policy Infrastructure Controllers), which are (usually) clustered.

The APIC is where we will start.

ACI is for the data center. It is a fabric (which is just a fancy name for the layout of the components) that can span data centers using OTV or similar overlay technologies, but it is not for the WAN. We can implement a similar level of programmability on our WAN links through APIC-EM (Application Policy Infrastructure Controller Enterprise Module), which uses ISR or ASR series routers along with the APIC-EM virtual machine to control and program them. APIC and APIC-EM are very similar; just the object of their focus is different. APIC-EM is outside the scope of this book, as we will be looking at data center technologies.

The APIC is our frontend. Through this, we can create and manage our policies, manage the fabric, create tenants, and troubleshoot. Most importantly, the APIC is not associated with the data path. If we lose the APIC for any reason, the fabric will continue to forward the traffic.

To give you the technical elevator pitch, ACI uses a number of APIs (application programming interfaces) such as REST (Representational State Transfer) using languages such as JSON (JavaScript Object Notation), and XML (eXtensible Markup Language), as well as the CLI and the GUI to manage the fabric, and other protocols such as OpFlex to supply the policies to the network devices. The first set (those that manage the fabric) are referred to as northbound protocols. Northbound protocols allow lower-level network components talk to higher-level ones. OpFlex (which we will discuss later in this chapter) is a southbound protocol. Southbound protocols (such as OpFlex and OpenFlow, which is another protocol you will hear in relation to SDN) allow the controllers to push policies down to the nodes (the switches).

This is a very brief introduction to the how. Now let's look at the why. What does ACI give us that the traditional network does not?

In a multi-tenant environment, we have defined goals. The primary purpose is that one tenant remains separate from another. We can achieve this in a number of ways.

We could have each of the tenants in their own DMZ (demilitarized zone), with firewall policies to permit or restrict traffic as required. We could use VLANs to provide a logical separation between tenants. This approach has two drawbacks.

It places a greater onus on the firewall to direct traffic, which is fine for northbound traffic (traffic leaving the data center) but is not suitable when the majority of the traffic is east-west bound (traffic between applications within the data center; see Figure 2).

We could use switches to provide layer-3 routing and use access lists to control and restrict traffic; these are well designed for that purpose.

Also, in using VLANs, we are restricted to a maximum of 4,096 potential tenants (due to the 12-bit VLAN ID).

An alternative would be to use VRFs (virtual routing and forwarding). VRFs are self-contained routing tables, isolated from each other unless we instruct the router or switch to share the routes by exporting and importing route targets (RTs). This approach is much better for traffic isolation, but when we need to use shared services, such as an Internet pipe, VRFs can become much harder to keep secure.

One way around this would be to use route leaking. Instead of having a separate VRF for the Internet, this is kept in the global routing table and then leaked to both tenants. This maintains the security of the tenants, and as we are using VRFs instead of VLANs, we have a service that we can offer to more than 4,096 potential customers. However, we also have a much bigger administrative overhead. For each new tenant, we need more manual configuration, which increases our chances of human error.

ACI allows us to mitigate all of these issues.

By default, ACI tenants are completely separated from each other. To get them talking to each other, we need to create contracts, which specify what network resources they can and cannot see. There are no manual steps required to keep them separate from each other, and we can offer Internet access rapidly during the creation of the tenant. We also aren't bound by the 4,096 VLAN limit. Communication is through VXLAN, which raises the ceiling of potential segments (per fabric) to 16 million (by using a 24-bit segment ID).

VXLAN is an overlay mechanism that encapsulates layer-2 frames within layer-4 UDP packets, also known as MAC-in-UDP (Figure 3). Through this, we can achieve layer-2 communication across a layer-3 network. Apart from the fact that through VXLAN, tenants can be placed anywhere in the data center and that the number of endpoints far outnumbers the traditional VLAN approach, the biggest benefit of VXLAN is that we are no longer bound by the Spanning Tree Protocol. With STP, the redundant paths in the network are blocked (until needed). VXLAN, by contrast, uses layer-3 routing, which enables it to use equal-cost multipathing (ECMP) and link aggregation technologies to make use of all the available links, with recovery (in the event of a link failure) in the region of 125 microseconds.

With VXLAN, we have endpoints, referred to as VXLAN Tunnel Endpoints (VTEPs), and these can be physical or virtual switch ports. Head-End Replication (HER) is used to forward broadcast, unknown destination address, and multicast traffic, which is referred to (quite amusingly) as BUM traffic.

This 16M limit with VXLAN is more theoretical, however. Truthfully speaking, we have a limit of around 1M entries in terms of MAC addresses, IPv4 addresses, and IPv6 addresses due to the size of the TCAM (ternary content-addressable memory). The TCAM is high-speed memory, used to speed up the reading of routing tables and performing matches against access control lists. The amount of available TCAM became a worry back in 2014 when the BGP routing table first exceeded 512 thousand routes, which was the maximum number supported by many of the Internet routers. The likelihood of having 1M entries within the fabric is also pretty rare, but even at 1M entries, ACI remains scalable in that the spine switches let the leaf switches know about only the routes and endpoints they need to know about. If you are lucky enough to be scaling at this kind of magnitude, however, it would be time to invest in more hardware and split the load onto separate fabrics. Still, a data center with thousands of physical hosts is very achievable.

A fabric is a fancy term for how the computing, network, and software components of a data center are laid out. The name itself comes from the crisscrossing of the network, much like an item of weaved clothing. The ACI fabric is relatively simplistic. It employs a two-tier design made of spine and leaf switches, but very particular switches.

In Figure 4, we can see a typical deployment with two spines and three leaves. The Nexus 9500 modular switches are deployed at the top of the topology and act as spines, which in a traditional three-tiered network design would be the aggregation or core switches. Line cards are used to provide the ASICs (application-specific integrated circuitry) required for ACI. There are different line cards available, so make sure you are not purchasing a card that is NX-OS mode only.

The next component is the leaf switches. These are the Nexus 9300-series switches.

The spines connect to the leaves through 40 GE ports, but the spines and leaves are never connected (spine to spine or leaf to leaf).

We can also extend the network, offering greater port density through Nexus 2000-series fabric extenders:

We can also add storage directly to the leaves themselves:

Alternatively, we could use another pair of switches, such as Cisco Nexus 5000-series switches, which would connect to the leaf switches:

The APIC controllers connect to the leaf switches.

The APIC controllers are completely separate from the day-to-day running of ACI. We need them to push new policies. They do not play a part in the functioning of the ACI fabric's data plane, however--only in the control plane.

If we lose one controller, then our tenants’ traffic still flows. If we lose all our controllers, then our tenants’ traffic still flows; we are just unable to push new policies until the controllers are reinstated into the network.

ACI best practice states that we should have a minimum of three controllers. Having three controllers offers high availability; it offers physical redundancy as well as database redundancy. Why not just two controllers, though? Three controllers (well, just an odd number greater than one) work better in a split-brain scenario (one controller disagreeing with another). In such an event, the majority would rule. The controllers use LLDP to find each other, which is part of the process discussed in the ACI fabric overlay section later on in this chapter. We will look at how to use multiple controllers in the troubleshooting section, as the majority of this book uses a much simpler design with just one controller, one spine, and two leaf switches, as seen when we look at the fabric menu later on in this chapter.

One of the most attractive reasons to deploy ACI is the ease of integration with other Cisco products (such as the ASAv firewall) and third-party systems.

This integration is performed through OpFlex. OpFlex is an open standards-based southbound protocol, designed to facilitate multi-vendor integration in both data center and cloud networks. OpFlex is important as it distinguishes ACI from other SDN models, which have integration but do not support the full feature set. The easiest way to try and explain this would be to look at it in the context of SNMP.

SNMP (Simple Network Management Protocol) allows monitoring network hardware, and all devices support the most basic MIB (Management Information Base) of iso.org.dod.internet.mgmt, so at the most basic level, you can pull out data such as interfaces and IP addresses. We are getting data but at the lowest common denominator. We need extra information, by way of specific MIBs, to be able to monitor our firewall’s VPN tunnels or the nodes in our load balancers. OpFlex gives us all the information, but the data is not bound to any particular format. It is a declarative model, which benefits any interested party. This declarative model is based on promise theory.

Promise theory, developed by Mark Burgess in the 1990s, sets ACI apart from other SDN implementations. They use imperative control, in which we have a controlling system, and the system being controlled is relieved of the burden of doing the thinking. While this does offer more autonomy to the controller, it can also create a bottleneck within the system. ACI, however, uses a declarative model. This model states what should happen but not how it should be done (leaving that up to the node being controlled). The node then makes a promise to achieve the desired state and, importantly, communicates back to the controller the success or failure of the task, along with the reason why. The controller is no longer a bottleneck in the system, and the commands are simpler; instead of separate commands to implement the same function on different vendor equipment, we have one command set understandable by both vendors' equipment. This is the benefit of open standards.

Even with open standards, though, there can be some ulterior motive. It is all well and good having the next best thing for integrating different technologies, but when this is designed for those technologies to run under one particular company's product, there can be some hesitation. However, there is a large backing from several renowned companies, such as Microsoft, IBM, Citrix, Red Hat, F5, SunGard Availability Services, and Canonical. So why has OpFlex gathered such a wide backing?

With the traditional SDN model, there is a bottleneck: the SDN controller. As we scale out, there is an impact on both performance and resilience. We also lose simplicity and agility; we still need to make sure that all the components are monitored and safeguarded, which invariably means bolting on more technology to achieve this.

OpFlex takes a different approach. A common language ensures that we do not need to add any extras that are not part of the original design. There is still complexity, but it is moved toward the edges of the network, and we maintain resilience, scalability, and simplicity. If we lose all of the controllers, then the network continues to operate--we may not be able to make policy changes until we restore the controllers, but the tenant’s data still flows, uninterrupted.

The protocol itself uses XML or JSON as the transmission medium. It allows us to see each node as a managed object (MO). Each MO consists of the following:

• Properties
• Child relations
• Parent relations
• MO relations
• Statistics
• Faults
• Health

While the ins and outs of these are beyond the scope of this book, you can read about them more in the IETF drafts. The first one in 2014 (https://tools.ietf.org/html/draft-smith-opflex-00) listed all these seven items, but subsequent drafts--the most recent being October 27, 2016 (https://tools.ietf.org/html/draft-smith-opflex-03)--compress the last four items into one, labeled observables.

What all this means is that for third-parties, OpFlex means greater integration across SDN platforms. If and when OpFlex does become a truly open standard, different vendors, equipment will be able to speak the same language using a simple JSON file.

To use ACI, we need to make sure that we are running our switches in ACI mode. We can check which version we are running by using the show version command:

BIOS: version 08.06
NXOS: version 6.1(2)I3(3)
BIOS compile time: 12/03/2014
NXOS image file name is: bootflash:///n9000-dk9.6.1.2.I3.3.bin
NXOS compile time: 12/05/2014 10:50:20 [12/05/2014 2:25]

We can tell that we are running an NX-OS mode switch as the image filename begins with n9000. ACI image filenames begin with aci-n9000.

The following instructions are for NX-OS release 6.1(2)l3(3) and later, and ACI image version 11.0(2x) or later. There are slight differences with earlier releases, so it is best to make sure you are on these releases before attempting the switch from NX-OS mode to ACI mode.

Check whether your hardware is supported: look in the release notes for Cisco Nexus 9000 Series ACI-mode switches.

Remove or turn off any unsupported module (poweroff module <module> command). If you do not do this step, the software will use a recovery/retry mechanism before powering down the unsupported module, which can cause delays.

If you have a dual-supervisor system, then confirm that the standby supervisor module is in the ha-standby state using the show module command.

Use it like this: show install all impact epld <epld-image-name>. This will check that the switch does not require any EPLD image upgrade. EPLD stands for electronic programmable logic device, and these enhance hardware functionality or resolve known issues. EPLD upgrades are quite infrequent, but they should not be overlooked.

We have a number of ways of performing the upgrade. We can use SCP to copy the image from the APIC to the switch, upgrade from another SCP server, or copy it directly from a USB port. We will look at all three methods, and are assuming that the Nexus switch has already been introduced into the network and has connectivity.

A word of warning when using USB drives, though: smaller is better. Taking a 1 TB drive loaded with all your favorite Nexus images and expecting it to work will only leave you hunting around for a 2 GB drive that has sat in a drawer gathering dust for a few years. This is due to the level of file system support. Older IOS versions only supported FAT16, which has a file size limit of 2 GB, while newer ones support FAT32 (such as IOS 15.1). Sometimes, it is easier to play it safe and go with FAT16.

Method 1 - Using SCP to copy the ACI image from the APIC

1. Enable SCP on the Nexus switch:

      switch(config)# features scp-server

2. Copy the image from the APIC server to the Nexus switch using the CLI:

      scp –r /firmware/fwrepos/fwrepo/<switch-image-name> 
      admin@switch-ip-address:switch-image

1. Copy the file from the SCP server using the switch's command line:

      Switch# copy scp: bootflash:

You will be prompted for the details of the SCP server and filenames.

We can copy an image from a USB drive to bootflash, using the dir command first so that we can cut and paste the filename in the copy command:

Switch# dir usb1:
(or dir usb2: depending on which USB slot you have plugged the drive into)
Switch# copy usb1:<ACI-image-name> bootflash:

If we have a dual-supervisor system, we have an additional step, which is to copy the ACI image to the standby supervisor module:

Switch(config)# copy bootflash:aci-image bootflast://sup-standby/

The next step is to upgrade the image.

In the following code, we first turn off NX-OS mode. We then make sure that the first change survives a reboot. In the third line, we boot the supervisor modules using the ACI image specified. Lastly, we perform a reload of the switch.

Switch(config)# no boot nxos
Switch(config)# copy running-config startup-config
Switch(config)# boot aci bootflash:aci-image-name
Switch(config)# reload

Once the switch has rebooted with the new image, we can log in.

We log in using the username admin and the password specified during setup. Notice that the fabric discovery process has been started at this point. It may be some minutes before the services start and we are able to access the switch via the console.

User Access Verification
(none) login: admin
*****************************************************************************
    Fabric discovery in progress, show commands are not fully functional
    Logout and Login after discovery to continue to use show commands.
*****************************************************************************
(none)#

If, for any reason, you need to revert to NX-OS mode from ACI mode, then follow these steps:

1. Reload the switch:

      admin@apic1:aci> reload

2. Access the bootloader:

      Ctrl+]
      loader>

3. Boot using the NX-OS image:

      loader> boot nxos-image-name

This can take a little while (usually under half an hour) while the filesystem is reformatted to make subsequent reloads faster.

As you can see, from the previous code, the switch performs a fabric discovery. We will look at this in the next section.

ACI uses inter-fabric messaging (IFM) to communicate between the different nodes. IFM uses TCP packets, which are secured by 1024-bit SSL encryption, and the keys are stored on secure storage. The Cisco Manufacturing Certificate Authority (CMCA) signs the keys.

Issues with IFM can prevent fabric nodes communicating and from joining the fabric. We will cover this in greater depth in the SSL Troubleshooting recipe in Chapter 9, Troubleshooting ACI, but we can look at the output of the checks on a healthy system:

apic1# netstat -ant | grep :12
tcp       0    0 10.0.0.1:12151         0.0.0.0:*         LISTEN
tcp       0    0 10.0.0.1:12215         0.0.0.0:*         LISTEN
tcp       0    0 10.0.0.1:12471         0.0.0.0:*         LISTEN
tcp       0    0 10.0.0.1:12279         0.0.0.0:*         LISTEN    
<truncated>
tcp       0    0 10.0.0.1:12567         10.0.248.29:49187 ESTABLISHED 
tcp       0    0 10.0.0.1:12343         10.0.248.30:45965 ESTABLISHED 
tcp       0    0 10.0.0.1:12343         10.0.248.31:47784 ESTABLISHED 
tcp       0    0 10.0.0.1:12343         10.0.248.29:49942 ESTABLISHED 
tcp       0    0 10.0.0.1:12343         10.0.248.30:42946 ESTABLISHED 
tcp       0    0 10.0.0.1:50820         10.0.248.31:12439 ESTABLISHED 
apic1# openssl s_client -state -connect 10.0.0.1:12151
CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
depth=1 O = Cisco Systems, CN = Cisco Manufacturing CA
verify error:num=19:self signed certificate in certificate chain
verify return:0
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server certificate request A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client certificate A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL3 alert read:fatal:handshake failure
SSL_connect:failed in SSLv3 read server session ticket A
139682023904936:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1300:SSL alert number 40
139682023904936:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
0 s:/CN=serialNumber=PID:APIC-SERVER-L1 SN:TEP-1-1, CN=TEP-1-1
   i:/O=Cisco Systems/CN=Cisco Manufacturing CA
1 s:/O=Cisco Systems/CN=Cisco Manufacturing CA
   i:/O=Cisco Systems/CN=Cisco Manufacturing CA
---
Server certificate
-----BEGIN CERTIFICATE-----
<runcated>
-----END CERTIFICATE-----
subject=/CN=serialNumber=PID:APIC-SERVER-L1 SN:TEP-1-1, CN=TEP-1-1
issuer=/O=Cisco Systems/CN=Cisco Manufacturing CA
---
No client certificate CA names sent
---
SSL handshake has read 2171 bytes and written 210 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: NONE
SSL-Session:
   Protocol : TLSv1.2
   Cipher   : DHE-RSA-AES256-GCM-SHA384
   Session-ID:
   Session-ID-ctx:
   Master-Key: 419BF5E19D0A02AA0D40BDF380E8E959A4F27371A87EFAD1B
   Key-Arg   : None
   PSK identity: None
   PSK identity hint: None
   SRP username: None
   Compression: 1 (zlib compression)
   Start Time: 1481059783
   Timeout   : 300 (sec)
   Verify return code: 19 (self signed certificate in certificate chain)
---
apic1#

IFM is essential in the success of the discovery process. A fabric node is only considered active when the APIC and the node can exchange heartbeats through IFM. Going forward, though, we still need IFM once we have active nodes, as it is also used by the APIC to push policies to the fabric leaf nodes.

The fabric discovery process has three stages and uses IFM, LLDP (Link Layer Discovery Protocol), DHCP (Dynamic Host Configuration Protocol), and TEPs (tunnel endpoints):

1. Stage 1: A second discovery brings in any spines connected to initial "seed" leaf.
2. Stage 2: The leaf node that is directly connected to APIC is discovered.
3. Stage 3: In this stage, we have the discovery of other leaf nodes and other APICs in the cluster.

The process can be visualized as follows:

The node can transition through a number of different states during the discovery process:

• Unknown: Node discovered but no node ID policy configured
• Undiscovered: Node ID configured but not yet discovered
• Discovering: Node discovered but no IP address assigned
• Unsupported: Node is not a supported model
• Disabled: Node has been decommissioned
• Inactive: No IP connectivity
• Active: Node is active

Using the acidiag fnvread command, you can see the current state. In the following command output, the leaf node is in the unknown state (note that I have removed the final column in the output, which was LastUpdMsg, the value of which was 0):

apic1# acidiag fnvread
ID Pod ID Name     Serial Number   IP Address     Role          State
---------------------------------------------------------------------
 0      0              TEP-1-101      0.0.0.0  unknown        unknown
Total 1 nodes
apic1#

During fabric registration and initialization, a port may transition to an out-of-service state. In this state, the only traffic permitted is DHCP and CDP or LLDP. There can be a number of reasons why we would transition to this state, but these are generally due to human error, such as cabling or LLDP not being enabled; again, these are covered in the Layer-2 troubleshooting recipe in Chapter 9, Troubleshooting ACI.

There are a couple of ways in which we can check the health of our controllers and nodes. We can use the CLI to check LLDP (show lldp neighbors), or we can use the GUI (System | Controllers | Node | Cluster as Seen by Node):

This shows us the APIC, and we can look at our leaf nodes from the Fabric menu. In the code output from acidiag fnvread, we saw a node named TEP-1-101. This is a leaf node, as we can see from the GUI (Fabric | Inventory | Fabric Membership):

We will look at the GUI in the next section.

On accessing the APIC, we are presented with the login page. It can take a few minutes for the system to fully initialize before we can log in.

Once we have successfully logged in, we are shown the System page. We have the main menu at the top, and each menu item has a submenu.

Here, we can see the System menu, with its submenu showing Quickstart, Dashboard, Controllers, Faults, and Config Zones:

The system page shows us the health of the system, separated into the overall system health and nodes and tenants with under 99% health. Faults counts are listed on the right-hand side, by domain, and by type. We can see the controller status in the bottom right-hand corner. 

Moving on to the Controllers submenu we can see the screen from the previous section.

We have one controller (apic1); its IP address is 10.0.0.1, it is in service and available, and the health state is Fully Fit. If we click on any of the column headings, we can sort them by ascending or descending order, which is useful if you have a large number of controllers.

We can see the interfaces present on our controller (apic1) from the Interfaces menu on the left-hand side:

We can keep track of how much storage we have used from the Storage menu option, and again, this is sortable by clicking on the column heading:

You will notice that the screen flickers every few seconds as it refreshes.

Also, in this section, we can see stats on our NTP servers, our fans and power supply units, and the equipment sensors (which in the simulator are all empty). Under Processes, we can see how much memory we are currently using from the Stats option:

Figure 19

We can also check on the CPU usage from the same window:

We can also see current and historical faults and events from the Processes menu.

The last menu option under Controllers is Controller Policies, where we can set policies for exporting diagnostic information. We will look at this in the troubleshooting section.

The final two options are Faults, in which we can see a couple of examples and Config Zones.

We do not have any zones, but we can create one from the drop-down menu. Configuration zones allow us to subdivide our ACI fabric, meaning that we can make configuration changes to one zone at a time, reducing the chances of an error affecting the entire fabric. Note that if you get access to the Cisco Devnet sandbox environment, the Config Zones option is unavailable. Devnet is a great place to learn the developer side of Cisco's products as well as getting access to hands-on labs and virtual and hardware-based sandbox environments. The virtual ones are fairly limited, but available all the time. The physical rack equipment offers the full range of functionality but does get booked up months in advance. You can find more about Devnet by going to the Devnet site: https://developer.cisco.com/site/devnet/home/index.gsp.

The Tenants tab shows us all our tenants. We have three preconfigured (common, infra, and mgmt):

If we select a tenant and go through the options, we can see the application profiles assigned to it, the networking configuration, layer 4-layer 7 service parameters, and all of the policies. We will go through these in greater detail in the next chapter when we set up some tenants.

This is where we will create new tenants.

From the Fabric menu, in the Inventory submenu, we can see our topology:

If we expand the pod out, we can see all of our leaf and spine nodes:

Going through these, we can see our interfaces, routing tables, processes, pools, and rules. One thing to note here is that we have many more routing options with a leaf node than we do with a spine node:

Under the Fabric Membership option, we have a list of our leaf and spine nodes, which shows us the serial numbers, ID, name, model, role, and assigned IP address. It also gives us the certificate information, so we know that SSL is healthy and IFM can function.

The last three options in this menu are for the nodes we may be having issues with, whether they are currently unmanaged, unreachable, or disabled and decommissioned.

The other options in the Fabric submenu are our policies. Here, we can set up callhome policies, monitoring, troubleshooting, spanning tree and VPC policies, whether we want to have CDP turned off or on, and various layer-2 policies. Many of these options have three options; take LLDP for example. We have an option for LLDP-OFF (disabled), LLDP-ON (enabled), and default (Receive State is enabled, and Transmit State is enabled). Similarly, for CDP we have CDP-OFF (disabled), CDP-ON (enabled), and default (where the Admin State is Disabled).

Under the VM Networking menu is where we would start connecting ACI into our third-party vendors. The default options are Microsoft, OpenStack, and VMWare.

L4-L7 Services allows us to further extend our ACI fabric with additional third-party solutions. This is performed through the addition of packages, which we can import from the Packages submenu's Quick Start option. This is something we will look at in a later chapter.

Under the Admin menu is where we configure AAA (Authentication, Authorization, and Accounting). Here, we can set up RBAC and also connect to authentication providers, such as LDAP; for example, Microsoft Active Directory, RADIUS, or TACACS+. We can also set up PKI to use certificate chains.

We can create maintenance schedules, either one-off tasks or regular ones. We can configure our log retention policies, upgrade our firmware, and configure callhome (after setting up the policies in the Fabric menu), SNMP, and Syslog. The Admin menu is also where we would perform configuration rollbacks and the importing of configuration files and exporting of technical support files.

The final menu option is Operations. This is where will can perform most of our troubleshooting, should the need arise. From here, we find endpoints and look at the traffic path, along with any faults along the path. We can perform a traceroute as well to check the data plane. This is something we will look at in Chapter 9, Troubleshooting ACI.

We can also check out usage with the Capacity Dashboard, create Optimizer configuration templates, track our endpoints, and even look at a traffic map.

For now, though, it’s time to start configuring!