Active Directory Disaster Recovery: Expert guidance on planning and implementing Active Directory disaster recovery plans with this book and eBook

Disaster Recovery (DR) is, or should be part of your Business Continuity plan. It is defined as the way of recovering from a disturbance to, or a destructive incident in, your daily operations. In the context of Information Systems and Technology, this means that if an incident completely destroys data, slows down productivity, or causes any other major interruptions of your operations or your business, the process of reverting to normal operations with minimum outage from that incident is called Business Continuity. Disaster Recovery is, or should be, a part of that process.

You could say that Business Continuity and Disaster Recovery go hand in hand, but they do vary depending on the area and subject. For example, if your WAN connection goes offline, it means that your business units can no longer communicate via email or share documents with each other, although each local unit can still operate and continue to work. This scenario would definitely be outlined in your Business Continuity Plan. However, if your server room burns down in one location, the rebuilding of the server room and the data housed in it would be Disaster Recovery.

The problem with Disaster Recovery is that the approach varies for different domains and applications. Also, the urgency and criticality vary across areas and subjects. A lot of companies have a very superficial Business Continuity plan, if they have any plan at all, and have Disaster Recovery plans that are just as superficial. A visual outline of a sample Business Continuity plan is shown below:

As you can see, DR is only a part of the greater picture. It is, however, one of the most crucial parts that many IT departments forget, or decide to overlook. Some even seem to think that DR is not an important step at all.

A lot of people may ask themselves: "Why would we need a 'guide' for Disaster Recovery? If a Domain Controller (DC) has a critical failure, we just install another one". This might seem to work at first, and even for a longer period in small organizations, but in the long run, there would be problems, and a lot of error messages. Correct recovery is crucial to ensure a stable AD environment. The speed at which problems appear, grows exponentially if there are multiple locations of various sizes across different time zones and countries. For example, let's say a company called Nail Corporation (www.nailcorp.com) has its headquarters in Los Angeles, California, and branch offices with several hundred employees in Munich, and Germany, in addition to branch offices in Brazil and India.

NailCorp has one big AD domain and a data center in Brazil having a 512 kilobit link to the headquarters. Let's suppose that the data center in Brazil is partially destroyed due to an earthquake. Network connectivity is restored fairly quickly, but both DCs are physically broken and have therefore become non-functional. The company has around 10,000 employees and, according to Microsoft's AD Sizer software, the space requirement for each Global Catalog server is about 5GB.

As you have to start the rebuild process from scratch, and you have no other DC at the site, you have to replicate 5GB over a 512 kilobit link. Assuming that you get maximum connectivity speed, and no other traffic is flowing at the same time, which is nearly impossible because your users will inadvertently boot their machines and want to start working, you would need over a day to replicate the database. This will increase your restoration time even further-in this case, by at least a day.

In the event of a disastrous event for a company such as NailCorp, you would want to replicate and rebuild as fast as possible. During that time, since you have machines authenticating against the other domain controllers in your company—assuming your DNS service is globally configured to support failover—your replication will be much slower. In this case, you should have different plans in place than just installing another DC.

Another good example is an application that authenticates against a specific DC, or pulls specific information from one. If that DC breaks, the DC will have to be rebuilt with the same name. If you do not do this the right way, you may see strange things happening This is not very far fetched especially in, for example, a software development company.

The need for Disaster Recovery is ever-increasing, and there are several books that touch upon the subject. But none of them are dedicated to different scenarios, and certainly none of them explain the entire process.

Recovering AD from any kind of disaster is trickier then most people think. If you do not understand the processes associated with recovery, you can damage more than you fix.

In order to prevent any kind of major interruptions, and to speed up recovery in the event of an disaster, there are several things that can be done.

For example, AD relies extremely heavily on DNSes. So you need to make sure that if you use AD Integrated (ADI) DNS zones, you should have a standard backup DNS server that has a complete copy of your zones in a non-integrated form. This DNS server should be on an isolated network, and should contain only the records and zones relating to AD, and not all existing dynamic updates.

You should also have a Delayed Replication Site (DRS), also called a lag site . This is a standard part of your AD domain. This should have one or two DCs, maybe a DNS server, and even a standby Exchange server in case one is needed. However, the AD replication is set up with a high link cost in order to prevent replication for a longer time period. Or, you can make it a completely isolated site with a firewall and force a replicate once every one to three months only. This will allow you to have a stable infrastructure. This state may be three months old, but if anything happens you can have a running AD within a few hours, instead of days.

Virtualization can be a boon, especially in this case. Buying a server is fairly cheap nowadays, and as for a DRS, you only need a lot of memory in the machine. VMWare server (http://vmware.com/products/server/) and Microsoft Virtual Server (http://www.microsoft.com/windowsserversystem/virtualserver/) can be downloaded and used for free nowadays. Both of these systems allow the DRS to be run in a virtualized, isolated environment.

Having a DRS can reduce restore time tremendously because, even if there is a global failure, the old DCs can be removed and new ones installed to replicate the DRS.

To avoid repetition, acronyms have been used wherever possible in this book. The following is a list of acronyms, with their respective explanations, used in this book:

We have established that DR is an important part of a Business Continuity plan. But now, we can go further and say that, DR for AD is only a part of a Disaster Recovery plan, and not the whole plan by itself.

You are correct if you think that you should have different DR guides for different things. While writing good DR documentation, it is important to take the standpoint that the person who performs the recovery has little or no knowledge of the system. If you roll out your own hardened and customized version of Windows 2003, some things might differ during the installation and someone who has no clear guide will install a system that differs from your actual DC install guidelines. This can cause incompatibility or result in an improperly-functioning system, later on. This happens say, when you have specific policies that are applied to DCs, and during an install process, the selection of policies is called in a manner different from the dictats of the DC policy.

You might think that this situation will never arise, but hurricane Katrina in the U.S., and the tsunami that struck Thailand, India, and others, proves that it can. Situations may arise when a knowledgeable person is not around at the time of crisis, so the guide needs to be as clear as possible. It may also be possible that the person doing the actual recovery is an external IT consultant or junior IT staff member because the senior and trained staff are not available. In this case, the person handling the recovery may not at familiar with your environment all be.

AD is a great system, but it is also very complex. Performing correct DR is therefore crucial. If AD forms a part of, or is the backbone of, your network and IT infrastructure, a proper guide to bringing it back online in the event of an incident needs to be as clear and concise as possible.

The Business Continuity plan, and the DR guides, especially the AD DR guides, should be practiced and tested at regular intervals. This effectively means that once a year or so, you need to test that your guides are working and that they will actually bring your business back online. In order to test all kinds of scenarios, building a test environment—preferably virtualized because it gives you much more flexibility such as rollbacks and snapshots—is a necessity.

It may be difficult to convince the top management that your systems could actually fail, but replicating your systems, or even just a crucial portion of your server infrastructure, and testing that would definitely be acceptable to them.

Since this book is meant as a reference, and we discuss different scenarios here, an overview of these scenarios is necessary. The following types of disasters or incidents are covered in this book. Illustrations and flowcharts are provided to visualize the disasters more easily, wherever necessary.

Recovery of Deleted Objects

The most common scenario (more common than a single DC hardware failure) is the accidental deletion of objects, computer accounts, users or Organizational Units (OU) within the AD. This is a possible scenario where no proper change management controls are in place, or where testing is not done properly. The restore can take some time, even if the backup tapes are immediately at hand, because the object relationship in AD is quite complex, and simply restoring the deleted objects will not work.

The real fun starts when you have a "safe" replication schedule due to various time zones and other reasons, such as office locations and line speeds. There are, and have been, scenarios where the deletion or modification of a critical service account, such as the Exchange service group, gets replicated in the course of 12 hours to all locations within the organization. The service that uses the account then stops working, and as it is probably a mission-critical service, gets noticed, fixed, and force-replicated to the closest DC. If things proceed smoothly, all locations will have their service restored, one after another, to the point where one of the last locations starts replicating forward in the chain to the first DC again, before it gets the restored information applied. Then, a vicious circle forms, as shown in the following diagram, giving way to some interesting possibilities. One possibility is that the service in different locations goes from working to non-working and back within a few hours, or returns to step one while the account remains deleted. This addresses the need for proper restoration of lost objects, and the proper process of forced replication.

Complete Site Hardware Failure

This scenario, describing an AD site and not necessarily a single physical site, is already quite drastic as it describes a total loss of AD service due to a complete hardware failure at a specific site. A site is a branch in your organization that is connected to your domain forest via a LAN or a WAN connection. This could also mean that a site includes two or more buildings, possibly distributed across an entire city. This scenario assumes that you have at least one other DC in your organization at another location that is unaffected. This scenario can be caused by anything that affects the whole server room, and is most likely to be physical. Fire and water, as well as storms or explosions, are very high on the probability list.

In this scenario, it is most likely that you have other servers that are also affected. This scenario will address the issue of how to get a complete site back up and running as quickly as possible. This is a critical scenario that needs to be fixed as soon as possible. You can, of course, re-route your users to another site for authentication if your WAN link gets backed up quickly, but if the links are not very fast, this can cause extreme slowness and precipitate incidents such as timeouts, and domain controller not found messages to the clients.

This is even worse if you have mission-critical systems authenticating against the AD as illustrated in the following diagram:

This chapter provided you with an overview of AD DR and how a DR guide fits into your organization's Business Continuity plan. It also provided you with a brief overview of the scenarios that will be addressed by this book. As you will have noticed by now, the subject of DR with an AD is much deeper and more crucial than you might think.

Microsoft's focus on pointing out the de-centralized architecture of AD is true, and it really does work for small incidents, for connection error, and so on. The picture is very different, however, when we look at how complex and devastating AD failure can be in some situations, and how crucial it is to have a proper recovery guide in place.

It is important to understand that the risks and threats discussed here exist in very real forms. In today's multi-platform environments and heterogeneous networks, there may be services and systems that authenticate against AD which probably didn't figure in your initial designs, but had to be added to the actual schema.

All these things make your infrastructure all the more mission-critical. If you want to use an analogy would you rather have insurance for your house that covers all your valuable items but not the house or one that covers the house as well?