One of the most popular questions on the Software Engineering Stack Exchange (https://softwareengineering.stackexchange.com/) website is this:

Whilst this should never be regarded as normal, for many developers, it is their reality. So, why do so many projects end up in an unmaintainable state? After all, every project starts off with a blank slate.

Some may say that it's because most programmers are inherently lazy, but most also take pride in their work, and value quality over speed. Others may say it's because the developers are incompetent, but even companies that employ very talented technical teams fall victim to this.

My theory is that during the lengthy development process, it's too easy to make little concessions along the way, where code quality is sacrificed to save other resources, usually time. For instance, you may stop writing tests to meet a deadline, or forgo refactoring because your manager assures you that the project is just a PoC or Minimum Viable Product (MVP). Little by little, these small concessions build up. Oftentimes, the deadlines become ever more unreasonable, and the MVP becomes the company's flagship product. That's how we end up with so many unmaintainable projects in this world.

These compromises, although small at the time, have a knock-on effect on the code that is written afterward. This cumulative effect is described using the metaphor of technical debt, which plays on the analogy of financial debt, where you incur compound interest on your existing debts.

Technical debt is a metaphor created by Ward Cunningham, an American computer programmer:

For example, if you want to start your own business, but do not have enough personal savings, you may opt to take out a loan with a bank. In this case, you incur a small debt now in order to acquire a potentially larger reward later, when your business generates a profit.

Likewise, you may decide to incur some technical debt in order to capture the First-Mover Advantage (FMA) to ship a feature before your competitors go to market. The debt comes in the form of poorly-written code; for instance, you may write everything into a single file (colloquially called a kitchen sink) with no modularization or tests.

In both cases, the debt is incurred with the expectation that it will be repaid, with interest, at a later date.

For development, repayment comes in the form of refactoring. This is where time is re-invested to revise the poorly-written code back to an acceptable standard. As this requires time and manpower, by incurring the technical debt, you are, in essence, trading a moderate increase in development speed now for a significant decrease later.

The problem arises when the debt is not repaid sufficiently quickly. At some point, the amount of maintenance done on the project is so great that no more features can be added, and the business may opt for a complete rewrite instead.

Before we discuss how to tackle technical debt, let's first examine some of its most common causes:

• Lack of talent: Inexperienced developers may not follow best practices and write unclean code.
• Lack of time: Setting unreasonable deadlines, or adding new features without allotting additional time, means developers do not have enough time to follow proper processes of writing tests, conducting code reviews, and so on.
• Lack of morale: We should not overlook the human aspect of development. If requirements change all the time, or developers are required to work overtime, then they're not likely to produce good work.

All of these causes can easily be mitigated. The problem of inexperienced developers can be tackled through mentoring, code reviews, and general training. The problem of morale can be tempered by providing better working environments. The issue of lack of time can be remedied by reducing the scope of the project to something more achievable; this may mean pushing non-essential features to a subsequent phase. Besides this, the business can employ more staff and/or outsource the development of well-defined modules to external contractors.

The real problem lies in the reluctance to tackle technical debt, since the biggest cause of technical debt is the existing technical debt. Any new code that depends on the bad code will very soon become part of the technical debt and incur further debt down the line.

When you talk with product managers or business owners, most of them understand the concept of technical debt; however, most managers or business owners I've encountered also tend to overestimate the short-term returns and underestimate the long-term consequences. They believe that technical debt works like personal loans issued by banks, with an interest rate of around 3% Annual Percentage Rate (APR); in reality, it works more like payday loans that charge you 1500% APR.

In fact, the debt metaphor isn't completely accurate. This is because, unlike a formalized loan, when you incur technical debt, you don't actually know the interest rate or repayment period beforehand.

The debt may require one week of refactoring time that you can delay indefinitely, or it may cost you a few months' time just a few days down the line. It is very hard to predict and quantify the effect of technical debt.

Furthermore, there's no guarantee that by incurring the debt, the current set of features are actually going to be finished earlier. Often, the consequences of technical debt are close to immediate; therefore, by rushing, it may actually slow you down within the same development cycle. It is very hard to predict and quantify the short-term benefits of incurring technical debt. In that sense, incurring technical debt resembles more of a gamble than a loan.

Next, let's examine the consequences of technical debt. Some are obvious:

• Development speed will slow down
• More manpower (and thus money) and time will need to be spent to implement the same set of features
• More bugs, which consequently means poorer user experience, and more personnel required for customer service

On the other hand, the human cost of technical debt is often overlooked; so let's spend some time discussing it here.

Most developers want to work on greenfield projects where they can develop new features, rather than to inherit legacy brownfield projects riddled with bugs and technical debt. This will likely reduce the morale of the developers.

In some cases, those working on brownfield projects may even show animosity toward their colleagues who work on greenfield projects. This is because newer frameworks, libraries, and paradigms will eventually replace older ones, making them obsolete. Those working on legacy projects know that the skills they develop will be worthless in a few years' time, making them less competitive on the job market. In comparison, their colleagues are gaining valuable experience on more modern frameworks that will increase their market value. I can't imagine a developer being happy knowing their skills are becoming less and less relevant.

Furthermore, having technical debt would likely ignite disagreement between developers and their managers about the best time to repay the debt. Typically, developers demand immediate repayment, while the (inexperienced) managers would try to push it further down the line.

Overall, having technical debt in the project tends to lower the morale of its developers.

In turn, low morale leads to the following:

• Lower productivity: Unmotivated developers are more likely to work slower, take longer breaks, and be less engaged in the business.
• Lower code quality: Development is a creative process—there is more than one way to implement a feature. Developers with low morale are unlikely to conjure up the willingness to figure out the best approach—they'll simply select for the approach that requires the least effort.
• High Turnover: Unhappy developers are going to be looking for better jobs, leading to a high turnover of staff for the company. This means the time invested to train the developer and integrate him/her into the team is wasted. Furthermore, it may cause other members of staff to lose confidence in the company, creating a snowball effect of people leaving.

Some managers may argue that the business is not responsible for the happiness of its developers—they pay them to produce work and value, not to be happy. Whilst this is true, an experienced project manager should remember that a development team is not a machine—it consists of people, each with their individual ambitions and emotions. Thus, the manager would be wise to consider the human costs of technical debt when making a business decision.

Despite its negative repercussions, incurring technical debt is often inevitable. In those cases, you must ensure that the decision is an informed and conscious one, and remember to repay the debt as soon as possible. So how do we actually pay back the debt? We do this through refactoring—or making our code cleaner without changing the existing behavior.

Whilst there are no formal definitions on what clean means, here are some signs of clean code:

• Well-structured: Code should consist of modules, separated by domains
• Well-documented: For example, include unit tests, inline comments, automatically generated documentation, and README files
• Succinct: Be concise, but not to the point of obfuscation
• Well-formatted and readable: Other developers must be able to review and work on the same code base, so it should be easy to understand and not deviate too far from well-established conventions

As you gain more experience, you'll be able to detect code that deviates from these signs. In programming, we call these deviations code smells. Code smells are weaknesses within the code that violate well-established design principles, paradigms, and patterns. While they are not bugs themselves, they may slow down development and make the code base more prone to errors later.

Therefore, refactoring is simply a process that moves the current code base from having a lot of code smells to one that is cleaner. As we have mentioned before, there is more than one way to achieve the same results, and developers need to be creative and figure out the best solutions to problems that arise.

The important point here is that developers should be given time to refactor; in other words, refactoring should be the core part of a development process, and be included in the time estimates that the developers provide.

Prevention is better than cure. Instead of incurring technical debt, how about avoiding it in the first place? Here, we outline some easy tactics that you can adopt to prevent technical debt.

Most decision makers, especially those without a technical background, greatly underestimate the effects of technical debt. Furthermore, in their view, developers do not understand the business costs of repaying technical debt in terms of manpower, salaries, and time.

That's why it is important for a professional developer to understand the situation from the decision maker's perspective and the constraints that they must work within. One of the most relevant models is the triple constraint model.

The classic project management triangle (also known as triple constraint or the iron triangle) coined the popular saying Time, Quality, Cost. Pick two. The triangle is shown as follows:

The triple constraint is a model used in project management to visualize the constraints on any projects, and to consider how optimizing the project for one area would cause another area to suffer:

• Time and Quality: You can design and build a high-quality platform in a short time, but you'll need to hire a lot of experienced developers, which will be expensive.
• Time and Cost: You can build a platform quickly with a few inexperienced developers, but the quality will be low.
• Quality and Cost: You can tell a few inexperienced developers to design and plan a platform properly. It'll be of good quality, but it's going to take a long time because they'll need time to learn the principles and apply them.

Most businesses are limited largely by their time and cost: by time, because for each day the product is not launched, the greater the chance their competitor delivers a similar product and captures the first-mover advantage (FMA); by cost, because the company still has to pay their staff salaries while the product is not generating any revenue.

To exacerbate the problem, many managers and business owners are focused more on tangible, immediate results, rather than long-term rewards. For these reasons, when given the choice, most decision-makers pick time and cost over quality.

The fallacy here is that by neglecting quality and incurring debt, they'll eventually be increasing both the time and cost requirements many times over.

Therefore, it is the duty of the developer to inform the product manager and business owner of the unpredictable effects of incurring technical debt to give them all of the advice they need to make an informed decision. You may want to turn the tables and approach it from a positive perspective—cleaning up technical debt would allow future development of new features to be completed more quickly.

Do this to prevent the worst-case scenario where the effort required to fix the code is greater than rewriting everything from scratch.

If the code base is so bad that it's close to FUBAR (a variation on the military slang that stands for 'Fucked Up Beyond Any Repair'), then a more drastic approach may be to refuse further development until refactoring is done. This may seem extreme, given that the people you're disobeying are paying your salary. While this is an easy way to forgo responsibility, it's not what a professional developer should do.

To paraphrase an analogy from The Clean Code by Robert C. Martin: Let's suppose you are a doctor and a patient asks you to perform open heart surgery on him/her in order to relieve a sore throat, what would you do? Of course, you'd refuse! Patients do not know what are best for them, that's why they must rely on your professional opinion.

Likewise, most business owners do not know what is best for them technically, which is why they hired you to make the best possible technical decisions for their business. They pay you not simply to code; they pay you because they want you to bring value to the business. As a professional, you should think about whether your actions are beneficial or detrimental to the business, in both the short and long term.

Business owners also need to trust the advice of their developers. If they do not respect their professional opinion, they shouldn't hire them in the first place.

However, it's not always the business owner's fault for making unreasonable demands; the developer who commits to those demands is equally at fault.

Remember, it is the business owner's, or your manager's, role to get as much out of you as possible. But more importantly, it is your duty to inform them of what is and isn't possible; so, when asked to complete features under a deadline that you cannot meet without sacrificing on quality, do not accept the deadline.

You may think the business would appreciate you for going the extra mile and making the impossible possible, but there are four problems with this line of thinking:

1. You may not actually complete the feature in time, while the business has planned a strategy that depends on that deadline being met.
2. You've demonstrated to the manager that you're willing to accept these deadlines, so they may set even tighter deadlines next time, even if they don't need to.
3. Rushing through code will likely incur technical debt.
4. Your fellow developers may resent you, since they may have to work overtime in order to keep up with your pace; otherwise, their manager may view them as slow. It also means they'll have to develop on top of your rushed code, making everyday work less enjoyable.

There's a time to stick your head out to save a business, but by doing it too often, you are actually hurting the team. The danger is that neither you nor the business owner will realize this; in fact, you may even naïvely celebrate the rapid progress being made.

The solution here is to manage your business owner's expectations. If you believe there's a 50% chance of meeting an optimistic deadline, then ask for the scope to be reduced further until you can be more confident in your estimate. Speaking from experience, business owners would rather hear it's not possible a month in advance than a promise of everything will be done that was not delivered.

This brings me back to the topic of defining and documenting processes. Good code starts with good planning, design, and management, and is maintained by good processes. Many of the problems outlined previously can be mitigated if there are clear guidelines outlining the following issues:

• Situations where incurring technical debt is appropriate, for example, to meet a legal requirement such as GDPR compliance.
• Occasions when developers can expect to receive time to repay these debts, for example, before the next feature is started, or two weeks at the end of each quarter.
• The distribution of work on greenfield/brownfield projects within the team, for example, with a rotation system.
• The Definition of Done – a list of criteria which must be met before a feature is considered "done", for example, code passes all tests and is peer-reviewed, and documentation is updated.

Software development paradigms such as Agile and Waterfall, as well as their implementations such as Scrum and Kanban, provide different ways to define and enforce these processes. For example, in Scrum, development happens in short iterations (typically one and four weeks) called sprints. At the beginning of each sprint, a meeting is held to review pending tasks and select features to be tackled in this sprint. At the end of each sprint, a retrospective meeting is held to review the progress of the sprint and identify lessons that can be learned and applied to subsequent sprints.

Although these paradigms and methodologies are popular in software development, they are not coupled to any technical processes at all. Instead, they deal with the entire development process, including gathering requirements and specifications, communicating with the client, design, development, and deployment.

Therefore, of more relevance to developers are development techniques, which specify how a developer should develop a feature. The most prominent technique is TDD.

Test-Driven Development is a development practice created by Kent Beck, it requires the developer to write tests for a feature before that feature is implemented. This provides some immediate benefits:

• It allows you to validate that your code works as intended.
• It avoids errors in your test suite, if you write your test first, then run it, and it does not fail, that's a prompt for you to check your test again. It might just be that you have inadvertently implemented this feature by chance, but it could also be an error in your test code.
• Since existing features would be covered by existing tests, it allows a test runner to notify you when a previously functional piece of code is broken by the new code (in other words, to detecting regressions). This is especially important for developers when they inherit old code bases they are not familiar with.

So, let's examine the principles of TDD, outline its process, and see how we can incorporate it into our workflow.

TDD consists of a rapid repetition of the following steps:

1. Identify the smallest functional unit of your feature that has not yet been implemented.
2. Identify a test case and write a test for it. You may want to have test cases that cover the happy path, which is the default scenario that produces no errors or exceptions, as well as unhappy paths, including dealing with edge cases.
3. Run the test and see it fail.
4. Write the minimum amount of code to make it pass.
5. Refactor the code.

For example, if we want to build a math utility library, then our first iteration of the TDD cycle may look like this:

1. Pick a feature: For this example, let's pick the sum function, which simply adds numbers together.
2. Define a test case: When running the sum function with 15 and 19 as the arguments, it should return 34:

var assert = require('assert');
var sum = require('sum');
describe('sum', function() {
 it('should return 34 when 15 and 19 are passed in', function() {
   assert.equal(34, sum(15, 19));
 });
});

3. Run the test: It fails because we haven't written the sum function yet.

4. Write the code: Write the sum function that will allow us to pass the test:

const sum = function(x, y) {
  return x + y;
}

5. Refactor: No refactoring needed.

This completes one cycle of the TDD process. In the next cycle, we will work on the same function, but define additional test cases:

1. Pick a feature: we'll continue developing the same sum function. 
2. Define a test case: this time, we will test it by supplying three arguments, 56, 32 and 17, we expect to receive the result 105:

describe('sum', function() {
 ...
 it('should return 105 when 56, 32 and 17 are passed in', function() {
   assert.equal(105, sum(56, 32, 17));
 });
});

3. Run the test: it fails because our current sum function only takes into account the first two parameters. 

4. Write the code: update the sum function to take into account the first three parameters:

const sum = function(x, y, z) {
  return x + y + z;
}

5. Refactor: improve the function by making it work for any number of function parameters:

const sum = function(...args) => [...args].reduce((x, y) => x + y, 0);

Note that calling with just two arguments would still work, and so the original behavior is not altered.

Once a sufficient number of test cases have been completed, we can then move on to the next function, such as multiply.

By following TDD, the number of bugs should reduce drastically; however, no process can guarantee error-free code. There will always be edge cases that were overlooked. Previously, we outlined the TDD process for implementing a new feature; now, let's look at how can we can apply the same process to fixing bugs.

In TDD, when a bug is encountered, it is treated the same way as a new feature—you'd first write a (failing) test to reproduce the bug, and then update the code until the test passes. Having the bug documented as a test case ensures the bug stays fixed in the future, preventing regression.

When you first learn to code, no one ever starts with writing tests. This means that for many developers, having tests in the code is an afterthought—a luxury if time permits. But what they don't realize is that everyone tests their code, consciously or otherwise.

After you've written a function, how do you know it works? You may open the browser console and run the function with some dummy test parameters, and if the output matches your expectations, then you may assume it's working. But what you're doing here is actually manually testing a function that has already been implemented.

The advantage of manual testing is that it requires no upfront costs—you just run the function and see if it works. However, the downside is that it cannot be automated, eating up more time in the long run.

Instead, you should formally define these manual tests as code, in the form of unit, integration and end-to-end (E2E) tests, among others.

Formally defining tests has a higher initial cost, but the benefit is that the tests can now be automated. As we will cover in Chapter 5, Writing End-to-End Tests, once a test is defined as code, we can use npm scripts to run it automatically every time the code changes, making the cost to run the tests in the future virtually zero.

The truth is that you'll need to test your code anyways; it's just a choice of whether you invest time to automate it now, saving time in the future, or save the time now but waste more time repeating each test manually in the future.

Mike Cohn developed the concept of the Testing Pyramid, which shows that an application should have a lot of unit tests (as they are fast and cheap to run), fewer integration tests, and even fewer UI tests, which take the most amount of time and are the most expensive to define and run. Needless to say, manual testing should only be done after unit, integration, and UI tests have been thoroughly defined:

Whilst avoiding manual testing is a benefit of TDD, it certainly is not the only one. A developer can still write their unit, integration and E2E tests after implementation of the feature. So what are the benefits of writing tests before implementation?

The answer is that it forces you to think about your requirements and break them down into atomic units. You can then write each test case around a specific requirement. The end result is that the test cases form the specification for your feature. Writing tests first helps you structure your code around the requirements, rather than retrofitting requirements around your code.

This also helps you to abide by the You Aren't Gonna Need It (YAGNI) principle, which prevents you from implementing features that aren't actually needed.

Lastly, writing the tests (and thus the specifications) forces you to think about the interface that consumers of your function would have to use to interact with your function—should everything be defined as properties inside a generic options object, or should it be a plain list of arguments?

// Using a generic options object
User.search(options) {
  return db.users.find(options.name, {
    limit: options.limit,
    skip: options.skip
  })
}

// A list of arguments
User.search(name, limit, skip) {
  return db.users.find(name, {limit, skip});
}

When developers want to use a tool or library, they learn by reading the documentation or guides that contain code samples they can try, or by following tutorials to build a basic application.

Test cases can essentially act as code samples and form part of the documentation. In fact, tests are the most comprehensive set of code samples there are, covering every use case that the application cares about.

Because TDD focuses on a single functional block at a time, its development cycles are usually very short (minutes to hours). This means small, incremental changes can be made and released rapidly.

When TDD is implemented within the framework of a software development methodology such as Scrum, small development cycles allow the methodology practitioner to capture fine-grained metrics on the progress of the team.

While TDD is the gold standard amongst development techniques, there are many obstacles preventing its implementation:

• Inexperienced team: TDD only works when the whole development team adopts it. Many junior developers, especially self-taught developers, never learned to write tests.
  The good news is that TDD is not hard; given a day or so, a developer can realistically learn about the different types of tests, including how to spy on functions and mock data. It's wise to invest time training a developer so that he/she can write more reliable code for the entire duration of his/her employment.
• Slower initial development speed: TDD requires the product owner to create a specification document and for the developers to write the tests before any functional code is written. This means the end product will likely take more time to complete. This goes back to a recurring theme in this chapter: pay the price now, or pay the interest later. If you've been reading everything so far, it'll be obvious the first option is the better one.
• Legacy code: Many legacy code bases do not have tests, or the tests are incomplete; worse still, there may be insufficient documentation to understand what each function is designed to do. We can write tests to verify functionality that we know, but we cannot be certain that it'll cover all cases. This is a tricky one because TDD means you write your tests first; if you already have all the code, then it can't be TDD. If the code base is large, you may continue to fix bugs (documenting them as unit tests as you do so) while starting on a rewrite.
• Slow tests: TDD is only practical when the tests can be run quickly (within a few seconds). If the test suite takes a few minutes to run, then developers would not receive quick enough feedback for those tests to be useful.
  The simplest way to mitigate this issue is by breaking the code into smaller modules and running tests on them individually. However, some tests, such as large integration and UI tests, are inevitably slow. In these cases, you can run them only when the code is committed and pushed, probably by integrating them into a Continuous Integration (CI) system, which is something we will cover in Chapter 8, Writing Unit/Integration Tests.

Although I encourage you to incorporate TDD into your workflow, I should add a disclaimer that it is not a silver bullet. TDD does not magically make your code performant or modular; it's just one technique that forces you to design your system better, making it more testable and maintainable.

Furthermore, TDD induces a high initial cost, so there are a few cases where this investment is not advisable:

• Firstly, when the project is a Proof-of-Concept (PoC). This is where the business and developers are only concerned with whether the idea is possible, not about its implementation. Once the concept is proven to be possible, the business may then agree to approve additional resources for the proper development of this feature.
• Secondly, when the product owner has not defined clear requirements (or does not want to), or the requirements change every day. This is more common than you think, since many early startups are constantly pivoting to find the right market fit. Needless to say, this is a bad situation for the developer, but if you do find yourself in this situation, then writing tests would be a waste of time, as they may become obsolete as soon as the requirements change.

In this chapter, we've looked at technical debt, its causes, consequences, and ways to prevent it. Then, we introduced TDD as a process to avoid technical debt; we outlined its benefits, and how to implement it in your workflow. In Chapter 5, Writing End-to-End Tests and Chapter 6, Storing Data in Elasticsearch, we will cover in more depth the different types of tests (unit, integration, and E2E / acceptance tests).

Good code, whatever its definition, takes less time to write than bad code in the long run. It would be wise to realize this fact and have the discipline to build a strong foundation from the get-go. You can build a house on weak foundations, and it may stand for a hundred years, but build a skyscraper on a weak foundation, it'll come tumbling down quicker than you can imagine.