One of the simplest ways to have faster builds is sometimes to upgrade your compiler. For instance, by upgrading Clang to 7.0.0, you could shave up to 30% off of build times using Precompiled Header (PCH) files. Since Clang 9, it has gained the -ftime-trace option, which can provide you with information on the compilation times of all the files it processes. Other compilers have similar switches, too: check out GCC's -ftime-report or MSVC's /Bt and /d2cgsummary. Often you can get faster compiles by switching the compiler, which is especially useful on your development machine; for example, Clang usually compiles code faster than GCC.
Once you have a fast compiler, let's take a look at what it needs to compile.