Optimizing compile time of a large C++ project

The codebase of our PointLine CAD is certainly quite large. sloccount calculated roughly  770’000 lines of C++ code. I know, this is not a very good metric to describe a project, but it gives an idea. Over time the compile time steadily increased. Of course we also added a lot of new stuff to the product. We also used advanced techniques to reduce the risk of bugs, that have to be paid with compile time. But still, the increase was disproportionate. We mitigated it by using IncrediBuild. Just like distcc, it distributes the compilation load across different machines on the LAN. If I’m lucky, I get about 20 cores compiling for me.

About once a year, one of us does some compile time optimization and tunes the precompiled headers. I did so about three years ago, and then this week it was my turn again. Reading what I could find about precompiled headers on the internet and applying that, I could get only a small speedup, roughly 10%. So I cleaned up the physical structure of the codebase. Here are some things I performed:

• Read this and this and this and this
• Removed the “include everything” header from all the places where it was not really needed, especially from the precompiled headers.
• Removed the includes of external libraries that are used only in a small fraction of the code from the precompiled headers.
• There are some template container classes that derive from the MFC containers to improve type safety. At the bottom of the file there are some typedefs for the most used instantiations. Now by the design of the classes, this implies that the value types have to be known, and thus be included in the file. So I refactored the typedefs into separate files to decrease the coupling. Don’t get me wrong here. Less files are generally better for compilation times, but this is to reduce the coupling. If only one of them is required in a frequently used interface class, all the others get dragged along.
• One frequently used class has a member function that is templated on a range type, so it can operate on any container that satisfies some concepts. The implementation had dependencies to stuff that was otherwise not so frequently used. So I separated this template function into a free function to reside in it’s own header file.
• Generally I totally prefer members by-value over by-pointer. But in frequently used classes that could almost be called interface classes, this brings all the dependencies with it. So in this case it is better to look after the pointers and only forward declare the classes in the header. That way, you can get rid of a lot of dependencies. This points in the direction of the PIMPL idiom, but that would be farther along that path. And I didn’t quite yet want to go that far.
• Export specific template instantiations for frequently used stuff as described here.
• Look at the bars in the IncrediBuild progress window. It visualizes the compilation times of the different units. This is a good indication of where the most potential for further optimization lies.
• Use the IncludeManager VisualStudio plugin to examine the physical structure. Is also approximates which of the included header have the biggest impact.

With all this I could reduce the time of a full rebuild to two thirds of what it was before. But more important to me than a full rebuild is the time to build after a simple change in one of the fundamental geometry classes. This took twice the time of a full rebuild before my changes. After my changes it’s 10% less than the reduced rebuild time. But it is important to note that with all the de-coupling, the actual compile time is only 20% the rest is linking. And I’m pretty sure, the size of the precompiled headers is responsible for that. So there is still a lot of potential to optimize further. The build still takes considerably longer than the version from two years ago. But I’m optimistic, that if we also get the linker time down, then we’re in pretty good shape.