Some linux distributions defaulted to use the –as-needed linker flag for a while. Ubuntu tried it in natty, but then reverted. Now with oneiric, it really is enabled by default.
I ran into this when one of my packages wouldn’t compile on oneiric. I always got linker errors with boost::filesystem and boost::system. Between natty and oneiric, the default version of the packaged libboost changed from 1.42 to 1.46, thus switching from filesystem v2 to v3. Obviously my first thought was that it must have to do with that. Also libwt which I use in the project had the same error in a previous version. So, I reduced my app until I was sure that couldn’t be the cause here. Also all my other packages didn’t have any problems with the transition to filesystem v3 apart from the regular changes for adapting to the new interfaces. But for these changes the compiler helps.
The project in question is organised more or less like that:
lib1.so <——  lib2.so <—— executable
All the three use boost::filesystem
Here and there I saw some posts that stated the change in ubuntu to default to the –as-needed linker flag. I did quite some reading up. With this flag, an application or library needs to link to all libs from which it needs symbols, and can no longer use them indirectly. There seem to be also other implications for intermediate libraries, of which I don’t know all yet. The linker also tries to determine which libs are not needed. For this to work, you have to pass the object files first and then the libraries. Also the order of the libraries is important if they have dependencies. As I use cmake, I usually don’t care about the exact linker commandline. After some digging, I found a link.txt file for every library or executable deep in the cmake build folder. That seems to contain the linker commandline.
For lib1.so, the linker command seems correct according to what I read so far. It lists all the required libraries after the object files. filesystem and system come last, which is good. Issuing an “ldd lib1.so” shows all other dependencies, but no reference to boost::filesystem or boost::system. Maybe I use only template and inline functions from boost::filesystem… ? “nm lib1.so | grep filesystem” shows some hits. So it looks indeed like I only used template functions out of boost::filesystem in my library.
For lib2.so, the linker commandline seems to be correct as well. filesystem and system are last again. But again, these two libs are missing in the ldd output. The output of nm shows even more symbols from filesystem than lib1, probably because I use the filesystem::iostreams as well.
For the executable, I get the linker errors when trying to build. The linker commandline has an oddity: Some boost libraries appear twice. No idea why that is and what the implications are… Anyway, taking the cmake generated linker commandline and removing all duplication still leads to the same linker errors.
I found a very interesting blog post about the –as-needed flag, which very well explains what it does, what problems it can cause and how to fix them. Too bad, it didn’t help in my case…
Now I tried to make a minimal project with the same structure to reproduce the error. But this builds nicely and doesn’t produce the error. So, I started adding things to it, and removing things from the original app simultaneously, while checking with ldd everytime for changes in the dependency to boost::filesystem.
- Adding stream writing to the minimal project.
- Removing most cpp files from the lib1 of the original app .
- Adding boost::serialize to the minimal app.
- Removing boost::serialize from the original app.
I continued until almost nothing was left of the lib1 in the original app. And all of a sudden, filesystem appeared in the dependency list. The lines that I commented out last came from a temporary hack that I applied a while ago:
We used some features from Boost 1.43 in the lib1 namely boost::random. But ubuntu versions up to natty shipped only boost 1.42. The features used were from header only libraries that are completely missing in 1.42. I didn’t want to compile and package everything involved with a newer boost version myself. So I just copied the header directories involved from boost 1.43 to a directory near the application, and used them from there in conjunction with the rest of the default boost1.42. This worked well until oneiric, which comes with boost1.46 that also has the libs missing in 1.42.
Sure, for local development it would have been cleaner and safer to use a newer boost library in a custom directory. That’s easy to do with cmake. But I wanted to always be able to generate debian packages for the new versions of the application. That would require packaging at least boost and libwt myself. And It would have led to conflicts with other dev packages that require libboost-dev or libwt-dev. So I thought it was the lesser evil to mix the missing headers in locally. Another approach that I might have considered would be to package a libboost_random-dev package that conflicts with libboost-dev (>= 1.43). Then at least I would have been warned at the right time.
Now that reminds me that I will have to be careful the next time when the boost version in ubuntu is updated as well. I’ve been using the precursor of boost::geometry in other projects (flugbuch and flightpred) for a while. And with boost1.47 it became part of the official distribution.
So, adding a condition to include the local boost headers only if the found version of boost is less than 1.43 solved the issue in the end. I always try to avoid such kludges in the first place, but sometimes I just choose the lesser evil. Well, next time I might try even harder to avoid such temporary hacks.
Leave a Reply