Fun with platform linker inconsistencies (AIX vs. Linux)

Imagine we have three source files, to be built into a pair of shared libs (one with a dependency on the other) and an exe as in:

// s1.C
extern "C" void foo(void){}

// s2.C
extern "C" void foo(void) ;
extern "C" void bar(void){foo();}

// m.C
extern "C" void foo(void) ;
extern "C" void bar(void) ;

int main()
{
   bar() ;
   foo() ;
   return 0 ;
}

On Linux, we can compile and link these with command line of the following sort

# g++ -shared s1.C -o libs1.so -fpic
# g++ -shared s2.C -o libs2.so -fpic -L. -ls1
# g++ -L. -ls2 m.C -Wl,-rpath-link,`pwd` -Wl,-rpath,`pwd`

Notice that we’ve not explicitly linked to libs1.so on Linux, even though we are using a symbol from it explictly. The linker picks up dependencies from other things that you choose to link to.

On AIX the equivalent commands to create a pair of shared libraries and link the exe to it fails at that exe link

# xlC -q64 -qmkshrobj s1.C -o shr1.o
# ar -X64 -crv libs1.a shr1.o
# xlC -q64 -qmkshrobj s2.C -o shr2.o -L. -ls1
# ar -X64 -crv libs2.a shr2.o
# xlC -q64 -L. -ls2 m.C
ld: 0711-317 ERROR: Undefined symbol: .foo
ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information.

You’ve got to add -ls1 to those link flags to get the exe to find its dependencies. I wonder which of these two link time behaviours is more common?

name mangling link errors with mixed C and C++

I’ve been scratching my head trying to figure out one last link error after I made some code changes. From the linker I got:

ld: 0711-317 ERROR: Undefined symbol: .ossCacheSysFunctionPointers

yet my symbol appeared to be there:

$ rm -rf .t
$ mkdir .t
$ cd .t
$ ar -xX64 ../libSTdb2osse.a
$ nm -X64 *.o | grep ossCacheSysFunctionPointers
.ossCacheSysFunctionPointers U           -
.ossCacheSysFunctionPointers() T        1600
ossCacheSysFunctionPointers() D        2120          24
ossCacheSysFunctionPointers() d        1984           8

Do you spot the problem? I’ve got the symbol with C++ mangling, but referenced using a C prototype. This is much easier to see with name mangling disabled

$ nm -C -X64 *.o | grep ossCacheSysFunctionPointers
.ossCacheSysFunctionPointers U           -
.ossCacheSysFunctionPointers__Fv T        1600
ossCacheSysFunctionPointers__Fv D        2120          24
ossCacheSysFunctionPointers__Fv d        1984           8

ie: my symbol is there with C++ mangling. That little () in the nm output was hard to spot today! Perhaps I should dig out my glasses?

the lesson to learn here, which I’ve “learned” a few times, apparently not well enough, is to not put prototypes in .C files. Put them in an appropriate header where all the files that include the header will see the same linkage. Of course this doesn’t help if the file that defines the function doesn’t include the function and ends up with a different linkage.

A linker option to try out.

Note to self:

Saw a linker .map file generated as a side effect of a bld today (some non-DB2 code) that was generated by:

g++ ... -Xlinker -Map -Xlinker foo.map

(linux, and the gnu-binutils linker).

I’ve used linker .map files on AIX countless times, and wished for the same in our linux builds (but didn’t think of looking for the option to generate something equivalent).

I want to try this out next time I have a link error where the unresolved symbol comes from one of the many static archives that we bundle into our product shared libs. Would this verbosity help with understanding the origin of the link error better?

using nm to track down the origin of an undefined symbol.

The shared library for our product for historical reasons is built with the (evil) -Bsymbolic flag.

This has a number of unfortunate side effects, one of which is deferring link errors due to unresolved symbols to executable link time, instead of when we build the shared library itself.

Having screwed up in my own build over the weekend, I see a link error in my build summary this morning:

/view/peeterj_m20/vbs/engn/lib/libdb2.so: undefined reference to `XXYY'

I happen to know exactly what caused this (this time), but just the other day I was asked by somebody to help them figure out something similar. So the topic seems worthy of a quick blog post.

Suppose one collects all the archive names that contribute to the shared library in question (for DB2 builders one can relink the library having done an ‘export VERY_VERBOSE=true’ on the command line).

A good way to get this is to redirect that build output to a file, grab just the link line and run something like:

# cat filewithRawLinkLine.txt | tr ' ' '\n' | grep -F -e .a -e .o > archivesAndObjectNames.txt

In my case, this looked like:

# head -5 archivesAndObjectNames.txt    
../../common/crypt/iccsdk/ci_Linux_AMD64/libicc.a     
/view/peeterj_m20/vbs/common/crypt/src/libSTcrypt.a     
/view/peeterj_m20/vbs/common/sdbin/src/libSTdb2sdbin.a     
/view/peeterj_m20/vbs/install/src/libSTjdk_path.a     
/view/peeterj_m20/vbs/engn/atm/alibatm.a     

This now leaves you in the position to search for the archive(s) that contributed the undefined symbol. This can be done with:

# for i in `cat archivesAndObjectNames.txt` ; do echo $i ; nm $i | grep XXYY ; done | tee nm.out | grep -B1 XXYY
 /view/peeterj_m20/vbs/engn/sqe/alibsqe.a        U XXYY

So, the archive supplying a reference to this symbol is ‘alibsqe.a’ This completes the task of doing a first localization of where the link error is coming from. One can continue to do this in a brute force way finding the object file within the archive, or solve it by source inspection once one knows where to look a bit better. In an extremely large source base, where nobody really knows any sizeable portion of it, narrowing down the problem is often an important first step.