Binary is bigger on centos 5 than centos 7

bigdogs · July 7, 2023, 10:14am

hello,

Due to certain project requirements, I need to support CentOS 5. After conducting some research, I have chosen the nightly-2020-07-01 version as my compilation toolchain. However, I have encountered a peculiar issue where the binary compiled on CentOS 5(6.2M) is significantly larger in size compared to CentOS 7(1.2M).
bulid command:

cargo +nightly-2020-07-01-x86_64-unknown-linux-gnu build --release --target x86_64-unknown-linux-gnu

centos7:

[root@localhost release]# ls -lh libsqlfilter.so 
-rwxr-xr-x 2 root root 1.2M Jul  7 18:10 libsqlfilter.so (after stripped)
[root@localhost release]# gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

centos5:

[root@localhost release]# ls -lh libsqlfilter.so 
-rwxr-xr-x 2 root root 6.2M Jul  7 18:13 libsqlfilter.so
[root@localhost rustspace]# docker run -v $PWD:/build -w /build --rm -t rasp:2 gcc --version
gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-55)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

This issue feels very strange to me, and I would appreciate it if anyone could provide me with some investigation ideas.

kornel · July 7, 2023, 10:57am

Have you tried stripping the debug info? That's usually the main source of bloat in Rust executables.

nerditation · July 7, 2023, 11:18am

one possible cause is that, the way how system libraries like libc, libgcc, etc links have changed between different OS versions. for instance, static linking vs dynamic linking.

what's the output of the command "ldd libsqlfilter.so" on different platforms?

cuviper · July 7, 2023, 7:44pm

readelf -WS libsqlfilter.so may also be helpful -- that will let you see differences in .debug sections, for example. If the .text and .data sizes are very different, then maybe you're getting a vendored library in one build and not the other, which might be visible in readelf -s symbols, but it would also lack the external linkage in readelf -d (or ldd as previously suggested).

bigdogs · July 8, 2023, 3:05am

sure, it is already stripped

[root@localhost www]# ls -lh libsqlfilter.centos*
-rwxr-xr-x 1 root root 6.2M Jul  8 10:45 libsqlfilter.centos5.so
-rwxr-xr-x 1 root root 1.2M Jul  8 10:54 libsqlfilter.centos7.so
[root@localhost www]# file libsqlfilter.centos*
libsqlfilter.centos5.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped
libsqlfilter.centos7.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=ed37cfb49c285b8c9396184757ff7a9438bb52dc, stripped

the ldd command shows they are also extactly same

[root@localhost www]# ldd libsqlfilter.centos5.so 
        linux-vdso.so.1 =>  (0x00007ffd139fd000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007ff3ff034000)
        librt.so.1 => /lib64/librt.so.1 (0x00007ff3fee2c000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff3fec10000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff3fe9fa000)
        libc.so.6 => /lib64/libc.so.6 (0x00007ff3fe62c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff3ff611000)
[root@localhost www]# ldd libsqlfilter.centos7.so 
        linux-vdso.so.1 =>  (0x00007ffc09229000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f7956861000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f7956659000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f795643d000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f7956227000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f7955e59000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7956d91000)

the `readelf -WS shows that every section size of centos5 also. biggger than centos7.. wired! (also I notice that there are some llvm sections in centos5 which are not exists in centos7)

[root@localhost www]# readelf -WS libsqlfilter.centos5.so 
There are 30 section headers, starting at offset 0x628560:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .gnu.hash         GNU_HASH        00000000000001c8 0001c8 000030 00   A  2   0  8
  [ 2] .dynsym           DYNSYM          00000000000001f8 0001f8 0010f8 18   A  3   3  8
  [ 3] .dynstr           STRTAB          00000000000012f0 0012f0 000968 00   A  0   0  1
  [ 4] .gnu.version      VERSYM          0000000000001c58 001c58 00016a 02   A  2   0  2
  [ 5] .gnu.version_r    VERNEED         0000000000001dc8 001dc8 000130 00   A  3   6  8
  [ 6] .rela.dyn         RELA            0000000000001ef8 001ef8 024ee8 18   A  2   0  8
  [ 7] .rela.plt         RELA            0000000000026de0 026de0 0002a0 18  AI  2  23  8
  [ 8] .init             PROGBITS        0000000000027080 027080 000018 00  AX  0   0  4
  [ 9] .plt              PROGBITS        0000000000027098 027098 0001d0 10  AX  0   0  4
  [10] .text             PROGBITS        0000000000027270 027270 11acb6 00  AX  0   0 16
  [11] .fini             PROGBITS        0000000000141f28 141f28 00000e 00  AX  0   0  4
  [12] .rodata           PROGBITS        0000000000141f40 141f40 020a0c 00   A  0   0 16
  [13] .eh_frame_hdr     PROGBITS        000000000016294c 16294c 00ee14 00   A  0   0  4
  [14] .eh_frame         X86_64_UNWIND   0000000000171760 171760 04aacc 00   A  0   0  8
  [15] .gcc_except_table PROGBITS        00000000001bc22c 1bc22c 007b40 00   A  0   0  4
  [16] .tbss             NOBITS          00000000003c4500 1c4500 0000f8 00 WAT  0   0 32
  [17] .init_array       INIT_ARRAY      00000000003c4500 1c4500 000008 08  WA  0   0  8
  [18] .ctors            PROGBITS        00000000003c4508 1c4508 000010 00  WA  0   0  8
  [19] .dtors            PROGBITS        00000000003c4518 1c

[root@localhost www]# readelf -WS libsqlfilter.centos7.so 
There are 28 section headers, starting at offset 0x12b180:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .note.gnu.build-id NOTE            0000000000000200 000200 000024 00   A  0   0  4
  [ 2] .gnu.hash         GNU_HASH        0000000000000228 000228 000030 00   A  3   0  8
  [ 3] .dynsym           DYNSYM          0000000000000258 000258 0006c0 18   A  4   1  8
  [ 4] .dynstr           STRTAB          0000000000000918 000918 0004e1 00   A  0   0  1
  [ 5] .gnu.version      VERSYM          0000000000000dfa 000dfa 000090 02   A  3   0  2
  [ 6] .gnu.version_r    VERNEED         0000000000000e90 000e90 000100 00   A  4   5  8
  [ 7] .rela.dyn         RELA            0000000000000f90 000f90 01a1f0 18   A  3   0  8
  [ 8] .rela.plt         RELA            000000000001b180 01b180 000258 18  AI  3  23  8
  [ 9] .init             PROGBITS        000000000001b3d8 01b3d8 00001a 00  AX  0   0  4
  [10] .plt              PROGBITS        000000000001b400 01b400 0001a0 10  AX  0   0 16
  [11] .text             PROGBITS        000000000001b5a0 01b5a0 0ae243 00  AX  0   0 16
  [12] .fini             PROGBITS        00000000000c97e4 0c97e4 000009 00  AX  0   0  4
  [13] .rodata           PROGBITS        00000000000c97f0 0c97f0 01a274 00   A  0   0 16
  [14] .eh_frame_hdr     PROGBITS        00000000000e3a64 0e3a64 0077f4 00   A  0   0  4
  [15] .eh_frame         PROGBITS        00000000000eb258 0eb258 02a10c 00   A  0   0  8
  [16] .gcc_except_table PROGBITS        0000000000115364 115364 007b40 00   A  0   0  4
  [17] .tbss             NOBITS          000000000031d380 11d380 000098 00 WAT  0   0 32
  [18] .init_array       INIT_ARRAY      000000000031d380 11d380 000010 08  WA  0   0  8
  [19] .fini_array       FINI_ARRAY      000000000031d390 1

nerditation · July 8, 2023, 4:15am

you can try the "cargo bloat" tool to check whether any of the dependencies is being built differently on different operating systems, but I highly doubt it, since both are built with the same version of rust toolchain. I can only imagine the difference is due to the platform linker. it might be related to reachability analysis, and/or it might be related to lto, just a guess.

if possible, can you test with other linkers such as lld or rust-lld, instead of gnu ld? you can change the setting in .cargo/config.toml or using an environment variable, see:
https://doc.rust-lang.org/cargo/reference/config.html#targettriplelinker

bigdogs · July 8, 2023, 4:28am

Thanks for the suggestion, Since centos5 and the toolchain is too old, I'm stucking at making clang(rust-lld) working at cetnos5.