debugging labs fail building bpftool with BR2_CCACHE=y is in the shell environment (+ a possible leak of system llvm?)

[lucaceresoli]

Building the debugging labs image on Ubuntu 22.04 fails under certain conditions. Superficially this looks like a Buildroot bug.

Steps to reproduce:

1. Use Ubuntu 22.04. @Tropicao said the build works on Arch. Other distros/releases might work or not.
2. Ensure the llvm package is not installed
3. Build the debugging training image according to the labs instructions (currently the debugging-training/2024.02 branch is at commit cdb4b6e)

bpftool fails building. This line is logged:

/bin/sh: 1: llvm-config: not found

Installing llvm on the host and rebuilding the build succeeds:

sudo apt install llvm
make bpftool-dirclean bpftool

Maybe just a missing dependency of the bpftool package on host-llvm?

[lucaceresoli]

Adding host-llvm to BPFTOOL_DEPENDENCIES makes the /bin/sh: 1: llvm-config: not found message disappear, and the build seems to fail a bit later, but still fails, this time ending with:

CC       struct_ops.o
CC       tracelog.o
CC       xlated_dumper.o
CC       disasm.o
LINK     bpftool
/.../buildroot/output/host/lib/libLLVM-15.so: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:233: bpftool] Error 1
make[1]: *** [package/pkg-generic.mk:283: /.../buildroot/output/build/bpftool-v7.1.0/.stamp_built] Error 2

[lucaceresoli]

Closing because:

• build succeeds on another Ubuntu 22.04 (same distribution, no apparent reason for the different behaviour)
• the failure is on Ubuntu 22.04 but the training is supported for Ubuntu 24.04
• the host which showed the failure has been upgraded to a more recent distribution where the build succeeds

[lucaceresoli]

Happened again, this time on Fedora 42, and at Builroot commit 34d4cac01534 ("package/e2fsprogs: bump version to 1.47.2")

It seems to be two issues together.

Issue 1: I run make with BR2_CCACHE in the environment, which leads to:

make[1]: Entering directory '/.../buildroot/output/build/bpftool-v7.1.0/src'
/bin/sh: line 1: llvm-config: command not found
...                        libbfd: [ on  ]
...               clang-bpf-co-re: [ on  ]
...                          llvm: [ OFF ]
...                        libcap: [ OFF ]

[...]

  CC       perf.o
make[2]: Entering directory '/.../buildroot/output/build/bpftool-v7.1.0/libbpf/src'
make[2]: Nothing to be done for '/usr/lib64/ccache/gcc'.
  CC       /.../buildroot/output/build/bpftool-v7.1.0/src/bootstrap/libbpf/staticobjs/bpf.o
/.../buildroot/output/host/bin/ccache: invalid option -- 'I'

Running instead BR2_CCACHE= make we get to the next build failure.

Issue 2: clang not found?

The new error message is:

make[1]: Entering directory '/.../buildroot/output/build/bpftool-v7.1.0/src'
/bin/sh: line 1: llvm-config: command not found
...                        libbfd: [ on  ]
...               clang-bpf-co-re: [ on  ]
...                          llvm: [ OFF ]
...                        libcap: [ OFF ]

[...]

clang \
        -I. \
        -I/.../buildroot/output/build/bpftool-v7.1.0/include/uapi/ \
        -I/.../buildroot/output/build/bpftool-v7.1.0/src/bootstrap/libbpf/include \
        -g -O2 -Wall -target bpf -c skeleton/profiler.bpf.c -o profiler.bpf.o
llvm-strip -g profiler.bpf.o
make[1]: llvm-strip: No such file or directory
make[1]: *** [Makefile:205: profiler.bpf.o] Error 127
make[1]: Leaving directory '/.../buildroot/output/build/bpftool-v7.1.0/src'
make: *** [package/pkg-generic.mk:283: //buildroot/output/build/bpftool-v7.1.0/.stamp_built] Error 2

And after installing llvm (sudo dnf install llvm) the build succeeds. Apparently system llvm is leaking?

[lucaceresoli]

Happened again, this time on Fedora 42, and at Builroot commit 34d4cac01534 ("package/e2fsprogs: bump version to 1.47.2")

Same with the current debugging-training/2025.02 branch, based on Buildroot 2025.02.3, commit 916b74de56be ("kernel: enable CONFIG_BPF_JIT_ALWAYS_ON"), still on Fedora 42.