Transitive dependency `include`s missing during build step

[ocaisa]

In #5023 we ensured that the include headers are added to C_INCLUDE_PATH and friends for direct dependencies. This does not cover the case though where transitive dependencies are needed. I saw this for a build of VTE:

In file included from /cvmfs/software.eessi.io/versions/2025.06/software/linux/aarch64/nvidia/grace/software/GTK4/4.20.2-GCCcore-14.3.0/include/gtk-4.0/gdk/wayland/gdkwayland.h:31,
                 from ../vte-0.82.2/src/app/app.cc:48:
/cvmfs/software.eessi.io/versions/2025.06/software/linux/aarch64/nvidia/grace/software/GTK4/4.20.2-GCCcore-14.3.0/include/gtk-4.0/gdk/wayland/gdkwaylanddevice.h:26:10: fatal error: wa
yland-client.h: No such file or directory
   26 | #include <wayland-client.h>

I entered the failing step and saw

eb-shell> echo $C_INCLUDE_PATH
/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/GTK4/4.20.2-GCCcore-14.3.0/include:
/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/GTK3/3.24.43-GCCcore-14.3.0/include:
/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/GnuTLS/3.8.10-GCCcore-14.3.0/include:
/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/fmt/11.2.0-GCCcore-14.3.0/include:
/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/lz4/1.10.0-GCCcore-14.3.0/include:
/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/Pango/1.57.0-GCCcore-14.3.0/include:
/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/cairo/1.18.4-GCCcore-14.3.0/include:
/cvmfs/software.eessi.io/versions/2025.06/software/linux/x86_64/amd/zen2/software/Vala/0.56.18-GCCcore-14.3.0/include

The full set of include paths should be far longer, and include Wayland (where the header can be found)

eb-shell> module list

Currently Loaded Modules:
  1) EESSI/2025.06                    21) libarchive/3.8.1-GCCcore-14.3.0     41) fmt/11.2.0-GCCcore-14.3.0           61) OpenGL/2025.09-GCCcore-14.3.0
  2) EasyBuild/5.2.0                  22) CMake/4.0.3-GCCcore-14.3.0          42) GMP/6.3.0-GCCcore-14.3.0            62) libepoxy/1.5.10-GCCcore-14.3.0
  3) EESSI-extend/2025.06-easybuild   23) PCRE2/10.45-GCCcore-14.3.0          43) nettle/3.10.2-GCCcore-14.3.0        63) GTK3/3.24.43-GCCcore-14.3.0
  4) GCCcore/14.3.0                   24) GLib/2.85.3-GCCcore-14.3.0          44) libtasn1/4.20.0-GCCcore-14.3.0      64) GStreamer/1.26.5-GCCcore-14.3.0
  5) libtommath/1.3.0-GCCcore-14.3.0  25) Vala/0.56.18-GCCcore-14.3.0         45) p11-kit/0.25.10-GCCcore-14.3.0      65) Graphene/1.10.8-GCCcore-14.3.0
  6) Tcl/9.0.1-GCCcore-14.3.0         26) libpng/1.6.50-GCCcore-14.3.0        46) GnuTLS/3.8.10-GCCcore-14.3.0        66) alsa-lib/1.2.14-GCCcore-14.3.0
  7) SQLite/3.50.1-GCCcore-14.3.0     27) Brotli/1.1.0-GCCcore-14.3.0         47) ATK/2.38.0-GCCcore-14.3.0           67) libogg/1.3.6-GCCcore-14.3.0
  8) libffi/3.5.1-GCCcore-14.3.0      28) freetype/2.13.3-GCCcore-14.3.0      48) at-spi2-core/2.56.2-GCCcore-14.3.0  68) libopus/1.5.2-GCCcore-14.3.0
  9) OpenSSL/3                        29) pixman/0.46.4-GCCcore-14.3.0        49) at-spi2-atk/2.38.0-GCCcore-14.3.0   69) libvorbis/1.3.7-GCCcore-14.3.0
 10) Python/3.13.5-GCCcore-14.3.0     30) expat/2.7.1-GCCcore-14.3.0          50) libjpeg-turbo/3.1.1-GCCcore-14.3.0  70) GST-plugins-base/1.26.6-GCCcore-14.3.0
 11) Meson/1.8.2-GCCcore-14.3.0       31) fontconfig/2.17.0-GCCcore-14.3.0    51) jbigkit/2.1-GCCcore-14.3.0          71) GST-plugins-bad/1.26.6-GCCcore-14.3.0
 12) Ninja/1.13.0-GCCcore-14.3.0      32) xorg-macros/1.20.2-GCCcore-14.3.0   52) libdeflate/1.24-GCCcore-14.3.0      72) GObject-Introspection/1.84.0-GCCcore-14.3.0
 13) libidn2/2.3.8-GCCcore-14.3.0     33) libpciaccess/0.18.1-GCCcore-14.3.0  53) LibTIFF/4.7.0-GCCcore-14.3.0        73) PyCairo/1.28.0-GCCcore-14.3.0
 14) libiconv/1.18-GCCcore-14.3.0     34) X11/20250608-GCCcore-14.3.0         54) Gdk-Pixbuf/2.42.12-GCCcore-14.3.0   74) PyGObject/3.54.3-GCCcore-14.3.0
 15) libunistring/1.3-GCCcore-14.3.0  35) cairo/1.18.4-GCCcore-14.3.0         55) Z3/4.15.1-GCCcore-14.3.0            75) librsvg/2.61.0-GCCcore-14.3.0
 16) libpsl/0.21.5-GCCcore-14.3.0     36) ICU/77.1-GCCcore-14.3.0             56) LLVM/20.1.8-GCCcore-14.3.0          76) libcroco/0.6.13-GCCcore-14.3.0
 17) cURL/8.14.1-GCCcore-14.3.0       37) HarfBuzz/11.4.1-GCCcore-14.3.0      57) Wayland/1.24.0-GCCcore-14.3.0       77) GTK4/4.20.2-GCCcore-14.3.0
 18) libxml2/2.14.3-GCCcore-14.3.0    38) FriBidi/1.0.16-GCCcore-14.3.0       58) elfutils/0.193-GCCcore-14.3.0
 19) gzip/1.14-GCCcore-14.3.0         39) Pango/1.57.0-GCCcore-14.3.0         59) libdrm/2.4.125-GCCcore-14.3.0
 20) zstd/1.5.7-GCCcore-14.3.0        40) lz4/1.10.0-GCCcore-14.3.0           60) libunwind/1.8.2-GCCcore-14.3.0

Doing a module refresh allowed the build step to succeed.

[ocaisa]

The issue arises in https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/tools/toolchain/toolchain.py#L1240 but I wonder where is the right place to address this?

[ocaisa]

For EESSI/software-layer#1398, I can use a hook to explicitly add Wayland as a dependency of VTE to get around the problem, but I do not think that should be required.

[ocaisa]

I've been looking into this for 2 hours and I'm not sure what the correct fix is. The problem is that you might actually want to be emptying out the environment variables as there are two relevant settings:

    --module-search-path-headers=MODULE-SEARCH-PATH-HEADERS
                        Environment variable set by modules on load with search paths to header files (type choice; default: cpath) (choices: cpath, include_paths)
    --search-path-cpp-headers=SEARCH-PATH-CPP-HEADERS
                        Search path used at build time for include directories (type choice; default: flags) (choices: flags, cpath, include_paths)

So module files could be using cpath or include_paths but there are multiple options for how you search for headers. I think the safest thing is to assume you want to listen to everything in both avenues (cpath and include_paths) and include all those directories in whatever SEARCH-PATH-CPP-HEADERS is set to.

[ocaisa]

I can do that, but I cannot pick apart where the conversion to CPPFLAGS happens

[ocaisa]

I have tracked down the source of the problem. The issue arises due to using anything other than the default value for

--search-path-cpp-headers=SEARCH-PATH-CPP-HEADERS
                        Search path used at build time for include directories (type choice; default: flags) (choices: flags, cpath, include_paths)

When using environment variables, any existing value in those variables is ignored and they are hard-reset. What should probably happen is

• Any existing values of the environment variables relevant to cpath and include_paths should be taken into account as fall back values (with locations from listed dependencies being preferred).
• All environment variables that are not explicitly expected should be cleared.