MEX files are invalid on Apple Silicon

[karimi-ali]

Dear WK,

I get the following error when running a WKWLOADROI command from MATLAB:

Invalid MEX-file '/Users/karimia/code/alik/auxiliaryMethods/wkw/matlab/wkwLoadRoi.mexmaci64':
dlopen(/Users/karimia/code/alik/auxiliaryMethods/wkw/matlab/wkwLoadRoi.mexmaci64, 0x0006): Library not loaded:
/usr/local/opt/lz4/lib/liblz4.1.dylib
Referenced from:
/Users/karimia/code/alik/auxiliaryMethods/wkw/matlab/wkwLoadRoi.mexmaci64
Reason: tried: '/usr/local/opt/lz4/lib/liblz4.1.dylib' (no such file),
'/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/lz4/lib/liblz4.1.dylib' (no such file),
'/usr/local/opt/lz4/lib/liblz4.1.dylib' (no such file), '/usr/local/lib/liblz4.1.dylib' (no such file),
'/usr/lib/liblz4.1.dylib' (no such file, not in dyld cache)

Trying to build using WKWBUILD gives the following error:

wkwBuild();
Updating crates.io index
Compiling libc v0.2.141
Compiling wkwrap v1.5.0 (/Users/karimia/code/alik/auxiliaryMethods/wkw/rust)
Compiling wkw_compress v1.4.0 (/Users/karimia/code/alik/auxiliaryMethods/wkw/matlab/rust/wkw_compress)
Compiling wkw_mex v1.4.0 (/Users/karimia/code/alik/auxiliaryMethods/wkw/matlab/rust/wkw_mex)
warning: unused #[macro_use] import
--> src/lib.rs:3:1
|
3 | #[macro_use]
| ^^^^^^^^^^^^
|
= note: #[warn(unused_imports)] on by default

error: linking with cc failed: exit status: 1
|
= note: LC_ALL="C" PATH="/Users/karimia/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/karimia/.cargo/bin:/Users/karimia/miniconda3/bin:/Users/karimia/miniconda3/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin" VSLANG="1033" ZERO_AR_DATE="1" "cc" "-Wl,-exported_symbols_list,/var/folders/d_/4gtzwhhd5x31zlwmwg2nfnfw0000gp/T/rustcAM18jy/list" "-arch" "arm64" "/var/folders/d_/4gtzwhhd5x31zlwmwg2nfnfw0000gp/T/rustcAM18jy/symbols.o" "/Users/karimia/code/alik/auxiliaryMethods/wkw/matlab/rust/wkw_compress/target/release/deps/wkw_compress.wkw_compress.6ce36766-cgu.5.rcgu.o" "-L" "/Users/karimia/code/alik/auxiliaryMethods/wkw/matlab/rust/wkw_compress/target/release/deps" "-L" "/Applications/MATLAB_R2021b.app/bin/maci64" "-L" "/usr/local/lib" "-L" "../lz4/lib" "-L" "/Users/karimia/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib" "/Users/karimia/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcompiler_builtins-30535ac27d951dda.rlib" "-lmx" "-lmex" "-llz4" "-liconv" "-lSystem" "-lc" "-lm" "-L" "/Users/karimia/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib" "-o" "/Users/karimia/code/alik/auxiliaryMethods/wkw/matlab/rust/wkw_compress/target/release/deps/libwkw_compress.dylib" "-Wl,-dead_strip" "-dynamiclib" "-Wl,-dylib" "-nodefaultlibs"
= note: ld: warning: directory not found for option '-L../lz4/lib'
ld: library not found for -llz4
clang: error: linker command failed with exit code 1 (use -v to see invocation)

It is a Apple M1 Max chip on the Macos Ventura (13.0.1)

[hotzenklotz]

You are missing the LZ4 compression library on your system. Have you tried installing it with brew, e.g. brew install lz4?

[karimi-ali]

Yeah, I already installed the LZ4 package. Here's the output of running it againg:

==> Downloading https://formulae.brew.sh/api/formula.jws.json
######################################################################## 100.0%
==> Downloading https://formulae.brew.sh/api/cask.jws.json
######################################################################## 100.0%
Warning: lz4 1.9.4 is already installed and up-to-date.
To reinstall 1.9.4, run:
brew reinstall lz4

[amotta]

Hi @karimi-ali,

MATLAB used to ship its own version of LZ4, which we linked against on macOS and Linux. That's what's happening here:

webknossos-wrap/matlab/wkwBuild.m

Line 11 in cee9b72

extraLinkPaths = {fullfile(matlabRoot, 'bin', arch)};

However, in more recent versions of MATLAB (at least in R2021a) LZ4 is gone.

Three possibilities for addressing this issue that came to mind:

• Compile LZ4 for each platform and check it into git. That's what we used to do, but is rather dirty.
• As for the Python packages that are built by continuous integration, the build script for the MATLAB code could be extended such that the sources of a recent version of LZ4 are downloaded, compiled and then "bundled".
• Require the users to manually install LZ4 on their system (as you've done) and then link against this version.

You could try the last approach by finding the directory in which homebrew has installed liblz4.so and then add this path to the extraLinkPaths cell array in wkwBuild.m.