Segmentation fault

[FatihSarigol]

Hello,
We've been receiving segmentation fault with the cloud analysis and we guess it is because the MPI is not set up correctly during installation but we tried several things but nothing helped. Do you have any suggestions? The collinear analysis works fine.
Thanks!

[FatihSarigol]

I just installed it again, using GCC 7.4 this time, instead of 10.2, and did not receive the warning messages that I received before, so probably that was the reason.

[FatihSarigol]

But yet again we get another error with SyntenicCloud:

size of pack(buffer) statement = 17
i-adhore: /i-adhore-3.0.01/src/SynthenicCloud.cpp:320: int SynthenicCloud::pack(char*) const: Assertion `bufSize==getPackSize()' failed.
[vlogin2:00602] *** Process received signal ***
[vlogin2:00602] Signal: Aborted (6)
[vlogin2:00602] Signal code:  (-6)
[vlogin2:00602] [ 0] /lib64/libpthread.so.0(+0xf630)[0x2b2e6c579630]
[vlogin2:00602] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b2e6d4f8387]
[vlogin2:00602] [ 2] /lib64/libc.so.6(abort+0x148)[0x2b2e6d4f9a78]
[vlogin2:00602] [ 3] /lib64/libc.so.6(+0x2f1a6)[0x2b2e6d4f11a6]
[vlogin2:00602] [ 4] /lib64/libc.so.6(+0x2f252)[0x2b2e6d4f1252]
[vlogin2:00602] [ 5] /bin/i-adhore(_ZNK14SynthenicCloud4packEPc+0x696)[0x428a86]
[vlogin2:00602] [ 6] /bin/i-adhore(_ZN14SynthenicCloud19packSynthenicCloudsERKSt6vectorIPS_SaIS1_EEPc+0x38)[0x428e28]
[vlogin2:00602] [ 7] /bin/i-adhore(_ZN7DataSet23parallelLevel2ADHoReDynEv+0x278)[0x41fbc8]
[vlogin2:00602] [ 8] /bin/i-adhore(main+0x29e)[0x41a19e]
[vlogin2:00602] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b2e6d4e4555]
[vlogin2:00602] [10] /bin/i-adhore[0x41a6cf]
[vlogin2:00602] *** End of error message ***
Aborted

[bedroesb]

@FatihSarigol I am glad you found a solution for the first problem. Did you already try running I-adhore using our container? There is one available on dockerhub which can also be used with singularity.

I would also like to know more info about you OS, version of cmake and which command you are running with which type of data.

[FatihSarigol]

Thank you for your reply Bert,
I also thought about trying docker but we don't have admin access on our HPCC.
Do you happen to have a list of versions of things that you used when it worked for you? Maybe we can already try to bring together the same things on our side if possible and it already works I hope.

Here are the versions in our system:

OS: CentOS Linux 7
cmake version: 2.8.12.2
Open MPI version: 3.1.3
To install this one, the version of gcc that I used: gcc/7.4.0

We used the dataset II. ini , just added three lines to the config file:

(cluster_type=hybrid
cloud_gap_size=20
cloud_cluster_gap=25)

Here is the complete report with parameters:

This is i-ADHoRe v3.0.
Copyright (c) 2002-2010, Flanders Interuniversity Institute for Biotechnology, VIB.
Algorithm designed by Klaas Vandepoele, Cedric Simillion, Jan Fostier, Dieter De Witte,
Koen Janssens, Sebastian Proost, Yvan Saeys and Yves Van de Peer.

Process 1/1 is alive on compute node.

WARNING: Maximum allowed number of gaps in the alignment not specified.  Setting to cluster_gap.
WARNING: Tandem gap size not correct in settings file. Using default (gap_size / 2)


************* i-ADHoRe parameters *************
        Number of genelists = 420
        Blast table = iADHoRe_PLAZA.table
        Output path = output/
        Gap size = 30
        Cluster gap size = 35
        Cloud gap size = 20
        Cloud cluster gap size = 25
        Max gaps in alignment = 35
        Tandem gap = 15
        Flush output = 1000
        Q-value = 0.75
        Anchorpoints = 3
        Probability cutoff = 0.01
        Cloud filtering method = Binomial
        Level 2 only = true
        Use family = true
        Write statistics = false
        Alignment method = GreedyGraphbased4
        Multiple hypothesis correction = Bonferroni
        Number of threads = 4
        Compare aligners = false
        Combined search (first collinear, then cloud)
        Visualize GHM.png = false
        Visualize Alignment = false
        Verbose output = true
************ END i-AdDHoRe parameters *********

Creating dataset...                     done. (time: 0.523861s)
Mapping gene families...                done. (time: 0.253818s)
Remapping tandem duplicates...  done. (time: 0.0680161s)
Writing genelists file...               done. (time: 0.177472s)
Hybrid Search (first Collinear, then Cloud Search on what is remaining in GHM)
Level 2 multiplicon detection...bufSize= 223
getPackSize()= 0

anchorPoints.size= 11

sizeof(x_objectID)=4
sizeof(y_objectID)= 4
sizeof(begin_x)= 4
sizeof(end_x)= 4
sizeof(begin_y)= 4
sizeof(end_y)= 4
sizeof(random_probability)= 8

sizeof(size)= 4
sizeof(int)= 4

size of pack(buffer) statement = 17
i-adhore: /i-adhore-3.0.01/src/SynthenicCloud.cpp:320: int SynthenicCloud::pack(char*) const: Assertion `bufSize==getPackSize()' failed.
[vlogin2:15756] *** Process received signal ***
[vlogin2:15756] Signal: Aborted (6)
[vlogin2:15756] Signal code:  (-6)
[vlogin2:15756] [ 0] /lib64/libpthread.so.0(+0xf630)[0x2ab66d559630]
[vlogin2:15756] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2ab66e4d8387]
[vlogin2:15756] [ 2] /lib64/libc.so.6(abort+0x148)[0x2ab66e4d9a78]
[vlogin2:15756] [ 3] /lib64/libc.so.6(+0x2f1a6)[0x2ab66e4d11a6]
[vlogin2:15756] [ 4] /lib64/libc.so.6(+0x2f252)[0x2ab66e4d1252]
[vlogin2:15756] [ 5] /bin/i-adhore(_ZNK14SynthenicCloud4packEPc+0x696)[0x428a86]
[vlogin2:15756] [ 6] /bin/i-adhore(_ZN14SynthenicCloud19packSynthenicCloudsERKSt6vectorIPS_SaIS1_EEPc+0x38)[0x428e28]
[vlogin2:15756] [ 7] /bin/i-adhore(_ZN7DataSet23parallelLevel2ADHoReDynEv+0x278)[0x41fbc8]
[vlogin2:15756] [ 8] /bin/i-adhore(main+0x29e)[0x41a19e]
[vlogin2:15756] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2ab66e4c4555]
[vlogin2:15756] [10] /bin/i-adhore[0x41a6cf]
[vlogin2:15756] *** End of error message ***
Aborted

And below I will add the complete output of my cmake command.
I wanted to add this specifically because I saw the same report from someone on the internet and I noticed that I don't get the Found MPI: TRUE line. Here are those lines from the report I found (which I can not be sure if clouds worked for them but still something looks better there) :

-- Found MPI_C: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so (found version "3.1") 
-- Found MPI_CXX: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")

So I am not sure if cmake actually correctly finds my MPI or not.

And on this line, that report finds only the first libmpi_cxx.so, but mine also finds libmpi.so as well following a ;
-- Found MPI_CXX: /usr/lib64/openmpi3/lib/libmpi_cxx.so;/usr/lib64/openmpi3/lib/libmpi.so

And my Found MPI lines don't state the MPI version, while that other report always states the version.

Before starting the install, I only load the openmpi module's bin folder using this command
export PATH=$PATH:/usr/lib64/openmpi3/bin/
Then it doesn't give the MPI not found errors.
Normally we used MPI before on our cluster and it worked fine.
Here is my cmake output:

-- The C compiler identification is GNU 7.4.0
-- The CXX compiler identification is GNU 7.4.0
-- Check for working C compiler: /apps/gcc/7.4.0/bin/cc
-- Check for working C compiler: /apps/gcc/7.4.0/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /apps/gcc/7.4.0/bin/c++
-- Check for working CXX compiler: /apps/gcc/7.4.0/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found MPI_C: /usr/lib64/openmpi3/lib/libmpi.so
-- Found MPI_CXX: /usr/lib64/openmpi3/lib/libmpi_cxx.so;/usr/lib64/openmpi3/lib/libmpi.so
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.7")
-- Found PNG: /usr/lib64/libpng.so (found version "1.5.13")
-- Configuring done
-- Generating done
-- Build files have been written to: /proj/fatih/IADORE3/i-adhore-3.0.01/build

And following this, make and make install commands don't raise any warning or error messages, either.

Thanks!

[bedroesb]

@FatihSarigol Thank you very much for this in depth information. I only was involved in making this public and generating the Docker image. I requested people that have experience with I-adhore to have a look at this issue.

Small remark on my side: if you use singularity you don't need root access to run the container.

[PSB-mibel]

We can't really replicate the issue locally I'm afraid. However, I would like to note that this cloud-mode was only added as a 'last' feature to the software release of I-ADHoRe 3.0, and that nobody (internally where the software was developed) has used the cloud-mode due to the too generalized output it produces. So if possible, I would suggest limiting yourself to the alignment-mode of I-ADHoRE, which is the mode used by everyone (we know) who uses the software.

[FatihSarigol]

@bedroesb Thank you both for your reply! I think I figured out how to use Singularity and pulled i-adhore from docker hub, but eventually failed to find a solution to this issue below, which is as far as I understand related to the way Singularity is installed to our HPC requiring root access. If you already know how to solve it, that would be great help:

FATAL: while extracting i-adhore_latest.sif: root filesystem extraction failed: extract command failed: ERROR : Failed to create user namespace: user namespace disabled

@PSB-mibel Thank you for your reply! We were mainly interested in i-adhore because of the cloud mode. But it should still be working fine if I could manage to install it without any issue, right?

Thanks both!!

[taprs]

Hi all,

I am getting the same assertion error as here with local installation, in single-threaded mode as well. Not sure what data might be helpful for debugging, but I am happy to provide anything upon request.

Singularity approach worked well for me like that:

$ singularity pull docker://vibpsb/i-adhore
$ singularity shell -B /path/to/data i-adhore_latest.sif
Singularity> i-adhore file.conf 

Cheers,
Nikita