RAM usage

[davidfstein]

I installed DeepMetaPSICOV locally, and it appears to be running correctly. However, it steadily consumes my RAM until all RAM and swap are in use and my system kills the process. I have 64GB of RAM on my local machine. Do you have an idea of the system requirements for the tool? I can try running on my cluster as well, but that will make installation more challenging.

Output as follows:

./testDMP.sh
*** Running DMP...
Using existing PSI-BLAST mtx file at 1guuA.mtx.
Using existing input alignment at 1guuA.aln.
Using existing PSIPRED stage 1 file.
Using existing PSIPRED stage 2 file.
Using existing SOLVPRED file.
Using existing alignment statistics files.
Using existing PSICOV predictions.
Using existing CCMpred predictions.
Using existing FreeContact predictions.
Using existing DMP map input.
Using existing DMP covariances.
Predicting contacts...
Killed
DMP ERROR 16 (DMP predictor failure) - please report error to psipred@cs.ucl.ac.uk
*** DMP test run failed; please check the error message(s) above, and your installation. If there are still problems, please raise an issue on the GitHub page or email psipred@cs.ucl.ac.uk.

[shaunmk]

That's weird; I've tested the example on a Linux system with 32G of memory and it runs fine. Although memory requirements will scale with the length of your target sequence, the included example is fairly small and should be able to run without requiring too many resources. From that output it looks like the memory leak is occurring in the neural network code.

What operating system and versions of Python and various Python packages you are using? The output of pip list or conda list should be a start. I can see if I can replicate the behaviour on my system. It could just be down to a bad version of PyTorch or some other package.

[davidfstein]

Thanks for the quick response! Here's my conda list output,

# packages in environment at /home/david/.bin/miniconda3/envs/deeppsicov:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             4.5                       1_gnu  
blas                      1.0                         mkl  
ca-certificates           2021.5.25            h06a4308_1  
certifi                   2021.5.30        py37h06a4308_0  
cudatoolkit               10.2.89              hfd86e86_1  
intel-openmp              2021.2.0           h06a4308_610  
ld_impl_linux-64          2.35.1               h7274673_9  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.3.0               h5101ec6_17  
libgomp                   9.3.0               h5101ec6_17  
libstdcxx-ng              9.3.0               hd4cf53a_17  
libuv                     1.40.0               h7b6447c_0  
mkl                       2021.2.0           h06a4308_296  
mkl-service               2.3.0            py37h27cfd23_1  
mkl_fft                   1.3.0            py37h42c9631_2  
mkl_random                1.2.1            py37ha9443f7_2  
ncurses                   6.2                  he6710b0_1  
ninja                     1.10.2               hff7bd54_1  
numpy                     1.20.2           py37h2d18471_0  
numpy-base                1.20.2           py37hfae3a4d_0  
openssl                   1.1.1k               h27cfd23_0  
pip                       21.1.3           py37h06a4308_0  
python                    3.7.10               h12debd9_4  
pytorch                   1.9.0           py3.7_cuda10.2_cudnn7.6.5_0    pytorch
readline                  8.1                  h27cfd23_0  
setuptools                52.0.0           py37h06a4308_0  
six                       1.16.0             pyhd3eb1b0_0  
sqlite                    3.36.0               hc218d9a_0  
tk                        8.6.10               hbc83047_0  
typing_extensions         3.10.0.0           pyh06a4308_0  
wheel                     0.36.2             pyhd3eb1b0_0  
xz                        5.2.5                h7b6447c_0  
zlib                      1.2.11               h7b6447c_3  

This is running on ubuntu 20.04

[shaunmk]

Hmmm, I can't reproduce this on a CentOS 7 system. A few things to try/check:
Try deleting the __pycache__ directory in deepmetapsicov_consens
Is this Ubuntu a virtual machine by any chance?
Can you show me the file sizes for the intermediate files created in the test directory?

[davidfstein]

I tried deleting the pycache dir but still ran into the same issue. And I'm not running a virtual machine.

These are the versions of the dependencies I've installed in case its relevant:
ccmpred=0.1.0
freecontact=1.0.21
blast=2.2.26
hhsuite=3.1.0

Here's the contents of the test directory.

total 6.0M
-rwxr-xr-x 1 david david 439K Jul  1 10:13 1guuA.aln
-rw-r--r-- 1 david david   72 Jul  6 10:16 1guuA.aux
-rw-r--r-- 1 david david 492K Jul  6 10:16 1guuA.blast
-rw-r--r-- 1 david david  66K Jul  7 12:37 1guuA.ccmpred
-rw-r--r-- 1 david david 7.9K Jul  6 10:15 1guuA.chk
-rw-r--r-- 1 david david 6.8K Jul  1 10:32 1guuA.colstats
-rw-r--r-- 1 david david 4.3M Jul  1 10:32 1guuA.deepmetapsicov.21c
-rw-r--r-- 1 david david    0 Jul  7 12:37 1guuA.deepmetapsicov.con
-rw-r--r-- 1 david david  116 Jul  1 10:32 1guuA.deepmetapsicov.fix
-rw-r--r-- 1 david david 586K Jul  1 10:32 1guuA.deepmetapsicov.map
-rw-r--r-- 1 david david  35K Jul  7 12:37 1guuA.evfold
-rwxr-xr-x 1 david david   58 Jul  1 10:13 1guuA.fasta
-rw-r--r-- 1 david david   95 Jul  6 10:16 1guuA.hhblog
-rw-r--r-- 1 david david   10 Jul  6 10:16 1guuA.mn
-rwxr-xr-x 1 david david 8.8K Jul  1 10:13 1guuA.mtx
-rw-r--r-- 1 david david  41K Jul  1 10:32 1guuA.pairstats
-rw-r--r-- 1 david david   10 Jul  6 10:16 1guuA.pn
-rw-r--r-- 1 david david  20K Jul  7 12:37 1guuA.psicov
-rw-r--r-- 1 david david   12 Jul  6 10:16 1guuA.sn
-rw-r--r-- 1 david david  700 Jul  1 10:32 1guuA.solv
-rw-r--r-- 1 david david 1.6K Jul  1 10:32 1guuA.ss
-rw-r--r-- 1 david david 1.6K Jul  1 10:32 1guuA.ss2
drwxr-xr-x 2 david david 4.0K Jul  1 10:13 example_con
-rwxr-xr-x 1 david david 1.5K Jul  1 10:13 testDMP.sh

[shaunmk]

I must say I'm stumped. I cannot work out what's going wrong. I notice that your test directory contains PSIBLAST output files that shouldn't be created for the test job. If you ran the example in a different running mode, that should be fine. The file sizes all look correct.

This is the point where I'd try the code on a different system or with a fresh Anaconda/Miniconda install (not just a new environment). While trying to match your package list I did find differences in the dependencies that PyTorch etc. pick up, though I think it's unlikely that that's what's causing the problem. Worth a try though.

[davidfstein]

Thanks for your help! I'll probably give it a shot on another system and see if it goes better