Python cant find executables

[casperkaae]

Hi there

I'm trying to install rgf/fastrgf and use the python wrapper to launch the executables.

I've installed using
pip install rgf_python

However when i import the rgf module i get a user warning

UserWarning: Cannot find FastRGF executable files. FastRGF estimators will be unavailable for usage.
  warnings.warn("Cannot find FastRGF executable files. FastRGF estimators will be unavailable for usage.")

To fix this issue i've compiled the rgf and fastrgf binaries* and added them to my $PATH variable (confirmed in bash that they are in the PATH) however i still get the same error. I've looked a bit into the rgf/utils get_paths and is_fastrgf_executable functions however i'm not completely sure why it fails?

*binaries: i was not sure which binaries are needed so i've added the following rgf, forest_predict, forest_train, discretized_trainer, discretized_gendata, auc

System
Python: conda 3.6.1
OS: ubuntu 16.04

[StrikerRUS]

Hi @casperkaae !

Let's start from the beginning: please try to install through pip again but with -v option now pip install rgf_python -v and post the whole log here.

[casperkaae]

Thanks for the quick response!, the log is below

(py36) caskaae@hyperion:~$ pip install rgf_python -v
Created temporary directory: /tmp/pip-ephem-wheel-cache-hrq_c3ya
Created temporary directory: /tmp/pip-install-gdn0ohun
Collecting rgf_python
  1 location(s) to search for versions of rgf-python:
  * https://pypi.org/simple/rgf-python/
  Getting page https://pypi.org/simple/rgf-python/
  Looking up "https://pypi.org/simple/rgf-python/" in the cache
  Current age based on date: 10278
  Freshness lifetime from max-age: 600
  Freshness lifetime from request max-age: 600
  Starting new HTTPS connection (1): pypi.org
  https://pypi.org:443 "GET /simple/rgf-python/ HTTP/1.1" 304 0
  Analyzing links from page https://pypi.org/simple/rgf-python/
    Found link https://files.pythonhosted.org/packages/38/b9/cccb61463017359d3d725ee84d4900b5f90f4ed60e4b9946bd084161f986/rgf_python-1.2.0.tar.gz#sha256=a029c6a5bad9391cf3e2c0b9a5cf98c6e34a760de4774460be81d92e1dd853a9 (from https://pypi.org/simple/rgf-python/), version: 1.2.0
    Found link https://files.pythonhosted.org/packages/a8/82/620c00f985bf37e8a500f8d55d9dbae5fbd0a457e1032d3bb9678b0681a5/rgf_python-1.2.1.tar.gz#sha256=e168687745424520c73cc614557a44f0bb09678374b15f1448a4d0f77280a918 (from https://pypi.org/simple/rgf-python/), version: 1.2.1
    Found link https://files.pythonhosted.org/packages/f1/d5/216df0c9d30b2126dd658547d1ec0f6f1b883202d50298c270f95645e1c1/rgf_python-1.2.2.tar.gz#sha256=56609b18c3d0fc5c01c9044770b0f075d10436d1c796df50ed06888a573688c4 (from https://pypi.org/simple/rgf-python/), version: 1.2.2
    Found link https://files.pythonhosted.org/packages/c6/64/324ad7f9b043e6094e548773dccb10fc363bf920a99fd1d7da93be0f3a4f/rgf_python-1.2.3.tar.gz#sha256=2125ff16890ac93d9181c9f4b0b701c7b12de05a8df53aa4b7daa5c883d2c5f7 (from https://pypi.org/simple/rgf-python/), version: 1.2.3
    Found link https://files.pythonhosted.org/packages/19/87/b1c5c72c589059b89c397092ab6bb3105e499982274ae7d6e5e53d01c5a3/rgf_python-1.2.5.tar.gz#sha256=4156040e8db8a16c397518f4dbb78f5a309b9c964355ed3f4362d4a0fdf11583 (from https://pypi.org/simple/rgf-python/), version: 1.2.5
    Found link https://files.pythonhosted.org/packages/8f/5f/81730e1aeaa54e6038902d7b0b464ffcd6fe8dfbdad93b9d010deef65a59/rgf_python-1.3.0.tar.gz#sha256=43d6edf700eaa720340f2aa78a8e9c7ea0468eb0df99bcd82a1c2d74e940dc85 (from https://pypi.org/simple/rgf-python/), version: 1.3.0
    Found link https://files.pythonhosted.org/packages/e2/36/767ad3555af45f69171bfd1d997ab1054871e61aff6b76a900765eb7dec2/rgf_python-1.3.1.tar.gz#sha256=7ff3738cb08356a055ccaa395c453d35088dfe61ca2b5f02e82da9fbea5439f1 (from https://pypi.org/simple/rgf-python/), version: 1.3.1
    Found link https://files.pythonhosted.org/packages/3a/4d/e4a7d32aad0400ace51a8005505fd5f0a1ce4e45199899989ea7a65e2c47/rgf_python-2.0.0.tar.gz#sha256=e40bf5cd76fe1515e5f0a7ea7fcdb5a9505405d5626b177ec8d21826313f81dc (from https://pypi.org/simple/rgf-python/), version: 2.0.0
    Found link https://files.pythonhosted.org/packages/70/e7/00446a94b5f4ab1d56554152dbc3e6d13fe611ab9b9c51ef6bc7a0ff8b61/rgf_python-2.0.1.tar.gz#sha256=bd4f73a142680e719c3be80375ba0447064c6e1e067a0a35de01ed2e042dc9ef (from https://pypi.org/simple/rgf-python/), version: 2.0.1
    Found link https://files.pythonhosted.org/packages/7b/88/18be049ac6ffbb368affa964c00089818c561a58c049b577e75385e1995f/rgf_python-2.0.2.tar.gz#sha256=ce527ec6501b29bfd97ae4156aeeebf450230aa37d84a773af6642a1c68dde41 (from https://pypi.org/simple/rgf-python/), version: 2.0.2
    Found link https://files.pythonhosted.org/packages/cf/27/515562eab19e8d001596652f8b6a48c97580b94cdae956cccca9abe095de/rgf_python-2.0.3.tar.gz#sha256=40d0e79652bd99cd075b653693467e711e78e7319872c089b9ffcdf387156e22 (from https://pypi.org/simple/rgf-python/), version: 2.0.3
    Found link https://files.pythonhosted.org/packages/1c/de/3409d5ecd095fb628aaf84927cada5248f06edba214ab549bc6bb45a495a/rgf_python-2.1.0.tar.gz#sha256=dcd086e0a116e3a7b786360ad00c33e0276896802329c384908986337d62ac8b (from https://pypi.org/simple/rgf-python/), version: 2.1.0
    Skipping link https://files.pythonhosted.org/packages/27/1c/af21c48dcb0c2aa25e3537d110f56eaa2cfa30ef33d35112951dcb29a77d/rgf_python-2.1.2-py2.py3-none-macosx_10_6_x86_64.macosx_10_7_x86_64.macosx_10_8_x86_64.macosx_10_9_x86_64.macosx_10_10_x86_64.macosx_10_11_x86_64.macosx_10_12_x86_64.macosx_10_13_x86_64.whl#sha256=f5c341e6cdb6457c645a8903c1853864e6bf2a41f49be34864b5d8b463e61f4a (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/8a/62/1862aa41399970d273d50fa19996969a298e1ca0d58cab94b9eccd439292/rgf_python-2.1.2-py2.py3-none-manylinux1_i686.whl#sha256=fad54ab32641a0a274517e0775459f0811e2debffd8c47ee6b785cb81fc9af53 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/d8/0d/e409a98d86c20efd171d07c3841c2a64e276996d8c0844096d043f8a89e7/rgf_python-2.1.2-py2.py3-none-manylinux1_x86_64.whl#sha256=e2b0e074889815fb9cf2b4412c375254492ebb74311569dfe7370b7419991f73 (from https://pypi.org/simple/rgf-python/), version: 2.1.2
    Skipping link https://files.pythonhosted.org/packages/98/ea/07ea0cd3d07b4a32fc94f2d4b734bc4d195e21b28cd1fbdee44c6d1f355f/rgf_python-2.1.2-py2.py3-none-win32.whl#sha256=52982ae01d775b37829891c4725b9f50a298fe780e51261f51ef9cc1309319d4 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/cb/83/e095b828f497460428becd30c307b1b29218360867cf729a8368948058f1/rgf_python-2.1.2-py2.py3-none-win_amd64.whl#sha256=c83f54c40e7ee5928227fb0cd6240e651a88ae8e91aee2ae9e287a941d1b9227 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/07/8e/a88bd589bef0843180a9c7254b0ce465ea2342d7840c3881e2134b7d40df/rgf_python-2.1.2.tar.gz#sha256=5ba150e00a166121c80e4e35ae5e573e082231cf2f5b772b1e8a7478d7bbfede (from https://pypi.org/simple/rgf-python/), version: 2.1.2
    Skipping link https://files.pythonhosted.org/packages/fe/6e/77ed62c5d01b4e365652dbfb42e5a638af1f927f92c77cc0408443e3fedd/rgf_python-2.2.0-py2.py3-none-macosx_10_6_x86_64.macosx_10_7_x86_64.macosx_10_8_x86_64.macosx_10_9_x86_64.macosx_10_10_x86_64.macosx_10_11_x86_64.macosx_10_12_x86_64.macosx_10_13_x86_64.whl#sha256=92d93dcb915c03f32e6a81011483569a39dada25e51f619555699c5eb294b6d8 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/39/95/7db1cf6097899c62c98b39437eba5713942ecaf11c4254a67d868afef928/rgf_python-2.2.0-py2.py3-none-manylinux1_i686.whl#sha256=cb3a7db0961a8db7f77a42f42605b702e0a8e142c8f90396862e69892d1c81e7 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/8d/1b/2cfa4fb5d45cf532c289e6d84cf2d5c5848016245ddcc7d21154c6b3989a/rgf_python-2.2.0-py2.py3-none-manylinux1_x86_64.whl#sha256=7a8672fe97417b26844629c21926339062a8c9a20622b1e9f911790c744500a6 (from https://pypi.org/simple/rgf-python/), version: 2.2.0
    Skipping link https://files.pythonhosted.org/packages/50/9e/5465dd83facaaeedf09b06feec5167276cb0a9361e6c8e9bf5e28e73130c/rgf_python-2.2.0-py2.py3-none-win32.whl#sha256=224a20cacbda901c3a9542975bea2c0b8784aa7ae0de78e357cef87ec375f96f (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/cb/4d/0b4504e73f59f239bee317762944ffe077b078dbb57aa17aa178ef67d5b2/rgf_python-2.2.0-py2.py3-none-win_amd64.whl#sha256=bc60da4e0b299fce98b62fe255bc8390787993e56a124796ac59500e1312ad3e (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/0c/28/b7282f39d0b811aa0857aace516c681cbab03a885cc66e6afefb04ed8d36/rgf_python-2.2.0.tar.gz#sha256=b9dec479a177277c08bb511515bf1bc299e29ba21063f754480c41e6eb8ac4a7 (from https://pypi.org/simple/rgf-python/), version: 2.2.0
    Skipping link https://files.pythonhosted.org/packages/d4/ba/93a63d7777fc03048e0c3200eb6063daef934b47fd6057b14ff3e357de98/rgf_python-2.3.0-py2.py3-none-macosx_10_6_x86_64.macosx_10_7_x86_64.macosx_10_8_x86_64.macosx_10_9_x86_64.macosx_10_10_x86_64.macosx_10_11_x86_64.macosx_10_12_x86_64.macosx_10_13_x86_64.whl#sha256=7a7e09161ddb5caeac4e2e2bcd79b4f2c3068a2d77781fb1986e8defeeaae9ce (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/25/ab/335fa23bc031c10d1dab0e678cd0d8e6f2da78609b0de70f7d57b7c10405/rgf_python-2.3.0-py2.py3-none-manylinux1_i686.whl#sha256=3e2e01ab24f8a24be9289c3bbbc47ba5324b7dc3559968e9c1d776ed163b22d3 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/6e/9a/5a7c608201b45f32e172cd77c1c1e2a5c1162808cb1d4151b4c0d0085403/rgf_python-2.3.0-py2.py3-none-manylinux1_x86_64.whl#sha256=9568618ffd5c46454f6750bd362fbf2022662726ba91fc043cb8e2d1796d28ad (from https://pypi.org/simple/rgf-python/), version: 2.3.0
    Skipping link https://files.pythonhosted.org/packages/87/89/d1e37f86f9fc0d8993670f0cb29212b159ed2fbc3c2ce731a229de35cdbc/rgf_python-2.3.0-py2.py3-none-win32.whl#sha256=381a0d293df96a8207db3c4403161983e22f4734d010273a31a60ac095d05396 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/1d/5b/9f37c9e5be8210c06c48621ee5100c2d62144e1043f474ec0598b2cd62d4/rgf_python-2.3.0-py2.py3-none-win_amd64.whl#sha256=810dddc9d57f0168babcf3baf42ef02ce70c7a0e618a4a5bebf26638de70a42e (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/c5/d7/633d28ac0b98deb21ee92780e292026e684cd0ae7a22e2a88cac5caa8480/rgf_python-2.3.0.tar.gz#sha256=d9b83e79e38d5e50e313d3cea27af989a1f96cbb9176c4067e71aebbb77181a2 (from https://pypi.org/simple/rgf-python/), version: 2.3.0
    Skipping link https://files.pythonhosted.org/packages/fc/74/bae2bad70302916f10954f6a0698f9dd4550b8adc10eecd30c79b3334fca/rgf_python-3.0.0-py2.py3-none-macosx_10_6_x86_64.macosx_10_7_x86_64.macosx_10_8_x86_64.macosx_10_9_x86_64.macosx_10_10_x86_64.macosx_10_11_x86_64.macosx_10_12_x86_64.macosx_10_13_x86_64.whl#sha256=42b4a9454facf6b2caa2f113765eed2e905c5cac70e8c9307c5b071a67b7bd40 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/61/ee/d8e917ebe9b9c15e0034d17af8aff82f0afb476d2c2c97a92a45b26d6152/rgf_python-3.0.0-py2.py3-none-manylinux1_i686.whl#sha256=b7c2bad1e28ca129f40d9410d09e0a332d4f5c0c5efde7a0442afe79b9a02641 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/e9/68/d0fddc3068c9b973bed1f5f8a4f0a9b25248d133685183d5235315d0d085/rgf_python-3.0.0-py2.py3-none-manylinux1_x86_64.whl#sha256=5b68c9b9de491fe975887617340f2dd28781a69bbdecdaaf9c0a563718877241 (from https://pypi.org/simple/rgf-python/), version: 3.0.0
    Skipping link https://files.pythonhosted.org/packages/59/70/d9b5ce24c4a0a3923d95df38000b213c5e29b95601a20b8de87f69f4c582/rgf_python-3.0.0-py2.py3-none-win32.whl#sha256=2bd5f871e386ec72c13fc1e8273f17ef5972a2402b08a2f4e675cf31a0151964 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/b5/09/75d7fa21efe3298da8613192e740265d3c526bff5b45631c2bb64d66f6e1/rgf_python-3.0.0-py2.py3-none-win_amd64.whl#sha256=faf3f6ab043ae9186b13505f5e629c06f5739d56f4237cb44d4f4884da39802f (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/49/3f/5e72ad095c44a80f9878f34e62c4a6d541aa36772453b3c06c66a3bb8c77/rgf_python-3.0.0.tar.gz#sha256=f82bf4b429787bcf0f19e60a90f9e1faadf9e984d24c961098bd68a9f1ae9351 (from https://pypi.org/simple/rgf-python/), version: 3.0.0
    Skipping link https://files.pythonhosted.org/packages/52/1a/b9dee7540ee1da9be3a4c5008f2c57b45b4585bb20e2bdce0834787d8043/rgf_python-3.1.0-py2.py3-none-macosx_10_6_x86_64.macosx_10_7_x86_64.macosx_10_8_x86_64.macosx_10_9_x86_64.macosx_10_10_x86_64.macosx_10_11_x86_64.macosx_10_12_x86_64.macosx_10_13_x86_64.whl#sha256=d1c0163303b1ff773786d69e613d5bc99bc1cf364d2af6bea6f6075ee4bfdb1b (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/7f/9b/f488ff1547129261bc5e8c248c4813c6b3f30d1b0696829c838f20223cb4/rgf_python-3.1.0-py2.py3-none-manylinux1_i686.whl#sha256=ceba72ba952211210ce486ddfb42130fa2eb4a19bb5d707f404f29ba0dc590eb (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/33/c4/7c73917d9a250f1d476b7f585e6b8955179e916f8410ed8f77418665aabf/rgf_python-3.1.0-py2.py3-none-manylinux1_x86_64.whl#sha256=23f525d874772da297cddef9fb664de18f9727c4b9a55a9f8ff4ab3f18ab2475 (from https://pypi.org/simple/rgf-python/), version: 3.1.0
    Skipping link https://files.pythonhosted.org/packages/7b/13/2460bc545d347aa6d9e5c8b87a49771812b3b8cb288e8cb401f1b881ada9/rgf_python-3.1.0-py2.py3-none-win32.whl#sha256=d8260578b36d546d44976fd0b4662d482bccf7e868cc254dc0e82eb08310d616 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/15/5e/9a96a8ff4da1a68935b32ffc8a1a7d8ebcff9ee999a4668ef7deeb326b2a/rgf_python-3.1.0-py2.py3-none-win_amd64.whl#sha256=32bc342444340fcf8d6119f140d6d0b946b0f9ea60d05f11a35bf4cd80bd6fe1 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/ae/22/ad7e7d0ff9304d757170e62403e792334ee432b43c6a246a14830b5e1a54/rgf_python-3.1.0.tar.gz#sha256=811c6d78dc6056884f3d225459a77195689bfb963c4cce42c2b5e060dde39826 (from https://pypi.org/simple/rgf-python/), version: 3.1.0
    Skipping link https://files.pythonhosted.org/packages/48/47/5468d6c7c636c3614a5ea6ae717596f65094b9845c9cd037cf2e7f403f05/rgf_python-3.2.0-py2.py3-none-macosx_10_6_x86_64.macosx_10_7_x86_64.macosx_10_8_x86_64.macosx_10_9_x86_64.macosx_10_10_x86_64.macosx_10_11_x86_64.macosx_10_12_x86_64.macosx_10_13_x86_64.whl#sha256=e3548997362562bc8e208dec31b90cbea42697b4d5f5bdcc52f62c604aa59f7e (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/24/4d/015b4d8e8d0f673880921778b98155548e017e439ce778c0a3b7c070b573/rgf_python-3.2.0-py2.py3-none-manylinux1_i686.whl#sha256=c14e55c8d40fed6d5f4526bff2cd7b8597780c3ea62d942c2f8f7d7b1d40f3a7 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/fe/d1/4dec9c38a61ac6eb159fe5161b9e6f54a9255d2f0585f9f0ca9b31d0bf71/rgf_python-3.2.0-py2.py3-none-manylinux1_x86_64.whl#sha256=0995ab3038054b611b4fa97d2bdb87aac4a59310a0fb6b86b5e500beb9e94da1 (from https://pypi.org/simple/rgf-python/), version: 3.2.0
    Skipping link https://files.pythonhosted.org/packages/00/fc/bf212db2a1235b993f11dae289115180e2dcb023715adff2674bac85792d/rgf_python-3.2.0-py2.py3-none-win32.whl#sha256=35281bdffb527234fed60944dd39a6219e2d29951ddc005be8267fe6d0054bd9 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Skipping link https://files.pythonhosted.org/packages/c8/ba/eaddd7d4afa2e2cf7ac08595be70853b99e312f2b8d02a339de4e60b83d4/rgf_python-3.2.0-py2.py3-none-win_amd64.whl#sha256=9f6f70286f868d6a0ab31ee4b8a429b7664f3fbc01a999267af43688ec1cff66 (from https://pypi.org/simple/rgf-python/); it is not compatible with this Python
    Found link https://files.pythonhosted.org/packages/f4/b9/d25a51f0a51d07d6d9d0d67f9ffeb1a057bc2113e281573c8160ce45b7ea/rgf_python-3.2.0.tar.gz#sha256=9a9f497c3e3f558257af20c43e00738d84969fa646780d5dc0fea43e5192ebb8 (from https://pypi.org/simple/rgf-python/), version: 3.2.0
  Using version 3.2.0 (newest of versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.2.5, 1.3.0, 1.3.1, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.1.0, 2.1.2, 2.2.0, 2.3.0, 3.0.0, 3.1.0, 3.2.0)
  Created temporary directory: /tmp/pip-unpack-nn17gdm4
  Looking up "https://files.pythonhosted.org/packages/fe/d1/4dec9c38a61ac6eb159fe5161b9e6f54a9255d2f0585f9f0ca9b31d0bf71/rgf_python-3.2.0-py2.py3-none-manylinux1_x86_64.whl" in the cache
  Ignoring unknown cache-control directive: 
  Current age based on date: 10278
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 10278
  Using cached https://files.pythonhosted.org/packages/fe/d1/4dec9c38a61ac6eb159fe5161b9e6f54a9255d2f0585f9f0ca9b31d0bf71/rgf_python-3.2.0-py2.py3-none-manylinux1_x86_64.whl
  Downloading from URL https://files.pythonhosted.org/packages/fe/d1/4dec9c38a61ac6eb159fe5161b9e6f54a9255d2f0585f9f0ca9b31d0bf71/rgf_python-3.2.0-py2.py3-none-manylinux1_x86_64.whl#sha256=0995ab3038054b611b4fa97d2bdb87aac4a59310a0fb6b86b5e500beb9e94da1 (from https://pypi.org/simple/rgf-python/)
Requirement already satisfied: scikit-learn>=0.18 in ./.conda/envs/py36/lib/python3.6/site-packages (from rgf_python) (0.19.1)
Installing collected packages: rgf-python

Successfully installed rgf-python-3.2.0
Cleaning up...

[StrikerRUS]

OK, it uses cached wheel...

Using cached https://files.pythonhosted.org/packages/fe/d1/4dec9c38a61ac6eb159fe5161b9e6f54a9255d2f0585f9f0ca9b31d0bf71/rgf_python-3.2.0-py2.py3-none-manylinux1_x86_64.whl

But the version is fresh.

Then please do the couple of things:

• provide your gcc version: gcc --v
• copy the content of ./.conda/envs/py36/lib/python3.6/site-packages/rgf_python here
• try to run forest_train -h if it is in that folder

And does the warning appear only for FastRGF or for RGF too?

[casperkaae]

Hi again

provide your gcc version: gcc --v

(py36) caskaae@hyperion:~$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.9' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) 

copy the content of ./.conda/envs/py36/lib/python3.6/site-packages/rgf_python here

.conda/envs/py36/lib/python3.6/site-packages/rgf_python-3.2.0.dist-info/ contains the following files: INSTALLER, METADATA, RECORD, top_level.txt, WHEEL

.conda/envs/py36/lib/python3.6/site-packages/rgf/ contains the following files:
fastrgf_model.py, forest_predict, forest_train, __init__.py, __pycache__/, rgf, rgf_model.py, sklearn.py, utils.py, VERSION

try to run forest_train -h if it is in that folder

.conda/envs/py36/lib/python3.6/site-packages/rgf/forest_train -h
.conda/envs/py36/lib/python3.6/site-packages/rgf/forest_train version 0.4 (Aug 2017) by Tong Zhang
  Read a dataset in appropriate format from a file.
  Train a decision forest, and save the trained model to a file.

usage: .conda/envs/py36/lib/python3.6/site-packages/rgf/forest_train [options]

 options:
  -h [-help | --help] :   print this help

....[discriptions of all the input arguments]

And does the warning appear only for FastRGF or for RGF too?

I have only seen the warning for the FastRGF executeables. If i run from rgf.sklearn import RGFRegressor it also only complains about missing FastRGF executeables

Thanks a lot for the help
Casper

[StrikerRUS]

Hmm...

Please try to grant executable permission for forest_predict and forest_train by

chmod a+x .conda/envs/py36/lib/python3.6/site-packages/rgf/forest_predict .conda/envs/py36/lib/python3.6/site-packages/rgf/forest_train

Also, you've said

To fix this issue i've compiled the rgf and fastrgf binaries* and added them to my $PATH variable (confirmed in bash that they are in the PATH) however i still get the same error.

Replace forest_predict and forest_train executables in .conda/envs/py36/lib/python3.6/site-packages/rgf/ by those you've compiled on yourself.

BTW,

*binaries: i was not sure which binaries are needed so i've added the following rgf, forest_predict, forest_train, discretized_trainer, discretized_gendata, auc

only rgf (RGF), forest_predict and forest_train (FastRGF) are used.

[casperkaae]

Hi again

I've tried to (regrant) executeable permissions as well as using my own compiled versions of the executeables, however no luck yet. Can there be anything in the python script detecting the files that doesn't work?

[StrikerRUS]

It's a pity.

Python package tries to find FastRGF as follows:

rgf/python-package/rgf/utils.py

Lines 188 to 196 in 0dd8273

	if is_fastrgf_executable(CURRENT_DIR):
	FASTRGF_PATH = CURRENT_DIR
	elif is_fastrgf_executable(DEFAULT_FASTRGF_PATH):
	FASTRGF_PATH = DEFAULT_FASTRGF_PATH
	elif is_fastrgf_executable(FASTRGF_PATH):
	pass
	else:
	FASTRGF_AVAILABLE = False
	warnings.warn("Cannot find FastRGF executable files. FastRGF estimators will be unavailable for usage.")

1. directory where script is placed;
2. directory from PATH ;
3. specified in .rgfrc config file directory or $HOME directory.

I suppose the package finds executables correctly but they are broken. To be more precise

rgf/python-package/rgf/utils.py

Line 137 in 0dd8273

def is_fastrgf_executable(path):

function returns False.

To check this, you can call this function with path to the folder where FastRGF executables are located right now and could be run by console command (forest_train -h).

It often happens when the gcc version is less than 5, but it's not your case.

Maybe something wrong with the OpenMP library?..

[casperkaae]

I've tested the is_fastrgf_executable function (slightly modded last the try-except to actually raise the error)

from rgf.utils import is_fastrgf_executable
path = '/home/caskaae/.conda/envs/py36/lib/python3.6/site-packages/rgf/'
is_fastrgf_executable(path)

the subprocess call fails with

subprocess.check_output(cmd_train, stderr=subprocess.STDOUT)
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
<ipython-input-28-34524568c302> in <module>()
----> 1 subprocess.check_output(cmd_train, stderr=subprocess.STDOUT)

~/.conda/envs/py36/lib/python3.6/subprocess.py in check_output(timeout, *popenargs, **kwargs)
    334 
    335     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
--> 336                **kwargs).stdout
    337 
    338 

~/.conda/envs/py36/lib/python3.6/subprocess.py in run(input, timeout, check, *popenargs, **kwargs)
    416         if check and retcode:
    417             raise CalledProcessError(retcode, process.args,
--> 418                                      output=stdout, stderr=stderr)
    419     return CompletedProcess(process.args, retcode, stdout, stderr)
    420 

CalledProcessError: Command '['/home/caskaae/.conda/envs/py36/lib/python3.6/site-packages/rgf/forest_train', 'forest.ntrees=10', 'tst.target=BINARY', 'trn.x-file=/tmp/rgf/temp_fastrgf.train.data.x', 'trn.y-file=/tmp/rgf/temp_fastrgf.train.data.y', 'model.save=/tmp/rgf/temp_fastrgf.model']' died with <Signals.SIGSEGV: 11>.

.... so something is not working with the executeables.

Btw i get the same error if i reinstall the pacakage from scratch through pip, e.g. not using my own compiled versions of the binaries

[StrikerRUS]

Thank you very much!

Let's exclude or confirm that your problem is actually this issue #92.

How many processor cores do you have?

Please run is_fastrgf_executable() multiple times varying two lines: replace 14 with some other numbers (42, 140, 400, etc.)

rgf/python-package/rgf/utils.py

Lines 142 to 143 in 0dd8273

	X = np.tile(np.array([[1, 0, 1, 0], [0, 1, 0, 1]]), (14, 1))
	y = np.tile(np.array([1, -1]), 14)

4-5 times will be enough.

[StrikerRUS]

Hi @casperkaae !

Have you had a chance to check different #data?

[dslate]

I ran into this problem while trying to get the RGF R package working on some x86_64 workstations running Ubuntu Linux 16.04 and python3.6. I found that python reported errors trying to "import rgf":

dave@alicia:~$ python3
Python 3.6.3 (default, Oct 27 2017, 18:35:38)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.

import rgf
/usr/local/lib/python3.6/site-packages/rgf/utils.py:196: UserWarning: Cannot find FastRGF executable files. FastRGF estimators will be unavailable for usage.
warnings.warn("Cannot find FastRGF executable files. FastRGF estimators will be unavailable for usage.")

Of my six workstations, the error occurred on the three that had 16, 24, and 24 cores, but not on the machines with 2, 4, and 10 cores.

Applying the following mod to /usr/local/lib/python3.6/site-packages/rgf/utils.py "fixed" "import rgf" by limiting the number of threads:

  ~ ~ ~ ~ ~ ~ ~ ~
  *** utils.py.orig       Sun Aug 12 10:20:21 2018
  --- utils.py    Sun Aug 12 17:39:27 2018
  ***************
  *** 146,151 ****
  --- 146,152 ----
        UUIDS.append('temp_fastrgf')
        path_train = os.path.join(path, "forest_train")
        params_train = []
  +     params_train.append("set.nthreads=4")
        params_train.append("forest.ntrees=%s" % 10)
        params_train.append("tst.target=%s" % "BINARY")
        params_train.append("trn.x-file=%s" % temp_x_loc)

Of course this does not explain why the larger number of cores induces the failure.

Sorry about the formatting of my post; I'm not sure how to fix it. To clarify, the line I added to rgf/utils.py was:

params_train.append("set.nthreads=4")

[StrikerRUS]

@dslate Thank you very much for your information! It maybe be treated as another proof that the FastRGF error is caused by hidden multithreading issue (race condition?): #92.

Of my six workstations, the error occurred on the three that had 16, 24, and 24 cores ...

Did you want to say 16, 24 and 32, right?

We'll appreciate any help from you. Do you have some time and wish to dig deeper in this issue? Do you still have the access to those machines where the error happens? Before your post I could reproduce the error only on Windows, but your workstations run under Linux - it's interesting.

PS. I fixed the markdown in your post: just added the code block around the diff.

[dslate]

@StrikerRUS Thanks for your response. Actually I did mean 16, 24 and 24; two of my machines have 24 cores each. After installing my mod to rgf/utils.py I have been trying to get FastRGF_Regressor working for a current project in R. I am already using lightgbm, xgboost, and catboost with the same data, and recently got the command line version of rgf (version 1.3) running ok with it. This is my first attempt to use FastRGF_Regressor, and I am running into additional difficulties, such as predictions that are basically random numbers that have almost no correlation with the target, which is a 0-1 binary (by the way, should I be treating this as a regression or classification problem; the goal is to maximize AUC score?). I don't know yet whether this problem is due to bugs in my code or in FastRGF. This project has a deadline, so I need to either get FastRGF working pretty soon so I can add it to the algorithms whose predictions I ensemble, or just give up and go with the algorithms I'm already using.

I really don't have much time to try to debug this, but if you come up with some potential fixes I may be able to help test them in my environment. I couldn't tell from the posts about this issue whether the errors occur intermittently with the same data or are data dependent. If the latter, perhaps the problem is not a "race condition" but some other bug(s) in the multi-threading algorithm.

[fukatani]

@dslate
Thanks for your report.
Debugging FastRGF will be a tough job, so it is better to lower threads and if you use it.

I agree that there is a possibility that it is not a data race.
Reproduction should be unstable if it is a data race, but it always happens first cycle of training.

[StrikerRUS]

@dslate AUC is a classification metric. For

the goal is to maximize AUC score

please take a look at this #199.

Unfortunately, FastRGF multithreading issue is rather old and I'm not sure we'll fix it soon. So, for your project you'd better use RGF, since FastRGF is unstable in your environment.

R wrapper (#208) will be integrated into this repo very soon (just after @fukatani's review).