davidsongroup/jaffa:2.5 Docker image fails under Singularity — writes to absolute root paths

[amarinderthind]

The davidsongroup/jaffa:2.5 container fails under Singularity/Apptainer because the pipeline writes to absolute root-level paths, which are read-only in Singularity.

Issues encountered:

run_check stage writes to /checks (echo "..." > /checks) — fails with Read-only file system

minimap2_transcriptome stage writes output to /<input_basename>/<input_basename>.paf — an absolute root path. Even with --writable-tmpfs, this fills the small tmpfs overlay (No space left on device)

Root cause:
Bpipe resolves output paths relative to / instead of the working directory. In Docker this may works because the filesystem is fully writable. In Singularity the root filesystem is read-only by default, and --writable-tmpfs only provides a small RAM-backed overlay unsuitable for real data output.

Environment:

Singularity 3.11.3 on HPC (NCI Gadi)
davidsongroup/jaffa:2.5 pulled via Nextflow
JAFFA 2.5, bpipe 0.9.13
Workaround:
Using --writable-tmpfs plus manually symlinking root-level paths back to the working directory before running bpipe:

mkdir -p "$PWD/${input_base}"
ln -sf "$PWD/${input_base}" /${input_base}
ln -sf "$PWD/checks" /checks

Suggestion:
Could the pipeline use relative paths (or respect jaffa_output) for intermediate files instead of writing to absolute root paths? This would make the container Singularity-compatible out of the box.

[amarinderthind]

Biocontainers (jaffa_2.5--hdfd78af_0.sif): This bioconda-built container has completely different internal paths — bpipe at /usr/local/opt/bpipe-0.9.13/bin/bpipe and JAFFAL at /usr/local/share/jaffa-2.5-0/JAFFAL.groovy. It failed because tools.groovy is missing from the container entirely, meaning the pipeline can't even start. The bioconda wrapper scripts (jaffal, jaffa-direct) likely generate this file at runtime.

[olliecheng]

Hi Amarinder, thanks for your error report. Are you able to share your command (including apptainer run or singularity run) that you are using? We tested JAFFA on Singularity on WEHI's Milton HPC prior to release with no issues here. In particular, I note that our testing also used Apptainer (v1.4.1) without --writable-tmpfs. As far as I'm aware, JAFFA should not be writing to the root directory, so I'm keen to get down to the bottom of what's going on here.

Re: biocontainers, we don't produce the biocontainers image so I'm unclear on the internals.

[amarinderthind]

Hi @olliecheng Thank for getting back to me. I may not have have those error report anymore. But here i am sharing a solution, which is currently working for us. But it would be great, if this is fixed in container itself.

[olliecheng]

Hi Amarinder, thanks for your reply. Unfortunately the link you shared is a 404 for me. Are you able to copy it here? Thanks!

[amarinderthind]

Hi, we implemented jaffal in our Nextflow pipeline and here is how currently we patched it.

    script:
    // include any user-supplied custom args:
    def args = task.ext.args ?: ''

    def input_name = "${fasta.baseName}.jaffal-input.fasta"
    def input_base = "${fasta.baseName}.jaffal-input"

"""
export HOME="\$PWD"
export BPIPE_HOME="\$PWD/.bpipe"
mkdir -p "\$HOME" "\$BPIPE_HOME"

# Required, don't delete
cp ${fasta} ${input_name}
input_fasta="\$PWD/${input_name}"

# bpipe writes outputs to absolute root paths (/${input_base}/ and /checks)
# symlink them back to workdir so they land on scratch, not the tmpfs overlay
mkdir -p "\$PWD/${input_base}"
ln -sf "\$PWD/${input_base}" /${input_base}
ln -sf "\$PWD/checks" /checks

# symlink csv output so bpipe can read it back from root path
ln -sf "\$PWD/${meta.sample}_jaffal_results.csv" /${meta.sample}_jaffal_results.csv

/JAFFA/tools/bin/bpipe run \
    -p refBase=${jaffal_db} \
    -p jaffa_output=./ \
    -p outputName=${meta.sample}_jaffal_results \
    ${genome_params} \
    -p threads=${task.cpus} ${args} \
    /JAFFA/JAFFAL.groovy \
    \${input_fasta} || true

# bpipe's compile_all_results writes fasta to root (after rm'ing any symlink)
# then fails output validation because it expects it at ./
# recover by copying from root to workdir
if [ ! -f "./${meta.sample}_jaffal_results.fasta" ] && [ -f "/${meta.sample}_jaffal_results.fasta" ]; then
    cp "/${meta.sample}_jaffal_results.fasta" ./
fi

# verify actual outputs exist
[ -f "./${meta.sample}_jaffal_results.csv" ] || { echo "ERROR: JAFFAL results CSV not found"; exit 1; }

"""

}

[olliecheng]

Hi @amarinderthind, we've investigated this issue and it seems to be occuring upstream in the pipeline software used, bpipe. I've reported the issue over there at ssadedin/bpipe#299. Specifically, it appears that the output directory string "./" can cause an issue where bpipe will try to write to root. This is mitigated by using a single dot without the trailing slash. As far as I could tell from my testing, there were no issues with other custom directory strings such as ./output.

In the meantime, can you try replacing -p jaffa_output=./ with -p jaffa_output=. and see if that fixes the issue?

[amarinderthind]

Hi @olliecheng Thanks for the updates and Suggestions. We will try this option in our next version of the pipeline, if not fixed in bpipe.