Skip to content

Fix build error with --build-template conda/micromamba:v2 for singularity #972

Open
munishchouhan wants to merge 6 commits intomasterfrom
971-build-error-with-build-template-condamicromambav2-for-singularity
Open

Fix build error with --build-template conda/micromamba:v2 for singularity #972
munishchouhan wants to merge 6 commits intomasterfrom
971-build-error-with-build-template-condamicromambav2-for-singularity

Conversation

@munishchouhan
Copy link
Member

@munishchouhan munishchouhan commented Jan 27, 2026

Closes #971

Problem

Singularity builds using --build-template conda/micromamba:v2 (and conda-pixi-v1) fail with tar permission errors during the multi-stage build process. The error occurs in the
second stage when extracting the conda environment archive:

tar: conda/conda-meta: Cannot change mode to rwxrwxrwx: No such file or directory

Similar errors cascade across multiple directories (pkgs/cache, fortomp, etc.), making the build unusable.

Root Cause

The Singularity templates for conda-micromamba-v2 and conda-pixi-v1 used a two-stage build pattern:

  1. Stage build: Install conda/pixi environment, then tar czf the entire environment into an archive
  2. Stage final: Start from a clean base image ({{base_image}}), copy the tar archive via %files from build, then tar xzf to extract it

The tar extraction in the final stage failed because:

  • File permissions and metadata from the conda environment couldn't be preserved across the stage boundary
  • The tar extract encountered permission-related errors (Cannot change mode to rwxrwxrwx) when restoring files into the final stage's filesystem
  • Unlike Docker multi-stage builds, Singularity's multi-stage %files from build mechanism doesn't handle complex permission structures in tar archives reliably

Solution

Convert both Singularity templates from multi-stage to single-stage builds. The conda/pixi environment is installed directly in the single stage — no tar archiving or extraction
needed.

Signed-off-by: munishchouhan <hrma017@gmail.com>
@munishchouhan munishchouhan self-assigned this Jan 27, 2026
Signed-off-by: munishchouhan <hrma017@gmail.com>
Copy link
Collaborator

@pditommaso pditommaso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there any tar options that could provide an alternative solution? bit concerned about shallowing stderr in case of error (and returning

@munishchouhan
Copy link
Member Author

is there any tar options that could provide an alternative solution? bit concerned about shallowing stderr in case of error (and returning

I tired multiple flags but no luck

--delay-directory-restore 
        --no-same-owner 
        --no-same-permissions 

This error could be because of proot
I think we should use single stage build for singularity builds

@munishchouhan
Copy link
Member Author

because even i make a workaround where we just ignore this exception, may lead to brocker env

@munishchouhan
Copy link
Member Author

and not using tar also decreases the build time

munishchouhan and others added 3 commits January 29, 2026 17:51
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
@munishchouhan
Copy link
Member Author

@pditommaso should we make it single stage build for singularity or keep the multi stage build which is giving error because of proot?

@pditommaso
Copy link
Collaborator

Sorry I was missing this, the solution suggest

Solution:
Suppress harmless tar errors with 2>/dev/null || true. The extraction succeeds and conda environments function correctly.

is it really needed to change the build structure?

@munishchouhan
Copy link
Member Author

Sorry I was missing this, the solution suggest

Solution:
Suppress harmless tar errors with 2>/dev/null || true. The extraction succeeds and conda environments function correctly.

is it really needed to change the build structure?

Apologies, I havenot changed the PR summary, its for the old solution, where we ignore atr errors
But the problem is that tar doesn't work in this case because of proot because of the permissions issue when untar the files in build stage.

So I suggested single build, I will update the pr summary

@munishchouhan
Copy link
Member Author

@pditommaso I have updated the pr summary

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build error with --build-template conda/micromamba:v2 for singularity

2 participants