Building custom AMI on Rocky 8.10 with 3.13.0 fails due to rustc version being 1.79 instead of 1.82

[gwolski]

Building a parallelcluster AMI overlaying parallecluster on my custom Rocky8.10 AMI.
I take my custom AMI and then run the pcluster build-image process.

The process fails at the ParallelClusterImage step. Looking at the Cloudwatch logs, I see:

According to Redh8.10, 1.75 is base rev of rustc for RHEL8/Rocky8.10.
That is what I have running on my 3.11.1 cluster head node.

% rustc --version
rustc 1.75.0 (82e1608df 2023-12-21) (Red Hat 1.75.0-1.module+el8.10.0+1668+84166756)

Nonetheless, I just confirmed that if I try to install rustc on a Rocky 8 machine, I get 1.79. So you must be installing it somehow, using latest release for RHEL8.10/Rocky8.10.

Nonetheless, ParallelClusterImage step fails.

I also tried to build with the community Rocky8.10 ami and it too failed. Here is my config file that I have in pcluster-3-13-0-Rocky-8-10-x86-64-ami-04310224db1f2a278.yml:

# parallelcluster template build file
# This comes from the parallelcluster documentation.
# This file is fed to pcluster to build a parallecluster AMI.
#
# ami-04310224db1f2a278 is generic community Rocky 8.10
#
Image:
  Name: pcluster-3-13-0-Rocky-8-10-x86-64-ami-04310224db1f2a278
  RootVolume:
    Size: 45

Build:
  Imds:
    ImdsSupport: v2.0
  InstanceType: m6a.large # 2 vcpus/8GB
  SubnetId: subnet-0d4e3914a7XXXXXX
  SecurityGroupIds:
    - sg-04f99035XXXXXXX # can be any security group that allows outbound connections. Don't need inbound.
  ParentImage: ami-04310224db1f2a278
  UpdateOsPackages:
    Enabled: false

DevSettings:
  TerminateInstanceOnFailure: true

The command I used to submit the above file is:

pcluster build-image --image-id pcluster-3-13-0-Rocky-8-10-x86-64-ami-04310224db1f2a278 --image-configuration pcluster-3-13-0-Rocky-8-10-x86-64-ami-04310224db1f2a278.yml --region us-west-2

Any workaround/remediation?

[cartalla]

See aws/efs-utils#280 (comment) for a workaround. I've confirmed that it fixes the install of efs-utils. Once I did that on the instance for the failed build I created an AMI of that and updated my build template to use the new ami instead of the Rocky8 ami. Testing underway to see if that fixes the pcluster issue.

[gwolski]

Not sure how we can use aws/efs-utils#280 (comment) as a work around for a pcluster build-image. Best I can tell is that workaround only works if you are building efs-utils standalone. pcluster will still try to build efs-utils and fail. efs-utils team has commented aws/efs-utils#280 (comment) they will be releasing the permanent solution soon.

I'm just going to wait.....

[rems0]

Hi all,
Apparently efs-utils-2.3.1 soon to be released includes the fix for this issue.
The current ParallelCluster 3.13.0 uses efs-utils-2.1.0.
Is there a way to force the pcluster build-image process to use efs-utils-2.3.1 or is a new ParallelCluster version needed that updates this package?

Thanks,
Richard

[rems0]

See aws/efs-utils#280 (comment) for a workaround. I've confirmed that it fixes the install of efs-utils. Once I did that on the instance for the failed build I created an AMI of that and updated my build template to use the new ami instead of the Rocky8 ami. Testing underway to see if that fixes the pcluster issue.

Did your test work? How? I'm trying to build a Rocky9 image but it fails with this efs-utils rustc issue.

[cartalla]

Fixing efs-utils install isn't a workaround. ParallelCluster tries to install the broken version anyway. Will have to wait for the new version of ParallelCluster as noted above.

[hanwen-cluster]

Please see our wiki for more information on the workaround https://github.com/aws/aws-parallelcluster/wiki/(3.13.0)-Build-Image-Fails-Due-to-Dependency-Version-Incompatibility

[gwolski]

@hanwen-cluster this workaround will not work. Sure, you'll build the efs-utils but parallelcluster will still try to install the broken version. Please see @cartalla's comment above.

[gwolski]

It looks like a new efs-utils was released that fixes this issue. https://github.com/aws/efs-utils/releases/tag/v2.3.1
Do you have an ETA for rolling this into a patch or a workaround on how to insert this into the build process. As noted earlier, your above wiki information will not solve the issue. parallelcluster still tries to build from scratch based on @cartalla work.

[rems0]

I followed the instructions above and was able to create a Rocky 9.5 image and create a new Parallelcluster using that image. efs-utils did not get compiled again or at least - if it tried to - it did not crash/stop the image build process at it. The creation of the image showed FAILED at the end, but the image was there and I the creation of the ParallelCluster succeeded.

…

On Tue, 27 May 2025 at 08:56, Guntram Wolski ***@***.***> wrote: *gwolski* left a comment (aws/aws-parallelcluster#6819) <#6819 (comment)> It looks like a new efs-utils was released that fixes this issue. Do you have an ETA for rolling this into a patch or a workaround on how to insert this into the build process. As noted earlier, your above wiki information will not solve the issue. parallelcluster still tries to build from scratch based on @cartalla <https://github.com/cartalla> work. — Reply to this email directly, view it on GitHub <#6819 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALFZOOGLQJV7BU6PXRKXXDT3ARHGPAVCNFSM6AAAAAB465BPFCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSMJSGIZTONZTGU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[gwolski]

Issue is resolved by moving to 3.13.1. I had a successful build.