CUDA tests in QUICK-25.03 fail on NVIDIA GeForce RTX 5090

[yuxt0261-ops]

---

Description

Dear QUICK Developers,

Thank you very much for your continued development and maintenance of QUICK.
We are writing to report a serious issue encountered when running the CUDA version of QUICK on a new-generation NVIDIA GPU (GeForce RTX 5090).

---

Version

• QUICK: 25.03

---

Build Environment

• CMake: 4.2.0
• Compiler: GNU GCC 13.4.0
• CUDA: 12.8.1
• MPI: OpenMPI 5.0.9

---

Test Environment

• OS: Ubuntu 24.04.3 LTS (GNU/Linux 6.8.0-87-generic x86_64)
• CPU: 2 × AMD EPYC 9354 (32 cores each)
• GPU: NVIDIA GeForce RTX 5090
• Scheduler: Slurm 23.11.4

---

Test Command

./runtest --cuda

---

Observed Behavior

Most CUDA tests fail. Representative cases include:

grad_psb3_b3lyp_631g

Difference file: grad_psb3_b3lyp_631g.grad.dif
The test terminates normally, but the numerical results differ significantly from the reference.

grad_psb3_b3lyp_631gss

Error file: test_quick_cuda_grad_psb3_b3lyp_631gss.err
The test terminates abnormally with the following error:
[ERROR] GPU error: cudaMemcpy failure
[INFO] At line 798 in file /home/xtyu/software/src/QUICK-QUICK-25.03/src/gpu/cuda/gpu_type.h
[INFO] Error code: 700
[INFO] Error message: an illegal memory access was encountered

All CUDA test outputs have been archived and attached as:

test_quick_cuda.tar.gz

---

Additional Verification

For comparison, we also ran the CPU version:

./runtest

All tests pass successfully without any errors.

---

Notes

We are aware that the QUICK documentation states that QUICK-25.03 CUDA has been tested on the following GPUs: H200, H100, A100, RTX3080Ti, RTX2080Ti, RTX8000, RTX6000, RTX2080, T4, V100, Titan V, P100, M40, GTX1080, K80, and K40.

Therefore, the RTX5090 is not included in the list of tested GPUs. Nonetheless, we would appreciate it if you could investigate this issue so that we can use QUICK on the RTX5090 for our scientific computations.

---

Closing

We sincerely appreciate your time and effort in looking into this issue.
Please let us know if any additional information, tests, or debugging output would be helpful.

Thank you very much for your work on QUICK, and we look forward to your response.

Best regards,
Xingtai Yu

[ohearnk]

Hello,

Thank you for reporting this issue. Currently, we do not have access to a Blackwell series NVIDIA GPU to attempt to reproduce this issue on. Though, we may be able to obtain access to such hardware in the future to debug and resolve this issue.

In the meantime, I have a few comments / thoughts:

• It appears support for Blackwell GPUs (compute capability >= 12.0) was added with CUDA v12.8. It may be that there have been bugfixes since this CUDA version specifically for Blackwell (though I have not confirmed this by reading the CUDA release notes). It might be worth testing newer drivers and versions of CUDA beyond v12.8 to see if this could resolve this issue somewhat easily.
• Can you double-check via the CUDA release notes for v12.8 that the drivers, compilers, and libraries all meet the minimum required versions?
• Does the GPU which the tests were run on have at least 4 GB of memory free? Also, is the hardware shared or dedicated to single use? My question here is concerned if the GPU is being shared amongst multiple codes, which could lead to out of resource conditions.

[yuxt0261-ops]

Hello,

Thank you very much for your detailed comments and suggestions. They were extremely helpful in guiding the debugging process.
I would like to provide an update with additional findings. The issue is now resolved on my side, and I hope the details below may be useful for investigating Blackwell support.

---

Findings related to CUDA version and build

I rebuilt QUICK using CUDA 13.0.0. When compiling with CUDA 13.0.0, the following error occurred:

/home/xtyu/software/src/QUICK-QUICK-25.03/src/gpu/cuda/gpu.cu(425): error: class "cudaDeviceProp" has no member "clockRate"
*gpu_core_freq = (double) (prop.clockRate * 1e-6f);

To proceed, I temporarily set gpu_core_freq to 0.0. After this minimal change, QUICK compiled successfully. This workaround avoids the removed clockRate field but does not attempt to provide a replacement implementation.

This build runs stably and all tests pass successfully without any errors. But I would like to emphasize that this only demonstrates that CUDA 13.0.0 works in this environment. It does not prove that CUDA 12.8 or other CUDA version is inherently supported or unsupported.

---

Check related to driver / CUDA compatibility

Based on your suggestion, I double-checked that all components meet the minimum requirements:

GLIBC: 2.39

nvidia-smi reports:
Driver: 580.95.05
Maximum supported CUDA version: 13.0

nvcc --version:
CUDA compilation tools 12.8.93 (for the original build)

Runtime linking confirms:
quick.cuda links against CUDA 12.8 libraries

From a versioning perspective, the CUDA 12.8 toolchain appears consistent with the installed driver.

---

GPU memory and resource check

Each RTX 5090 provides 32 GB of memory.

Jobs were submitted with:
#SBATCH --gres=gpu:1
#SBATCH --exclusive

nvidia-smi monitoring during execution showed:
~5.2 GB peak GPU memory usage
No competing processes
GPU utilization near 0% at failure time

This suggests that the failures observed under CUDA 12.8 are unlikely to be caused by memory pressure or resource contention.

---

Summary

In summary:
QUICK built with CUDA 12.8.1 shows incorrect results or runtime failures on RTX 5090.
Rebuilding with CUDA 13.0.0, plus a minimal workaround for cudaDeviceProp::clockRate, results in a build that passes all CUDA tests.
Driver, CUDA, and runtime library versions appear consistent.
GPU memory availability and exclusivity are sufficient.

I hope this information helps narrow down whether the issue lies in CUDA 12.8 Blackwell support or in how QUICK interacts with it.

Thank you again for your time and for maintaining QUICK.
Please let me know if further testing on my side would be helpful.

Best regards,
Xingtai Yu