option to select flavour of GPU

[ijiraq]

Currently we have V100 and A100 GPU but the two types have different drivers:

A100 report NVIDIA-SMI 515.65.01 Driver Version: 515.65.01 CUDA Version: 11.7

V100 report NVIDIA-SMI 470.141.03 Driver Version: 470.141.03 CUDA Version: 11.4

This causes trouble as a container that runs on one node might not run on the other. Having the images on the hardware be the same would be helpful.

[ijiraq]

Or, Skaha could have a flag that allows selecting the GPU flavour desired.

[ijiraq]

Both is what is really needed. Same DRIVERS so containers can launch and selection ability to enable playing fair.

[fraserw]

I wonder if it's simpler to have all GPU nodes running the same driver, with that driver # presented somewhere (especially if it isn't the most recent) so people could compile their containers accordingly?

[at88mph]

CADC-13476

Currently tracked in CADC JIRA