Idea for RFC to support URIs like https:// and s3:// paths as inputs to PATH job parameters

[mwiebe]

In a Deadline Cloud feature request, aws-deadline/deadline-cloud#616, the request is to directly support data already stored in S3 as job input and output. In this turntable render sample, the instructions to try it out are to download an obj file from github, and then submit it from the local file system with the job.

If Open Job Description directly supported URIs like https:// and s3:// in job parameters, both of these cases could be handled with simpler input and fewer processing steps.

An input file parameter could accept an S3 object path like s3://<my-bucket>/assets/teapot.obj or https://github.com/alecjacobson/common-3d-test-models/raw/refs/heads/master/data/stanford-bunny.obj. Then the runtime can either download the file before running the task, or present a virtual file system that lazy-loads the file's data. The path mapping rule will be from s3://<my-bucket>/assets or the https URI prefix to a local temporary path.

Similarly, the runtime for the job would upload any output file/directory contents after a task run is complete.

An implementation of this in the OpenJD CLI could use the curl command for http file parameters, aws s3 cp for S3 file parameters, and aws s3 sync for S3 directory parameters.

[leongdl]

I like this - one idea I was thinking of. Can we think about custom scheme handlers like fus3:// to signal systems like mount point would be used. This would require the a scheme handler, or plugin system to be added. I would not want to be limited to https:// or s3:// only.

This can then enable other uses cases on CMF such as fsx:// and future extendable the evolution of deadline too?