Incomplete draft for batch HPC submission

[davidorme]

So, we ideally want the snakemake system up and running, but this is just a quick draft of a simpler system that we might use. It does require the manual creation of a batch file specifying parameter combinations for different jobs, but that may be a useful starting point.

The basic idea is that a user creates a batch file in an empty directory on the HPC and then runs this tool, pointing to the batch file. That then sets multiple array jobs running, iterating over the batches. Linking into the discussion on #165, the basic idea here is to have a file structure like this:

site_directory/
    config/
    data/
        lots_of_files
path/to/experiments
    experiment_one/
        experiment_one_batch.toml
        scenario_one/
            outputs_from_scenario_one
        scenario_two/
            outputs_from_scenario_two

You would run the job by doing:

hpc_jobs/hpc_ve_run_batch_submit.sh path/to/experiments/experiment_one/experiment_one_batch.toml

That would then load the TOML config, work out how many array jobs to run and then queue those jobs. Each job would then run the specific config options and direct the outputs to the correct directories.

I'm mostly banking my quick thoughts here as a sketch for now, so this is either a post MVP task or a sketch for someone else to turn into a complete picture. Although the discussion in #165 makes the directory structure and approach a much more live question.