Streamline (input) data file names for scenario runs

[arne-exe]

As discussed with the data team, we should find a way to streamline file names so that they are easy to understand across users.
File names likely need to be unique for each scenario, although meta data (like grid and runtime) can be specified in either the sites folder or the assumptions folder (see #163). As an example, for Maliau we could potentially use something related to "Maliau scenario first run".

[nickwctan]

One thing to be aware when creating multiple files and folder is to follow a good file naming convention. I am horrible at it, but it is good practice. For example, we could start with

(Site)(scenario)(scenario_ID)

Maliau_scenario_1
Maliau_scenario_2
Danum_scenario_1
...

[davidorme]

I'd use named scenarios that have some meaning - it makes it much easier to review the folders. That way you can also use numbers to show repeats of runs, if stochastic differences between runs become important. E.g.

Maliau_single_pft_run_1

You can also put the date on, but at least initially we're going to be producing wonky results that need a model fix. We wouldn't want to keep those results - we'd want to repeat it with the fixed code. So, the date isn't really meaningful.

We haven't started enforcing the idea of per folder manifest files, but those are intended to provide a way to log all the directories and provide information about them.

[hrlai]

So, we have sort of settled on a folder naming convention of scenario/maliau/maliau_1 and I like this. But what about the input file names? Currently we have a mixed bunch of namings:

data/scenarios/maliau/maliau_1/data
├── animal_functional_groups_Maliau_level1.csv
├── animal_functional_groups_Maliau_level2.csv
├── elevation_maliau_2010_2020_100m.nc
├── era5_maliau_2010_2020_100m.nc
├── litter_maliau.nc
├── plant_cohort_data_Maliau_50x50.csv
├── plant_constants_Maliau_50x50.csv
├── plant_input_data_Maliau_50x50.nc
├── plant_pft_definitions_Maliau_50x50.csv
└── soil_maliau.nc

I have gone all spartan by soil_maliau.nc etc. but there are reasons for longer names. Do we want a consistent filename?

[nickwctan]

So, we have sort of settled on a folder naming convention of scenario/maliau/maliau_1 and I like this. But what about the input file names? Currently we have a mixed bunch of namings:

Does soil and litter input files differ according to grid size? If yes, perhaps its good to have that info in to avoid conflicts with other input files you might create in the future?

Do we want a consistent filename?

Thanks for raising it up, consistent file names throughout would be good to have. Animal input files are not affected by grid size and time at all, so I don't see a reason to add them?

What about "file name"_"site"_"grid config"_"grid size"_"duration"

[hrlai]

Does soil and litter input files differ according to grid size?

Yes, it would, but if we change the grid size, are we going to put it in the same maliau_1 folder, or in a new scenario called maliau_2?

Same question for duration.

[davidorme]

The grid size is so fundamental that you'd need a complete set of new files really. I think only animals would be unaffected. Also, I think we want the scenario folders to be portable - so should contain the complete set of files needed to run that scenario, so duplication here is fine.

[hrlai]

Just double checking @davidorme , what you meant was to put, for example, soil_maliau_50x50.nc and soil_maliau_100x100.nc in different folders, right? That would be my inclination as well. In other words, we would have maliau_1/soil_maliau.nc and maliau_2/soil_maliau.nc without the unnecessary filename suffix. The grid size info should be self-explanatory in a metadata file and/or the config file.

[davidorme]

Important

@arne-exe @hrlai @Baizurah @lelavathy @annarallings @nickwctan I think that the way in which we have written the ve_run command setup and examples have conflated two different things, and I'm wondering if this is not helping how you are thinking about the file structures and usage.

So:

1. Apologies if this is all completely clear and is already being handled!
2. The rest of this message is how I think we should think about this structure. It's definitely not the last word or necessarily best, but from a user experience point of view, I'd really like to know if this description changes how you think about how setting up VE works. If it does then I think we've made a bad mistake in how we describe it and how the VE run command works that should (post MVP) become a priority!

So, I think we need to be clearer about different possible uses of "scenario" here.

• A fundamental input level thing (e.g. 50x50 vs 100x100). Here any run uses a fundamentally different set of input files (except for e.g. animals).
• A set of compatible files that could be shared by runs using different permutations of parameters/constants etc. Note that scenarios here might use different data files too - if we had temperature files for different carbon scenarios for example.

So, I think our file structures need to reflect this:

1. Fundamental directories that are "sites" and where we might have more than one "site" directory for the same physical location (maliau_50m and maliau_100m) if they are using a different spatial scale or timesteps.

   One thing here is that I don't think that we should have to create new sites for spatial or temporal subsets within the same spatial scale and timestep. So running a 9x9 block for 2 months rather than a 50x50 block for 120 months should be a model configuration choice, not a whole new "site", but we aren't there yet.

2. Structures within "sites" that allow us to run different scenarios, where we might run multiple scenarios out of the same fundamental location and provide data that supports different scenarios, but all run within the same fundamental spatial and temporal context.

   I think a really important usage point here is that you are all used to simply pointing to an entire directory containing a fixed suite of config files that are used for that set of data files. I think that naturally leads to thinking of a "scenario" as being a fixed config and data files, and you need to duplicate those files to a new scenario directory to change the config.

   However we don't have to do that! We could for example have a directory containing core.toml, plants.toml, ..., abiotic_rcp26.toml, abiotic_rcp46.toml. Then you can use ve_run core.toml plants.toml abiotic_rcp26.toml ..... So your data files for this "site" could include both abiotic_rcp25.nc and abiotic_rcp46.nc and then you can point different invocations of VE to different parts of your config library to run different scenarios within the same site.

   This avoids duplication of files and allows us to ship a single "site" that gives users a lot more freedom to play around. I do wonder it we want a new shorthand way of selecting config files - a scenario file that simple points to a particular set of config files.

@jacobcook1995 - I'm wondering here if we should simply retire the "point to a config directory" option and just have "multiple explicit config file paths" and "one single scenario path". The point to a directory option naturally conflates these two levels (site/scenario) and I think in retrospect it is unhelpful. Replacing it with a specific minimal scenario spec that just identifies a suite of config files is far more flexible and makes a clear distinction between site and scenario.

[hrlai]

Ah I see what you mean now. I also do not have the best solution now but this opens up a good starting for us to discuss during the next team meeting.

a new shorthand way of selecting config files

This would be good, or what you're considering "multiple explicit config file paths". Especially the latter. That way one could in principal dump all input data files in one folder, and then what we've been calling "scenarios" are just separate sets of config files that point to specific data paths.

[davidorme]

That way one could in principal dump all input data files in one folder, and then what we've been calling "scenarios" are just separate sets of config files that point to specific data paths.

Exactly this. It is far more concise and avoids a huge amount of duplication. One thing we'd have to think about though is the output directory - that would need to be configurable at the scenario level, otherwise the underlying config file would send everything to the same destination. @jacobcook1995 I even think we should break the output config out of core.output and into a separate output section.

[davidorme]

@hrlai Again - apologies if this is something that you are all already aware of!

An important thing about thinking about this kind of structure for sites and scenarios is the ability to override config options on a per-run basis. If we think of site+scenario as a fixed set of config and data files, then the natural thing is to think that you have to duplicate everything if you want to alter (e.g.) groundwater_storage.

However the config overrides mean that you can do something like:

ve_run path_to_config_dir \
    --config soils.constants.groundwater_storage=0.2 \
    --config core.outputs.output_directory=../outputs/gw_0.2

Warning

Do not trust the syntax or option names there!

This fits with the separate 'site' and 'location' thinking - you can have a single set of site files and permute the inputs from the same directory to run more than one scenario. In fact, this is what underpins the snakemake sensitivity engine that we're occasionally talking about - it takes a single "site" directory and sets up hundreds of different permutations by overriding sets of configurations.

https://github.com/ImperialCollegeLondon/virtual_ecosystem_snakemake_template

[jacobcook1995]

@jacobcook1995 - I'm wondering here if we should simply retire the "point to a config directory" option and just have "multiple explicit config file paths" and "one single scenario path". The point to a directory option naturally conflates these two levels (site/scenario) and I think in retrospect it is unhelpful. Replacing it with a specific minimal scenario spec that just identifies a suite of config files is far more flexible and makes a clear distinction between site and scenario.

Yes I think this would make sense, the current way that information is spread across the example config files doesn't contain much meaning. So narrowing it down to a smaller number of files with defined purposes would be useful.

I guess one thing we would have to watch is that climate input data can belong either "site" or "scenario" (e.g. drought scenarios would involve rainfall inputs), so we probably wouldn't want to only allow input data paths to be provided as part of the site config

[davidorme]

I guess one thing we would have to watch is that climate input data can belong either "site" or "scenario" (e.g. drought scenarios would involve rainfall inputs), so we probably wouldn't want to only allow input data paths to be provided as part of the site config

The way I think about this, is that a "site" directory might contain multiple different climate data files. They'd all be sampled to the same grid and timesteps but are all there ready to use. The user can then have different config files available to switch between them. I don't think there is "site config" per se - just site data and then config files that utilise different bits of that site data in different combinations.