Guidance on "multiple versions" issue

[sashakames]

This concern has been discussed among several folks involved. I don't think we have this documented to track though. I recommend creating a FAQ so we can document this clearly as it has been a recurring issue.
"When CMOR runs to the next day and creates multiple versions for a single output"

Suggestion to provide solutions and a reference implementation so groups can conveniently implement.

[taylor13]

@mauzey1 See question at bottom of this comment. I agree with @sashakames that we should provide better guidance concerning this. (Will try to propose something specific later.)

Here is some background discussion of this and a need for improvement of its treatment.

First see #267 .

Then note (especially text in bold) the conclusion of a discussion in #937 (comment)

Hi @matthew-mizielinski . You said:

This is via the drs entries in the cvs and it will take the version from the input JSON if supplied (note that specifying the version will add this to the global attributes of the file produced).

All I find in the cmor-cvs.json file is the template, which includes , indicating where the version should appear in the directory path. How does the version value get assigned by CMOR? In the sample input file you shared with me, the "version" does not appear.

Yes, CMOR will add the version to the end of the template as an extra directory and will use today's date by default. This can be overwritten by adding it into the input JSON file, but this has the unfortunate side-effect of adding a version global attribute.

You say if it is supplied by the data provider in the input JSON file, then it will be written by CMOR as a global attribute. I don't think that's a good idea, because the same file might be included unaltered in an updated dataset version (because some other files were replaced), and then the global attribute version would not be correct. Am I right?

Agreed, version in the files is not helpful/misleading.

Would it be difficult to prevent a user-specied "version" from being written as a global attribute in a file and simply use it to construct the filename?

[mauzey1]

Would it be difficult to prevent a user-specied "version" from being written as a global attribute in a file and simply use it to construct the filename?

The user could create a hidden attribute used only as the "version" in the directory path. They would also have to replace version in the directory path template with their attribute. Something like the following.

"_dir_version": "v20260420"
"output_path_template": "<drs_specs><mip_era><activity_id><institution_id><source_id><experiment_id><variant_label><region><frequency><variable_id><branding_suffix><grid_label><dir_version>",

This would resolve the issue of files being written into two directories if the CMORizing spans across two days. However, this would cause an issue with a version mismatch between the global attribute version and directory path version.

[mauzey1]

Thinking more on it, the attribute-directory mismatch would only be an issue for the CMOR user. The publisher of the data only cares about the global attributes and file name, right? As long as the version used in the directory path is not used in the file name than it should be okay.

[taylor13]

As I understand it (@sashakames please confirm!) the publisher doesn't read or record the global attribute "version" (or "dir_version"). The publisher extracts what it considers the dataset version from the directory path.

The potential problem I'm trying to prevent is:

1. A data provider uses CMOR to write a dataset comprising multiple files, and CMOR places those files in a single directory labeled, for example, v20260420. (It currently also records version as a global attribute in the file.)
2. ESGF publishes the dataset.
3. The data provider subsequently discovers an error in one of the published files, and rewrites the file with CMOR assigning it a new date, say, "v20260501", and it is placed in a directory v20260501.
4. The previously written files that were unflawed, are moved into this new directory since they are considered part of the updated dataset (and the data provider should not have to rewrite them).
5. The version global attribute in those unflawed files remains as "v2026060420", but the files now are in the v20260501 directory, and the global attribute is now wrong.

We don't want the "version" to be recorded in the file as a global attribute because it can't be easily updated when it becomes part of an updated dataset version. If we can prevent version from being written as a global attribute to being with, then the inconsistency can be prevented. [Replacing version with dir_version won't really help, since that value too can become inconsistent with the directory name.]

[sashakames]

@taylor13 Thats exactly the issue with using a global attribute to track the version: it's extrinsic to the file.
I thought there was a best practice external to CMOR that could override the internal date that increments daily without setting a global attribute.

[taylor13]

Yes, CMOR will generate the current date, or the data provider can override that by specifying a different date in their cmor input file (by assigning a date to the "version" input item). So there's no problem with the filename and the publisher will work just fine.

The problem is that those reading the files will see "version" recorded as a global attribute, and that date could indicate that the file was written as part of an earlier version dataset (as described in #947 (comment)).

So I'd like CMOR to not write the version as a global attribute. There are other things in the input file that don't get written as global attributes; for example outpath. Can we handle version like outpath? Also, can version be omitted entirely from the input file when we don't want to override the version generated by CMOR, based on the current date?

[mauzey1]

The _version attribute generated by CMOR is already a hidden attribute that doesn't get placed in the global attributes of a dataset. Setting version in the input JSON will override the CMOR-generated _version when writing templates but with the side effect of being stored as a global attribute in a dataset. You can't really set _version in the user JSON since whatever value it has will get overwritten by CMOR.

So we just want to be able to set version in the input JSON without making it a global attribute? This will require some code changes.

Alternatively, we could use the method I provided in #947 (comment) to use a another hidden attribute _dir_version that can be used in templates as <dir_version> and will not be added as a global attribute. This will require updating guidance on the doc site.

[taylor13]

Now I understand. Thanks. I agree that we should proceed as you suggested earlier (and I didn't really follow). Let's name it _directory_date (similar to the name we use in https://docs.google.com/document/d/1dcDBu3gOXqdEBIgZo4i5OQkAhgPWLhAU4qI8uAw3NOg/edit?tab=t.0 ). As you say, we will have to update the guidance document.

@matthew-mizielinski would that be o.k. with you?

[matthew-mizielinski]

Now I understand. Thanks. I agree that we should proceed as you suggested earlier (and I didn't really follow). Let's name it _directory_date (similar to the name we use in https://docs.google.com/document/d/1dcDBu3gOXqdEBIgZo4i5OQkAhgPWLhAU4qI8uAw3NOg/edit?tab=t.0 ). As you say, we will have to update the guidance document.

@matthew-mizielinski would that be o.k. with you?

Fine by me, but note that I don't use this functionality of CMOR (we use specify create_subdirectories=0 in all our cmor_setup() calls and construct the directory structure independently), so I might suggest asking @wachsylon or @sol1105 for a more informed opinion.

[taylor13]

thanks @matthew-mizielinski . Good idea to ping @wachsylon and @sol1105

[mauzey1]

I've updated the guidance for setting a "version" for the output path on the doc site.

PCMDI/cmor3_documentation#192 will add this change.

[wachsylon]

We modified the CMOR behaviour by just ovewriting output_path_template with a specified keyvalue in cdo cmor.

Sth like
"output_path_template":"<activity_id><institution_id><source_id><experiment_id><member_id><table><variable_id><grid_label>v20260402"

So that there was no attribute in the CMOR output.