Automatically read the data download export description to extract all value mappings

[pt-kkraemer]

Description of the issue

When downloading the bulk data some non-integer values are stored as integers in order to reduce file size. Most of these integer-text-mappings can be found in the Katalogwerte.xml from the bulk download. But some of the mapping values are not present there, see for example #653.

Ideas of solution

We could try to download the export description PDF from the bulk download which can be found here and use that.

Workflow checklist

• Find out if the link of the export description has a pattern or if it changes
• Find a suitable python package to read PDF or extract tables from PDF
• Update the values in colums_to_replace.py
• I am aware of the workflow in CONTRIBUTING.md

[FlorianK13]

Do you intend this as part of the python package? Or as a script to create PRs for updates of the python package?

[nesnoj]

• Min solution: Alert message only
• Better: Bot via GH action that compares values from repo with (downloaded and scraped) the PDF and creates a PR

[nesnoj]

• Better: Bot via GH action that compares values from repo with (downloaded and scraped) the PDF and creates a PR

Unfortunately I wasn't able to find an alternative source for the Katalogwerte (they are not contained in the PDF).

I see 2 checks that could be done:

1. Verification of columns to be replaced: compare XSD files with columns_replace_list
   Example: "Hauptbrennstoff" in EinheitenVerbrennung.xsd -> yes, included in columns_replace_list
2. Verification if columns from 1. to be mapped are included in Katalogwerte.xml
   Example: "Hauptbrennstoff" in EinheitenVerbrennung.xsd -> yes, included in Katalogwerte.xml

If these 3 sources are not in sync, a PR (or at least an issue) is created.

On the value mapping:
To verify if all values in the data are included in Katalogwerte.xml (e.g. Katalogwert=1503 -> "Steinkohlenbriketts") and decide on the need of adding them to system_catalog would require a full download of the Gesamtdatenexport, right? This will probably fail due to GH actions limits. The data download, extraction and collection of unique Katalogwerte could also take place somewhere else though.
Ideas for effort reduction:

• It might be sufficient to extract only the first file of multiple technology files (e.g. AnlagenEegSolar_1.xml)
• Sync the execution to the MaStR format release cycles: the automation is only triggered on new format releases

[pt-kkraemer]

We can download only Katalogwerte.xml via GH actions, it works just like in row 251 here:

remote_zip_file.extractzip("Katalogwerte.xml", path=Path(save_path))

[nesnoj]

We can download only Katalogwerte.xml via GH actions, it works just like in row 251 here:

remote_zip_file.extractzip("Katalogwerte.xml", path=Path(save_path))

Sorry for might being too unspecific! Yes, extracting single files works well due to your partial feature 👍 .
I was referring to the detection of actual occurring values in the data files (e.g. EinheitenVerbrennung.xml -> "Hauptbrennstoff" -> values 2456..2483) which are to be checked against the values from Katalogwerte.xml (e.g. for this example Katalogwerte.xml says 2457->"Steinkohlen") to answer our original question: "Does the catalog contain mappings ID->value for all data tables?"