Skip to content

516: Dynamically create database tables based on XSD files#684

Closed
Simon-Will wants to merge 7 commits intoOpenEnergyPlatform:622-deprecate-api-downloadfrom
Simon-Will:516-dynamic-tables-and-columns
Closed

516: Dynamically create database tables based on XSD files#684
Simon-Will wants to merge 7 commits intoOpenEnergyPlatform:622-deprecate-api-downloadfrom
Simon-Will:516-dynamic-tables-and-columns

Conversation

@Simon-Will
Copy link

@Simon-Will Simon-Will commented Dec 7, 2025

Summary of the discussion

A first attempt at creating tables based on the MaStR XSD files.

Checklist until this is really ready:

  • Download current documentation and create database model from it
  • Use new database model for all the insertion code
  • Use fallback XSD for when using the current documentation fails for some reason
  • Implement CSV export
  • Give the user an easy way to use the mastr_table_to_db_model returned from Mastr.generate_data_model. E.g. by adding a function that generates a Python code snippet with the SQLALchemy models/tables.
  • Clear up the date situation. I made a couple of changes to utils_download_bulk.py because I found date handling unnecessarily complex. Add interactive download functionality for MaStR date selection #696 #697 changes the same code and adds support for retrieving available XML download dates. If Add interactive download functionality for MaStR date selection #696 #697 is merged, we have to update this retrieval logic to also retrieve the documentation download dates.
  • Think about how we handle the transition from users' existing databases. Especially w.r.t. to translated databases and also all the renamed columns.
  • Create usage examples
  • Agree on a couple of open points and implement them
    • How to determine primary key of tables? By hardcoding it for MaStR tables we know? Or by checking the available columns and choosing the most likely one based on some hierarchy (e.g. "Id > MastrNummer > EinheitMastrNummer > …") Cf. this code
    • How much do we want to adjust/normalize column names? Cf. this code
    • Do we want to handle the case where adding only some columns to a table fails? Cf. this code
  • Go through the library and remove newly obsolete code

Type of change (CHANGELOG.md)

Added

  • Add the new method Mastr.generate_data_model that downloads the newest MaStR documentation and uses the XSD file to build SQLAlchemy models from the contained definitions

Updated

  • Update the method Mastr.download with two optional new arguments mastr_table_to_db_table, with which the user can pass their own database schema, and alter_database_tables, with which the user can prevent open-mastr from issuing any DDL statements.

Removed

  • Remove the method Mastr.translate because the user can now pass a dict mapping MaStR table names to DB tables. By adjusting their DB table names, they can achieve a similar result.

Workflow checklist

Automation

Closes #516

PR-Assignee

Reviewer

  • 🐙 Follow the Reviewer Guidelines
  • 🐙 Provided feedback and show sufficient appreciation for the work done

@Simon-Will Simon-Will marked this pull request as draft December 7, 2025 21:47
@Simon-Will Simon-Will changed the base branch from develop to 622-deprecate-api-download December 7, 2025 21:48
@FlorianK13 FlorianK13 deleted the branch OpenEnergyPlatform:622-deprecate-api-download January 13, 2026 14:16
@FlorianK13 FlorianK13 closed this Jan 13, 2026
@FlorianK13
Copy link
Member

@Simon-Will The "deprecate" branch was merged and does not exist anymore, hence this PR should be directed to the "develop" branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants