Importing UK DNO CIM Files

[DrWatsonApps]

Hello,

The UK distribution companies are presently in the process of exporting their networks, under a strict use license, in a CIM format.

They have presently released the 'EQ' part of their CIM model, with further data coming in May and November.

However, I am having some issues importing into Veragrid. There seems to be an incompatibility in different versions of CIM/CGMES.

I think these models could provide a lot of beneficial data for users. So please consider introducing compatibility with the UK CIM standards.

IG

[SanPen]

Hi!

CIM and CGMES are, as time passes more and more separated.
CIM could be the base thing, and CGMES is the EU adaptation plus an inexplicably large amount of questionable design choices.

On top of that, neither CIM nor CGMES are standards because they do not define basic behaviors like connectivity properly. Hence, every implementation is what the implementer thought it should be (the open world assumption).
There are deeper explanations for this, both culturally and technically, but it would derail the message into why are we as a society buying into this madness, having perfectly good open formats.

The only successful case of CIM is ERCOT because they defined all that is undefined in the standard and they did a software implementation that they later use as a guide for the entities communicating with them. That's the way.

That beign said, believe it or not, the VeraGrid implementation is the "closest" to the standard, since it was designed by one of the fathers of the CGMES. But again, what UK DSO's thought the standard should be is another matter.

So, to try to do something about the issue, I'd need said files to test.

Alternativelly you can open the rosetta browser and inspect the probaly hundreds of thousands of "standard mismatches" in the incoming files.

let me know and I'll try my best.

[DrWatsonApps]

Hey SanPen,

Thanks for getting back so fast!

The backstory is quite interesting. The electrical engineering world is a small one at times.

To my knowledge the standard they are saying they are going to use is CGMES v3 (Ofgem chat here). However, call me a cynic, I do also believe we will see a lot of inconsistency in the CIM files submitted by each DNO/DSO.

As of right now - the only files available are the 'EQ' equipment registers, with further tranches of data coming in May, and the final full CIM files in December this year. I've tried importing the EQ only to look through a nice tidy database - to no avail.

I would love to share the files with you and make it a bit easier, but part of the license agreement you sign up to when you request access to these files is that you are not allowed to share them.

That being said you can download the files yourself for NGEDs area here.

Btw, I only discovered Veragrid yesterday and I think all of this is very cool. Please keep up the good work!

IG

[DrWatsonApps]

From my present forays into trying to fix this myself, I've found that there are errors when generator p/pmax/pmin in the CGMES data is 'None', and also that synchronous machine have no 'pmax' attribute in the CGMES file for Veragrid to use. Potentially also the same with 'pmin' but I never got that far.

Mind you I'm really not very savvy on the software side of the fence so take it with a grain of salt please!

IG

[SanPen]

that's a good start. Since in CIM everything seems to be optional, you need to protect every single property against being None, which is madness.

I'll register and try to get the files.

[DrWatsonApps]

cgmes_utils.py

cgmes_to_veragrid.py

import log nged_em.xlsx

I managed to get the database for East Mids imported. I was very happy about it. :)

I made up my own TP and SSH from the LTDS data, attached below. I used AI for much of this so expect mistakes and madness. It does import- but with many errors (see the log). And sadly no diagram. I tried to make one with the 132kV buses, but its my first time using Veragrid I just had the buses in the diagram and the loadflow says its a dead network 0 V, 0 W, 0 VArs, 0 joy. :'(

Probably going to need to wait until the real TP and SSH come out.

EMIDS_SSH_generated_v4_lumped_balanced_supply_veragrid_profilepatch_fixmissingPQ_fixgenPQ_eps.xml
EMIDS_TP_generated_v2_no_boundary_prefix_veragrid_profilepatch.xml

Please have a look through and use anything you find helpful.

[griddigit-ci]

Hello IG, Santiago,

I somehow landed to this issue. Didn't check the detail, but a few things to mention the DNO exchange is still in heavy implementation and there are issues that are cleaned. Although it is based on CGMES v3, there were some regulatory consideration which required the model to be stripped from some information to meet the LTDS requirements. There are other implementation milestones in 2026 where other profiles will also be implemented. Don't expect to run any power flow only with EQ. In addition, there is other efforts in UK for the interface between NESO and DNO which will utilize CGMES v3 (and maybe 3+). So there is more to come also in termas of precised definitions. At that point when things get cleared it will also make sense to improve VeraGrid, but now it is a bit fluid. Another point is if there is a way to fund/sponsor the development in some way so that the work is not voluntary which makes the development a long lasting effort

Best regards
Chavdar

[DrWatsonApps]

Chavdar,

I fully described the situation in the first post.

The aim was not to have a running load flow, it was/is a request to start introducing compatibility with the Ofgem cim models.

Ie. You should be able to import and see the database from the EQ. Yes we could start this process much later. But why? The EQ will still have same problems now as it will then.

Much of the issues are these 'None' values everywhere, plus a change needed to import an ofgem model rather than an EU model. Those issues and others wont go away with future releases.

As for funding. Well personally I'm skint. 😂 But I've done what I can to help with my limited expertise in software. I know that Innovate UK has an Energy Digitisation programme. There might be some alignment there. I could go talk to them about it and see what they think?

[griddigit-ci]

Yes, I get that :)
there might be something to be done with the importer as for the use cases that were covered I do not think there was EQ import only. This is maybe what is causing you problem
But lets see - if any of these programmes can find some spare funds to support could be great :)

[SanPen]

Hi, I improved the CGMES import by addig some more choosing dialogues.

You'll see it prompts you to a window (Roseta explorer) where you can inspect the CGMES you're loading and not only the conversion.

cgmes_utils.py

cgmes_to_veragrid.py

import log nged_em.xlsx

I managed to get the database for East Mids imported. I was very happy about it. :)

I made up my own TP and SSH from the LTDS data, attached below. I used AI for much of this so expect mistakes and madness. It does import- but with many errors (see the log). And sadly no diagram. I tried to make one with the 132kV buses, but its my first time using Veragrid I just had the buses in the diagram and the loadflow says its a dead network 0 V, 0 W, 0 VArs, 0 joy. :'(

Probably going to need to wait until the real TP and SSH come out.

EMIDS_SSH_generated_v4_lumped_balanced_supply_veragrid_profilepatch_fixmissingPQ_fixgenPQ_eps.xml EMIDS_TP_generated_v2_no_boundary_prefix_veragrid_profilepatch.xml

Please have a look through and use anything you find helpful.

If you'd like to contribute the code we can have an introductory call ;)

[SanPen]

Hi,

I've added some dialogues to handle better the random .zip/.xml files dropped.
After selecting CGMES 3.0.0 you may see the import model window (not the conversion)

[DrWatsonApps]

SanPen,

Thanks for your work so far!

I would love to be involved with a bit of the coding if you would like to chat a bit. I'm much more involved on the electrical engineering side than the software side, but hopefully I can contribute.

[SanPen]

The latest version of VeraGrid has improved speed and synthax quite substantially. Please have a look.

You're also welcome to send me an email to spenate @ eroots . tech to schedule a call.