Remove asset and sensor IDs from flex-config and sensors_to_show on deletion

[nhoening]

When we delete an asset or sensor, its ID remains in the flex-config (flex_model and flex_context) and graph config (sensors_to_show), which are all JSON. I just ran into a broken graph dialogue locally because of that.

Addressing this on the db level seems too hard, as the data is in the postgres database, but withn a custom JSON structure.
We should address it when the API or CLI do the deletion. If someone deletes right in the db, we might not easily safeguard that, indeed.

We have API endpoints that deal with deletion of assets and sensors.
For asset deletion, even the logic for assets is in a service method that the CLI also uses. So that is a good spot.

TODO:

• create a service method delete_sensor() in data/services/sensors.py, akin to delete_asset() in data/services/generic_assets.py. It should be used from both the AI endpoint that deletes sensors, and the CLI command that deletes sensors (replicating the logic found in both if them)
• In both of the service methods for deletion, we should add logic (maybe in a util function), which searches for asset IDs and sensor IDs in the sensors_to_show, flex-context and flex-model attributes of a set of assets.
  • For that, we need to traverse the JSON structure (which is described in data/schemas/generic_assets.py::SensorsToShowSchema, data/schemas/scheduling/__init__py::FlexContextSchema and data/schemas/scheduling/storage.py::StorageFlexModelSchema. Traversing these JSON structures might require new code - maybe it should live next to the respective schema. (Can we have generic traversal function for each schema?). The SensorsToShow schema also has some older variants, which might still exist in the data. If possible, we should cover them, as well. Next to traversal by code, we could also use SQLAlchemy queries to find relevant assets with IDs (see [1]) - but it seems that is not much shorter.
  • Sensor IDs are always referenced by the "sensor" or "sensors" keys in sensors_to_show, flex_context and flex_model. Asset IDs are in sensors_to_show when a flex-config field is referenced, e.g. {"asset": 4239, "flex-model": "soc-min"}. So, much work can be done checking for those keys.
  • Also, the set of assets to check should be restricted to assets in the same account as the deleted asset/sensor. If a public asset is being deleted, we should check all assets.
• write a test for both of these new functionalities (test that deleting an asset/sensor will delete their IDs in flex-config of other assets and also from sensors_to_show)

[1] Inspiration for a query finding assets where IDs of the deleted sensor are in sensors_to_show

assets = session.query(GenericAsset).filter(
    cast(GenericAsset.attributes, JSONB)['sensors_to_show'].contains([delete_asset_id])  # still needs work, the sensors, are in a "sensors" key
).all()

[Flix6x]

Thanks for working this out. Even if we complete this, we still won't be robust against deletions directly in the db, as you already pointed out. I suggest that we first make the UI robust against deleted sensors and assets, by showing a toast notification. And check that scheduling jobs show up in RQ Dashboard with an informative error. That seems like less work to me, and leading to a more robust solution (covering db deletions).

Having the functionality you envision is still worth pursuing, though, also for being able to warn users about any references they are about to cut.

Finally, I think that each reference being altered should lead to an appropriate audit log entry. There is no undo button for this action, so we should at least have good logs.