Skip to content

Commit db4f7f8

Browse files
committed
Merge remote-tracking branch 'origin/main' into copilot/shorten-coordinator-test-instructions
# Conflicts: # documentation/changelog.rst # flexmeasures/api/v3_0/assets.py
2 parents 4da0c85 + 20f8110 commit db4f7f8

15 files changed

Lines changed: 632 additions & 42 deletions

File tree

documentation/api/notation.rst

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,13 @@ For example, to obtain data originating from data source 42, include the followi
274274
275275
Data source IDs can be found by hovering over data in charts.
276276

277+
For the ``GET /api/v3_0/sensors/<id>/data`` endpoint specifically, source filtering supports:
278+
279+
- ``source``: filter by data source ID
280+
- ``account``: filter by the account ID linked to data sources
281+
282+
Filtering that endpoint by source type is currently not supported.
283+
277284
.. _units:
278285

279286
Units

documentation/changelog.rst

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,12 @@ New features
1010
-------------
1111
* Added API and UI support for copying assets and their subtrees [see `PR #2017 <https://www.github.com/FlexMeasures/flexmeasures/pull/2017>`_ and `PR #2120 <https://www.github.com/FlexMeasures/flexmeasures/pull/2120>`_]
1212
* Improve UX after deleting a child asset through the UI [see `PR #2119 <https://www.github.com/FlexMeasures/flexmeasures/pull/2119>`_]
13+
* Improve source filtering in the sensor data GET endpoint by exposing the documented query parameters in Swagger and allowing filtering by the account linked to data sources [see `PR #2083 <https://www.github.com/FlexMeasures/flexmeasures/pull/2083>`_]
1314

1415
Infrastructure / Support
1516
----------------------
1617
* Upgraded dependencies [see `PR #2114 <https://www.github.com/FlexMeasures/flexmeasures/pull/2114>`_]
18+
* Run ``flexmeasures jobs run-worker`` with RQ's embedded scheduler on by default so jobs created with ``enqueue_in`` are promoted from the scheduled registry when due; pass ``--without-scheduler`` to disable [see `PR #2112 <https://www.github.com/FlexMeasures/flexmeasures/pull/2112>`_]
1719

1820
Bugfixes
1921
-----------
@@ -35,7 +37,6 @@ New features
3537
* Button on sensor page to delete data (by date range and source) [see `PR #2095 <https://www.github.com/FlexMeasures/flexmeasures/pull/2095>`_]
3638
* Support inferring ``soc-at-start`` from configured ``state-of-charge`` sources and fail early when those values are stale or missing near schedule start [see `PR #2026 <https://www.github.com/FlexMeasures/flexmeasures/pull/2026>`_]
3739
* UI support for editing JSON attributes on sensors, assets and accounts [see `PR #2093 <https://www.github.com/FlexMeasures/flexmeasures/pull/2093>`_]
38-
* Support coupling data sources to accounts, and preserve user ID and account ID references in audit logs and data sources for traceability and compliance [see `PR #2058 <https://www.github.com/FlexMeasures/flexmeasures/pull/2058>`_]
3940
* Support fetching a schedule in a different unit still compatible to the sensor unit [see `PR #1993 <https://www.github.com/FlexMeasures/flexmeasures/pull/1993>`_]
4041
* Support saving state-of-charge schedules to sensors with ``"%"`` unit, using the ``soc-max`` flex-model field as the capacity for unit conversion [see `PR #1996 <https://www.github.com/FlexMeasures/flexmeasures/pull/1996>`_]
4142
* Version headers (for server and API) in API responses [see `PR #2021 <https://www.github.com/FlexMeasures/flexmeasures/pull/2021>`_]
@@ -44,11 +45,13 @@ New features
4445
* Separate the ``StorageScheduler``'s tie-breaking preference for a full :abbr:`SoC (state of charge)` from its reported energy costs [see `PR #2023 <https://www.github.com/FlexMeasures/flexmeasures/pull/2023>`_ and `PR #2108 <https://www.github.com/FlexMeasures/flexmeasures/pull/2108>`_]
4546
* Improve asset graph hover interaction with a vertical ruler across subcharts, while keeping hover dots for easier visual tracking [see `PR #2079 <https://www.github.com/FlexMeasures/flexmeasures/pull/2079>`_]
4647
* Improve asset audit log messages for JSON field edits (especially ``sensors_to_show`` and nested flex-config values) [see `PR #2055 <https://www.github.com/FlexMeasures/flexmeasures/pull/2055>`_]
48+
* Clean up stale sensor references from ``flex-config`` and ``sensors_to_show`` when deleting a sensor, using JSONB queries to find affected assets before pruning those references [see `PR #2106 <https://www.github.com/FlexMeasures/flexmeasures/pull/2106>`_]
4749

4850
Infrastructure / Support
4951
----------------------
50-
* Migrate from ``pip`` to ``uv`` for dependency management, and from ``make`` to ``poe`` [see `PR #1973 <https://github.com/FlexMeasures/flexmeasures/pull/1973>`_]
52+
* Support coupling data sources to accounts, and preserve user ID and account ID references in audit logs and data sources for traceability and compliance [see `PR #2058 <https://www.github.com/FlexMeasures/flexmeasures/pull/2058>`_]
5153
* Stop creating new toy assets when restarting the docker-compose stack [see `PR #2018 <https://www.github.com/FlexMeasures/flexmeasures/pull/2018>`_]
54+
* Migrate from ``pip`` to ``uv`` for dependency management, and from ``make`` to ``poe`` [see `PR #1973 <https://github.com/FlexMeasures/flexmeasures/pull/1973>`_]
5255
* Improve contact information to get in touch with the FlexMeasures community [see `PR #2022 <https://www.github.com/FlexMeasures/flexmeasures/pull/2022>`_]
5356
* Expand audit logging for password life-cycle: password_reset, password_changed and reset_password_instructions_sent events [see `PR #2036 <https://github.com/FlexMeasures/flexmeasures/pull/2036>`_]
5457
* Upgraded some dependencies for security reasons [see `PR #2037 <https://www.github.com/FlexMeasures/flexmeasures/pull/2037>`_]
@@ -661,7 +664,6 @@ Infrastructure / Support
661664
* Add new annotation types: ``"error"`` and ``"warning"`` [see `PR #1131 <https://github.com/FlexMeasures/flexmeasures/pull/1131>`_ and `PR #1150 <https://github.com/FlexMeasures/flexmeasures/pull/1150>`_]
662665
* When deleting a sensor, asset or account, delete any annotations that belong to them [see `PR #1151 <https://github.com/FlexMeasures/flexmeasures/pull/1151>`_]
663666
* Removed deprecated ``app.schedulers`` and ``app.forecasters`` (use ``app.data_generators["scheduler"]`` and ``app.data_generators["forecaster"]`` instead) [see `PR #1098 <https://github.com/FlexMeasures/flexmeasures/pull/1098/>`_]
664-
* Save beliefs faster by bulk saving [see `PR #1159 <https://github.com/FlexMeasures/flexmeasures/pull/1159>`_]
665667
* Introduced dynamic, JavaScript-generated toast notifications [see `PR #1152 <https://github.com/FlexMeasures/flexmeasures/pull/1152>`_]
666668

667669
Bugfixes

flexmeasures/api/common/schemas/sensor_data.py

Lines changed: 36 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
SensorEntityAddressField,
1717
SensorIdField,
1818
)
19+
from flexmeasures.api.common.schemas.users import AccountIdField
1920
from flexmeasures.api.common.utils.api_utils import upsample_values
2021
from flexmeasures.data.models.planning.utils import initialize_index
2122
from flexmeasures.data.schemas import AwareDateTimeField, DurationField, SourceIdField
@@ -149,9 +150,33 @@ def check_schema_unit_against_sensor_unit(self, data, **kwargs):
149150
)
150151

151152

152-
class GetSensorDataSchema(SensorDataDescriptionSchema):
153-
resolution = DurationField(required=False)
154-
source = SourceIdField(required=False)
153+
class GetSensorDataFilterSchemaMixin:
154+
"""Shared filters for GET sensor data request parsing and docs."""
155+
156+
resolution = DurationField(
157+
required=False,
158+
metadata=dict(
159+
description="Resolution of the returned sensor data in ISO 8601 duration format.",
160+
example="PT15M",
161+
),
162+
)
163+
source = SourceIdField(
164+
required=False,
165+
metadata=dict(
166+
description="Filter by a specific data source ID.",
167+
example=42,
168+
),
169+
)
170+
account = AccountIdField(
171+
required=False,
172+
metadata=dict(
173+
description="Filter by the account linked to data sources.",
174+
example=3,
175+
),
176+
)
177+
178+
179+
class GetSensorDataSchema(GetSensorDataFilterSchemaMixin, SensorDataDescriptionSchema):
155180

156181
# Optional field that can be used for extra validation
157182
type = fields.Str(
@@ -202,6 +227,7 @@ def load_data_and_make_response(sensor_data_description: dict) -> dict:
202227
unit = sensor_data_description["unit"]
203228
resolution = sensor_data_description.get("resolution")
204229
source = sensor_data_description.get("source")
230+
account = sensor_data_description.get("account")
205231

206232
# Post-load configuration of event frequency
207233
if resolution is None:
@@ -231,6 +257,7 @@ def load_data_and_make_response(sensor_data_description: dict) -> dict:
231257
horizons_at_least=horizons_at_least,
232258
horizons_at_most=horizons_at_most,
233259
source=source,
260+
source_account_ids=account.id if account else None,
234261
beliefs_before=sensor_data_description.get("prior", None),
235262
one_deterministic_belief_per_event=True,
236263
resolution=resolution,
@@ -264,6 +291,12 @@ def load_data_and_make_response(sensor_data_description: dict) -> dict:
264291
return response
265292

266293

294+
class GetSensorDataQuerySchema(
295+
GetSensorDataFilterSchemaMixin, SensorDataTimingDescriptionSchema
296+
):
297+
"""Document the actual query parameters for GET /sensors/<id>/data."""
298+
299+
267300
class PostSensorDataSchema(SensorDataDescriptionSchema):
268301
"""
269302
This schema includes data (values) and still describes it.

flexmeasures/api/v3_0/sensors.py

Lines changed: 6 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
from flexmeasures.api.common.schemas.sensor_data import ( # noqa F401
3232
SensorDataDescriptionSchema,
3333
GetSensorDataSchema,
34+
GetSensorDataQuerySchema,
3435
PostSensorDataSchema,
3536
)
3637
from flexmeasures.api.common.schemas.sensors import SensorId # noqa F401
@@ -64,6 +65,7 @@
6465
from flexmeasures.data.schemas.scheduling import GetScheduleSchema
6566
from flexmeasures.data.schemas.units import UnitField
6667
from flexmeasures.data.services.sensors import get_sensor_stats
68+
from flexmeasures.data.services.sensors import delete_sensor as delete_sensor_and_data
6769
from flexmeasures.data.services.scheduling import (
6870
create_scheduling_job,
6971
get_data_source_for_job,
@@ -588,7 +590,8 @@ def get_data(self, id: int, **sensor_data_description: dict):
588590
- "resolution" (read [the docs about frequency and resolutions](https://flexmeasures.readthedocs.io/latest/api/notation.html#frequency-and-resolution))
589591
- "horizon" (read [the docs about belief timing](https://flexmeasures.readthedocs.io/latest/api/notation.html#tracking-the-recording-time-of-beliefs))
590592
- "prior" (the belief timing docs also apply here)
591-
- "source" (read [the docs about sources](https://flexmeasures.readthedocs.io/latest/api/notation.html#sources))
593+
- "source" (filter by data source ID, read [the docs about sources](https://flexmeasures.readthedocs.io/latest/api/notation.html#sources))
594+
- "account" (filter by the account ID linked to data sources)
592595
593596
An example query to fetch data for sensor with ID=1, for one hour starting June 7th 2021 at midnight, in 15 minute intervals, in m³/h:
594597
@@ -605,7 +608,7 @@ def get_data(self, id: int, **sensor_data_description: dict):
605608
required: true
606609
schema: SensorId
607610
- in: query
608-
schema: SensorDataTimingDescriptionSchema
611+
schema: GetSensorDataQuerySchema
609612
610613
responses:
611614
200:
@@ -1348,19 +1351,8 @@ def delete(self, id: int, sensor: Sensor):
13481351
- Sensors
13491352
"""
13501353

1351-
"""Delete time series data."""
1352-
db.session.execute(delete(TimedBelief).filter_by(sensor_id=sensor.id))
1353-
1354-
AssetAuditLog.add_record(
1355-
sensor.generic_asset, f"Deleted sensor '{sensor.name}': {sensor.id}"
1356-
)
1357-
13581354
sensor_name = sensor.name
1359-
AssetAuditLog.add_record(
1360-
sensor.generic_asset,
1361-
f"Deleted sensor '{sensor_name}': {id}",
1362-
)
1363-
db.session.execute(delete(Sensor).filter_by(id=sensor.id))
1355+
delete_sensor_and_data(sensor)
13641356
db.session.commit()
13651357
current_app.logger.info("Deleted sensor '%s'." % sensor_name)
13661358
return {}, 204

flexmeasures/api/v3_0/tests/conftest.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,9 @@
1313
from flexmeasures.data.models.generic_assets import GenericAssetType, GenericAsset
1414
from flexmeasures.data.models.time_series import TimedBelief
1515

16+
# 10-minute resolution values for "some gas sensor"
17+
GAS_MEASUREMENTS_10MIN = [91.3, 91.7, 92.1]
18+
1619

1720
@pytest.fixture(scope="module")
1821
def setup_api_test_data(
@@ -203,7 +206,7 @@ def add_gas_measurements(db, source: Source, sensor: Sensor, values=None):
203206
pd.Timestamp("2021-05-02T00:00:00+02:00") + timedelta(minutes=minutes)
204207
for minutes in range(0, 30, 10)
205208
]
206-
event_values = list(values) if values else [91.3, 91.7, 92.1]
209+
event_values = list(values) if values else GAS_MEASUREMENTS_10MIN
207210
beliefs = [
208211
TimedBelief(
209212
sensor=sensor,

flexmeasures/api/v3_0/tests/test_sensor_data.py

Lines changed: 55 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
from sqlalchemy.engine import Engine
88

99
from flexmeasures import Sensor, Source, User
10+
from flexmeasures.api.v3_0.tests.conftest import GAS_MEASUREMENTS_10MIN
1011
from flexmeasures.api.v3_0.tests.utils import make_sensor_data_request_for_gas_sensor
1112

1213

@@ -73,9 +74,17 @@ def test_get_sensor_data(
7374
print("Server responded with:\n%s" % response.json)
7475
assert response.status_code == 200
7576
values = response.json["values"]
76-
# We expect two data points (from conftest) followed by 2 null values (which are converted to None by .json)
77-
# The first data point averages [91.3, 91.7], and the second data point averages [92.1, None].
78-
assert all(a == b for a, b in zip(values, [91.5, 92.1, None, None]))
77+
# GAS_MEASUREMENTS_10MIN stores 10-minute values; resampled to 20-minute resolution:
78+
# - 1st interval: average of [91.3, 91.7] = 91.5
79+
# - 2nd interval: average of [92.1, None] = 92.1 (only one value present)
80+
# - 3rd and 4th intervals: no data → None
81+
expected = [
82+
sum(GAS_MEASUREMENTS_10MIN[0:2]) / 2, # 91.5
83+
GAS_MEASUREMENTS_10MIN[2], # 92.1
84+
None,
85+
None,
86+
]
87+
assert all(a == b for a, b in zip(values, expected))
7988

8089

8190
@pytest.mark.parametrize(
@@ -114,6 +123,49 @@ def test_get_instantaneous_sensor_data(
114123
assert all(a == b for a, b in zip(values, [815, 818, None, None]))
115124

116125

126+
@pytest.mark.parametrize(
127+
"requesting_user", ["[email protected]"], indirect=True
128+
)
129+
def test_get_sensor_data_filtered_by_source_account(
130+
client,
131+
setup_api_test_data: dict[str, Sensor],
132+
setup_roles_users: dict[str, User],
133+
requesting_user,
134+
db,
135+
):
136+
"""Check that GET /sensors/<id>/data can filter by the account linked to a source."""
137+
sensor = setup_api_test_data["some gas sensor"]
138+
source_user = db.session.get(User, setup_roles_users["Test Supplier User"])
139+
assert source_user.account_id is not None
140+
message = {
141+
"start": "2021-05-02T00:00:00+02:00",
142+
"duration": "PT1H20M",
143+
"horizon": "PT0H",
144+
"unit": "m³/h",
145+
"account": source_user.account_id,
146+
"resolution": "PT20M",
147+
}
148+
response = client.get(
149+
url_for("SensorAPI:get_data", id=sensor.id),
150+
query_string=message,
151+
)
152+
print("Server responded with:\n%s" % response.json)
153+
assert response.status_code == 200
154+
values = response.json["values"]
155+
# The fixture also stores data from an accountless "Other source".
156+
# Filtering by the user account should exclude those points.
157+
# GAS_MEASUREMENTS_10MIN stores 10-minute values; resampled to 20-minute resolution:
158+
# - 1st interval: average of [91.3, 91.7] = 91.5
159+
# - 2nd interval: average of [92.1, None] = 92.1 (only one value present)
160+
expected = [
161+
sum(GAS_MEASUREMENTS_10MIN[0:2]) / 2, # 91.5
162+
GAS_MEASUREMENTS_10MIN[2], # 92.1
163+
None,
164+
None,
165+
]
166+
assert all(a == b for a, b in zip(values, expected))
167+
168+
117169
@pytest.mark.parametrize(
118170
"requesting_user, status_code",
119171
[

flexmeasures/api/v3_0/tests/test_sensors_api.py

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
import pytest
44
import math
55
import io
6+
import json
67

78
from flask import url_for
89
from sqlalchemy import select, func
@@ -536,6 +537,29 @@ def test_delete_a_sensor(client, setup_api_test_data, requesting_user, db):
536537
existing_sensor_id = existing_sensor.id
537538
sensor_count = db.session.scalar(select(func.count()).select_from(Sensor))
538539

540+
asset = existing_sensor.generic_asset
541+
asset.flex_model = {
542+
"soc-max": {"sensor": existing_sensor_id},
543+
"static-limit": "10 kW",
544+
}
545+
asset.flex_context = {
546+
"consumption-price": {"sensor": existing_sensor_id},
547+
"inflexible-device-sensors": [existing_sensor_id],
548+
}
549+
asset.sensors_to_show = [
550+
{"title": "Power", "plots": [{"sensor": existing_sensor_id}]},
551+
existing_sensor_id,
552+
]
553+
asset.sensors_to_show_as_kpis = [
554+
{
555+
"sensor": existing_sensor_id,
556+
"title": "Temperature KPI",
557+
"function": "sum",
558+
}
559+
]
560+
db.session.add(asset)
561+
db.session.commit()
562+
539563
delete_sensor_response = client.delete(
540564
url_for("SensorAPI:delete", id=existing_sensor_id),
541565
)
@@ -552,12 +576,46 @@ def test_delete_a_sensor(client, setup_api_test_data, requesting_user, db):
552576
db.session.scalar(select(func.count()).select_from(Sensor)) == sensor_count - 1
553577
)
554578

579+
asset_after = db.session.get(GenericAsset, asset.id)
580+
assert asset_after.flex_model.get("soc-max") is None
581+
assert asset_after.flex_model.get("static-limit") == "10 kW"
582+
assert asset_after.flex_context.get("consumption-price") is None
583+
assert asset_after.flex_context.get("inflexible-device-sensors") == []
584+
assert str(existing_sensor_id) not in json.dumps(asset_after.sensors_to_show)
585+
assert str(existing_sensor_id) not in json.dumps(
586+
asset_after.sensors_to_show_as_kpis
587+
)
588+
555589
check_audit_log_event(
556590
db=db,
557591
event=f"Deleted sensor '{existing_sensor.name}': {existing_sensor.id}",
558592
user=requesting_user,
559593
asset=existing_sensor.generic_asset,
560594
)
595+
check_audit_log_event(
596+
db=db,
597+
event=f"Removed sensor reference '{existing_sensor.name}': {existing_sensor.id} from flex-model (because sensor has been deleted).",
598+
user=requesting_user,
599+
asset=existing_sensor.generic_asset,
600+
)
601+
check_audit_log_event(
602+
db=db,
603+
event=f"Removed sensor reference '{existing_sensor.name}': {existing_sensor.id} from flex-context (because sensor has been deleted).",
604+
user=requesting_user,
605+
asset=existing_sensor.generic_asset,
606+
)
607+
check_audit_log_event(
608+
db=db,
609+
event=f"Removed sensor reference '{existing_sensor.name}': {existing_sensor.id} from sensors-to-show (because sensor has been deleted).",
610+
user=requesting_user,
611+
asset=existing_sensor.generic_asset,
612+
)
613+
check_audit_log_event(
614+
db=db,
615+
event=f"Removed sensor reference '{existing_sensor.name}': {existing_sensor.id} from sensors-to-show-as-kpis (because sensor has been deleted).",
616+
user=requesting_user,
617+
asset=existing_sensor.generic_asset,
618+
)
561619

562620

563621
@pytest.mark.parametrize(

0 commit comments

Comments
 (0)