Support coupling data sources to accounts, and preserve user ID and account ID references in audit logs and data sources by Copilot · Pull Request #2058 · FlexMeasures/flexmeasures

Copilot · 2026-03-24T22:04:42Z

Description

Add account_id to data_source table, with migration to populate based on data_source.user_id.
Set account_id upon POSTing data via the API.
Extend flexmeasures add source with the capability of assigning the new data source to an account.
Preserve user IDs and account IDs in audit logs and data sources upon deletion.
Added changelog item in documentation/changelog.rst.
Updated various agent instructions.

Look & Feel

...

How to test

New and adapted tests:

pytest -k test_post_sensor_data_sets_account_id_on_data_source
pytest -k test_data_delete
pytest -k test_delete_user

Manual tests of db migration (assuming account ID 3 is created):

To upgrade/downgrade, run

flexmeasures db upgrade
flexmeasures db downgrade 8b62f8129f34

To create/delete account:

flexmeasures add account --name DeleteMe
flexmeasures delete account --id 3

To create/delete user:

flexmeasures add user --account 3 --email deleteme@flexmeasures.io --roles admin --username deleteme
flexmeasures delete user --email deleteme@flexmeasures.io

To create a data source tied to an account:

flexmeasures add source --name delme --type forecaster --account 3

To inspect data sources and audit log:

select * from data_source;
select * from audit_log;

Further Improvements

...

Related Items

Closes #2047.

read-the-docs-community · 2026-03-24T22:08:59Z

Documentation build overview

📚 flexmeasures | 🛠️ Build #32097903 | 📁 Comparing 4724959 against latest (1ce87c1)

🔍 Preview build

Show files changed (3 files in total): 📝 3 modified | ➕ 0 added | ➖ 0 deleted

File	Status
changelog.html	📝 modified
_autosummary/flexmeasures.data.services.data_sources.html	📝 modified
api/v3_0.html	📝 modified

- Add account_id column (FK to account.id, nullable) to DataSource - Add account relationship (backref 'data_sources' on Account) - Update __table_args__ UniqueConstraint to include account_id - Set self.account from user.account in __init__ when user is provided

- Add account_id column (nullable, FK to account.id) - Data migration: populate account_id from user's account where user_id IS NOT NULL - Update UniqueConstraint to include account_id column - down_revision: 8b62f8129f34

…backfill

…ng sensor data Context: - DataSource.__init__ now sets self.account = user.account when a user is provided - The POST sensor data endpoint calls get_or_create_source(current_user) - Need to verify account_id is populated correctly on the resulting data source Change: - Added test_post_sensor_data_sets_account_id_on_data_source - Posts sensor data as test_supplier_user_4@seita.nl - Asserts data_source.account_id == user.account_id

… changelog entry Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/7969297e-88c2-4ace-99c9-ec74109f03c8

… unflushed users Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/7969297e-88c2-4ace-99c9-ec74109f03c8

…d invariants Context: - PR #2058 added account_id FK to DataSource table - No migration checklist existed for reviewing/writing Alembic migrations - New domain invariant (user-type DataSources now have account_id) was undocumented - Agent failed to self-update during the PR session Change: - Added Alembic migration checklist with safe backfill pattern - Added DataSource entity to Domain Knowledge with account_id invariant - Documented invariants #5 and #6 for DataSource UniqueConstraint and account_id - Added Lessons Learned section with PR #2058 case study

…essons Context: - PR #2058 added a test for account_id on DataSource after API POST - No guidance existed for testing data source properties after API calls - Agent failed to self-update during the PR session Change: - Added 'Testing DataSource Properties After API Calls' section with pattern - Added Lessons Learned section documenting PR #2058 self-improvement failure

…Coordinator failure Context: - PR #2058 session: Coordinator was not invoked (3rd session in a row) - API Specialist was not engaged despite endpoint behavior change - No agents updated their instructions (same failure as 2026-02-06) Change: - Added 'Agent Selection Checklist' mapping code change types to required agents - Documented PR #2058 as lessons learned (3 distinct failures) - Reinforced that Coordinator must ALWAYS be last agent in every session - Clarified that endpoint behavior changes → API Specialist must be engaged

…lure pattern Context: - PR #2058 repeated the same 3 failures from sessions 2026-02-06 and 2026-02-08 - Coordinator not invoked, no agent self-updates, API Specialist not engaged - Same failures now documented 3 times in Review Lead instructions with no change in behavior Change: - Added 'Additional Pattern Discovered (2026-03-24)' with root cause analysis - Documented missed API Specialist engagement for endpoint behavior change - Flagged governance escalation if Coordinator invocation fails a 4th time - Noted code observation: or-pattern for account_id and empty Initial plan commit

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/7969297e-88c2-4ace-99c9-ec74109f03c8

…a data source before POSTing sensor data Signed-off-by: F.N. Claessen <claessen@seita.nl>

…l time (UTC+1) Signed-off-by: F.N. Claessen <claessen@seita.nl>

Signed-off-by: F.N. Claessen <claessen@seita.nl>

Flix6x

Done:

Code review
Agent review + improvements
Test review + improvements

To do:

Test db migration upgrade + downgrade
Failing test for deleting an account via the CLI: disallow, orphan or cascade?
- Opened discussion in #2047 (comment)
- Set copilot to implement orphaning the data source from the user and account (to test out consequences)

To consider:

Let documentation expert review, too?

…lineage preservation Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/004f0bb9-133d-45c1-9e06-2fedbf25b258

…n delete tests Context: - PR #2058 dropped FK constraints on DataSource.user_id and account_id to preserve lineage when users/accounts are deleted - The new tests only checked one of the two lineage fields per test Change: - test_delete_user: also assert data_source.account_id is preserved after user deletion (user deletion should not nullify account_id) - test_delete_account: also save and assert data_source.user_id per data source; account deletion internally calls delete_user so both user_id and account_id lineage must survive the full deletion chain

…eservation Context: - The existing entry only mentioned adding account_id to data_source - PR #2058 also drops FK constraints on user_id and account_id to preserve historical references when users/accounts are deleted (data lineage) Change: - Changelog now states both the column addition AND the behavior change: 'historical user/account IDs are no longer nullified when users or accounts are deleted'

…s lesson Context: - PR #2058 dropped FK constraints on DataSource.user_id and account_id for data lineage preservation (a deliberate design decision) - The original changelog entry omitted the FK drop and behavior change - No existing agent pattern covered 'intentional FK removal for lineage' Change: - Added checklist for no-FK lineage pattern (passive_deletes, tests, changelog) - Added lesson: changelog must describe behavior changes, not just schema changes - Documented FK drop design decision for future reviewers

…plementation The note previously described the 'or' anti-pattern as a concern, but the final code already uses 'if account_id is not None' correctly. Clarified to point to the implemented solution.

Signed-off-by: F.N. Claessen <claessen@seita.nl>

…d-to-data-source

…either is deleted Signed-off-by: F.N. Claessen <claessen@seita.nl>

Signed-off-by: F.N. Claessen <claessen@seita.nl>

- Resolve conflicts in agent documentation files - Keep both DataSource and Annotation documentation sections - Accept deletion of lead.md from origin/main (governance consolidated) - Integrate annotation API implementation from main branch - All conflict markers resolved

Update all agent documentation to use 'Lead' instead of 'Review Lead', reflecting the consolidated governance model where lead.md has been removed and its responsibilities are now documented in AGENTS.md. Changes: - AGENTS.md: All 31 'Review Lead' references updated to 'Lead' - coordinator.md: Updated monitoring patterns to reference Lead - test-specialist.md: Updated coordination examples to reference Lead - tooling-ci-specialist.md: Updated coordination examples to reference Lead This ensures consistent terminology across all agent instructions.

Update b2e07f0dafa1_merge.py migration to specify both parent revisions that are being merged: - 3f4a6f9d2b11 (increase_audit_event_length_to_500) - b2c3d4e5f6a7 (Drop FK constraint from audit_log.active_user_id) This resolves the multiple heads issue and creates a proper mergepoint in the migration history. The migration now correctly shows as a (mergepoint) in the history output.

Clean up leftover git merge conflict markers (<<<<<<< HEAD, =======, >>>>>>> origin/main) that were not properly resolved. Both pattern sections (2026-03-24 and 2026-02-10) have been preserved as they document complementary learning from different sessions.

Signed-off-by: F.N. Claessen <claessen@seita.nl>

nhoening

Happy with this!

I have minor comments, some about fewer comments, one question and I believe we should merge two db migrations.

nhoening · 2026-04-02T14:12:04Z

documentation/changelog.rst


 Infrastructure / Support
 ----------------------
+* Support coupling data sources to accounts, and preserve user ID and account ID references in audit logs and data sources for traceability and compliance [see `PR #2058 <https://www.github.com/FlexMeasures/flexmeasures/pull/2058>`_]


Suggested change

* Support coupling data sources to accounts, and preserve user ID and account ID references in audit logs and data sources for traceability and compliance [see `PR #2058 <https://www.github.com/FlexMeasures/flexmeasures/pull/2058>`_]

* Support coupling data sources to accounts (enabling to query data added by any source within an account, for more robustness), and preserve user ID and account ID references in audit logs and data sources for traceability and compliance [see `PR #2058 <https://www.github.com/FlexMeasures/flexmeasures/pull/2058>`_]

nhoening · 2026-04-02T14:22:05Z

flexmeasures/cli/tests/test_data_delete.py

+        (ds.id, ds.user_id, ds.account_id) for ds in data_sources_before
+    ]
+
    # Add creation audit log record


Suggested change

# Add creation audit log record

# Add creation audit log record, as that has not automatically been done when setting up test data

nhoening · 2026-04-02T14:31:17Z

flexmeasures/data/migrations/versions/a1b2c3d4e5f6_drop_fk_constraints_from_data_source.py

+    existing_fk_names = [fk["name"] for fk in existing_fks]
+
+    with op.batch_alter_table("data_source", schema=None) as batch_op:
+        # Drop the account_id FK if it exists


This FK is only created in this PR - in revision 9877450113f6.

Remove it there and also from this one.
We then have only the removal of the user_id FK left in this revision, so I'd like to see this revision merged with revision 9877450113f6.

nhoening · 2026-04-02T14:35:06Z

flexmeasures/data/models/audit_log.py

    active_user_name = Column(String(255))
-    active_user_id = Column(
-        "active_user_id", Integer(), ForeignKey("fm_user.id", ondelete="SET NULL")
+    # No DB-level FK with cascade for active_user_id so that deleting a user preserves the lineage reference in this column.


Let's make this lengthy comment only once (not three times), for "any user_id or account_id"

nhoening · 2026-04-02T14:44:13Z

flexmeasures/data/models/audit_log.py

+    active_user = db.relationship(
+        "User",
+        primaryjoin="AuditLog.active_user_id == User.id",
+        foreign_keys="[AuditLog.active_user_id]",


I don't understand this syntax - maybe add a small explanation. Is this not a database-level FK (which this PR deleted)?

Initial plan

4b37779

Copilot AI assigned Copilot and Flix6x Mar 24, 2026

Copilot started work on behalf of Flix6x March 24, 2026 22:04 View session

Copilot AI and others added 11 commits March 24, 2026 22:10

migration: add account_id to data_source table (9877450113f6)

6745921

- Add account_id column (nullable, FK to account.id) - Data migration: populate account_id from user's account where user_id IS NOT NULL - Update UniqueConstraint to include account_id column - down_revision: 8b62f8129f34

migration: replace N+1 loop with single UPDATE...FROM for account_id …

78fb04f

…backfill

migration: use correlated subquery for account_id data migration; add…

85778e4

… changelog entry Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/7969297e-88c2-4ace-99c9-ec74109f03c8

data_sources: fix account_id assignment to avoid SAWarning and handle…

06bbec9

… unflushed users Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/7969297e-88c2-4ace-99c9-ec74109f03c8

data_sources: use explicit is not None check for account_id assignment

4191ae3

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/7969297e-88c2-4ace-99c9-ec74109f03c8

Copilot AI changed the title ~~[WIP] Add account_id field to data_source table~~ Add account_id to data_source table Mar 24, 2026

Copilot AI requested a review from Flix6x March 24, 2026 22:33

Copilot finished work on behalf of Flix6x March 24, 2026 22:33

Flix6x added 4 commits March 25, 2026 00:25

feat: improve test by using a user that is not already registered as …

38db656

…a data source before POSTing sensor data Signed-off-by: F.N. Claessen <claessen@seita.nl>

fix: set approximate create datetime based on git info, assuming loca…

67eba6e

…l time (UTC+1) Signed-off-by: F.N. Claessen <claessen@seita.nl>

style: flake8

b7a4a5c

Signed-off-by: F.N. Claessen <claessen@seita.nl>

feat: add migration check to architecture agent instructions

44cff24

Signed-off-by: F.N. Claessen <claessen@seita.nl>

Flix6x marked this pull request as ready for review March 24, 2026 23:36

docs: add db upgrade warning and update agent instructions accordingly

95653f1

Signed-off-by: F.N. Claessen <claessen@seita.nl>

Flix6x approved these changes Mar 24, 2026

View reviewed changes

Copilot started work on behalf of Flix6x March 25, 2026 02:44 View session

Copilot AI requested a review from Flix6x March 25, 2026 02:44

Copilot stopped work on behalf of Flix6x due to an error March 25, 2026 02:44
The session was cancelled by the user.

Copilot started work on behalf of Flix6x March 25, 2026 02:48 View session

Copilot AI and others added 5 commits March 25, 2026 03:09

data_sources: drop FK constraints on user_id and account_id for data …

4e22e43

…lineage preservation Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/004f0bb9-133d-45c1-9e06-2fedbf25b258

coordinator: clarify account_id fallback pattern references actual im…

ebfba1d

…plementation The note previously described the 'or' anti-pattern as a concern, but the final code already uses 'if account_id is not None' correctly. Clarified to point to the implemented solution.

Copilot AI changed the title ~~Add account_id to data_source table~~ data_source: drop FK constraints on user_id/account_id to preserve lineage on deletion Mar 25, 2026

Copilot finished work on behalf of Flix6x March 25, 2026 03:33

Flix6x added 4 commits March 27, 2026 13:38

feat: add test covering CLI command to delete user

c64b4ee

Signed-off-by: F.N. Claessen <claessen@seita.nl>

Merge remote-tracking branch 'origin/main' into copilot/add-account-i…

531cb74

…d-to-data-source

feat: let audit log entries retain the account ID and user IDs after …

57c1f8e

…either is deleted Signed-off-by: F.N. Claessen <claessen@seita.nl>

docs: rewrite changelog entry

e1d6cdc

Signed-off-by: F.N. Claessen <claessen@seita.nl>

Flix6x changed the title ~~data_source: drop FK constraints on user_id/account_id to preserve lineage on deletion~~ Support coupling data sources to accounts, and preserve user ID and account ID references in audit logs and data sources Mar 27, 2026

fix: dropped foreign key name

7ec2cb1

Signed-off-by: F.N. Claessen <claessen@seita.nl>

Flix6x added this to the 0.32.0 milestone Mar 27, 2026

Flix6x added 10 commits March 27, 2026 15:27

fix: when deleting a user, still log the ID of the deleted user

0fefb2e

Signed-off-by: F.N. Claessen <claessen@seita.nl>

feat: allow tying a newly CLI-created data source to an account

6bcf9c6

Signed-off-by: F.N. Claessen <claessen@seita.nl>

fix: add missing drop_constraints

faa876f

Signed-off-by: F.N. Claessen <claessen@seita.nl>

chore: minimize diff

317c687

Signed-off-by: F.N. Claessen <claessen@seita.nl>

style: flake8

4401355

Signed-off-by: F.N. Claessen <claessen@seita.nl>

fix: test

d7a4ec1

Signed-off-by: F.N. Claessen <claessen@seita.nl>

Flix6x added the Data label Apr 2, 2026

Flix6x approved these changes Apr 2, 2026

View reviewed changes

style: flake8

4724959

Signed-off-by: F.N. Claessen <claessen@seita.nl>

Flix6x requested a review from nhoening April 2, 2026 13:38

nhoening requested changes Apr 2, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support coupling data sources to accounts, and preserve user ID and account ID references in audit logs and data sources#2058

Support coupling data sources to accounts, and preserve user ID and account ID references in audit logs and data sources#2058
Copilot wants to merge 38 commits intomainfrom
copilot/add-account-id-to-data-source

Copilot AI commented Mar 24, 2026 •

edited by Flix6x

Loading

Uh oh!

read-the-docs-community bot commented Mar 24, 2026 •

edited

Loading

Uh oh!

Flix6x left a comment •

edited

Loading

Uh oh!

nhoening left a comment

Uh oh!

nhoening Apr 2, 2026

Uh oh!

nhoening Apr 2, 2026

Uh oh!

nhoening Apr 2, 2026

Uh oh!

nhoening Apr 2, 2026

Uh oh!

nhoening Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	* Support coupling data sources to accounts, and preserve user ID and account ID references in audit logs and data sources for traceability and compliance [see `PR #2058 <https://www.github.com/FlexMeasures/flexmeasures/pull/2058>`_]
	* Support coupling data sources to accounts (enabling to query data added by any source within an account, for more robustness), and preserve user ID and account ID references in audit logs and data sources for traceability and compliance [see `PR #2058 <https://www.github.com/FlexMeasures/flexmeasures/pull/2058>`_]

	# Add creation audit log record
	# Add creation audit log record, as that has not automatically been done when setting up test data

Conversation

Copilot AI commented Mar 24, 2026 • edited by Flix6x Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Look & Feel

How to test

Further Improvements

Related Items

Uh oh!

read-the-docs-community bot commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

Flix6x left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nhoening left a comment

Choose a reason for hiding this comment

Uh oh!

nhoening Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

nhoening Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

nhoening Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

nhoening Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

nhoening Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Mar 24, 2026 •

edited by Flix6x

Loading

read-the-docs-community bot commented Mar 24, 2026 •

edited

Loading

Flix6x left a comment •

edited

Loading