PRD: migrate authenticated data flows from Electric collections to query collections

[Vijayabaskar56]

Problem Statement

The application currently relies heavily on Electric-backed TanStack DB collections for authenticated client data. In practice, the sync layer has been unreliable, produces frequent sync errors in the browser console, and adds ongoing bandwidth cost through polling-style behavior that is poorly suited to a serverless billing model. The current data flow also leaks Electric-specific contracts into routes, hooks, and workflows, such as readiness semantics, transaction confirmation behavior, and collection cache assumptions. This makes the app harder to reason about, harder to test, and harder to evolve.

From the user's perspective, the product should feel fast and reliable when creating, editing, publishing, restoring, favoriting, and browsing forms and workspaces. Data should be consistent after user actions, but the system does not need true live multi-user sync. Eventual consistency is acceptable as long as optimistic interactions remain responsive, mutation failures roll back correctly, and important workflows confirm successfully. The migration should reduce console noise, reduce network overhead, preserve core UX, and make the architecture easier to maintain.

Solution

Replace the Electric-backed collection strategy with query-backed TanStack DB read models and TanStack Start server functions as the read/write transport. Instead of one global sync mechanism, the application will use smaller purpose-built query collections and paginated server queries where appropriate. The solution will preserve optimistic UX for core mutations, use direct canonical writes or refetch-based confirmation depending on the workflow, and split read models by usage so lightweight views do not fetch full form content unnecessarily.

The migration will happen in a coexistence phase. New query-based collections and orchestration modules will be introduced alongside the existing Electric implementation, then hooks and routes will be switched incrementally. This allows the app to preserve behavior while reducing risk. Authorization semantics will be corrected as part of the migration so organization membership, not creator ownership, determines access for the target flows.

User Stories

1. As an organization member, I want to open the dashboard without background sync instability, so that I can trust the app to load my workspaces and forms consistently.
2. As an organization member, I want workspace and form listings to load without full document payloads, so that the dashboard and sidebar stay fast.
3. As an organization member, I want to create a workspace and see it appear immediately, so that the app feels responsive.
4. As an organization member, I want to create a new form and enter the editor right away, so that I can start working without waiting on background sync.
5. As an organization member, I want failed creates, updates, and deletes to roll back cleanly, so that the UI never lies about saved state.
6. As an organization member, I want the editor to load full form detail only when needed, so that routine navigation does not waste bandwidth.
7. As an organization member, I want form cards, sidebar items, and summary views to use lightweight read models, so that common screens stay efficient.
8. As an organization member, I want route redirects based on form status to remain correct even when the form is not already cached locally, so that navigation stays predictable.
9. As an organization member, I want published forms to route to submissions and draft forms to route to edit mode reliably, so that status-based flows remain intuitive.
10. As an organization member, I want to duplicate a form and immediately work on the copy, so that I can iterate quickly.
11. As an organization member, I want archiving and restoring a form to feel immediate but still be trustworthy, so that cleanup workflows remain safe.
12. As an organization member, I want favorites to toggle instantly and roll back on failure, so that small interactions still feel polished.
13. As an organization member, I want favorite state to be derived from a simple reliable model, so that favorite-related UI is easy to trust.
14. As an organization member, I want version history lists to load quickly without pulling every version's full content, so that version browsing is lightweight.
15. As an organization member, I want a version preview to load when I ask for it, so that detailed content is fetched on demand instead of preloaded unnecessarily.
16. As an organization member, I want publish, restore, and discard version actions to remain optimistic where appropriate, so that version workflows stay fast.
17. As an organization member, I want unpublished-change indicators to remain accurate after publishing, restoring, and discarding, so that I know when the draft differs from the published state.
18. As an organization member, I want local draft sync after login to remain available, so that unauthenticated local work is not lost.
19. As an organization member, I want local drafts to be removed only after cloud sync is actually confirmed, so that no draft disappears prematurely.
20. As an organization member, I want submissions tables to remain paginated and scalable, so that large forms do not force giant client-side collections.
21. As an organization member, I want submission counts and lightweight summaries to remain available in the broader UI, so that I can understand activity without loading the full submissions table.
22. As an organization member, I want fewer sync-related console errors, so that the application feels healthier and less brittle.
23. As an organization member, I want less background bandwidth usage, so that the application is cheaper and more efficient to operate.
24. As an engineer maintaining the app, I want data access semantics to be explicit instead of hidden behind Electric-specific readiness behavior, so that the app is easier to reason about.
25. As an engineer maintaining the app, I want route logic to depend on stable read interfaces instead of collection internals, so that navigation code is easier to test.
26. As an engineer maintaining the app, I want write orchestration centralized, so that optimistic behavior and reconciliation stay consistent across screens.
27. As an engineer maintaining the app, I want version workflows encapsulated in a dedicated module, so that publishing and restore semantics do not leak across components.
28. As an engineer maintaining the app, I want authorization semantics aligned across reads and writes, so that membership-based access behaves consistently.
29. As an engineer maintaining the app, I want a coexistence rollout path, so that I can migrate incrementally without destabilizing the app.
30. As an engineer maintaining the app, I want behavior-oriented tests around the new data modules, so that the migration is safe to evolve after launch.

Implementation Decisions

• The migration will use a coexistence rollout instead of a big-bang replacement.
• The migration will correct authorization semantics as part of the work, not as a separate future effort.
• Organization membership will be treated as the write/read boundary for this phase; stricter role-based controls may be added later.
• Query-backed TanStack DB collections will replace Electric-backed collections for most authenticated read models.
• TanStack Start server functions will be the read transport for query-backed collections rather than introducing raw REST endpoints.
• Form reads will be split into at least two read models: a lightweight listing model and a full detail model.
• Workspace reads may include a richer summary model that includes lightweight form information when beneficial for dashboard/sidebar experiences.
• Favorites will remain a small dedicated model, with richer UI data derived from other already-loaded form read models.
• Form version data will be split into a lightweight version list model and an on-demand version content fetch path.
• Large submissions tables will remain paginated server-query driven rather than being fully mirrored into a general-purpose collection.
• Lightweight submission summaries or counts may exist separately from the paginated submissions table flow.
• Route decisions that currently depend on synchronous collection cache inspection will move to an explicit read strategy: cached query-backed value when available, otherwise server-backed fetch, otherwise a loading state.
• Optimistic mutation behavior will be preserved, but confirmation strategy will vary by workflow.
• Canonical direct writes may be used for forms, workspaces, and favorites when the server returns sufficient authoritative data.
• Refetch-based confirmation will be used for version-sensitive flows and local-draft-to-cloud sync where stronger post-write confidence is needed.
• Local drafts will only be deleted after server success and subsequent read-model confirmation.
• Electric-specific readiness and transaction-confirmation assumptions will be removed from routes, hooks, and orchestration code.
• The architecture will center on a small number of deep modules:
  • an authenticated read model module
  • a form version workflow module
  • a client command/orchestration module
  • an authorization access module or revised helper surface
• Query collection adapters and route-level consumers should remain thin wrappers around those deeper modules.
• Existing screens should preserve user-visible behavior as much as possible while moving away from Electric-specific contracts.

Testing Decisions

• Good tests should verify external behavior and business guarantees, not storage internals or adapter implementation details.
• Tests should focus on observable outcomes such as returned read-model shape, route decision inputs/outputs, optimistic state transitions, rollback behavior, and post-mutation confirmation behavior.
• Tests should follow the TanStack DB testing guidance from the official testing guide, especially:
  • using isolated collections per test
  • using mock sync behavior or controlled collection setup
  • using behavior-focused assertions such as toMatchObject
  • waiting for readiness explicitly where needed
• Broad behavior coverage is desired for the first pass.
• The primary modules to test are:
  • the authenticated read model
  • the form version workflow
  • the client command layer
  • authorization helper/access behavior
• Thin adapter modules should only receive tests where they contain meaningful logic beyond wiring.
• Prior art in the repository is limited, with existing lightweight Vitest coverage around utility behavior; new tests should establish stronger behavioral patterns for data modules.
• Tests should validate optimistic create/update/delete behavior, rollback on mutation failure, fallback-from-cache-to-server behavior, version workflow semantics, and local-draft sync confirmation behavior.
• Tests should avoid depending on Electric-specific implementation details, since the goal is to validate the target architecture rather than the old transport.

Out of Scope

• Reintroducing real-time multi-user synchronization.
• Designing a role-based authorization model beyond the broad membership semantics chosen for this phase.
• Reworking the public form delivery path unless it is directly affected by shared module changes.
• Large schema redesigns unrelated to the migration.
• Perfect parity of every internal Electric-specific utility if the user-visible workflow can be preserved through simpler mechanisms.
• Full removal of every Electric artifact in the earliest coexistence step; cleanup happens after functional parity is reached.
• Unrelated UI redesigns or interaction rewrites outside the data-flow migration.

Further Notes

• The strongest architectural risk is the mismatch between current Electric read visibility and current creator-based server authorization behavior.
• The most important migration fan-out point is the hook layer that currently centralizes live collection queries.
• The current route layer relies on cache-first behavior for form-status redirects, so preserving route correctness during coexistence is critical.
• The local-to-cloud draft sync flow currently relies on Electric transaction confirmation semantics and will need an explicit replacement strategy.
• The migration should prioritize reliability, lower network overhead, and maintainable module boundaries over preserving every old internal mechanism.