Merge branch 'main' into 306_callbacks_on_change

Author: shihab-dls

docs/explanations/decisions/0003-controller-initialization-on-main-event-loop.md

@@ -0,0 +1,42 @@
+# 3. Controller Initialization on Main Event Loop
+
+Date: 2024-08-01
+
+**Related:** [PR #49](https://github.com/DiamondLightSource/FastCS/pull/49)
+
+## Status
+
+Accepted
+
+## Context
+
+The Backend accepts a pre-created Mapping with no async initialization hook for controllers. Each backend subclass manually orchestrates initialization steps, leading to inconsistent lifecycle patterns. Controllers need a way to perform async setup before their API is exposed to transports.
+
+## Decision
+
+Move initialisation logic into Backend so that it:
+- Creates the event loop
+- Runs `controller.initialise()` before creating the Mapping
+- Creates the Mapping from the initialized controller
+- Runs initial tasks including `controller.connect()`
+
+Controller then has two hooks: `initialise()` for pre-API setup (hardware introspection, dynamic attribute creation) and `connect()` for post-API connection logic.
+
+## Consequences
+
+The new design the initialisation of the application is easier to understand and makes the API for writing controllers and transports simpler and more flexible.
+
+### Migration Pattern
+
+For controller developers:
+
+```python
+class MyController(Controller):
+    async def initialise(self) -> None:
+        # Async setup before API creation (introspect hardware, create attributes)
+        await self._hardware.connect()
+
+    async def connect(self) -> None:
+        # Async setup after API creation (establish connections)
+        await self._hardware.initialize()
+```

docs/explanations/decisions/0004-backend-to-transport-refactoring.md

@@ -0,0 +1,55 @@
+# 4. Rename Backend to Transport
+
+Date: 2024-11-29
+
+**Related:** [PR #67](https://github.com/DiamondLightSource/FastCS/pull/67)
+
+## Status
+
+Accepted
+
+## Context
+
+The original FastCS architecture used the term "backend" ambiguously to describe both the overall framework that managed controllers and the specific communication protocol implementations (EPICS CA, PVA, REST, Tango). This dual usage created confusion:
+
+- It was unclear whether "backend" referred to the framework itself or the protocol layer
+- The terminology didn't clearly differentiate between the abstract framework and the underlying communication mechanisms
+- The inheritance pattern made it difficult to compose multiple transports or swap them dynamically
+
+## Decision
+
+Rename "backend" to "transport" for all protocol/communication implementations to clearly differentiate them from the framework.
+
+The term "backend" can now refer to the overall FastCS framework/system, while "transport" specifically refers to protocol implementations (EPICS CA, PVA, REST, GraphQL, Tango). FastCS accepts Transport implementations as plugins, enabling flexible composition and loose coupling.
+
+Key architectural changes:
+- Introduce `TransportAdapter` abstract base class with standardized interface
+- Move to composition-based architecture where transports are passed to `FastCS` rather than being subclasses
+- Introduce `FastCS` class as the programmatic interface for running controllers with transports
+- Add `launch()` function as the primary entry point for initializing controllers
+
+## Consequences
+
+### Benefits
+
+- **Clear Terminology:** The separation between framework (backend) and protocol layer (transport) is now explicit
+- **Consistent Architecture:** All transports follow the adapter pattern with a standardized interface
+- **Flexible Composition:** Transports can be added, removed, or swapped at runtime
+- **Improved Extensibility:** Adding new transport protocols is straightforward with the adapter pattern
+
+### Migration Pattern
+
+**Before (Inheritance hierarchy):**
+```python
+class EpicsBackend(Backend):
+    def run(self):
+        # Protocol-specific implementation
+
+fastcs = FastCS(controller)  # Tightly coupled to framework
+```
+
+**After (Composition with Transport plugins):**
+```python
+transport = EpicsTransport(controller_api)
+fastcs = FastCS(controller, [transport])
+```

docs/explanations/decisions/0005-background-thread-removal.md

@@ -0,0 +1,32 @@
+# 5. Remove Background Thread from Backend
+
+Date: 2025-01-24
+
+**Related:** [PR #98](https://github.com/DiamondLightSource/FastCS/pull/98)
+
+## Status
+
+Accepted
+
+## Context
+
+The FastCS Backend implementation uses `asyncio.run_coroutine_threadsafe()` to execute controller operations on a background event loop thread managed by `AsyncioDispatcher`. The system needs a simpler concurrency model that uses native async/await patterns and allows transports to manage their own threading if needed.
+
+## Decision
+
+Remove the background thread from Backend, making it fully async, while allowing specific transports to use a background thread if required. Backend should accept an event loop from the caller and use native async/await throughout. Transports that need threading (like Tango) manage their own threading explicitly.
+
+Key architectural changes:
+- Backend receives event loop from caller (no background dispatcher)
+- Initialization uses `loop.run_until_complete()` instead of cross-thread scheduling
+- Backend exposes `async serve()` method using native async/await patterns
+- Scan tasks use `Task` objects from `loop.create_task()` instead of `Future` objects
+- Transports that need threading create their own `AsyncioDispatcher` when needed
+
+## Consequences
+
+### Benefits
+
+- **Simpler Concurrency Model:** Single event loop for most operations, no cross-thread coordination needed
+- **Task-Based API:** Consistent use of `Task` objects with standard cancellation support
+- **Composability:** `async serve()` can be composed with other async operations

docs/explanations/decisions/0006-controller-api-abstraction-layer.md

@@ -0,0 +1,67 @@
+# 6. Create ControllerAPI Abstraction Layer
+
+Date: 2025-03-10
+
+**Related:** [PR #87](https://github.com/DiamondLightSource/FastCS/pull/87)
+
+## Status
+
+Accepted
+
+## Context
+
+Transports currently access `Controller` instances directly to extract attributes, methods, and metadata for serving over their protocols. This creates a few problems:
+
+- **Tight Coupling:** Transports are coupled to internal Controller structure, making evolution difficult
+- **Code Duplication:** Every transport re-implemented similar traversal logic for discovering attributes and methods
+- **No Encapsulation:** Transports have direct access to mutable controller state
+- **No Static View:** No complete, immutable snapshot of controller API after initialization
+
+## Decision
+
+Introduce `ControllerAPI` as an abstraction layer that provides transports with a complete, static, read-only representation of a controller's capabilities after initialization.
+
+All transports now work with `ControllerAPI` instead of direct `Controller` access. A single `create_controller_api()` function handles all API extraction, replaces custom traversal logic in each transport.
+
+Key architectural changes:
+- `ControllerAPI` dataclass represents the complete, hierarchical structure of what a controller exposes
+- Separate dictionaries for attributes, command_methods, put_methods, and scan_methods
+- `walk_api()` method provides depth-first traversal of the API tree
+- Backend creates ControllerAPI during initialization and passes to transports
+
+## Consequences
+
+### Benefits
+
+- **Encapsulation:** Transports work with read-only API, cannot modify controller internals
+- **Single Source of Truth:** One canonical representation of controller capabilities
+- **Reduced Code Duplication:** Traversal and extraction logic written once, used by all transports
+- **Separation of Concerns:** Controllers focus on device logic, ControllerAPI handles representation, transports focus on protocol
+- **Testability:** Transports can be tested with synthetic ControllerAPIs; controllers tested independently
+- **Evolution Independence:** Controller internals can change without affecting transports
+
+### Migration Pattern
+
+**Before (Direct Controller access):**
+```python
+class EpicsCAIOC:
+    def __init__(self, pv_prefix: str, controller: Controller):
+        # Each transport traverses controller itself
+        for attr_name in dir(controller):
+            attr = getattr(controller, attr_name)
+            if isinstance(attr, Attribute):
+                self._create_pv(f"{pv_prefix}{attr_name}", attr)
+```
+
+**After (ControllerAPI abstraction):**
+```python
+class EpicsCAIOC:
+    def __init__(self, pv_prefix: str, controller_api: ControllerAPI):
+        # Transport receives ready-to-use API structure
+        for attr_name, attr in controller_api.attributes.items():
+            self._create_pv(f"{pv_prefix}{attr_name}", attr)
+
+        # Walk sub-controllers using standard method
+        for sub_api in controller_api.walk_api():
+            # Process sub-controllers with consistent structure
+```

docs/explanations/decisions/0007-transport-consolidation.md

@@ -0,0 +1,66 @@
+# 7. Merge TransportAdapter and TransportOptions into Transport
+
+Date: 2025-09-29
+
+**Related:** [PR #220](https://github.com/DiamondLightSource/FastCS/pull/220)
+
+## Status
+
+Accepted
+
+## Context
+
+FastCS transports are implemented using two separate classes: `TransportAdapter` for implementation and separate `*Options` classes for configuration. This pattern requires:
+
+- Two classes per transport
+- Pattern matching logic in FastCS to create the right adapter from options
+- Inconsistent constructor signatures across transports
+- Redundant Options classes that only carry configuration data
+
+## Decision
+
+Merge `TransportAdapter` and `*Options` classes into a single `Transport` dataclass that combines configuration and implementation.
+
+All transports should follow a unified pattern: configuration fields are dataclass attributes, and `connect()` and `serve()` methods handle initialization and execution. FastCS accepts Transport instances directly.
+
+Key architectural changes:
+- All transports use `@dataclass` decorator combining configuration and implementation
+- Standardized `connect(controller_api, loop)` method to load controller API into transport
+- Standardized `async serve()` method for running the transport service
+
+## Consequences
+
+### Benefits
+
+- **Reduced API Surface:** 5 classes instead of 10 (one Transport per protocol)
+- **Simpler Mental Model:** Configuration and implementation in one place
+- **Consistent Interface:** All transports follow same initialization pattern
+
+### Migration Pattern
+
+**Before (Options + Adapter pattern):**
+```python
+# Configuration separate from implementation
+@dataclass
+class MyOptions:
+    param1: str
+    param2: int = 42
+
+class MyTransport(TransportAdapter):
+    def __init__(self, controller_api: ControllerAPI, options: MyOptions):
+        self._options = options
+        # Setup using self._options.param1
+```
+
+**After (Unified Transport):**
+```python
+# Configuration and implementation unified
+@dataclass
+class MyTransport(Transport):
+    param1: str
+    param2: int = 42
+
+    def connect(self, controller_api: ControllerAPI, loop: asyncio.AbstractEventLoop):
+        self._controller_api = controller_api
+        # Setup using self.param1 directly
+```

docs/explanations/decisions/0008-transport-dependencies-as-optional-extras.md

@@ -0,0 +1,68 @@
+# 8. Split Transport Dependencies into Optional Extras
+
+Date: 2025-09-30
+
+**Related:** [PR #221](https://github.com/DiamondLightSource/FastCS/pull/221)
+
+## Status
+
+Accepted
+
+## Context
+
+Currently all transport dependencies are installed regardless of which transports users actually needed.
+
+Problems with required dependencies:
+- Minimal installations bloated by unused transport dependencies
+- Unclear dependency relationships for each transport
+- No way to install just the core FastCS functionality
+
+## Decision
+
+Split transport dependencies into optional extras in `pyproject.toml`, allowing users to install only what they need.
+
+The core FastCS package now requires only essential dependencies (pydantic, numpy, ruamel.yaml, IPython). Each transport is available as an optional extra, with convenience groups like `[all]`, `[epics]`, and `[dev]` for common installation patterns.
+
+Key architectural changes:
+- Core dependencies: pydantic, numpy, ruamel.yaml, IPython
+- Individual transport extras: `[epicsca]`, `[epicspva]`, `[tango]`, `[graphql]`, `[rest]`
+- Convenience groups: `[epics]`, `[all]`, `[dev]`, `[demo]`
+- Each transport declares its own dependencies explicitly
+
+## Consequences
+
+### Benefits
+
+- **Minimal Core Installation:** Users can install FastCS core without transport dependencies
+- **Explicit Dependency Relationships:** Each transport declares what it needs
+- **Flexible Installation:** Users choose exactly what they need: `pip install fastcs[epicspva,rest]`
+- **Development Convenience:** `pip install fastcs[dev]` includes everything for development
+- **Clear Documentation:** Installation commands are self-documenting
+
+### Installation Patterns
+
+**Minimal (core only):**
+```bash
+pip install fastcs
+```
+
+**Single transport:**
+```bash
+pip install fastcs[epicspva]  # EPICS PVA
+pip install fastcs[rest]      # REST API
+```
+
+**Multiple transports:**
+```bash
+pip install fastcs[epics,rest]  # EPICS CA + PVA + REST
+```
+
+**All transports:**
+```bash
+pip install fastcs[all]
+```
+
+**Development:**
+```bash
+pip install fastcs[dev]
+```