feat(virtio): add generic vhost-user frontend device

CHANGELOG.md

@@ -10,6 +10,11 @@ and this project adheres to
 
 ### Added
 
+- [#5687](https://github.com/firecracker-microvm/firecracker/issues/5687): Add
+  generic vhost-user frontend device support. This allows attaching any virtio
+  device type (e.g. virtio-fs, virtio-scsi) via the vhost-user protocol without
+  requiring a dedicated Firecracker frontend for each device type. Configured
+  via the `/vhost-user-devices/{id}` API endpoint.
 - [#5323](https://github.com/firecracker-microvm/firecracker/pull/5323): Add
   support for Vsock Unix domain socket path overriding on snapshot restore. More
   information can be found in the

docs/vhost-user.md

@@ -0,0 +1,102 @@
+# Using generic vhost-user devices
+
+## What is a vhost-user device
+
+The [vhost-user protocol](https://qemu-project.gitlab.io/qemu/interop/vhost-user.html)
+allows virtio device emulation to be offloaded to a separate backend
+process communicating over a Unix domain socket. The backend handles the
+actual device logic while Firecracker acts as the frontend, managing
+virtqueues and guest memory.
+
+A generic vhost-user frontend knows nothing about the specific virtio
+device type being implemented. The backend is fully responsible for the
+device configuration space. This allows using device types that
+Firecracker would never support natively (e.g. virtio-fs, virtio-scsi)
+without requiring a dedicated frontend for each.
+
+## Prerequisites
+
+- The vhost-user backend process must be running and listening on the
+  configured Unix domain socket **before** configuring the device in
+  Firecracker.
+- The backend must support the `VHOST_USER_PROTOCOL_F_CONFIG` protocol
+  feature, as Firecracker relies on the backend to provide the device
+  configuration space.
+- The guest kernel must include the driver for the virtio device type
+  being emulated (e.g. `CONFIG_VIRTIO_FS=y` for virtio-fs).
+
+## Configuration
+
+The following options are available:
+
+- `id` - unique identifier of the device.
+- `device_type` - the virtio device type ID as defined in the
+  [virtio specification](https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html#x1-1930005).
+  For example: `26` for virtio-fs, `8` for virtio-scsi.
+- `socket` - path to the vhost-user backend Unix domain socket.
+- `num_queues` - number of virtqueues to configure for this device.
+- `queue_size` (optional) - size of each virtqueue. Defaults to 256.
+
+### Config file
+
+```json
+"vhost-user-devices": [
+    {
+        "id": "fs0",
+        "device_type": 26,
+        "socket": "/tmp/virtiofsd.sock",
+        "num_queues": 1,
+        "queue_size": 256
+    }
+]
+```
+
+### API
+
+```console
+curl --unix-socket $socket_location -i \
+    -X PUT 'http://localhost/vhost-user-devices/fs0' \
+    -H 'Accept: application/json' \
+    -H 'Content-Type: application/json' \
+    -d "{
+         \"id\": \"fs0\",
+         \"device_type\": 26,
+         \"socket\": \"/tmp/virtiofsd.sock\",
+         \"num_queues\": 1,
+         \"queue_size\": 256
+    }"
+```
+
+## Example: virtio-fs with virtiofsd
+
+Start the [virtiofsd](https://gitlab.com/virtio-fs/virtiofsd) backend:
+
+```console
+virtiofsd \
+    --socket-path=/tmp/virtiofsd.sock \
+    --shared-dir=/path/to/shared \
+    --tag=myfs
+```
+
+> [!NOTE]
+>
+> The `--tag` flag is required to enable the `VHOST_USER_PROTOCOL_F_CONFIG`
+> protocol feature in virtiofsd.
+
+Then configure the device in Firecracker as shown above. Inside the
+guest, mount the shared directory:
+
+```console
+mount -t virtiofs myfs /mnt
+```
+
+## Limitations
+
+- **Snapshotting is not supported.** Creating or restoring snapshots of
+  a VM with generic vhost-user devices will fail.
+- **Configuration space writes are not yet forwarded** to the backend
+  via `VHOST_USER_SET_CONFIG`.
+- **The backend must be started before the device is attached.**
+  Firecracker connects to the socket when processing the
+  `PUT /vhost-user-devices/{id}` request and will return an error if the
+  backend is not available.

src/firecracker/src/api_server/parsed_request.rs

@@ -27,6 +27,7 @@ use super::request::net::{parse_patch_net, parse_put_net};
 use super::request::pmem::parse_put_pmem;
 use super::request::snapshot::{parse_patch_vm_state, parse_put_snapshot};
 use super::request::version::parse_get_version;
+use super::request::vhost_user_device::parse_put_vhost_user_device;
 use super::request::vsock::parse_put_vsock;
 use crate::api_server::request::hotplug::memory::{
     parse_get_memory_hotplug, parse_patch_memory_hotplug, parse_put_memory_hotplug,
@@ -107,6 +108,9 @@ impl TryFrom<&Request> for ParsedRequest {
                 parse_put_net(body, path_tokens.next())
             }
             (Method::Put, "snapshot", Some(body)) => parse_put_snapshot(body, path_tokens.next()),
+            (Method::Put, "vhost-user-devices", Some(body)) => {
+                parse_put_vhost_user_device(body, path_tokens.next())
+            }
             (Method::Put, "vsock", Some(body)) => parse_put_vsock(body),
             (Method::Put, "entropy", Some(body)) => parse_put_entropy(body),
             (Method::Put, "hotplug", Some(body)) if path_tokens.next() == Some("memory") => {

src/firecracker/src/api_server/request/mod.rs

@@ -18,5 +18,6 @@ pub mod pmem;
 pub mod serial;
 pub mod snapshot;
 pub mod version;
+pub mod vhost_user_device;
 pub mod vsock;
 pub use micro_http::{Body, Method, StatusCode};

src/firecracker/src/api_server/request/vhost_user_device.rs

@@ -0,0 +1,83 @@
+// Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+use vmm::logger::{IncMetric, METRICS};
+use vmm::rpc_interface::VmmAction;
+use vmm::vmm_config::vhost_user_device::VhostUserDeviceConfig;
+
+use super::super::parsed_request::{ParsedRequest, RequestError, checked_id};
+use super::{Body, StatusCode};
+
+pub(crate) fn parse_put_vhost_user_device(
+    body: &Body,
+    id_from_path: Option<&str>,
+) -> Result<ParsedRequest, RequestError> {
+    METRICS.put_api_requests.vhost_user_count.inc();
+    let id = if let Some(id) = id_from_path {
+        checked_id(id)?
+    } else {
+        METRICS.put_api_requests.vhost_user_fails.inc();
+        return Err(RequestError::EmptyID);
+    };
+
+    let device_cfg =
+        serde_json::from_slice::<VhostUserDeviceConfig>(body.raw()).inspect_err(|_| {
+            METRICS.put_api_requests.vhost_user_fails.inc();
+        })?;
+
+    if id != device_cfg.id {
+        METRICS.put_api_requests.vhost_user_fails.inc();
+        Err(RequestError::Generic(
+            StatusCode::BadRequest,
+            "The id from the path does not match the id from the body!".to_string(),
+        ))
+    } else {
+        Ok(ParsedRequest::new_sync(VmmAction::InsertVhostUserDevice(
+            device_cfg,
+        )))
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::api_server::parsed_request::tests::vmm_action_from_request;
+
+    #[test]
+    fn test_parse_put_vhost_user_device_request() {
+        parse_put_vhost_user_device(&Body::new("invalid_payload"), None).unwrap_err();
+        parse_put_vhost_user_device(&Body::new("invalid_payload"), Some("id")).unwrap_err();
+
+        let body = r#"{
+            "id": "bar",
+            "device_type": 26,
+            "socket": "/tmp/test.sock",
+            "num_queues": 2
+        }"#;
+        parse_put_vhost_user_device(&Body::new(body), Some("1")).unwrap_err();
+
+        let body = r#"{
+            "foo": "1"
+        }"#;
+        parse_put_vhost_user_device(&Body::new(body), Some("1")).unwrap_err();
+
+        let body = r#"{
+            "id": "fs0",
+            "device_type": 26,
+            "socket": "/tmp/virtiofsd.sock",
+            "num_queues": 2
+        }"#;
+        let r = vmm_action_from_request(
+            parse_put_vhost_user_device(&Body::new(body), Some("fs0")).unwrap(),
+        );
+
+        let expected_config = VhostUserDeviceConfig {
+            id: "fs0".to_string(),
+            device_type: 26,
+            socket: "/tmp/virtiofsd.sock".to_string(),
+            num_queues: 2,
+            queue_size: None,
+        };
+        assert_eq!(r, VmmAction::InsertVhostUserDevice(expected_config));
+    }
+}

src/firecracker/swagger/firecracker.yaml

@@ -371,6 +371,40 @@ paths:
           schema:
             $ref: "#/definitions/Error"
 
+  /vhost-user-devices/{id}:
+    put:
+      summary: Creates or updates a generic vhost-user device. Pre-boot only.
+      description:
+        Creates a new generic vhost-user device with ID specified by id parameter.
+        If a device with the specified ID already exists, it is replaced.
+        The backend process must be running and listening on the specified Unix
+        domain socket before making this request. The CONFIG protocol feature is
+        required.
+      operationId: putGuestVhostUserDeviceByID
+      parameters:
+        - name: id
+          in: path
+          description: The id of the guest vhost-user device
+          required: true
+          type: string
+        - name: body
+          in: body
+          description: Guest vhost-user device properties
+          required: true
+          schema:
+            $ref: "#/definitions/VhostUserDevice"
+      responses:
+        204:
+          description: Vhost-user device is created/updated
+        400:
+          description: Vhost-user device cannot be created/updated due to bad input
+          schema:
+            $ref: "#/definitions/Error"
+        default:
+          description: Internal server error.
+          schema:
+            $ref: "#/definitions/Error"
+
   /logger:
     put:
       summary: Initializes the logger by specifying a named pipe or a file for the logs output.
@@ -1256,6 +1290,41 @@ definitions:
         description:
           Flag to map backing file in read-only mode.
 
+  VhostUserDevice:
+    type: object
+    required:
+      - id
+      - device_type
+      - socket
+      - num_queues
+    properties:
+      id:
+        type: string
+        description:
+          Unique identifier of the device.
+      device_type:
+        type: integer
+        minimum: 0
+        maximum: 255
+        description:
+          The virtio device type ID as defined in the virtio specification.
+          For example, 26 for virtio-fs, 8 for virtio-scsi.
+          The backend is responsible for handling the corresponding device protocol.
+      socket:
+        type: string
+        description:
+          Path to the vhost-user backend Unix domain socket.
+      num_queues:
+        type: integer
+        minimum: 1
+        description:
+          Number of virtqueues to configure for this device.
+      queue_size:
+        type: integer
+        minimum: 1
+        description:
+          Queue size. Defaults to 256 if not specified.
+
   Error:
     type: object
     properties:
@@ -1298,6 +1367,11 @@ definitions:
         description: Configurations for all pmem devices.
         items:
           $ref: "#/definitions/Pmem"
+      vhost-user-devices:
+        type: array
+        description: Configurations for all generic vhost-user devices.
+        items:
+          $ref: "#/definitions/VhostUserDevice"
       vsock:
         $ref: "#/definitions/Vsock"
       entropy:

src/vmm/src/builder.rs

@@ -36,6 +36,7 @@ use crate::devices::virtio::mem::{VIRTIO_MEM_DEFAULT_SLOT_SIZE_MIB, VirtioMem};
 use crate::devices::virtio::net::Net;
 use crate::devices::virtio::pmem::device::Pmem;
 use crate::devices::virtio::rng::Entropy;
+use crate::devices::virtio::vhost_user_generic::device::VhostUserGeneric;
 use crate::devices::virtio::vsock::{Vsock, VsockUnixBackend};
 #[cfg(feature = "gdb")]
 use crate::gdb;
@@ -246,6 +247,13 @@ pub fn build_microvm_for_boot(
         vm_resources.pmem.devices.iter(),
         event_manager,
     )?;
+    attach_vhost_user_devices(
+        &mut device_manager,
+        &vm,
+        &mut boot_cmdline,
+        vm_resources.vhost_user.devices.iter(),
+        event_manager,
+    )?;
 
     if let Some(unix_vsock) = vm_resources.vsock.get() {
         attach_unixsock_vsock_device(
@@ -752,6 +760,28 @@ fn attach_pmem_devices<'a, I: Iterator<Item = &'a Arc<Mutex<Pmem>>> + Debug>(
     Ok(())
 }
 
+fn attach_vhost_user_devices<'a, I: Iterator<Item = &'a Arc<Mutex<VhostUserGeneric>>> + Debug>(
+    device_manager: &mut DeviceManager,
+    vm: &Arc<Vm>,
+    cmdline: &mut LoaderKernelCmdline,
+    vhost_user_devices: I,
+    event_manager: &mut EventManager,
+) -> Result<(), StartMicrovmError> {
+    for device in vhost_user_devices {
+        let id = device.lock().expect("Poisoned lock").id.clone();
+        // The device mutex mustn't be locked here otherwise it will deadlock.
+        device_manager.attach_virtio_device(
+            vm,
+            id,
+            device.clone(),
+            cmdline,
+            event_manager,
+            true, // vhost-user devices require memfd-backed memory
+        )?;
+    }
+    Ok(())
+}
+
 fn attach_unixsock_vsock_device(
     device_manager: &mut DeviceManager,
     vm: &Arc<Vm>,

src/vmm/src/device_manager/pci_mngr.rs

@@ -431,6 +431,12 @@ impl<'a> Persist<'a> for PciDevices {
                         transport_state,
                     })
                 }
+                VirtioDeviceType::VhostUserGeneric => {
+                    warn!(
+                        "Skipping generic vhost-user device. VhostUserGeneric does not support \
+                         snapshotting yet"
+                    );
+                }
             }
         }
 

src/vmm/src/device_manager/persist.rs

@@ -299,6 +299,12 @@ impl<'a> Persist<'a> for MMIODeviceManager {
                         device_info,
                     });
                 }
+                VirtioDeviceType::VhostUserGeneric => {
+                    warn!(
+                        "Skipping generic vhost-user device. VhostUserGeneric does not support \
+                         snapshotting yet"
+                    );
+                }
             };
 
             Ok(())