DAOS-16999 bio: Set LED on auto-faulty detection

docs/admin/administration.md

@@ -44,32 +44,35 @@ severity, message, description, and cause.
 
 |Event|Event type|Severity|Message|Description|Cause|
 |:----|:----|:----|:----|:----|:----|
-| device\_set\_faulty| INFO\_ONLY| NOTICE or ERROR| Device: <uuid\> set faulty / Device: <uuid\> set faulty failed: <rc\> / Device: <uuid\> auto faulty detect / Device: <uuid\> auto faulty detect failed: <rc\> | Indicates that a device has either been explicitly automatically set as faulty. Device UUID specified in event data. | Either DMG set nvme-faulty command was used to explicitly set device as faulty or an error threshold was reached on a device which has triggered an auto faulty reaction. |
-| device\_media\_error| INFO\_ONLY| ERROR| Device: <uuid\> <error-type\> error logged from tgt\_id:<idx\> | Indicates that a device media error has been detected for a specific target. The error type could be unmap, write, read or checksum (csum). Device UUID and target ID specified in event data. | Media error occurred on backing device. |
-| device\_unplugged| INFO\_ONLY| NOTICE| Device: <uuid\> unplugged | Indicates device was physically removed from host. | NVMe SSD physically removed from host. |
-| device\_plugged| INFO\_ONLY| NOTICE| Detected hot plugged device: <bdev-name\> | Indicates device was physically inserted into host. | NVMe SSD physically added to host. |
-| device\_replace| INFO\_ONLY| NOTICE or ERROR| Replaced device: <uuid\> with device: <uuid\> [failed: <rc\>] | Indicates that a faulty device was replaced with a new device and if the operation failed. The old and new device IDs as well as any non-zero return code are specified in the event data. | Device was replaced using DMG nvme replace command. |
-| device\_link\_speed\_changed| NOTICE or WARNING| NVMe PCIe device at <pci-address\> port-<idx\>: link speed changed to <transfer-rate\> (max <transfer-rate\>)| Indicates that an NVMe device link speed has changed. The negotiated and maximum device link speeds are indicated in the event message field and the severity is set to warning if the negotiated speed is not at maximum capability (and notice level severity if at maximum). No other specific information is included in the event data.| Either device link speed was previously downgraded and has returned to maximum or link speed has downgraded to a value that is less than its maximum capability.|
-| device\_link\_width\_changed| NOTICE or WARNING| NVMe PCIe device at <pci-address\> port-<idx\>: link width changed to <pcie-link-lanes\> (max <pcie-link-lanes\>)| Indicates that an NVMe device link width has changed. The negotiated and maximum device link widths are indicated in the event message field and the severity is set to warning if the negotiated width is not at maximum capability (and notice level severity if at maximum). No other specific information is included in the event data.| Either device link width was previously downgraded and has returned to maximum or link width has downgraded to a value that is less than its maximum capability.|
+| unknown\_ras\_event| INFO\_ONLY| NOTICE| Unknown RAS event| Indicates an unidentified or undefined RAS event has occurred.| Internal placeholder for unknown events.|
 | engine\_format\_required|INFO\_ONLY|NOTICE|DAOS engine <idx\> requires a <type\> format|Indicates engine is waiting for allocated storage to be formatted on formatted on instance <idx\> with dmg tool. <type\> can be either SCM or Metadata.|DAOS server attempts to bring-up an engine that has unformatted storage.|
 | engine\_died| STATE\_CHANGE| ERROR| DAOS engine <idx\> exited exited unexpectedly: <error\> | Indicates engine instance <idx\> unexpectedly. <error> describes the exit state returned from exited daos\_engine process.| N/A                          |
 | engine\_asserted| STATE\_CHANGE| ERROR| TBD| Indicates engine instance <idx\> threw a runtime assertion, causing a crash. | An unexpected internal state resulted in assert failure. |
 | engine\_clock\_drift| INFO\_ONLY   | ERROR| clock drift detected| Indicates CART comms layer has detected clock skew between engines.| NTP may not be syncing clocks across DAOS system.      |
-| engine\_join\_failed| INFO\_ONLY| ERROR | DAOS engine <idx\> (rank <rank\>) was not allowed to join the system | Join operation failed for the given engine instance ID and rank (if assigned). | Reason should be provided in the extended info field of the event data. |
 | pool\_corruption\_detected| INFO\_ONLY| ERROR | Data corruption detected| Indicates a corruption in pool data has been detected. The event fields will contain pool and container UUIDs. | A corruption was found by the checksum scrubber. |
-| pool\_destroy\_deferred| INFO\_ONLY| WARNING | pool:<uuid\> destroy is deferred| Indicates a destroy operation has been deferre. | Pool destroy in progress but not complete. |
 | pool\_rebuild\_started| INFO\_ONLY| NOTICE   | Pool rebuild started.| Indicates a pool rebuild has started. The event data field contains pool map version and pool operation identifier. | When a pool rank becomes unavailable a rebuild will be triggered.   |
 | pool\_rebuild\_finished| INFO\_ONLY| NOTICE| Pool rebuild finished.| Indicates a pool rebuild has finished successfully. The event data field includes the pool map version and pool operation identifier.  | N/A|
 | pool\_rebuild\_failed| INFO\_ONLY| ERROR| Pool rebuild failed: <rc\>.| Indicates a pool rebuild has failed. The event data field includes the pool map version and pool operation identifier. <rc\> provides a string representation of DER code.| N/A                          |
 | pool\_replicas\_updated| STATE\_CHANGE| NOTICE| List of pool service replica ranks has been updated.| Indicates a pool service replica list has changed. The event contains the new service replica list in a custom payload. | When a pool service replica rank becomes unavailable a new rank is selected to replace it (if available). |
-| pool\_durable\_format\_incompat| INFO\_ONLY| ERROR| incompatible layout version: <current\> not in [<min\>, <max\>]| Indicates the given pool's layout version does not match any of the versions supported by the currently running DAOS software.| DAOS engine is started with pool data in local storage that has an incompatible layout version. |
-| container\_durable\_format\_incompat| INFO\_ONLY| ERROR| incompatible layout version[: <current\> not in [<min\>, <max\>\]| Indicates the given container's layout version does not match any of the versions supported by the currently running DAOS software.| DAOS engine is started with container data in local storage that has an incompatible layout version.|
+| pool\_durable\_format\_incompatible| INFO\_ONLY| ERROR| incompatible layout version: <current\> not in [<min\>, <max\>]| Indicates the given pool's layout version does not match any of the versions supported by the currently running DAOS software.| DAOS engine is started with pool data in local storage that has an incompatible layout version. |
+| pool\_destroy\_deferred| INFO\_ONLY| WARNING | pool:<uuid\> destroy is deferred| Indicates a destroy operation has been deferred. | Pool destroy in progress but not complete. |
+| pool\_start\_failed| INFO\_ONLY| ERROR| Pool <uuid\> start failed: <rc\>| Indicates that a pool failed to start. The pool UUID and error code are specified in the event data.| Pool start operation encountered an error during startup.|
+| container\_durable\_format\_incompatible| INFO\_ONLY| ERROR| incompatible layout version[: <current\> not in [<min\>, <max\>\]| Indicates the given container's layout version does not match any of the versions supported by the currently running DAOS software.| DAOS engine is started with container data in local storage that has an incompatible layout version.|
 | rdb\_durable\_format\_incompatible| INFO\_ONLY| ERROR| incompatible layout version[: <current\> not in [<min\>, <max\>]] OR incompatible DB UUID: <uuid\> | Indicates the given RDB's layout version does not match any of the versions supported by the currently running DAOS software, or the given RDB's UUID does not match the expected UUID (usually because the RDB belongs to a pool created by a pre-2.0 DAOS version).| DAOS engine is started with rdb data in local storage that has an incompatible layout version.|
 | swim\_rank\_alive| STATE\_CHANGE| NOTICE| TBD| The SWIM protocol has detected the specified rank is responsive.| A remote DAOS engine has become responsive.|
 | swim\_rank\_dead| STATE\_CHANGE| NOTICE| SWIM rank marked as dead.| The SWIM protocol has detected the specified rank is unresponsive.| A remote DAOS engine has become unresponsive.|
 | system\_start\_failed| INFO\_ONLY| ERROR| System startup failed, <errors\>| Indicates that a user initiated controlled startup failed. <errors\> shows which ranks failed.| Ranks failed to start.|
 | system\_stop\_failed| INFO\_ONLY| ERROR| System shutdown failed during <action\> action, <errors\>  | Indicates that a user initiated controlled shutdown failed. <action\> identifies the failing shutdown action and <errors\> shows which ranks failed.| Ranks failed to stop.|
-| system\_fabric\_provider\_changed| NOTICE| System fabric provider has changed: <old-provider\> -> <new-provider\>| Indicates that the system-wide fabric provider has been updated. No other specific information is included in event data.| A system-wide fabric provider change has been intentionally applied to all joined ranks.|
+| device\_set\_faulty| INFO\_ONLY| NOTICE or ERROR| Device: <uuid\> set faulty / Device: <uuid\> set faulty failed: <rc\> / Device: <uuid\> auto faulty detect / Device: <uuid\> auto faulty detect failed: <rc\> | Indicates that a device has either been explicitly automatically set as faulty. Device UUID specified in event data. | Either DMG set nvme-faulty command was used to explicitly set device as faulty or an error threshold was reached on a device which has triggered an auto faulty reaction. |
+| device\_media\_error| INFO\_ONLY| ERROR| Device: <uuid\> <error-type\> error logged from tgt\_id:<idx\> | Indicates that a device media error has been detected for a specific target. The error type could be unmap, write, read or checksum (csum). Device UUID and target ID specified in event data. | Media error occurred on backing device. |
+| device\_unplugged| INFO\_ONLY| NOTICE| Device: <uuid\> unplugged | Indicates device was physically removed from host. | NVMe SSD physically removed from host. |
+| device\_plugged| INFO\_ONLY| NOTICE| Detected hot plugged device: <bdev-name\> | Indicates device was physically inserted into host. | NVMe SSD physically added to host. |
+| device\_replace| INFO\_ONLY| NOTICE or ERROR| Replaced device: <uuid\> with device: <uuid\> [failed: <rc\>] | Indicates that a faulty device was replaced with a new device and if the operation failed. The old and new device IDs as well as any non-zero return code are specified in the event data. | Device was replaced using DMG nvme replace command. |
+| system\_fabric\_provider\_changed| INFO\_ONLY| NOTICE| System fabric provider has changed: <old-provider\> -> <new-provider\>| Indicates that the system-wide fabric provider has been updated. No other specific information is included in event data.| A system-wide fabric provider change has been intentionally applied to all joined ranks.|
+| engine\_join\_failed| INFO\_ONLY| ERROR | DAOS engine <idx\> (rank <rank\>) was not allowed to join the system | Join operation failed for the given engine instance ID and rank (if assigned). | Reason should be provided in the extended info field of the event data. |
+| device\_link\_speed\_changed| INFO\_ONLY| NOTICE or WARNING| NVMe PCIe device at <pci-address\> port-<idx\>: link speed changed to <transfer-rate\> (max <transfer-rate\>)| Indicates that an NVMe device link speed has changed. The negotiated and maximum device link speeds are indicated in the event message field and the severity is set to warning if the negotiated speed is not at maximum capability (and notice level severity if at maximum). No other specific information is included in the event data.| Either device link speed was previously downgraded and has returned to maximum or link speed has downgraded to a value that is less than its maximum capability.|
+| device\_link\_width\_changed| INFO\_ONLY| NOTICE or WARNING| NVMe PCIe device at <pci-address\> port-<idx\>: link width changed to <pcie-link-lanes\> (max <pcie-link-lanes\>)| Indicates that an NVMe device link width has changed. The negotiated and maximum device link widths are indicated in the event message field and the severity is set to warning if the negotiated width is not at maximum capability (and notice level severity if at maximum). No other specific information is included in the event data.| Either device link width was previously downgraded and has returned to maximum or link width has downgraded to a value that is less than its maximum capability.|
+| device\_led\_set| INFO\_ONLY| NOTICE| LED on device <device\> set to state <state\>| Indicates that the LED state has been changed on a device. Device identifier and LED state are specified in the event message.| LED control command was issued to change device LED state for visual identification or fault indication.|
 
 ## System Logging
 

src/bio/bio_device.c

@@ -755,8 +755,8 @@ led_device_action(void *ctx, struct spdk_pci_device *pci_device)
 			opts->led_state = d_led_state;
 		else
 			/* Leave state as NA */
-			D_ERROR("LED state GET not supported for non-VMD device (type %s:%s)\n",
-				pci_dev_type, addr_buf);
+			D_INFO("LED state GET not supported for non-VMD device (type %s:%s)\n",
+			       pci_dev_type, addr_buf);
 		return;
 	case CTL__LED_ACTION__SET:
 		break;
@@ -816,6 +816,10 @@ led_device_action(void *ctx, struct spdk_pci_device *pci_device)
 			  addr_buf, LED_STATE_NAME(opts->led_state));
 }
 
+/**
+ * Clear timer if both input parameters are NULL otherwise set timer expiry if expiry_time or
+ * check faulty state if is_faulty.
+ */
 static int
 set_timer_and_check_faulty(struct bio_xs_context *xs_ctxt, struct spdk_pci_addr pci_addr,
 			   uint64_t *expiry_time, bool *is_faulty)
@@ -826,7 +830,7 @@ set_timer_and_check_faulty(struct bio_xs_context *xs_ctxt, struct spdk_pci_addr
 	int			 dev_list_cnt, rc;
 	char			 tr_addr[ADDR_STR_MAX_LEN + 1];
 
-	D_ASSERT((expiry_time != NULL) || (is_faulty != NULL));
+	D_ASSERT((expiry_time == NULL) || (is_faulty == NULL));
 
 	rc = spdk_pci_addr_fmt(tr_addr, ADDR_STR_MAX_LEN + 1, &pci_addr);
 	if (rc != 0) {
@@ -850,10 +854,10 @@ set_timer_and_check_faulty(struct bio_xs_context *xs_ctxt, struct spdk_pci_addr
 			D_ERROR("No transport address for dev:"DF_UUID", unable to verify state\n",
 				DP_UUID(dev_info->bdi_dev_id));
 		} else if (strcmp(dev_info->bdi_traddr, tr_addr) == 0) {
-			if ((is_faulty != NULL) && (dev_info->bdi_flags & NVME_DEV_FL_FAULTY) != 0)
+			if ((is_faulty) && ((dev_info->bdi_flags & NVME_DEV_FL_FAULTY) != 0)) {
+				/* Set is_faulty return value */
 				*is_faulty = true;
-
-			if (expiry_time != NULL) {
+			} else {
 				d_bdev = lookup_dev_by_id(dev_info->bdi_dev_id);
 				if (d_bdev == NULL) {
 					D_ERROR("Failed to find dev "DF_UUID"\n",
@@ -862,7 +866,16 @@ set_timer_and_check_faulty(struct bio_xs_context *xs_ctxt, struct spdk_pci_addr
 					goto out;
 				}
 
-				d_bdev->bb_led_expiry_time = *expiry_time;
+				if (expiry_time != NULL) {
+					/* Set timer if non-zero value supplied */
+					if (*expiry_time != 0)
+						d_bdev->bb_led_expiry_time = *expiry_time;
+					d_bdev->bb_led_identify_active = true;
+				} else {
+					/* Clear timer if no pointer supplied */
+					d_bdev->bb_led_expiry_time     = 0;
+					d_bdev->bb_led_identify_active = false;
+				}
 			}
 		}
 	}
@@ -881,6 +894,61 @@ set_timer(struct bio_xs_context *xs_ctxt, struct spdk_pci_addr pci_addr, uint64_
 	return set_timer_and_check_faulty(xs_ctxt, pci_addr, &expiry_time, NULL);
 }
 
+static int
+clear_timer(struct bio_xs_context *xs_ctxt, struct spdk_pci_addr pci_addr)
+{
+	return set_timer_and_check_faulty(xs_ctxt, pci_addr, NULL, NULL);
+}
+
+// static int
+// set_led_identify_flag(struct bio_xs_context *xs_ctxt, struct spdk_pci_addr pci_addr, bool active)
+//{
+//	struct bio_dev_info	*dev_info = NULL, *tmp;
+//	struct bio_bdev		*d_bdev = NULL;
+//	d_list_t		 dev_list;
+//	int			 dev_list_cnt, rc;
+//	char			 tr_addr[ADDR_STR_MAX_LEN + 1];
+//
+//	rc = spdk_pci_addr_fmt(tr_addr, ADDR_STR_MAX_LEN + 1, &pci_addr);
+//	if (rc != 0) {
+//		D_ERROR("Failed to format PCI address (%s)\n", spdk_strerror(-rc));
+//		return -DER_INVAL;
+//	}
+//
+//	D_INIT_LIST_HEAD(&dev_list);
+//
+//	rc = bio_dev_list(xs_ctxt, &dev_list, &dev_list_cnt);
+//	if (rc != 0) {
+//		D_ERROR("Error getting BIO device list\n");
+//		return rc;
+//	}
+//
+//	d_list_for_each_entry_safe(dev_info, tmp, &dev_list, bdi_link) {
+//		if (dev_info->bdi_traddr != NULL &&
+//		    strcmp(dev_info->bdi_traddr, tr_addr) == 0) {
+//			d_bdev = lookup_dev_by_id(dev_info->bdi_dev_id);
+//			if (d_bdev == NULL) {
+//				D_ERROR("Failed to find dev "DF_UUID"\n",
+//					DP_UUID(dev_info->bdi_dev_id));
+//				rc = -DER_NONEXIST;
+//				goto out;
+//			}
+//
+//			d_bdev->bb_led_identify_active = active;
+//			D_DEBUG(DB_MGMT, "Device "DF_UUID" LED identify active=%d\n",
+//				DP_UUID(d_bdev->bb_uuid), active);
+//		}
+//	}
+//
+// out:
+//	d_list_for_each_entry_safe(dev_info, tmp, &dev_list, bdi_link) {
+//		d_list_del(&dev_info->bdi_link);
+//		bio_free_dev_info(dev_info);
+//	}
+//
+//	return rc;
+// }
+
 static int
 check_faulty(struct bio_xs_context *xs_ctxt, struct spdk_pci_addr pci_addr, bool *is_faulty) {
 	return set_timer_and_check_faulty(xs_ctxt, pci_addr, NULL, is_faulty);
@@ -967,7 +1035,7 @@ led_manage(struct bio_xs_context *xs_ctxt, struct spdk_pci_addr pci_addr, Ctl__L
 			}
 		} else {
 			/* Clear LED start time to cancel any previously set timers */
-			rc = set_timer(xs_ctxt, pci_addr, 0);
+			rc = clear_timer(xs_ctxt, pci_addr);
 			if (rc != 0) {
 				D_ERROR("Clearing LED start time failed (%d)\n", rc);
 				return rc;
@@ -976,7 +1044,7 @@ led_manage(struct bio_xs_context *xs_ctxt, struct spdk_pci_addr pci_addr, Ctl__L
 		break;
 	case CTL__LED_ACTION__RESET:
 		/* Clear LED start time on bdevs as identify state has been reset */
-		rc = set_timer(xs_ctxt, pci_addr, 0);
+		rc = clear_timer(xs_ctxt, pci_addr);
 		if (rc != 0) {
 			D_ERROR("Clearing LED start time failed (%d)\n", rc);
 			return rc;

src/bio/bio_internal.h

@@ -319,17 +319,19 @@ struct bio_bdev {
 	 * saved and when it is reached the prior LED state will be restored.
 	 */
 	uint64_t		 bb_led_expiry_time;
-	unsigned int		 bb_removed:1,
-				 bb_replacing:1,
-				 bb_trigger_reint:1,
-	/*
-	 * If a faulty device is replaced but still plugged, we'll keep
-	 * the 'faulty' information here, so that we know this device was
-	 * marked as faulty (at least before next server restart).
-	 */
-				bb_faulty:1,
-				bb_tgt_cnt_init:1,
-				bb_unmap_supported:1;
+	unsigned int             bb_removed : 1, bb_replacing : 1, bb_trigger_reint : 1,
+	    /*
+	     * If a faulty device is replaced but still plugged, we'll keep
+	     * the 'faulty' information here, so that we know this device was
+	     * marked as faulty (at least before next server restart).
+	     */
+	    bb_faulty              : 1,
+	    /*
+	     * Track if LED is actively blinking for device identification.
+	     * Set when IDENTIFY/QUICK_BLINK state is applied, cleared when
+	     * identify completes (timer expires or manual reset).
+	     */
+	    bb_led_identify_active : 1, bb_tgt_cnt_init : 1, bb_unmap_supported : 1;
 	/* bdev roles data/meta/wal */
 	unsigned int		bb_roles;
 };
@@ -614,6 +616,8 @@ int xs_poll_completion(struct bio_xs_context *ctxt, unsigned int *inflights,
 void bio_bdev_event_cb(enum spdk_bdev_event_type type, struct spdk_bdev *bdev,
 		       void *event_ctx);
 struct spdk_thread *init_thread(void);
+struct bio_xs_context           *
+init_xs_context(void);
 void bio_release_bdev(void *arg);
 bool is_server_started(void);
 d_list_t *bio_bdev_list(void);