Add MCCL_SCUBA_LOG_LEVEL cvar for log filtering (#734)

Summary:
Pull Request resolved: #734

Adds a new configuration variable to control the verbosity of MCCL Scuba logging. This enables operators to filter out lower-priority logs and reduce noise in Scuba tables, improving observability by focusing on the most relevant events (critical errors, high-priority events, etc.) while reducing storage costs and query complexity.

Also making  MCCL_SCUBA_ENABLED=true by default so that we always have log for any job.

Reviewed By: dboyda

Differential Revision: D93675354

fbshipit-source-id: 323d3b1c2e5b50b005447a0e27b2567cc9c68e6f

Author: mlunar-meta

comms/utils/cvars/nccl_cvars.yaml

@@ -2807,6 +2807,21 @@ cvars:
      events to dedicated MCCL Scuba tables for observability and debugging.
      This is separate from NCCLX logging to minimize risk.
 
+ - name        : MCCL_SCUBA_LOG_LEVEL
+   type        : enum
+   default     : HIGH
+   choices     : CRITICAL, HIGH, NORMAL, LOW
+   description : |-
+     Minimum log priority level for MCCL Scuba logging. Only logs at or above
+     this priority level will be forwarded to Scuba; lower-priority logs are
+     dropped. The priority levels from highest to lowest are:
+       CRITICAL - Critical logs only (e.g., errors, failures)
+       HIGH     - Critical and high-priority logs (e.g., important events)
+       NORMAL   - Critical, high, and normal logs (default operational logs)
+       LOW      - All logs including low-priority (verbose/debug)
+     For example, setting MCCL_SCUBA_LOG_LEVEL=HIGH means only HIGH and
+     CRITICAL priority logs will be forwarded to Scuba.
+
  - name        : NCCL_GIN_GDAKI_NIC_HANDLER
    type        : int
    default     : 0