Merge #159046

159046: status: add node_id label to stores initialized after AddNode r=tbg a=tbg

Previously, when a node restarted with additional stores, the new stores
could be missing the node_id label in their metrics. This happened because
stores added to an existing node are initialized asynchronously via
initializeAdditionalStores, which can complete after recorder.AddNode has
already run. AddNode sets the node_id label on all stores currently in
storeRegistries, but AddStore didn't add the label for stores registered
later.

Fix this by having AddStore add the node_id label if AddNode has already
been called (i.e., if desc.NodeID is set). There's no risk of duplicate
labels since a store is either present when AddNode runs (and AddNode adds
the label) or added afterward (and AddStore adds it).

Fixes #159045

Release note (bug fix): Previously, when a node was restarted with additional stores for the first time, the metrics for these stores could be missing the node_id label.

Co-authored-by: Tobias Grieger <[email protected]>

Author: craig[bot]

pkg/server/status/recorder.go

@@ -345,6 +345,14 @@ func (mr *MetricsRecorder) AddStore(store storeMetrics) {
 	defer mr.mu.Unlock()
 	storeID := store.StoreID()
 	store.Registry().AddLabel("store", strconv.Itoa(int(storeID)))
+	// If AddNode has already been called, we need to add the node_id label here.
+	// This can happen when stores are initialized asynchronously after node start.
+	// There's no risk of duplicate labels: either the store is added before AddNode
+	// runs (in which case desc.NodeID is 0 here, and AddNode adds the label), or it's
+	// added after (in which case we add it here, and AddNode never saw this store).
+	if !disableNodeAndTenantLabels && mr.mu.desc.NodeID != 0 {
+		store.Registry().AddLabel("node_id", strconv.Itoa(int(mr.mu.desc.NodeID)))
+	}
 	mr.mu.storeRegistries[storeID] = store.Registry()
 	mr.mu.stores[storeID] = store
 }