Skip to content

Include watched keys and pubsub channel/pattern memory in client memory overhead#3362

Open
enjoy-binbin wants to merge 1 commit intovalkey-io:unstablefrom
enjoy-binbin:client_memory
Open

Include watched keys and pubsub channel/pattern memory in client memory overhead#3362
enjoy-binbin wants to merge 1 commit intovalkey-io:unstablefrom
enjoy-binbin:client_memory

Conversation

@enjoy-binbin
Copy link
Member

Previously, multiStateMemOverhead() and pubsubMemOverhead() only
counted the structure overhead (listNode, watchedKey, hashtable)
but not the actual watched_key/channel/pattern robj objects
themselves. This led to under-reporting of client memory usage.

This was problematic because:

  1. If a client watches many keys or watches keys with very large names,
    this memory was invisible in CLIENT INFO tot-mem, making it hard to
    identify which client is consuming memory.
  2. This memory was incorrectly attributed to used_memory_dataset instead
    of client memory, making memory analysis misleading.
  3. The maxmemory-clients eviction mechanism could not account for this
    memory, potentially allowing clients to bypass memory limits.

The same issues apply to SUBSCRIBE/PSUBSCRIBE/SSUBSCRIBE with large or
numerous channel/pattern names.

Since each client holds its own robj copy for watched keys and pubsub
subscriptions (via incrRefCount on argv), this memory should be attributed
to the client for accurate memory accounting.

Changes:

  • Add watched_keys_mem field to multiState struct to track watched key
    robj memory with O(1) lookup
  • Add pubsub_object_mem field to ClientPubSubData struct to track
    channel/pattern robj memory with O(1) lookup

The memory counters are updated on add/delete operations, making the
overhead calculation O(1) instead of O(N).

…ry overhead

Previously, multiStateMemOverhead() and pubsubMemOverhead() only
counted the structure overhead (listNode, watchedKey, hashtable)
but not the actual watched_key/channel/pattern robj objects
themselves. This led to under-reporting of client memory usage.

This was problematic because:
1. If a client watches many keys or watches keys with very large names,
   this memory was invisible in CLIENT INFO tot-mem, making it hard to
   identify which client is consuming memory.
2. This memory was incorrectly attributed to used_memory_dataset instead
   of client memory, making memory analysis misleading.
3. The maxmemory-clients eviction mechanism could not account for this
   memory, potentially allowing clients to bypass memory limits.

The same issues apply to SUBSCRIBE/PSUBSCRIBE/SSUBSCRIBE with large or
numerous channel/pattern names.

Since each client holds its own robj copy for watched keys and pubsub
subscriptions (via incrRefCount on argv), this memory should be attributed
to the client for accurate memory accounting.

Changes:
- Add watched_keys_mem field to multiState struct to track watched key
  robj memory with O(1) lookup
- Add pubsub_object_mem field to ClientPubSubData struct to track
  channel/pattern robj memory with O(1) lookup

The memory counters are updated on add/delete operations, making the
overhead calculation O(1) instead of O(N).

Signed-off-by: Binbin <[email protected]>
@codecov
Copy link

codecov bot commented Mar 13, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.53%. Comparing base (c9ce3e0) to head (29f77fa).

Additional details and impacted files
@@             Coverage Diff              @@
##           unstable    #3362      +/-   ##
============================================
+ Coverage     74.47%   74.53%   +0.06%     
============================================
  Files           130      130              
  Lines         72719    72728       +9     
============================================
+ Hits          54155    54206      +51     
+ Misses        18564    18522      -42     
Files with missing lines Coverage Δ
src/multi.c 97.01% <100.00%> (+0.03%) ⬆️
src/networking.c 91.49% <ø> (+0.15%) ⬆️
src/pubsub.c 97.73% <100.00%> (+0.03%) ⬆️
src/server.h 100.00% <ø> (ø)

... and 21 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

[BUG] WATCH can consume large amounts of memory and bypasses OOM

1 participant