Operator stops reconciling due to no incoming events in reflector watcher stream

[viggys]

Seeking help from k8s experts.

I’m facing an issue with my custom operator - the custom resources stop getting reconciled after a few hours (or days) for any CREATE/UPDATE/DELETE events. On debugging, I observed that my operator stops receiving events in the controller’s reflector watcher. There are no errors with respect to the watcher around the time when the issue starts to happen. Although, there are general watch errors, like “too old resource version“ or “unexpected EOF“ which are recoverable. To recover from this situation, I’m forced to restart my operator pod after which things start to work as expected, until I see the same issue happen again after few hours or days.

Is there any way I can recover from a “silent” dead watch in this situation? I do not have access to the reflector, as it’s internally managed by controller-runtime. If not in the operator, what can i check further to investigate this issue?

Kubernetes version: 1.34.1
Controller-Runtime version: v0.22.4

Note:

• I have tried using Cache SyncPeriod, but it only helps with reconciling existing resources in the operator’s cache. It does not help with reconciling newly created resources.

Cluster information:
Kubernetes version: 1.34.1
Cloud being used: OCI
Installation method: Oracle Kubernetes Engine
Host OS: Oracle Linux 8.10

[alvaroaleman]

This sounds like a bug and one that is very hard to reproduce. If this happen, could you maybe get a stackdump of your controller using pprof or by sending it a SIGQUIT?

[viggys]

Sure @alvaroaleman, will try to get this info. Please give me a few days until I'm able to repro this issue.

[viggys]

@alvaroaleman I have extracted the stackdump of my controllers from 2 operators by sending it a SIGQUIT after the issue started to happen.
The reflector for v1alpha1.FabricPeer in hlf-operator stopped receiving events after 2026-02-04T00:21:06Z based on some additional logging that has been excluded from the attached log file.
The reflector for v1alpha1.OBPConsole in obp-operator stopped receiving events after 2026-02-04T00:26:55Z.

Can you please take a look? Please let me know if you need the additional event logs. I would need to remove the user-data before sharing it.

hlf-operator-2026-02-04-controller-runtime-dump.log
obp-operator-2026-02-04-controller-runtime-dump.log

[viggys]

Hi @alvaroaleman, Just following up on this - wanted to see if the stack dump provided any clues?

As a temporary workaround, I’ve been restarting the watch if no events are received within a 3-minute idle timeout. I noticed that my OKE cluster sends BOOKMARK events at least every minute, so the silence was definitely unexpected.

I’m happy to help investigate this further under your guidance. If you can point me to the specific area where you suspect the issue originates, I can try to put together a fix or gather more detailed logs.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale