Skip to content

Comments

Add self kill when controller no longer messages#2699

Open
dulinriley wants to merge 1 commit intometa-pytorch:mainfrom
dulinriley:export-D92726630
Open

Add self kill when controller no longer messages#2699
dulinriley wants to merge 1 commit intometa-pytorch:mainfrom
dulinriley:export-D92726630

Conversation

@dulinriley
Copy link
Contributor

Summary:
Fixes: #2198

One of the principles of ownership we want to maintain in monarch is that if the owner of
a resource (meshes of hosts, procs, actors) is not alive, that all of the resources it owned
are also not alive. This should include if the owner hits a hard failure that prevents it from
running any cleanup.

Before this change, we didn't have that guarantee, an actor would keep running if its owner
went away, as long as the actor didn't message the owner.
This change tracks every time a GetState message is received by a ProcMeshAgent, and tracks
which controller sent it. We use this information to infer the controller is alive and well.
We use a new KeepaliveGetState message which is the same as a GetState but has an explicit
side effect. Only ProcMeshAgent uses this for actors at the moment, but we will expand to cover
procs and hosts too.

Benefits of this change:

  • Actors might be using resources like CPU, memory, or GPUs that are no
    longer needed and can be recovered.
  • If you disconnect the client, all of the actors it spawned
    will shut themselves down. And then any resources those actors spawned will also shut
    themselves down recursively. You don't need to run any special cleanup function on the client.

This feature is hidden behind a flag, and this flag defaults to off. It may have unexpected issues
when we start having actors stop themselves, so we want to test this thoroughly.

Differential Revision: D92726630

Summary:
Fixes: meta-pytorch#2198

One of the principles of ownership we want to maintain in monarch is that if the owner of
a resource (meshes of hosts, procs, actors) is not alive, that all of the resources it owned
are also not alive. This should include if the owner hits a hard failure that prevents it from
running any cleanup.

Before this change, we didn't have that guarantee, an actor would keep running if its owner
went away, as long as the actor didn't message the owner. 
This change tracks every time a GetState message is received by a ProcMeshAgent, and tracks
which controller sent it. We use this information to infer the controller is alive and well.
We use a new KeepaliveGetState message which is the same as a GetState but has an explicit
side effect. Only ProcMeshAgent uses this for actors at the moment, but we will expand to cover
procs and hosts too.

Benefits of this change:
* Actors might be using resources like CPU, memory, or GPUs that are no
longer needed and can be recovered.
* If you disconnect the client, all of the actors it spawned
will shut themselves down. And then any resources those actors spawned will also shut
themselves down recursively. You don't need to run any special cleanup function on the client.

This feature is hidden behind a flag, and this flag defaults to off. It may have unexpected issues
when we start having actors stop themselves, so we want to test this thoroughly.

Differential Revision: D92726630
@meta-codesync
Copy link

meta-codesync bot commented Feb 19, 2026

@dulinriley has exported this pull request. If you are a Meta employee, you can view the originating Diff in D92726630.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Feb 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot. fb-exported meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

When a parent actor has a fault, child actors should also fault and not keep running

1 participant