Any one used “—exit-signal-handler” ?

[HenryTangDev]

Any one enabled —exit-signal-handler successfully, it seems not working.

Here’s what I did:

1. Using mpirun to launch 8 instances ;
2. waiting until training happened after several iterations;
3. From other terminals, issue: pkill -15 -f pretrain_gpt to kill processes gracefully.
4. Noticed that the training code didn’t enter into the signal handler to save checkpoints, instead the dataloader workers intercepted the SIGTERM signal and hits runtime error.