sensu-client process is not getting monitored by the check-process.rb script

[gopalvd]

Team,
I have a strange behaviour. I am trying to monitor the sensu-client process running on my linux server. For that i have set up the check definition and installed the respective dependent gems. This is what i am seeing.

When i run the script manually i see CheckProcess OK and found 1 process matching.
[root@XXXXXXXX checks]# /opt/sensu/embedded/bin/ruby /etc/sensu/plugins/check-process.rb -p sensu-client
CheckProcess OK: Found 1 matching processes; cmd /sensu-client/

When sensu-client itself is running the check, I see this.
{"timestamp":"2017-11-09T03:53:50.181638-0600","level":"info","message":"publishing check result","payload":{"client":"xxxxxxxxxx","check":{"command":"/etc/sensu/plugins/check-process.rb -p sensu-client","subscribers":[],"standalone":true,"handlers":["default","mailer"],"interval":30,"type":"standard","mail_to":"xxxxxxxxxxxx@target.com","name":"check_process-test","issued":1510221230,"executed":1510221230,"duration":0.17,"output":"CheckProcess CRITICAL: Found 0 matching processes; cmd /sensu-client/\n","status":2}}}

Can some one help, what could be the issue?

[majormoses]

Hmm that is indeed strange, when I have the time I will try to replicate in my environment. I never really considered using sensu to monitor the sensu process as how would it even get the request from the transport (rabbitmq|redis) if the process is dead? Even in the case of standalone checks we still have the same problem of it being able to locally schedule itself.

I think you probably should leverage native functionality in your process supervisor such as upstart, runit, systemd, etc to attempt restarting the process if it dies. It's kinda a who watches the watcher problem. Here are some other suggestions which are not mutually exclusive with what was proposed above:

• Another monitoring system such as a monit to watch the clients and health of sensu server, api, transport, etc.
• Something that may make sense if you have a static setup is to use the ttl functionality to check that it is still receiving a "heartbeat" and will create an alert when the TTL expires without the check actually able to run. You can read about sensu check TTLs here: https://sensuapp.org/docs/latest/reference/checks.html#check-attributes.

That being said if the process is in fact up then there is definately a bug, config, or environment issue somewhere.

Can you please include the version of this plugin, sensu-server, and sensu-client?

[gopalvd]

@majormoses Thanks for looking in to this. I totally agree that monitoring sensu-client by sensu is not a great idea. Also we are using a diff tools for that. And yes this plugin is working perfectly when using other processes like crond, chef-client systemd etc.
I noticed this strange behaviour only when i did for sensu-client. And this was done with out any intention. I want to test the plugin and gave the sensu-client process to check and noticed this.

Yes to me seems like a bug.

Here are the details that you are looking for.
gem version for plugins ---- 2.5.0
sensu-client version --- "sensu-0.26.3-1.x86_64"
sensu-client version --- "sensu-1.0.2-1.el7.x86_64"

[gopalvd]

Sorry the last is the sensu-server version
sensu-server version --- "sensu-1.0.2-1.el7.x86_64"

[majormoses]

Hmm this is what I show in my environment and I can't explain it, I will look through the code when I have some time. This is what I got manually which matches your output:

babrams@ip-10-55-141-110:~$ /opt/sensu/embedded/bin/check-process.rb -p sensu-client
CheckProcess OK: Found 1 matching processes; cmd /sensu-client/

I was scratching my head for a minute (clearly tired) as this is what I initially did and got back:

babrams@ip-10-55-141-110:~$ /opt/sensu/embedded/bin/check-process.rb - sensu-client
CheckProcess OK: Found 121 matching processes

[gopalvd]

Means a bug? or some options needs to be used in the command?

[majormoses]

I was missing the p after the - so it was returning matching all processes!
which basically matches:

babrams@ip-10-55-141-110:~$ ps -Al | wc -l
122

[gopalvd]

I did the same
CheckProcess OK: Found 402 matching processes

[gopalvd]

Correct when i run the command, its giving one matching process. But when sensu-client is executing i see 0 processes running and throwing the error.
CheckProcess CRITICAL: Found 0 matching processes; cmd /sensu-client/