"watchdog-expired" on all masternodes after 12.1?

stan.distortion

Well-known Member
Oct 30, 2014
865
507
163
Yeah, I did it already. I hoped that this is cause of our problem but it is not. I change it to correct and masternodes go to ENABLED but minute later some of them go WATCHDOG state again...
A few things to try, go into the sentinel folder and run this:
Code:
SENTINEL_DEBUG=1 ./venv/bin/python bin/sentinel.py
That will run sentinel manually with debugging on, it should just run second with a lot of weird looking output, is so that part's ok.

Another to try is:
Code:
dash-cli masternode status
If all is good it will give a few lines of details and the last line should be:
Code:
"status": "Masternode successfully started"
 

Walter

Active Member
Masternode Owner/Operator
Jul 17, 2014
231
201
103
@moocowmoo helped me out with the problem I had. It ended up being a syntax issue. "RPCUSER= " needs to NOT have a space between the = sign and the username supplied. I'm pretty sure that's what he said... It was a really dumb mistake tbh. He was fantastic though, really appreciated him helping like that :)
 
Last edited:

AjM

Well-known Member
Foundation Member
Jun 23, 2014
1,341
575
283
Finland
Bug confirmed, i am 100% sure about this, all my nodes are ok now.

v12.1
If in dash.conf rpcuser and/or rpcpassword include any special character, like !"#¤%&/,
sentinel wont work like it should be, and masternode status become to watchdog_expired,
only normal chars are working ok.

I did have this problem, after fixed/removed all special characters, no problem any more.

Bug report: https://github.com/dashpay/dash/issues/1323
 

Stealth923

Well-known Member
Foundation Member
Mar 9, 2014
348
390
233
Bug confirmed, i am 100% sure about this, all my nodes are ok now.

v12.1
If in dash.conf rpcuser and/or rpcpassword include any special character, like !"#¤%&/,
sentinel wont work like it should be, and masternode status become to watchdog_expired,
only normal chars are working ok.

I did have this problem, after fixed/removed all special characters, no problem any more.

Bug report: https://github.com/dashpay/dash/issues/1323
Just to comment on this - I actually setup new MNs with brand new rpcuser and rpcpassword for 12.1 with special characters and sentinel test script was failing. It does not support special characters.

Went back to plain letters and numbers and all humming along smoothly now.
 

TreasureSeeker

New Member
Apr 28, 2014
25
5
3
I've noticed "Watchdog_expired" problems too. On the latest one I logged in to my masternode and ran in the sentinel directory:

venv/bin/python bin/sentinel.py

The status of that masternode changed from "Watchdog_expired" in my local wallet to "Enabled" so that seemed to fix it.

I'm wondering whether it's anything to do with the amount of time between Sentinel polls in the cron job. I've set mine to 5 minutes in accordance with the instructions at https://dashpay.atlassian.net/wiki/display/DOC/Updating+to+12.1+-+Masternodes , ie....

*/5 * * * * cd /home/YOURUSERNAME/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2>&1 >> sentinel-cron.log

but I see that the actual sentinel github page at https://github.com/dashpay/sentinel indicates 2 minutes, ie....

*/2 * * * * cd /home/YOURUSERNAME/sentinel && ./venv/bin/python bin/sentinel.py >/dev/null 2>&1

Does anyone know whether 5 minutes or 2 minutes or some other time is best, and whether this may be anything to do with the "Watchdog_expired" issues?
 

qwizzie

Well-known Member
Aug 6, 2014
1,609
764
183
I've noticed "Watchdog_expired" problems too. On the latest one I logged in to my masternode and ran in the sentinel directory:

venv/bin/python bin/sentinel.py

The status of that masternode changed from "Watchdog_expired" in my local wallet to "Enabled" so that seemed to fix it.

I'm wondering whether it's anything to do with the amount of time between Sentinel polls in the cron job. I've set mine to 5 minutes in accordance with the instructions at https://dashpay.atlassian.net/wiki/display/DOC/Updating+to+12.1+-+Masternodes , ie....

*/5 * * * * cd /home/YOURUSERNAME/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2>&1 >> sentinel-cron.log

but I see that the actual sentinel github page at https://github.com/dashpay/sentinel indicates 2 minutes, ie....

*/2 * * * * cd /home/YOURUSERNAME/sentinel && ./venv/bin/python bin/sentinel.py >/dev/null 2>&1

Does anyone know whether 5 minutes or 2 minutes or some other time is best, and whether this may be anything to do with the "Watchdog_expired" issues?
I changed the Sentinel cronjob today from 5 to 2 minutes as i had some troubles with my own masternodes, after they got disconnected somehow without Dash Central noticing (i only noticed they got kicked off the network by checking on Dashninja.pl). During cold restart i also came across the Watchdog_expired notification, setting the cronjob to 2 minutes seems to have helped with that.

It took a long time though before Dashninja.pl and later on Dash Central indicated them as active again.
(slow network propogation i guess).

Edit : i'm not using any special characters in my dash.conf by the way
Edit : also i'm using heavily modified and duplicate monit restart software, which helps with my own setup (which is one server with multiple dedicated IP addresses).
I need to keep an eye on my monit implementation to see if it is doing what i hope its doing :confused:
 
Last edited:

TreasureSeeker

New Member
Apr 28, 2014
25
5
3
Thanks for the replies. I've update my cron to 2 minutes tonight as an experiment - will see how it pans out.
 

Solaso

Member
Aug 20, 2015
134
20
68
*/2 * * * * cd /home/YOURUSERNAME/sentinel && ./venv/bin/python bin/sentinel.py >/dev/null 2>&1

Did you change YOURUSERNAME correct and the path is live?
 

TreasureSeeker

New Member
Apr 28, 2014
25
5
3
Yes, username changed and path to sentinel is correct. Haven't seen any more watchdogs expiring since changing to a 2 minute cronjob rather than 5 mins.