Welcome to the Dash Forum!

Please sign up to discuss the most innovative cryptocurrency!

"watchdog-expired" on all masternodes after 12.1?

Discussion in 'Masternode Questions and Help.' started by edificio, Feb 6, 2017.

  1. stan.distortion

    stan.distortion Active Member

    Joined:
    Oct 30, 2014
    Messages:
    840
    Likes Received:
    492
    Trophy Points:
    133
    A few things to try, go into the sentinel folder and run this:
    Code:
    SENTINEL_DEBUG=1 ./venv/bin/python bin/sentinel.py
    That will run sentinel manually with debugging on, it should just run second with a lot of weird looking output, is so that part's ok.

    Another to try is:
    Code:
    dash-cli masternode status
    If all is good it will give a few lines of details and the last line should be:
    Code:
    "status": "Masternode successfully started"
     
  2. Walter

    Walter Active Member
    Masternode Owner/Operator

    Joined:
    Jul 17, 2014
    Messages:
    231
    Likes Received:
    201
    Trophy Points:
    103
    @moocowmoo helped me out with the problem I had. It ended up being a syntax issue. "RPCUSER= " needs to NOT have a space between the = sign and the username supplied. I'm pretty sure that's what he said... It was a really dumb mistake tbh. He was fantastic though, really appreciated him helping like that :)
     
    #32 Walter, Feb 6, 2017
    Last edited: Feb 7, 2017
    • Like Like x 2
  3. AjM

    AjM Well-known Member
    Foundation Member

    Joined:
    Jun 23, 2014
    Messages:
    1,334
    Likes Received:
    571
    Trophy Points:
    283
    Bug confirmed, i am 100% sure about this, all my nodes are ok now.

    v12.1
    If in dash.conf rpcuser and/or rpcpassword include any special character, like !"#¤%&/,
    sentinel wont work like it should be, and masternode status become to watchdog_expired,
    only normal chars are working ok.

    I did have this problem, after fixed/removed all special characters, no problem any more.

    Bug report: https://github.com/dashpay/dash/issues/1323
     
    • Like Like x 3
  4. Stealth923

    Stealth923 Well-known Member
    Foundation Member

    Joined:
    Mar 9, 2014
    Messages:
    343
    Likes Received:
    370
    Trophy Points:
    233
    Just to comment on this - I actually setup new MNs with brand new rpcuser and rpcpassword for 12.1 with special characters and sentinel test script was failing. It does not support special characters.

    Went back to plain letters and numbers and all humming along smoothly now.
     
    • Like Like x 2
    • Agree Agree x 1
  5. TreasureSeeker

    TreasureSeeker New Member

    Joined:
    Apr 28, 2014
    Messages:
    25
    Likes Received:
    5
    Trophy Points:
    3
    I've noticed "Watchdog_expired" problems too. On the latest one I logged in to my masternode and ran in the sentinel directory:

    venv/bin/python bin/sentinel.py

    The status of that masternode changed from "Watchdog_expired" in my local wallet to "Enabled" so that seemed to fix it.

    I'm wondering whether it's anything to do with the amount of time between Sentinel polls in the cron job. I've set mine to 5 minutes in accordance with the instructions at https://dashpay.atlassian.net/wiki/display/DOC/Updating+to+12.1+-+Masternodes , ie....

    */5 * * * * cd /home/YOURUSERNAME/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2>&1 >> sentinel-cron.log

    but I see that the actual sentinel github page at https://github.com/dashpay/sentinel indicates 2 minutes, ie....

    */2 * * * * cd /home/YOURUSERNAME/sentinel && ./venv/bin/python bin/sentinel.py >/dev/null 2>&1

    Does anyone know whether 5 minutes or 2 minutes or some other time is best, and whether this may be anything to do with the "Watchdog_expired" issues?
     
  6. qwizzie

    qwizzie Well-known Member

    Joined:
    Aug 6, 2014
    Messages:
    1,540
    Likes Received:
    725
    Trophy Points:
    183
    I changed the Sentinel cronjob today from 5 to 2 minutes as i had some troubles with my own masternodes, after they got disconnected somehow without Dash Central noticing (i only noticed they got kicked off the network by checking on Dashninja.pl). During cold restart i also came across the Watchdog_expired notification, setting the cronjob to 2 minutes seems to have helped with that.

    It took a long time though before Dashninja.pl and later on Dash Central indicated them as active again.
    (slow network propogation i guess).

    Edit : i'm not using any special characters in my dash.conf by the way
    Edit : also i'm using heavily modified and duplicate monit restart software, which helps with my own setup (which is one server with multiple dedicated IP addresses).
    I need to keep an eye on my monit implementation to see if it is doing what i hope its doing :confused:
     
    #36 qwizzie, Feb 18, 2017
    Last edited: Feb 18, 2017
  7. AjM

    AjM Well-known Member
    Foundation Member

    Joined:
    Jun 23, 2014
    Messages:
    1,334
    Likes Received:
    571
    Trophy Points:
    283
    I have 5 mins in the cron time, and no problems so far.
     
  8. TreasureSeeker

    TreasureSeeker New Member

    Joined:
    Apr 28, 2014
    Messages:
    25
    Likes Received:
    5
    Trophy Points:
    3
    Thanks for the replies. I've update my cron to 2 minutes tonight as an experiment - will see how it pans out.
     
  9. Solaso

    Solaso Member

    Joined:
    Aug 20, 2015
    Messages:
    134
    Likes Received:
    20
    Trophy Points:
    68
    */2 * * * * cd /home/YOURUSERNAME/sentinel && ./venv/bin/python bin/sentinel.py >/dev/null 2>&1

    Did you change YOURUSERNAME correct and the path is live?
     
  10. TreasureSeeker

    TreasureSeeker New Member

    Joined:
    Apr 28, 2014
    Messages:
    25
    Likes Received:
    5
    Trophy Points:
    3
    Yes, username changed and path to sentinel is correct. Haven't seen any more watchdogs expiring since changing to a 2 minute cronjob rather than 5 mins.