Welcome to the Dash Forum!

Please sign up to discuss the most innovative cryptocurrency!

"watchdog-expired" on all masternodes after 12.1?

Discussion in 'Masternode Questions and Help.' started by edificio, Feb 6, 2017.

  1. edificio

    edificio New Member

    Joined:
    Jan 30, 2017
    Messages:
    28
    Likes Received:
    8
    Trophy Points:
    3
    Hello,

    I just upgraded to 12.1, followed Tao's guide for upgrading and have succesfully started my masternodes.
    But they all say "Watchdog_expired", both via VPS command and in the new Dash Core.

    What is the error exactly and how do i fix it?

    Any help would be much appriciated! :)
     
  2. stan.distortion

    stan.distortion Active Member

    Joined:
    Oct 30, 2014
    Messages:
    840
    Likes Received:
    492
    Trophy Points:
    133
    • Like Like x 1
  3. edificio

    edificio New Member

    Joined:
    Jan 30, 2017
    Messages:
    28
    Likes Received:
    8
    Trophy Points:
    3
    No i have not.. I'll give it a shot, thanks! :)
     
    • Like Like x 1
  4. edificio

    edificio New Member

    Joined:
    Jan 30, 2017
    Messages:
    28
    Likes Received:
    8
    Trophy Points:
    3
    I got to step 5, then i got this error:

    ~/.dash/sentinel$ venv/bin/python bin/sentinel.py
    -342: non-JSON HTTP response with '401 Unauthorized' from server
    Cannot connect to dashd. Please ensure dashd is running and the JSONRPC port is open to Sentinel.

    I really don't know what the JSONRPC port is :) But dashd is running and synced
     
  5. stan.distortion

    stan.distortion Active Member

    Joined:
    Oct 30, 2014
    Messages:
    840
    Likes Received:
    492
    Trophy Points:
    133
    Do you have rpcuser=<replace> and rpcpassword=<replace> in dash.conf? "<replace>" can be changed to anything but as far as I know they're needed to use the RPC interface. rpcallowip=127.0.0.1 should also be in there to only allow connections form the same machine.

    EDIT: A sample, might not need all the options but it runs ok:
    Code:
    rpcuser=user2468
    rpcpassword=87967849bvuiu9dfs6vg65db
    rpcallowip=127.0.0.1
    listen=1
    server=1
    daemon=1
    txindex=1
    logtimestamps=1
    maxconnections=256
    masternode=1
    masternodeprivkey=*********************
    
    EDIT2: If dash.conf edited then client needs to be restarted for changes to take effect.
     
    #5 stan.distortion, Feb 6, 2017
    Last edited: Feb 6, 2017
  6. edificio

    edificio New Member

    Joined:
    Jan 30, 2017
    Messages:
    28
    Likes Received:
    8
    Trophy Points:
    3
    My .dash.conf looks like this:

    rpcuser=wdwqdqwd
    rpcpassword=wd3f23f3d33
    rpcallowip=127.0.0.1
    listen=1
    server=1
    daemon=1
    logtimestamps=1
    maxconnections=256
    masternode=1
    masternodeprivkey=***********************
    addnode=23.23.186.131

    (I changed rpcuser, rpcpassword and masternodeprivkey)

    I seems like i have the things you mentioned?
     
  7. stan.distortion

    stan.distortion Active Member

    Joined:
    Oct 30, 2014
    Messages:
    840
    Likes Received:
    492
    Trophy Points:
    133
    I'm not sure, asked around and someone should be back before too long though. Try this command from inside the sentinel folder, it might give some more info:
    Code:
    ./venv/bin/py.test ./test
    It runs some sentinel self-tests and should come up ok almost immediately.
     
  8. edificio

    edificio New Member

    Joined:
    Jan 30, 2017
    Messages:
    28
    Likes Received:
    8
    Trophy Points:
    3
    Here you go:


    ============================= test session starts ==============================
    platform linux2 -- Python 2.7.6, pytest-3.0.1, py-1.4.31, pluggy-0.3.1
    rootdir: /home/tieb/.dashcore/sentinel, inifile:
    collected 20 items

    test/integration/test_jsonrpc.py F
    test/unit/test_dash_config.py .
    test/unit/test_dashd_data_shims.py ..
    test/unit/test_dashy_things.py .....
    test/unit/test_models.py ..
    test/unit/test_submit_command.py .
    test/unit/models/test_proposals.py ...
    test/unit/models/test_superblocks.py .....

    =================================== FAILURES ===================================
    __________________________________ test_dashd __________________________________

    def test_dashd():
    config_text = DashConfig.slurp_config_file(config.dash_conf)
    network = 'mainnet'
    is_testnet = False
    genesis_hash = u'00000ffd590b1485b3caadc19b22e6379c733355108f107a430458cdf3407ab6'
    for line in config_text.split("\n"):
    if line.startswith('testnet=1'):
    network = 'testnet'
    is_testnet = True
    genesis_hash = u'00000bafbc94add76cb75e2ec92894837288a481e5c005f6563d91623bf8bc2c'

    creds = DashConfig.get_rpc_creds(config_text, network)
    dashd = DashDaemon(**creds)
    assert dashd.rpc_command is not None

    assert hasattr(dashd, 'rpc_connection')

    # Dash testnet block 0 hash == 00000bafbc94add76cb75e2ec92894837288a481e5c005f6563d91623bf8bc2c
    # test commands without arguments
    > info = dashd.rpc_command('getinfo')

    test/integration/test_jsonrpc.py:34:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    lib/dashd.py:42: in rpc_command
    return self.rpc_connection.__getattr__(params[0])(*params[1:])
    venv/local/lib/python2.7/site-packages/bitcoinrpc/authproxy.py:139: in __call__
    response = self._get_response()
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    self = <bitcoinrpc.authproxy.AuthServiceProxy object at 0x7eff02bd7490>

    def _get_response(self):
    http_response = self.__conn.getresponse()
    if http_response is None:
    raise JSONRPCException({
    'code': -342, 'message': 'missing HTTP response from server'})

    content_type = http_response.getheader('Content-Type')
    if content_type != 'application/json':
    raise JSONRPCException({
    > 'code': -342, 'message': 'non-JSON HTTP response with \'%i %s\' from server' % (http_response.status, http_response.reason)})
    E JSONRPCException: -342: non-JSON HTTP response with '401 Unauthorized' from server

    venv/local/lib/python2.7/site-packages/bitcoinrpc/authproxy.py:187: JSONRPCException
    ===================== 1 failed, 19 passed in 0.73 seconds ======================
     
  9. stan.distortion

    stan.distortion Active Member

    Joined:
    Oct 30, 2014
    Messages:
    840
    Likes Received:
    492
    Trophy Points:
    133
    You've tried restarting dashd? Something else maybe worth trying, stop the client (dash-cli stop) and delete all the files with a *.dat extension from the .dashcore folder except wallet.dat, restart the client and test sentinel again. The client will take some time to rebuild those files and sentinel won't run until that's done, it can be checked with "dash-cli mnsync status" but the self-tests should run ok (iirc).

    EDIT: And please remove the "addnode=23.23.186.131" line from dash.conf, apparently it's very old :)
     
  10. edificio

    edificio New Member

    Joined:
    Jan 30, 2017
    Messages:
    28
    Likes Received:
    8
    Trophy Points:
    3
    I don't have a dashcore folder. I just replaced the old dashd with the new one as Tao's guide said. I also haven't removed the old blockchain.. That's probably it?
     
  11. stan.distortion

    stan.distortion Active Member

    Joined:
    Oct 30, 2014
    Messages:
    840
    Likes Received:
    492
    Trophy Points:
    133
    No, the blockchain needs to be reindexed for 12.1 but it's fine otherwise. The all the *.dat files are safe to remove and mncache.dat may be causing your problem, peers.dat can cause some issues, etc. Simpler to remove them all and start fresh if there are any doubts but certainly try deleting mncache.dat.

    Can you post a link to Tao's guide please? I've not read it and I might be getting confused on some points. You've got sentinel in your .dashcore folder and the rest of your files should be in there too but he may be organising folders differently.
     
  12. edificio

    edificio New Member

    Joined:
    Jan 30, 2017
    Messages:
    28
    Likes Received:
    8
    Trophy Points:
    3
  13. stan.distortion

    stan.distortion Active Member

    Joined:
    Oct 30, 2014
    Messages:
    840
    Likes Received:
    492
    Trophy Points:
    133
    Ah, that one. Ok, it's a little out of date for 12.1 and if you didn't copy or renamed any directories for the update the you've probably got a completely new installation, the old client installs in /home/USER/.dash and the new one installs in /home/USER/.dashcore

    Go to the home folder ("cd") and check the output of "cat .dash/dash.conf" and "cat .dashcore/dash.conf", they'll probably be different and the second might come up with nothing if the file exists but it's empty, it will say the file doesn't exist otherwise.
     
  14. edificio

    edificio New Member

    Joined:
    Jan 30, 2017
    Messages:
    28
    Likes Received:
    8
    Trophy Points:
    3
    I think i'm just gonna try to reinstall the nodes completely, i think I've really made a mess of it, and i have no idea which files are where :)
     
    • Agree Agree x 1
  15. stan.distortion

    stan.distortion Active Member

    Joined:
    Oct 30, 2014
    Messages:
    840
    Likes Received:
    492
    Trophy Points:
    133
    That could be the simplest thing to do, it will take a few hours to get the blockchain up to date but at least you'll know everything is as it should be. This script is very useful for managing masternodes btw but it's not up to date with 12.1 yet, it should be up to date in the next few days though so worth checking:
    https://github.com/moocowmoo/dashman
    When it's up to date it will handle installing both the client and Sentinal but the guy that wrote it is one of the core team and they've all been working flat out for about 26 hours, they'll need some sleep before getting to those things :)
     
  16. edificio

    edificio New Member

    Joined:
    Jan 30, 2017
    Messages:
    28
    Likes Received:
    8
    Trophy Points:
    3
    Jesus christ, yes, let them get some sleep :O I'll give it a shot, thank you so much for your help!
     
  17. stan.distortion

    stan.distortion Active Member

    Joined:
    Oct 30, 2014
    Messages:
    840
    Likes Received:
    492
    Trophy Points:
    133
    They're the ones that deserve it really but no problem at all :) I almost forgot to mention, server requirements have been raised for 12.1 so a server upgrade at the same time might make sense. 1GB RAM minimum and 40GB hard drive space, for now you might get away with less hard drive space (it was 20GB initially) but Sentinals requirements mean that space will be needed at some stage and the RAM is definitely needed.

    Feel free to PM if you've any issues, some parts of that update guide can get technical but it's fairly straight forward once you know what each part needs to do:
    https://dashpay.atlassian.net/wiki/display/DOC/Updating+to+12.1+-+Masternodes
     
    • Like Like x 1
  18. AjM

    AjM Well-known Member
    Foundation Member

    Joined:
    Jun 23, 2014
    Messages:
    1,334
    Likes Received:
    571
    Trophy Points:
    283
    Watchdog_expired problem also here, 50% of my masternodes are in Watchdog_expired status,
    first 1.5 hours was ok, enabled, then they changed to Watchdog_expired.

    All nodes and settings are same, and sentinel_debug.log is empty, no errors.
    All nodes are sync and 'dash-cli masternode status' gives "status": "Masternode successfully started"

    Any idea?
     
  19. stan.distortion

    stan.distortion Active Member

    Joined:
    Oct 30, 2014
    Messages:
    840
    Likes Received:
    492
    Trophy Points:
    133
    Not sure but they might be following different sides of a fork, they're at the same block height? "dash-cli getchaintips" might also give some indication if that's happening, if it is they should sort themselves out and all get back on the longest chain but starting with "- reindex" might sort things out quicker if they're being stubborn (can take up to a few hours).
     
  20. AjM

    AjM Well-known Member
    Foundation Member

    Joined:
    Jun 23, 2014
    Messages:
    1,334
    Likes Received:
    571
    Trophy Points:
    283
    All nodes are in the same block height.
    All nodes are installed from the scratch, new os and new dash install and new dowloaded blockchain.
    Current status: 50% Watchdog_expired and other 50% enabled.

    :mad:
     
  21. stan.distortion

    stan.distortion Active Member

    Joined:
    Oct 30, 2014
    Messages:
    840
    Likes Received:
    492
    Trophy Points:
    133
    Just asked around, it sounds like it's a connectivity issue due to the network upgrading and should sort it's self out once things stabilise. It wont effect payments while enforcement is off and if they all check out ok with 'dash-cli masternode status' then all should be good. Might be worth checking them here just to see what another part of the network is seeing:
    https://www.dashninja.pl/
     
    • Like Like x 1
  22. AjM

    AjM Well-known Member
    Foundation Member

    Joined:
    Jun 23, 2014
    Messages:
    1,334
    Likes Received:
    571
    Trophy Points:
    283
    Hmm... after 5 hours Watchdog_expired, they are now again enabled....
    Dashninja still shows 50% is expired but remote wallet say they are enabled all 100%.

    Edit: aaand now they are again expired, going to force start command from remote.
     
  23. AjM

    AjM Well-known Member
    Foundation Member

    Joined:
    Jun 23, 2014
    Messages:
    1,334
    Likes Received:
    571
    Trophy Points:
    283
    Ok, found bug, i am 95% sure, this is it.

    If dash.conf rpcuser and/or rpcpassword include any special character, like !"#¤%&/,
    sentinel wont work like it should be, my 50% nodes was expired because of this, others did not included special chars.

    Its bed time for me, tomorrow i put bug report into GIT.
     
    • Like Like x 2
  24. maky

    maky Member

    Joined:
    Nov 14, 2014
    Messages:
    70
    Likes Received:
    32
    Trophy Points:
    58
    Do not have special chars in rpcuser and rpcpassword and WATCHDOG_EXPIRED problem.
     
  25. Walter

    Walter Active Member
    Masternode Owner/Operator

    Joined:
    Jul 17, 2014
    Messages:
    231
    Likes Received:
    201
    Trophy Points:
    103
    I'm getting the same error:

    ~/.dash/sentinel$ venv/bin/python bin/sentinel.py
    -342: non-JSON HTTP response with '401 Unauthorized' from server
    Cannot connect to dashd. Please ensure dashd is running and the JSONRPC port is open to Sentinel.

    tried a number of things and still no result. tearing my bloody hair out here! lol
     
  26. stan.distortion

    stan.distortion Active Member

    Joined:
    Oct 30, 2014
    Messages:
    840
    Likes Received:
    492
    Trophy Points:
    133
    There shouldn't be any need to if they're ok on the servers
    At least you've got some left to tear out! ;) Are you in the .dash folder or the .dashcore folder? The new one is .dashcore.

    EDIT: Guessing you're using these instructions for Sentinel, linking them just in case:
    https://github.com/dashpay/sentinel

    Everything is the same except the new folder name and Sentinal installation.
     
  27. maky

    maky Member

    Joined:
    Nov 14, 2014
    Messages:
    70
    Likes Received:
    32
    Trophy Points:
    58
    Maybe only I'm so stupid but... do You modify this below or just copy paste into cron like I do?

    */5 * * * * cd /home/YOURUSERNAME/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2&>1 >sentinel-cron.log
    */5 * * * * cd /home/---------HERE---------/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2&>1 >sentinel-cron.log
     
  28. bertlebbert

    bertlebbert Active Member

    Joined:
    Jul 17, 2014
    Messages:
    669
    Likes Received:
    289
    Trophy Points:
    133
    You modify it... replace both 'YOURUSERNAME' and '---------HERE---------"
     
  29. stan.distortion

    stan.distortion Active Member

    Joined:
    Oct 30, 2014
    Messages:
    840
    Likes Received:
    492
    Trophy Points:
    133
    You need to modify it, just one line so delete the last and replace YOURUSERNAME with the name of your home folder, ie. /home/maky/.dashcore/sentinel
     
    • Like Like x 1
  30. maky

    maky Member

    Joined:
    Nov 14, 2014
    Messages:
    70
    Likes Received:
    32
    Trophy Points:
    58
    Yeah, I did it already. I hoped that this is cause of our problem but it is not. I change it to correct and masternodes go to ENABLED but minute later some of them go WATCHDOG state again...

    PS.
    It seems that for me problem is solved. If not I let You know.