"watchdog-expired" on all masternodes after 12.1?

edificio

New Member
Jan 30, 2017
28
8
3
Hello,

I just upgraded to 12.1, followed Tao's guide for upgrading and have succesfully started my masternodes.
But they all say "Watchdog_expired", both via VPS command and in the new Dash Core.

What is the error exactly and how do i fix it?

Any help would be much appriciated! :)
 

edificio

New Member
Jan 30, 2017
28
8
3
I got to step 5, then i got this error:

~/.dash/sentinel$ venv/bin/python bin/sentinel.py
-342: non-JSON HTTP response with '401 Unauthorized' from server
Cannot connect to dashd. Please ensure dashd is running and the JSONRPC port is open to Sentinel.

I really don't know what the JSONRPC port is :) But dashd is running and synced
 

stan.distortion

Active Member
Oct 30, 2014
842
492
133
I got to step 5, then i got this error:

~/.dash/sentinel$ venv/bin/python bin/sentinel.py
-342: non-JSON HTTP response with '401 Unauthorized' from server
Cannot connect to dashd. Please ensure dashd is running and the JSONRPC port is open to Sentinel.

I really don't know what the JSONRPC port is :) But dashd is running and synced
Do you have rpcuser=<replace> and rpcpassword=<replace> in dash.conf? "<replace>" can be changed to anything but as far as I know they're needed to use the RPC interface. rpcallowip=127.0.0.1 should also be in there to only allow connections form the same machine.

EDIT: A sample, might not need all the options but it runs ok:
Code:
rpcuser=user2468
rpcpassword=87967849bvuiu9dfs6vg65db
rpcallowip=127.0.0.1
listen=1
server=1
daemon=1
txindex=1
logtimestamps=1
maxconnections=256
masternode=1
masternodeprivkey=*********************
EDIT2: If dash.conf edited then client needs to be restarted for changes to take effect.
 
Last edited:

edificio

New Member
Jan 30, 2017
28
8
3
My .dash.conf looks like this:

rpcuser=wdwqdqwd
rpcpassword=wd3f23f3d33
rpcallowip=127.0.0.1
listen=1
server=1
daemon=1
logtimestamps=1
maxconnections=256
masternode=1
masternodeprivkey=***********************
addnode=23.23.186.131

(I changed rpcuser, rpcpassword and masternodeprivkey)

I seems like i have the things you mentioned?
 

stan.distortion

Active Member
Oct 30, 2014
842
492
133
My .dash.conf looks like this:

rpcuser=wdwqdqwd
rpcpassword=wd3f23f3d33
rpcallowip=127.0.0.1
listen=1
server=1
daemon=1
logtimestamps=1
maxconnections=256
masternode=1
masternodeprivkey=***********************
addnode=23.23.186.131

(I changed rpcuser, rpcpassword and masternodeprivkey)

I seems like i have the things you mentioned?
I'm not sure, asked around and someone should be back before too long though. Try this command from inside the sentinel folder, it might give some more info:
Code:
./venv/bin/py.test ./test
It runs some sentinel self-tests and should come up ok almost immediately.
 

edificio

New Member
Jan 30, 2017
28
8
3
Here you go:


============================= test session starts ==============================
platform linux2 -- Python 2.7.6, pytest-3.0.1, py-1.4.31, pluggy-0.3.1
rootdir: /home/tieb/.dashcore/sentinel, inifile:
collected 20 items

test/integration/test_jsonrpc.py F
test/unit/test_dash_config.py .
test/unit/test_dashd_data_shims.py ..
test/unit/test_dashy_things.py .....
test/unit/test_models.py ..
test/unit/test_submit_command.py .
test/unit/models/test_proposals.py ...
test/unit/models/test_superblocks.py .....

=================================== FAILURES ===================================
__________________________________ test_dashd __________________________________

def test_dashd():
config_text = DashConfig.slurp_config_file(config.dash_conf)
network = 'mainnet'
is_testnet = False
genesis_hash = u'00000ffd590b1485b3caadc19b22e6379c733355108f107a430458cdf3407ab6'
for line in config_text.split("\n"):
if line.startswith('testnet=1'):
network = 'testnet'
is_testnet = True
genesis_hash = u'00000bafbc94add76cb75e2ec92894837288a481e5c005f6563d91623bf8bc2c'

creds = DashConfig.get_rpc_creds(config_text, network)
dashd = DashDaemon(**creds)
assert dashd.rpc_command is not None

assert hasattr(dashd, 'rpc_connection')

# Dash testnet block 0 hash == 00000bafbc94add76cb75e2ec92894837288a481e5c005f6563d91623bf8bc2c
# test commands without arguments
> info = dashd.rpc_command('getinfo')

test/integration/test_jsonrpc.py:34:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
lib/dashd.py:42: in rpc_command
return self.rpc_connection.__getattr__(params[0])(*params[1:])
venv/local/lib/python2.7/site-packages/bitcoinrpc/authproxy.py:139: in __call__
response = self._get_response()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <bitcoinrpc.authproxy.AuthServiceProxy object at 0x7eff02bd7490>

def _get_response(self):
http_response = self.__conn.getresponse()
if http_response is None:
raise JSONRPCException({
'code': -342, 'message': 'missing HTTP response from server'})

content_type = http_response.getheader('Content-Type')
if content_type != 'application/json':
raise JSONRPCException({
> 'code': -342, 'message': 'non-JSON HTTP response with \'%i %s\' from server' % (http_response.status, http_response.reason)})
E JSONRPCException: -342: non-JSON HTTP response with '401 Unauthorized' from server

venv/local/lib/python2.7/site-packages/bitcoinrpc/authproxy.py:187: JSONRPCException
===================== 1 failed, 19 passed in 0.73 seconds ======================
 

stan.distortion

Active Member
Oct 30, 2014
842
492
133
You've tried restarting dashd? Something else maybe worth trying, stop the client (dash-cli stop) and delete all the files with a *.dat extension from the .dashcore folder except wallet.dat, restart the client and test sentinel again. The client will take some time to rebuild those files and sentinel won't run until that's done, it can be checked with "dash-cli mnsync status" but the self-tests should run ok (iirc).

EDIT: And please remove the "addnode=23.23.186.131" line from dash.conf, apparently it's very old :)
 

edificio

New Member
Jan 30, 2017
28
8
3
I don't have a dashcore folder. I just replaced the old dashd with the new one as Tao's guide said. I also haven't removed the old blockchain.. That's probably it?
 

stan.distortion

Active Member
Oct 30, 2014
842
492
133
I don't have a dashcore folder. I just replaced the old dashd with the new one as Tao's guide said. I also haven't removed the old blockchain.. That's probably it?
No, the blockchain needs to be reindexed for 12.1 but it's fine otherwise. The all the *.dat files are safe to remove and mncache.dat may be causing your problem, peers.dat can cause some issues, etc. Simpler to remove them all and start fresh if there are any doubts but certainly try deleting mncache.dat.

Can you post a link to Tao's guide please? I've not read it and I might be getting confused on some points. You've got sentinel in your .dashcore folder and the rest of your files should be in there too but he may be organising folders differently.
 

stan.distortion

Active Member
Oct 30, 2014
842
492
133
Ah, that one. Ok, it's a little out of date for 12.1 and if you didn't copy or renamed any directories for the update the you've probably got a completely new installation, the old client installs in /home/USER/.dash and the new one installs in /home/USER/.dashcore

Go to the home folder ("cd") and check the output of "cat .dash/dash.conf" and "cat .dashcore/dash.conf", they'll probably be different and the second might come up with nothing if the file exists but it's empty, it will say the file doesn't exist otherwise.
 

edificio

New Member
Jan 30, 2017
28
8
3
I think i'm just gonna try to reinstall the nodes completely, i think I've really made a mess of it, and i have no idea which files are where :)
 

stan.distortion

Active Member
Oct 30, 2014
842
492
133
I think i'm just gonna try to reinstall the nodes completely, i think I've really made a mess of it, and i have no idea which files are where :)
That could be the simplest thing to do, it will take a few hours to get the blockchain up to date but at least you'll know everything is as it should be. This script is very useful for managing masternodes btw but it's not up to date with 12.1 yet, it should be up to date in the next few days though so worth checking:
https://github.com/moocowmoo/dashman
When it's up to date it will handle installing both the client and Sentinal but the guy that wrote it is one of the core team and they've all been working flat out for about 26 hours, they'll need some sleep before getting to those things :)
 

edificio

New Member
Jan 30, 2017
28
8
3
Jesus christ, yes, let them get some sleep :O I'll give it a shot, thank you so much for your help!
 

stan.distortion

Active Member
Oct 30, 2014
842
492
133
They're the ones that deserve it really but no problem at all :) I almost forgot to mention, server requirements have been raised for 12.1 so a server upgrade at the same time might make sense. 1GB RAM minimum and 40GB hard drive space, for now you might get away with less hard drive space (it was 20GB initially) but Sentinals requirements mean that space will be needed at some stage and the RAM is definitely needed.

Feel free to PM if you've any issues, some parts of that update guide can get technical but it's fairly straight forward once you know what each part needs to do:
https://dashpay.atlassian.net/wiki/display/DOC/Updating+to+12.1+-+Masternodes
 
  • Like
Reactions: studioz

AjM

Well-known Member
Foundation Member
Jun 23, 2014
1,335
571
283
Finland
Watchdog_expired problem also here, 50% of my masternodes are in Watchdog_expired status,
first 1.5 hours was ok, enabled, then they changed to Watchdog_expired.

All nodes and settings are same, and sentinel_debug.log is empty, no errors.
All nodes are sync and 'dash-cli masternode status' gives "status": "Masternode successfully started"

Any idea?
 

stan.distortion

Active Member
Oct 30, 2014
842
492
133
Watchdog_expired problem also here, 50% of my masternodes are in Watchdog_expired status,
first 1.5 hours was ok, enabled, then they changed to Watchdog_expired.

All nodes and settings are same, and sentinel_debug.log is empty, no errors.
All nodes are sync and 'dash-cli masternode status' gives "status": "Masternode successfully started"

Any idea?
Not sure but they might be following different sides of a fork, they're at the same block height? "dash-cli getchaintips" might also give some indication if that's happening, if it is they should sort themselves out and all get back on the longest chain but starting with "- reindex" might sort things out quicker if they're being stubborn (can take up to a few hours).
 

AjM

Well-known Member
Foundation Member
Jun 23, 2014
1,335
571
283
Finland
Not sure but they might be following different sides of a fork, they're at the same block height? "dash-cli getchaintips" might also give some indication if that's happening, if it is they should sort themselves out and all get back on the longest chain but starting with "- reindex" might sort things out quicker if they're being stubborn (can take up to a few hours).
All nodes are in the same block height.
All nodes are installed from the scratch, new os and new dash install and new dowloaded blockchain.
Current status: 50% Watchdog_expired and other 50% enabled.

:mad:
 

stan.distortion

Active Member
Oct 30, 2014
842
492
133
Just asked around, it sounds like it's a connectivity issue due to the network upgrading and should sort it's self out once things stabilise. It wont effect payments while enforcement is off and if they all check out ok with 'dash-cli masternode status' then all should be good. Might be worth checking them here just to see what another part of the network is seeing:
https://www.dashninja.pl/
 
  • Like
Reactions: AjM

AjM

Well-known Member
Foundation Member
Jun 23, 2014
1,335
571
283
Finland
Hmm... after 5 hours Watchdog_expired, they are now again enabled....
Dashninja still shows 50% is expired but remote wallet say they are enabled all 100%.

Edit: aaand now they are again expired, going to force start command from remote.
 

AjM

Well-known Member
Foundation Member
Jun 23, 2014
1,335
571
283
Finland
Ok, found bug, i am 95% sure, this is it.

If dash.conf rpcuser and/or rpcpassword include any special character, like !"#¤%&/,
sentinel wont work like it should be, my 50% nodes was expired because of this, others did not included special chars.

Its bed time for me, tomorrow i put bug report into GIT.
 

maky

Member
Nov 14, 2014
71
32
58
Do not have special chars in rpcuser and rpcpassword and WATCHDOG_EXPIRED problem.
 

Walter

Active Member
Masternode Owner/Operator
Jul 17, 2014
231
201
103
I'm getting the same error:

~/.dash/sentinel$ venv/bin/python bin/sentinel.py
-342: non-JSON HTTP response with '401 Unauthorized' from server
Cannot connect to dashd. Please ensure dashd is running and the JSONRPC port is open to Sentinel.

tried a number of things and still no result. tearing my bloody hair out here! lol
 

stan.distortion

Active Member
Oct 30, 2014
842
492
133
Hmm... after 5 hours Watchdog_expired, they are now again enabled....
Dashninja still shows 50% is expired but remote wallet say they are enabled all 100%.

Edit: aaand now they are again expired, going to force start command from remote.
There shouldn't be any need to if they're ok on the servers
I'm getting the same error:

~/.dash/sentinel$ venv/bin/python bin/sentinel.py
-342: non-JSON HTTP response with '401 Unauthorized' from server
Cannot connect to dashd. Please ensure dashd is running and the JSONRPC port is open to Sentinel.

tried a number of things and still no result. tearing my bloody hair out here! lol
At least you've got some left to tear out! ;) Are you in the .dash folder or the .dashcore folder? The new one is .dashcore.

EDIT: Guessing you're using these instructions for Sentinel, linking them just in case:
https://github.com/dashpay/sentinel

Everything is the same except the new folder name and Sentinal installation.
 

maky

Member
Nov 14, 2014
71
32
58
Maybe only I'm so stupid but... do You modify this below or just copy paste into cron like I do?

*/5 * * * * cd /home/YOURUSERNAME/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2&>1 >sentinel-cron.log
*/5 * * * * cd /home/---------HERE---------/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2&>1 >sentinel-cron.log
 

stan.distortion

Active Member
Oct 30, 2014
842
492
133
Maybe only I'm so stupid but... do You modify this below or just copy paste into cron like I do?

*/5 * * * * cd /home/YOURUSERNAME/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2&>1 >sentinel-cron.log
*/5 * * * * cd /home/---------HERE---------/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2&>1 >sentinel-cron.log
You need to modify it, just one line so delete the last and replace YOURUSERNAME with the name of your home folder, ie. /home/maky/.dashcore/sentinel
 
  • Like
Reactions: bertlebbert

maky

Member
Nov 14, 2014
71
32
58
Yeah, I did it already. I hoped that this is cause of our problem but it is not. I change it to correct and masternodes go to ENABLED but minute later some of them go WATCHDOG state again...

PS.
It seems that for me problem is solved. If not I let You know.