TAO'S MASTERNODE SETUP GUIDE FOR DUMMIES (UPDATE GUIDE UPDATED FOR 13.0)

stan.distortion

Active Member
Oct 30, 2014
848
497
133

UdjinM6

Official Dash Dev
Dash Core Team
Moderator
May 20, 2014
3,638
3,538
1,183
Can anybody please let me know what the status "WATCHDOG_EXPIRED" means?

Thank you,
It means your node have not yet seen proper "watchdog" message (which is triggered by Sentinel) from the masternode in WATCHDOG_EXPIRED state. While in migrations period it's ok to see this for all 12.0 masternodes (they don't have one) and for some 12.1 masternodes (due to connectivity issues - 12.1 masternode could be surrounded by 12.0 nodes only which do not recognize this message and do not relay it).
 

Dusan

New Member
Oct 7, 2016
32
9
8
I have the same problem. Running 2 MN on 1 Server with 2 different IP's on VULTR (15GB SSD, 1 CPU 768MB Memory 1000GB Bandwidth).
On my local PC it showed ENABLED first but after a while is the status WATCHDOG_EXPIRED.
Also on dashninja.pl it was 100% active but now it shows the status Inactive(0%).

But if I look on Server, both 2 dash demons are running.
ps ax| grep dashd
22216 ? SLsl 0:51 ./dashd
22271 ? SLsl 0:57 ./dashd


Also staus shows
./dash-cli mnsync status
{
"AssetID": 999,
"AssetName": "MASTERNODE_SYNC_FINISHED",
"Attempt": 0,
"IsBlockchainSynced": true,
"IsMasternodeListSynced": true,
"IsWinnersListSynced": true,
"IsSynced": true,
"IsFailed": false
}


So should I not care about it in migration period? Or is something wrong with it please?
 
Last edited:

stan.distortion

Active Member
Oct 30, 2014
848
497
133
I have the same problem. Running 2 MN on 1 Server with 2 different IP's on VULTR (15GB SSD, 1 CPU 768MB Memory 1000GB Bandwidth).
On my local PC it showed ENABLED first but after a while is the status WATCHDOG_EXPIRED.
Also on dashninja.pl it was 100% active but now it shows the status Inactive(0%).

But if I look on Server, both 2 dasd demons are running.
ps ax| grep dashd
22216 ? SLsl 0:51 ./dashd
22271 ? SLsl 0:57 ./dashd


Also staus shows
./dash-cli mnsync status
{
"AssetID": 999,
"AssetName": "MASTERNODE_SYNC_FINISHED",
"Attempt": 0,
"IsBlockchainSynced": true,
"IsMasternodeListSynced": true,
"IsWinnersListSynced": true,
"IsSynced": true,
"IsFailed": false
}


So should I not care about it in migration period? Or is something wrong with it please?
What's the output of "dash-cli masternode status", same for both? Check for special characters in the RPC settings in dash.conf (RPC user and pass), that's caused trouble for some. looks like then need alphanumeric only.
 

UdjinM6

Official Dash Dev
Dash Core Team
Moderator
May 20, 2014
3,638
3,538
1,183
I have the same problem. Running 2 MN on 1 Server with 2 different IP's on VULTR (15GB SSD, 1 CPU 768MB Memory 1000GB Bandwidth).
On my local PC it showed ENABLED first but after a while is the status WATCHDOG_EXPIRED.
Also on dashninja.pl it was 100% active but now it shows the status Inactive(0%).

But if I look on Server, both 2 dash demons are running.
ps ax| grep dashd
22216 ? SLsl 0:51 ./dashd
22271 ? SLsl 0:57 ./dashd


Also staus shows
./dash-cli mnsync status
{
"AssetID": 999,
"AssetName": "MASTERNODE_SYNC_FINISHED",
"Attempt": 0,
"IsBlockchainSynced": true,
"IsMasternodeListSynced": true,
"IsWinnersListSynced": true,
"IsSynced": true,
"IsFailed": false
}


So should I not care about it in migration period? Or is something wrong with it please?
Masternodes in WATCHDOG_EXPIRED should be paid during migration period too. The issue most likely is caused by lack of connectivity (e.g. 12.1 mn is surrounded by 12.0 nodes only, which can't relay new messages).
 

Dusan

New Member
Oct 7, 2016
32
9
8
What's the output of "dash-cli masternode status", same for both? Check for special characters in the RPC settings in dash.conf (RPC user and pass), that's caused trouble for some. looks like then need alphanumeric only.
dash-cli masternode status
{
.
.
"status": "Masternode successfully started"
}

..both OK

The dash.conf worked fine for me in 12.0.58.
I just copied it and changed in it:
masternodeaddr=111.111.111.111:9999 to externalip=111.111.111.111 without 9999,
all other remained the same.
 
  • Like
Reactions: stan.distortion

Dusan

New Member
Oct 7, 2016
32
9
8
Masternodes in WATCHDOG_EXPIRED should be paid during migration period too. The issue most likely is caused by lack of connectivity (e.g. 12.1 mn is surrounded by 12.0 nodes only, which can't relay new messages).
Thanks for Info

When I look peers on my local PC there are more 12.1 MN's surrounded then 12.0 MN's.

How can I display peers in console on the Server please?
 
  • Like
Reactions: stan.distortion

UdjinM6

Official Dash Dev
Dash Core Team
Moderator
May 20, 2014
3,638
3,538
1,183
Thanks for Info

When I look peers on my local PC there are more 12.1 MN's surrounded then 12.0 MN's.

How can I display peers in console on the Server please?
"dash-cli getpeerinfo" to get full info or "dash-cli getpeerinfo | grep version" to get only list of their protocols
 

bhkien

Active Member
Mar 31, 2014
462
286
133
Canada
vietriches.com
It means your node have not yet seen proper "watchdog" message (which is triggered by Sentinel) from the masternode in WATCHDOG_EXPIRED state. While in migrations period it's ok to see this for all 12.0 masternodes (they don't have one) and for some 12.1 masternodes (due to connectivity issues - 12.1 masternode could be surrounded by 12.0 nodes only which do not recognize this message and do not relay it).
Thank you very much.
 
  • Like
Reactions: stan.distortion

Dusan

New Member
Oct 7, 2016
32
9
8
It seams to be 12.1 majority... similar by both MN's

dash-cli getpeerinfo | grep subver
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.0.58/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.0.58/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
 
  • Like
Reactions: bhkien

stan.distortion

Active Member
Oct 30, 2014
848
497
133
It seams to be 12.1 majority... similar by both MN's

dash-cli getpeerinfo | grep subver
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.0.58/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.0.58/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
"subver": "/Dash Core:0.12.1/",
It can still be isolated in other parts of the network, there are a lot of islands with one version or the other. It's perfectly normal, you don't realise it as a user but it's kind of scary from an admin perspective :) Try running "dash-cli getchaintips" sometime, they're places where the network has begun to fork, there will only be a few on a new installation but there are pages of them on an old one! They all fix themselves though, it's extremely fault-tolerant and that's what the network is doing at the mo, trying to go with one version or the other. The new version will accept the old one but the old can't accept the new so it can only go one way but it's chaotic while it sorts its self out.

"Masternode successfully started" on both clients means all is good. You could try checking them in dashninja to see what another part of the network is seeing, you may get the same results as your own client, maybe the reverse but it's a good service, chances are they'll both show up as active:
https://www.dashninja.pl/
 
  • Like
Reactions: Dusan

Dusan

New Member
Oct 7, 2016
32
9
8
Maybe I did not properly do the step 10 at Updating+to+12.1
*/5 * * * * cd /home/YOURUSERNAME/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2>&1 >> sentinel-cron.log

should it be run as root or user?

*/5 * * * * root cd /home/YOURUSERNAME/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2>&1 > sentinel-cron.log
*/5 * * * * MNuser cd /home/YOURUSERNAME/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2>&1 > sentinel-cron.log
 

Dusan

New Member
Oct 7, 2016
32
9
8
On my local PC it showed ENABLED first but after a while is the status WATCHDOG_EXPIRED.
The Problem was that this command did not run from crontab (advised to run every 5 min., I set it up every one minit):
cd /home/YOURUSERNAME/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2>&1 >> sentinel-cron.log
 

stan.distortion

Active Member
Oct 30, 2014
848
497
133
The Problem was that this command did not run from crontab (advised to run every 5 min., I set it up every one minit):
cd /home/YOURUSERNAME/.dashcore/sentinel && ./venv/bin/python bin/sentinel.py 2>&1 >> sentinel-cron.log
You've changed YOURUSERNAME to the name of your home folder? It shouldn't be run as root and if it has you may need to change permissions on some files before it can be run as a normal user.
 

GrandMasterDash

Well-known Member
Masternode Owner/Operator
Jul 12, 2015
2,740
979
183
In "Remote wallet -- in datacenter, has no funds, and runs Sentinel"
...
step 5) delete debug.log and all cache files new dashcore folder

rm -f .dashcore/{debug.log,peers.dat,fee_estimates.dat,mn*.dat}


... it should also remove budget.dat?

rm -f .dashcore/{budget.dat,debug.log,peers.dat,fee_estimates.dat,mn*.dat}
 

Mirage1981

New Member
Feb 10, 2017
1
0
1
39
Thanks for the update,

but I run into some problems with the sentinel at step 8:
venv/bin/python bin/sentinel.py
should return nothing but silence. This is how you know it's working.
but I receive:
Traceback (most recent call last):
File "bin/sentinel.py", line 209, in <module>
main()
File "bin/sentinel.py", line 144, in main
if not is_dashd_port_open(dashd):
it wont let me copy the full list

any help??
 

MrMime

Well-known Member
Foundation Member
Mar 9, 2014
169
36
188
Code:
Traceback (most recent call last):
  File "bin/sentinel.py", line 206, in <module>
    Transient.set(mutex_key, misc.now(), timeout_seconds)
  File "lib/models.py", line 662, in set
    setting, created = Setting.get_or_create(name=setting_name, defaults=setting_dikt)
  File "/home/mrmime/.dashcore/sentinel/venv/local/lib/python2.7/site-packages/peewee.py", line 4807, in get_or_create
    return cls.create(**params), True
  File "/home/mrmime/.dashcore/sentinel/venv/local/lib/python2.7/site-packages/peewee.py", line 4776, in create
    inst.save(force_insert=True)
  File "/home/mrmime/.dashcore/sentinel/venv/local/lib/python2.7/site-packages/playhouse/signals.py", line 69, in save
    ret = super(Model, self).save(*args, **kwargs)
  File "/home/mrmime/.dashcore/sentinel/venv/local/lib/python2.7/site-packages/peewee.py", line 4982, in save
    pk_from_cursor = self.insert(**field_dict).execute()
  File "/home/mrmime/.dashcore/sentinel/venv/local/lib/python2.7/site-packages/peewee.py", line 3435, in execute
    cursor = self._execute()
  File "/home/mrmime/.dashcore/sentinel/venv/local/lib/python2.7/site-packages/peewee.py", line 2826, in _execute
    return self.database.execute_sql(sql, params, self.require_commit)
  File "/home/mrmime/.dashcore/sentinel/venv/local/lib/python2.7/site-packages/peewee.py", line 3683, in execute_sql
    self.commit()
  File "/home/mrmime/.dashcore/sentinel/venv/local/lib/python2.7/site-packages/peewee.py", line 3507, in __exit__
    reraise(new_type, new_type(*exc_args), traceback)
  File "/home/mrmime/.dashcore/sentinel/venv/local/lib/python2.7/site-packages/peewee.py", line 3676, in execute_sql
    cursor.execute(sql, params or ())
peewee.OperationalError: attempt to write a readonly database
So I get this error when I try to run sentinel w/o sudo, seems to work okay with sudo however, in getting no response. MN is fully synced, but appears that crontab eventually fails, and I resort back to inactive status after a couple hours.. Any similar issues?

Got some assistance, likely I ran things as root, derp :/
 
Last edited:

Janus Anderson

New Member
Feb 13, 2017
3
0
1
48
I've been trying to figure out this problem for a couple of hours now - (I'm a relative linux noob so forgive me if I'm missing something obvious).

I've been trying to update my masternode to 12.1, but when I try to start the sentinel, I get:

> ~/dashcore/sentinel$ venv/bin/python bin/sentinel.py
> -342: non-JSON HTTP response with '401 Unauthorized' from server
> Cannot connect to dashd. Please ensure dashd is running and the JSONRPC port is open to Sentinel.

My masternode is running off of vultr as the guide suggests.

I have tried:
* verifying that my rpcuser and rpcpassword in dash.conf do not have any special characters
* tried adding :9999 to the externalip in dash.conf.
* starting an entirely new ubuntu 14.04 instance and going through the setup from scratch, this still happens.

Also, I noticed in the step where you start the dash masternode in the guide (./dashd), if you don't specify the -daemon flag it will not give you back shell control, it just hangs.

I can verify that the dash server is running:

:~/dashcore$ ./dash-cli mnsync status
{
"AssetID": 999,
"AssetName": "MASTERNODE_SYNC_FINISHED",
"Attempt": 0,
"IsBlockchainSynced": true,
"IsMasternodeListSynced": true,
"IsWinnersListSynced": true,
"IsSynced": true,
"IsFailed": false
}

I am also using the default firewall specified in the guide. What am I missing here? Thanks for any help :)
 

Bertrand256

Active Member
Feb 13, 2017
230
279
123
Dash Address
XwZzf7yqYoUBnDFqE7r3zuNmpwKo1CYLMC
@Janus, did you notice, that along with version 12.1, dashd configuration directory has changed from .dash to .dashcore? Make sure, that you modify .dashcore/dash.conf file, not the .dash/dash.conf.
 

Janus Anderson

New Member
Feb 13, 2017
3
0
1
48
Ok, I figured it out. That wasn't it exactly but I did make a mistake with the directory structure because I'm a linux noob.

Apparently my directory was named 'dashcore' and not '.dashcore' which are different :) Seems to be working now, thanks!
 

camosoul

Grizzled Member
Sep 19, 2014
2,263
1,130
1,183
Masternodes in WATCHDOG_EXPIRED should be paid during migration period too. The issue most likely is caused by lack of connectivity (e.g. 12.1 mn is surrounded by 12.0 nodes only, which can't relay new messages).
Does dashd have an option similar to the banhammer that qt has? I monitor the peers list and give any 12.0 node a 1 week banhammer in qt...
 

UdjinM6

Official Dash Dev
Dash Core Team
Moderator
May 20, 2014
3,638
3,538
1,183
Does dashd have an option similar to the banhammer that qt has? I monitor the peers list and give any 12.0 node a 1 week banhammer in qt...
Code:
help setban
Code:
setban "ip(/netmask)" "add|remove" (bantime) (absolute)

Attempts add or remove a IP/Subnet from the banned list.

Arguments:
1. "ip(/netmask)" (string, required) The IP/Subnet (see getpeerinfo for nodes ip) with a optional netmask (default is /32 = single ip)
2. "command"      (string, required) 'add' to add a IP/Subnet to the list, 'remove' to remove a IP/Subnet from the list
3. "bantime"      (numeric, optional) time in seconds how long (or until when if [absolute] is set) the ip is banned (0 or empty means using the default time of 24h which can also be overwritten by the -bantime startup argument)
4. "absolute"     (boolean, optional) If set, the bantime must be a absolute timestamp in seconds since epoch (Jan 1 1970 GMT)

Examples:

> dash-cli setban "192.168.0.6" "add" 86400
> dash-cli setban "192.168.0.0/24" "add"
> curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "setban", "params": ["192.168.0.6", "add" 86400] }' -H 'content-type: text/plain;' http://127.0.0.1:9998/
 

johnambush

New Member
Feb 19, 2015
6
1
3
Hi guys. I had masternode set up on vultr till new version came up. There were some hardware requirments, minimun and recommended. So my question is: is that that minimal vultr server enough for masternode or do I need to rent bigger? And What about raspberry pi2?