Problems after upgrading to 12.2

Apr 24, 2017
132
30
78
51
Italy
pietrosperoni.it
Dash Address
XsSU7489b1N3F2JCiJ6guBCk1cYuxAEhBQ
From when I upgraded to 12.2 I keep having problems with the masternodes.

They keep not working. One is refusing to even start syncing. After 12 hours it still says:

{
"AssetID": 0,
"AssetName": "MASTERNODE_SYNC_INITIAL",
"AssetStartTime": 1510777732,
"Attempt": 0,
"IsBlockchainSynced": false,
"IsMasternodeListSynced": false,
"IsWinnersListSynced": false,
"IsSynced": false,
"IsFailed": false
}

this one is on an Ubuntu 17.10 x64 with 1024 MB and 25 GB SSD.

On the other side the one on CentOS 7 x64 with 1024 MB and 25 GB SSD.
seem to start ok and now is:
{
"AssetID": 999,
"AssetName": "MASTERNODE_SYNC_FINISHED",
"AssetStartTime": 1510682349,
"Attempt": 0,
"IsBlockchainSynced": true,
"IsMasternodeListSynced": true,
"IsWinnersListSynced": true,
"IsSynced": true,
"IsFailed": false
}
But on dashcentral it says:
Error: Your masternode dropped off the masternode list and is not enabled!
and IP not available

I did everything that was told to do for the upgrade in the instructions. Is there anything I needed to do.

Thanks.
 
Apr 24, 2017
132
30
78
51
Italy
pietrosperoni.it
Dash Address
XsSU7489b1N3F2JCiJ6guBCk1cYuxAEhBQ
Dont trust dashcentral.
Look directly at the place where dashcentral gets its info, which is https://www.dashninja.pl/
so on dashcentral it says that the first one has the port open (but it is still not syncing and thus not working)
while the second is now working (but it has the port closed). If this did anything it added to the confusion o_O

So the first question would be, why is the first masternode not syncing?

Once that is handled I will see if the other is fixed or not
 

strophy

Administrator
Dash Core Team
Dash Support Group
Feb 13, 2016
781
489
133
You need to start from a solid base before we start looking at problems with your configuration. Your spec is currently probably too low, you may be running out of memory while while processing network votes. You should currently be aiming for around 2GB RAM + 4GB swap.
 

UdjinM6

Official Dash Dev
Core Developer
Dash Core Team
May 20, 2014
3,639
3,537
1,183
Use "dash-cli getinfo" and check if you node is on the same block as https://explorer.dash.org or https://chainz.cryptoid.info/dash/
If it's not, find the next block (block in getinfo output + 1) in either one of explorers, grab its hash and use these commands:
dash-cli mnsync reset
dash-cli reconsiderblock <blockhashhere>
dash-cli clearbanned

(<blockhashhere> - block hash from block explorer, without < >)

Then start it from your local 12.2 wallet again.
 

UdjinM6

Official Dash Dev
Core Developer
Dash Core Team
May 20, 2014
3,639
3,537
1,183
If you run masternode as a qt wallet you are doing it wrong, don't do this :)
For local wallet (the one you start your remote masternode from) - yes, ignore the dash-cli part
 
  • Like
Reactions: Pietro Speroni
Apr 24, 2017
132
30
78
51
Italy
pietrosperoni.it
Dash Address
XsSU7489b1N3F2JCiJ6guBCk1cYuxAEhBQ
Thank you very much.
I investigated a bit more,

First of all, I do have dash-cli!!!
It's just that to call it I need to call ./dash-cli and not just dash-cli.
Some things are so natural for some people that they forget to tell them :)

So I run the dash-cli and here is what I got:

{
"version": 120201,
"protocolversion": 70208,
"walletversion": 61000,
"balance": 0.00000000,
"privatesend_balance": 0.00000000,
"blocks": 769068,
"timeoffset": 0,
"connections": 8,
"proxy": "",
"difficulty": 35181684.73744042,
"testnet": false,
"keypoololdest": 1508498810,
"keypoolsize": 1001,
"paytxfee": 0.00000000,
"relayfee": 0.00010000,
"errors": ""
}

Now the block where it is blocked is 769068. So I should look for the hash of 769069
Which I think is here:
https://explorer.dash.org/block/000000000000006886127fe2947b433336715331e4afec2abbe039099b5e2cb1

The hash then is this:
000000000000006886127fe2947b433336715331e4afec2abbe039099b5e2cb1

I don't have a gui :).

Then:
~/.dashcore$ ./dash-cli mnsync reset
success

But then when I do:
:~/.dashcore$ ./dash-cli reconsiderblock 000000000000006886127fe2947b433336715331e4afec2abbe039099b5e2cb1
I get:

error code: -5
error message:
Block not found

:(

I also tried to add the next on one, but I got the same error.

Still blocked.
 

demo

Well-known Member
Apr 23, 2016
3,113
263
153
Dash Address
XnpT2YQaYpyh7F9twM6EtDMn1TCDCEEgNX
:(

I also tried to add the next on one, but I got the same error.

Still blocked.
What about the previous block 769067? Or the same 769068? Have you tried to reconsider them?

As long as you are stuck in 769068, the system correctly reports "Block not found" because you are trying to REconsider the non existent (for your system) block 769069. Obviously you cannot REconsider block 769069. You have to consider it first, before REconsidering it!!!!

So maybe you should try to reconsider block 769068, but dont do it unless @UdjinM6 says so.
 
Last edited:

UdjinM6

Official Dash Dev
Core Developer
Dash Core Team
May 20, 2014
3,639
3,537
1,183
Thank you very much.
I investigated a bit more,

First of all, I do have dash-cli!!!
It's just that to call it I need to call ./dash-cli and not just dash-cli.
Some things are so natural for some people that they forget to tell them :)

So I run the dash-cli and here is what I got:

{
"version": 120201,
"protocolversion": 70208,
"walletversion": 61000,
"balance": 0.00000000,
"privatesend_balance": 0.00000000,
"blocks": 769068,
"timeoffset": 0,
"connections": 8,
"proxy": "",
"difficulty": 35181684.73744042,
"testnet": false,
"keypoololdest": 1508498810,
"keypoolsize": 1001,
"paytxfee": 0.00000000,
"relayfee": 0.00010000,
"errors": ""
}

Now the block where it is blocked is 769068. So I should look for the hash of 769069
Which I think is here:
https://explorer.dash.org/block/000000000000006886127fe2947b433336715331e4afec2abbe039099b5e2cb1

The hash then is this:
000000000000006886127fe2947b433336715331e4afec2abbe039099b5e2cb1

I don't have a gui :).

Then:
~/.dashcore$ ./dash-cli mnsync reset
success

But then when I do:
:~/.dashcore$ ./dash-cli reconsiderblock 000000000000006886127fe2947b433336715331e4afec2abbe039099b5e2cb1
I get:

error code: -5
error message:
Block not found

:(

I also tried to add the next on one, but I got the same error.

Still blocked.
Hmmm... interesting.
What is the output of
Code:
./dash-cli getchaintips 3
?
 

UdjinM6

Official Dash Dev
Core Developer
Dash Core Team
May 20, 2014
3,639
3,537
1,183
Or maybe you have no blocks because you have no good peers around you? Pick some masternode from the list, grab its IP and use it
Code:
./dash-cli addnode <IPhere>:9999 onetry
 

demo

Well-known Member
Apr 23, 2016
3,113
263
153
Dash Address
XnpT2YQaYpyh7F9twM6EtDMn1TCDCEEgNX
Or maybe you have no blocks because you have no good peers around you? Pick some masternode from the list, grab its IP and use it
Code:
./dash-cli addnode <IPhere>:9999 onetry
If this is the case, maybe you should propose a DIP.

  1. The masternode who thinks he has bad peers, reports that.
  2. The rest masernodes check these supposed bad peers.
  3. If the majority also agree that they are bad peers, then these bad peers are automatically banned.
  4. If the majority founds that the peers are not bad, then they ban the one who reported them.
  5. The ban remains valid until the bad becomes good, and upon bad's request to 3 random good masternodes who are willing to confirm that badness is gone.
  6. The service of confirming the status change of a banned could require some fee paid by the banned, so that the good will have incentives to provide this service and to accept requests from the banned.
 
Last edited:

UdjinM6

Official Dash Dev
Core Developer
Dash Core Team
May 20, 2014
3,639
3,537
1,183
If this is the case, maybe you should propose a DIP.
The masternode who thinks he has bad peers, reports that.
The rest masernodes check these supposed bad peers.
If the majority also agree that they are bad peers, then these bad peers are automatically banned.
If they found that the peers are not bad, then they ban the one who reported them.
The ban remains valid until the bad becomes good, and upon bad's request to at least 3 random good masternodes to confirm that.
It could be that the whole cluster is isolated due to a fork, so banning more neighbors won't help, you won't discover new peers and will have to pick them manually anyway.
 

demo

Well-known Member
Apr 23, 2016
3,113
263
153
Dash Address
XnpT2YQaYpyh7F9twM6EtDMn1TCDCEEgNX
It could be that the whole cluster is isolated due to a fork, so banning more neighbors won't help, you won't discover new peers and will have to pick them manually anyway.
If the cluster is tottaly isolated due to a fork , then the peers from this cluster do not report bad blocks. They report good blocks which are forked blocks.
The issue resolved by the above proposed DIP is when some mnos report bad blocks (blocks that have corrupted signatures).
Bitcoin (and Dash) always assumes that the majority is good, there is not any dishonest majority protocol implemented yet.
 
Last edited:

UdjinM6

Official Dash Dev
Core Developer
Dash Core Team
May 20, 2014
3,639
3,537
1,183
If the cluster is tottaly isolated due to a fork , then the peers from this cluster do not report bad blocks. They report good blocks which are forked blocks.
he issue resolved by the above proposed DIP is when some mnos report bad blocks (blocks that have corrupted signatures).
Bitcoin (and Dash) always assumes that the majority is good, there is not any dishonest majority protocol implemented yet.
They only report good block if there is a miner. If this cluster consists of a non-mining nodes on a wrong chain and all miners switched to other peers and are mining on the right chain, then the whole cluster will just sit there waiting for blocks forever (well, for 24h actually, until the ban goes away).
 
  • Like
Reactions: demo

demo

Well-known Member
Apr 23, 2016
3,113
263
153
Dash Address
XnpT2YQaYpyh7F9twM6EtDMn1TCDCEEgNX
They only report good block if there is a miner. If this cluster consists of a non-mining nodes on a wrong chain and all miners switched to other peers and are mining on the right chain, then the whole cluster will just sit there waiting for blocks forever (well, for 24h actually, until the ban goes away).
In that case you should not call it "wrong chain".
You should call it "frozen chain", as long as the last block of the banned cluster is not wrong, is it?

For the "frozen chain" problem an improved DIP should be proposed, that takes into account the existence of a miner in the cluster.
 
Last edited:
Apr 24, 2017
132
30
78
51
Italy
pietrosperoni.it
Dash Address
XsSU7489b1N3F2JCiJ6guBCk1cYuxAEhBQ
Hmmm... interesting.
What is the output of
Code:
./dash-cli getchaintips 3
?
So, getinfo now gives:

{
"version": 120201,
"protocolversion": 70208,
"walletversion": 61000,
"balance": 0.00000000,
"privatesend_balance": 0.00000000,
"blocks": 769068,
"timeoffset": 0,
"connections": 23,
"proxy": "",
"difficulty": 35181684.73744042,
"testnet": false,
"keypoololdest": 1508498810,
"keypoolsize": 1001,
"paytxfee": 0.00000000,
"relayfee": 0.00010000,
"errors": ""
}

While getchaintips gives:
{
"height": 769079,
"hash": "000000000000008754811f3bbaacd16d0b3afef07b6a498626bab80fa7027155",
"difficulty": 15215184.12686991,
"chainwork": "0000000000000000000000000000000000000000000000598465deb4ec8673b1",
"branchlen": 11,
"status": "headers-only"
},

{
"height": 769068,
"hash": "00000000000000233e44da14628ac182b13df1774b709cb07ea2233444d92e40",
"difficulty": 35181684.73744042,
"chainwork": "000000000000000000000000000000000000000000000059756f5105d3d81a23",
"branchlen": 0,
"status": "active"
},

{
"height": 769066,
"hash": "00000000000000219eb0ff02c8ac196eeaceee1ce7453d228a1af77e129b607f",
"difficulty": 38353722.43502148,
"chainwork": "00000000000000000000000000000000000000000000005971283f0510761f8a",
"branchlen": 1,
"status": "invalid"
}

did I went off into a sidebranch?
 
Apr 24, 2017
132
30
78
51
Italy
pietrosperoni.it
Dash Address
XsSU7489b1N3F2JCiJ6guBCk1cYuxAEhBQ
I also added my other masternode as a connection with addnode. The reply was silent.
Then I run again the getinfo and the getchaintips. Getchaintips did not change. Getinfo lowered the number of connection from 23 to 21
{
"version": 120201,
"protocolversion": 70208,
"walletversion": 61000,
"balance": 0.00000000,
"privatesend_balance": 0.00000000,
"blocks": 769068,
"timeoffset": 0,
"connections": 21,
"proxy": "",
"difficulty": 35181684.73744042,
"testnet": false,
"keypoololdest": 1508498810,
"keypoolsize": 1001,
"paytxfee": 0.00000000,
"relayfee": 0.00010000,
"errors": ""
}

Thanks for the help. It appears to be still blocked.

Should I just keep adding masternodes?
 

demo

Well-known Member
Apr 23, 2016
3,113
263
153
Dash Address
XnpT2YQaYpyh7F9twM6EtDMn1TCDCEEgNX
As far as I understand it, he told you to add some random ips of masternodes as your peers, so that you increase the chance to discover a good masternode.
Adding your other masternode which was also a bad one, was meaningless.
 
Apr 24, 2017
132
30
78
51
Italy
pietrosperoni.it
Dash Address
XsSU7489b1N3F2JCiJ6guBCk1cYuxAEhBQ
As far as I understand it, he told you to add some random ips of masternodes as your peers, so that you increase the chance to discover a good masternode.
Adding your other masternode which was also a bad one, was meaningless.
no, no, my other one is working now. Sorry, I forgot to mention it.
 

UdjinM6

Official Dash Dev
Core Developer
Dash Core Team
May 20, 2014
3,639
3,537
1,183
So, getinfo now gives:

{
"version": 120201,
"protocolversion": 70208,
"walletversion": 61000,
"balance": 0.00000000,
"privatesend_balance": 0.00000000,
"blocks": 769068,
"timeoffset": 0,
"connections": 23,
"proxy": "",
"difficulty": 35181684.73744042,
"testnet": false,
"keypoololdest": 1508498810,
"keypoolsize": 1001,
"paytxfee": 0.00000000,
"relayfee": 0.00010000,
"errors": ""
}

While getchaintips gives:
{
"height": 769079,
"hash": "000000000000008754811f3bbaacd16d0b3afef07b6a498626bab80fa7027155",
"difficulty": 15215184.12686991,
"chainwork": "0000000000000000000000000000000000000000000000598465deb4ec8673b1",
"branchlen": 11,
"status": "headers-only"
},

{
"height": 769068,
"hash": "00000000000000233e44da14628ac182b13df1774b709cb07ea2233444d92e40",
"difficulty": 35181684.73744042,
"chainwork": "000000000000000000000000000000000000000000000059756f5105d3d81a23",
"branchlen": 0,
"status": "active"
},

{
"height": 769066,
"hash": "00000000000000219eb0ff02c8ac196eeaceee1ce7453d228a1af77e129b607f",
"difficulty": 38353722.43502148,
"chainwork": "00000000000000000000000000000000000000000000005971283f0510761f8a",
"branchlen": 1,
"status": "invalid"
}

did I went off into a sidebranch?
try
./dash-cli mnsync reset
./dash-cli reconsiderblock 00000000000000219eb0ff02c8ac196eeaceee1ce7453d228a1af77e129b607f
./dash-cli clearbanned
 

UdjinM6

Official Dash Dev
Core Developer
Dash Core Team
May 20, 2014
3,639
3,537
1,183
Thanks. Now it works. Do you think you could explain me what happened, and how to get out of it another time?
Code:
dash-cli masternode winners 5000 | grep 769066
12.1:
Code:
"769066": "XwkuyuADEqbwvJudgNVXTDvhDLSyKQhwns:4, XhJEHGdx7Dd7yZtpuaRY9R5VNDfgn9kfqS:6",
12.2:
Code:
"769066": "XwkuyuADEqbwvJudgNVXTDvhDLSyKQhwns:4, XhJEHGdx7Dd7yZtpuaRY9R5VNDfgn9kfqS:3",
actual block, https://explorer.dash.org/block/00000000000000219eb0ff02c8ac196eeaceee1ce7453d228a1af77e129b607f:
Code:
XvhExSNNr97U1ZenWFjJmgD8wc7v88ZUF7: 1.80849243
XwkuyuADEqbwvJudgNVXTDvhDLSyKQhwns: 1.8084924
i.e. miner paid to the other MN and 12.1 nodes rejected it and forked off.
That's a "normal" situation for migration period when spork8 is ON. We tried to recover from it quickly turning spork8 OFF and issuing spork12 to replay few recent blocks but some nodes isolated themselves too quickly it seems. https://www.dash.org/forum/threads/version-12-2-release.17807/page-3#post-146139
 
Apr 24, 2017
132
30
78
51
Italy
pietrosperoni.it
Dash Address
XsSU7489b1N3F2JCiJ6guBCk1cYuxAEhBQ
Code:
dash-cli masternode winners 5000 | grep 769066
12.1:
Code:
"769066": "XwkuyuADEqbwvJudgNVXTDvhDLSyKQhwns:4, XhJEHGdx7Dd7yZtpuaRY9R5VNDfgn9kfqS:6",
12.2:
Code:
"769066": "XwkuyuADEqbwvJudgNVXTDvhDLSyKQhwns:4, XhJEHGdx7Dd7yZtpuaRY9R5VNDfgn9kfqS:3",
actual block, https://explorer.dash.org/block/00000000000000219eb0ff02c8ac196eeaceee1ce7453d228a1af77e129b607f:
Code:
XvhExSNNr97U1ZenWFjJmgD8wc7v88ZUF7: 1.80849243
XwkuyuADEqbwvJudgNVXTDvhDLSyKQhwns: 1.8084924
i.e. miner paid to the other MN and 12.1 nodes rejected it and forked off.
That's a "normal" situation for migration period when spork8 is ON. We tried to recover from it quickly turning spork8 OFF and issuing spork12 to replay few recent blocks but some nodes isolated themselves too quickly it seems. https://www.dash.org/forum/threads/version-12-2-release.17807/page-3#post-146139
Thank you very much @UdjinM6 .

Actually few hours later the masternode broke again. So I started again wondering if I were going to find a way to solve everything by myself this time. But this time I worked fine. Also the other node this morning needed a restart on my computer while the server was running fine. It's all very mysterious still. :).

Maybe @strophy is right and I should upgrade to servers more powerful. But when I look at the statistics it still looks it is only using a fraction of the power and memory available.
 

strophy

Administrator
Dash Core Team
Dash Support Group
Feb 13, 2016
781
489
133
Upgrading your server to the recommended spec will likely cost you $5 extra per month. Not upgrading and continuously being sent to the back of the queue due to memory spikes will currently cost you well over $3000 per month. It's an easy choice ;)
 
  • Like
Reactions: splawik21