Inconsistency in Masternode - Upate to 0.12.0.58 asap !

kot

Administrator
Dash Core Team
Moderator
Foundation Member
Masternode Owner/Operator
Dash Support Group
Mar 17, 2015
710
1,878
263
Dear All,
Here is the incident root cause analysis done by @UdjinM6 (many thanks for that!)

Forked Blocks:
Hash: 00000000000047e6e7aebb3fa1a0565a283f05d11d9324c23252424fb6e6518d
Previous Block: 000000000002dc1c76bbac26dede4791e11565f8bed8030652d8823afd1443c8
Height: 523412 <-------------
Transaction Merkle Root: d106445cd5dc1bff22b37fc4fabcea78593a19aa9e93307231ab90355a5faf2a
Time: 1471728542 (2016-08-20 21:29:02)
Difficulty: 20 351.533 (Bits: 1b03385f)
Nonce: 2929999341
Payee: Xoi3AgTmuBGR2beRJsXKrLtsX8WAtZweZP <-------------

Hash: 000000000000e54f036576a10597e0e42cc22a5159ce572f999c33975e121d4d
Previous Block: 000000000002dc1c76bbac26dede4791e11565f8bed8030652d8823afd1443c8
Height: 523412 <-------------
Transaction Merkle Root: 9f8f2556ad9e488596adb21fb3fc161b9b10b744a68920806b72908ea49b7b01
Time: 1471728541 (2016-08-20 21:29:01)
Difficulty: 20 351.533 (Bits: 1b03385f)
Nonce: 1110724282
Payee: XtCicpfofmiPUzDQzAmdy6Sf5zCmr3cXhi <-------------

What happened :
1. We had an inconsistency in masternode list between pre-0.12.0.58 (57?) and 0.12.0.58 nodes, _probably_ due to a bug in ipv6 MN signatures itself or due to its fixes #847 and #858 (improved in #860) /facepalm here/ or both. Anyway there must be some inconsistency to trigger (2).
2. Masternode list inconsistency was the reason why results of voting for a winning masternode could look like this: { "523412": "XtCicpfofmiPUzDQzAmdy6Sf5zCmr3cXhi:5, Xoi3AgTmuBGR2beRJsXKrLtsX8WAtZweZP:5" } Note, that neither one of masternodes received enough votes (6+) to be declared a clear winner.
3. We have a rule which says that if there are not enough votes to declare a clear winner, network should accept a block with any of nodes (which were voted for) included as valid (https://github.com/dashpay/dash/blob/master/src/masternode-payments.cpp#L519)
4. However there probably was large enough number of nodes who had vote results like { "523412": "XtCicpfofmiPUzDQzAmdy6Sf5zCmr3cXhi:4, Xoi3AgTmuBGR2beRJsXKrLtsX8WAtZweZP:6" } or { "523412": "XtCicpfofmiPUzDQzAmdy6Sf5zCmr3cXhi:6, Xoi3AgTmuBGR2beRJsXKrLtsX8WAtZweZP:4" } and they thought that there was a clear winner but it was different for each side.
5. So when two large miners solved block with different winner every node which had a clear winner recieved a block it liked, rejected another block and later continued with choosen branch.
6. Miner who solved 00000000000047e6e7aebb3fa1a0565a283f05d11d9324c23252424fb6e6518d was minnig on pre-0.12.0.58 node and like probably most of pre-0.12.0.58 nodes had a clear winner, so he also rejected another block and continued minig on his chain.
7. Nodes that had no clear winner probably accepted both blocks they received but later just continued with the longest chain.

Temporary solution:
If suggestion about masternode list inconsistency (1) above is right then having most nodes on 0.12.0.58 should reduce inconsistency i.e. it will be much more unlikely to have a situation when there are two different winners with 6+ votes for different parts of network.

Current known issues:
We can't get rid of masternode list inconsistency completely right now due to the way it's implemented.

Thoughts on future solutions
:
Since masternode winner list is actually a part of the consensus, we need to implement a deterministic solution to avoid any inconsistency. This can be either some on-chain solution or off-chain protocol enhancement like another round of voting or some kind of vote synchronization among top10 masternodes.

Final thoughts:
Having most of masternodes and all (or at least all major) miners on 0.12.0.58 should be enough for now. In long-term a solution must be provided to address inconsistency issue.
 

rustycase

Active Member
Apr 19, 2016
495
116
113
Thanks to all the devoted humans who are right on top of discrepancies that may occur !
Much appreciation !!!
rc
 

camosoul

Grizzled Member
Sep 19, 2014
2,261
1,130
1,183
I noticed an oddity where dashninja says I got a payment, but it shows an even 1000 balance. Only happened twice... Related?
 

tungfa

Grizzled Member
Dash Core Team
Moderator
Foundation Member
Masternode Owner/Operator
Apr 9, 2014
8,902
6,735
1,283
We are "nearly" there but still .... these last % of old Mn's .....
PLEASE UPDATE !

 

qwizzie

Well-known Member
Aug 6, 2014
1,697
815
183
We are "nearly" there but still .... these last % of old Mn's .....
PLEASE UPDATE !

Couple of questions :

* how much is nearly there ? 85% or 90% or higher ?
* zpool.ca is currently paying MN fees so i assumed enforcement was "on" by now .. is it still "off" ?
* why are we not using more official channels like Twitter / Facebook / Slack to reach more masternodes owners and explain to them why exactly they need to upgrade ?

I cant help but feel if we still need 6.7% or more to switch to latest version it could actually take awhile, as those 0.12.0.55 owners dont seem to be aware of the enforcement problem
(which may have resulted to not much decline in the 0.12.0.55 %)
 
Last edited:

Balych

Active Member
Sep 12, 2015
365
211
113
Dash Address
Xba1ychX7CjgbRrCKE1LjHjT3jLUhcexs5
so i assumed enforcement was "on" by now .. is it still "off" ?
Code:
$ ./dash-cli spork show
{
    "SPORK_2_INSTANTX" : 978307200,
    "SPORK_3_INSTANTX_BLOCK_FILTERING" : 0,
    "SPORK_5_MAX_VALUE" : 2000,
    "SPORK_7_MASTERNODE_SCANNING" : 978307200,
    "SPORK_8_MASTERNODE_PAYMENT_ENFORCEMENT" : 1472238675,
    "SPORK_9_MASTERNODE_BUDGET_ENFORCEMENT" : 0,
    "SPORK_10_MASTERNODE_PAY_UPDATED_NODES" : 0,
    "SPORK_11_RESET_BUDGET" : 0,
    "SPORK_12_RECONSIDER_BLOCKS" : 0,
    "SPORK_13_ENABLE_SUPERBLOCKS" : 0
}
$ date -d @1472238675
Fri Aug 26 19:11:15 UTC 2016
 

qwizzie

Well-known Member
Aug 6, 2014
1,697
815
183
Code:
$ ./dash-cli spork show
{
    "SPORK_2_INSTANTX" : 978307200,
    "SPORK_3_INSTANTX_BLOCK_FILTERING" : 0,
    "SPORK_5_MAX_VALUE" : 2000,
    "SPORK_7_MASTERNODE_SCANNING" : 978307200,
    "SPORK_8_MASTERNODE_PAYMENT_ENFORCEMENT" : 1472238675,
    "SPORK_9_MASTERNODE_BUDGET_ENFORCEMENT" : 0,
    "SPORK_10_MASTERNODE_PAY_UPDATED_NODES" : 0,
    "SPORK_11_RESET_BUDGET" : 0,
    "SPORK_12_RECONSIDER_BLOCKS" : 0,
    "SPORK_13_ENABLE_SUPERBLOCKS" : 0
}
$ date -d @1472238675
Fri Aug 26 19:11:15 UTC 2016

okay, so this is not about enforcement anymore but its more about the masternode selection winners ? the higher % on 0.12.0.58
the less chance on accidental split offs due to multiple winners ?

Also crowning site needs updating i think with regards to enforcement, it still shows :
Payment Enforcement Status: 'On' since 2015-08-22-12:00
 
Last edited:

tungfa

Grizzled Member
Dash Core Team
Moderator
Foundation Member
Masternode Owner/Operator
Apr 9, 2014
8,902
6,735
1,283
Couple of questions :

* how much is nearly there ? 85% or 90% or higher ?
* why are we not using more official channels like Twitter / Facebook / Slack to reach more masternodes owners and explain to them why exactly they need to upgrade ?

I cant help but feel if we still need 6.7% or more to switch to latest version it could actually take awhile, as those 0.12.0.55 owners dont seem to be aware of the enforcement problem
(which may have resulted to not much decline in the 0.12.0.55 %)
- we are using all channels !
- 85-90% is nearly 100% - correct ?
(and better than the initial 67 or whatever it was)

we can survive without the last couple of % but it would be 'nice' to finally get these old nodes updated
so i keep pushing this a bit longer :rolleyes:
 
  • Like
Reactions: qwizzie

Darren

Active Member
Aug 3, 2016
196
153
103
New Hampshire
www.darrentapp.com
I noticed that the masternode payment enforcement number changed. Does that mean that enforcement is back on?

I thought a screen shot here, but I'm having some serious upload problems. Anyway:

0.12.0.58: 83.8%
0.12.0.59: 0.5%
 

tungfa

Grizzled Member
Dash Core Team
Moderator
Foundation Member
Masternode Owner/Operator
Apr 9, 2014
8,902
6,735
1,283
yes enforcement is back on
59 ? might be a testrun by some devs :rolleyes:
 

tungfa

Grizzled Member
Dash Core Team
Moderator
Foundation Member
Masternode Owner/Operator
Apr 9, 2014
8,902
6,735
1,283
84% - Not Bad !

not sure what to do about these other 16% - they seem to be officially MIA and do NOT care ??

 
Sep 1, 2016
59
7
48
41
84% - Not Bad !

not sure what to do about these other 16% - they seem to be officially MIA and do NOT care ??

Isn't that alot of nodes still though? 16% of 4000 is like a 500+ nodes...

100-16 =84, so MNOs get a B grade for upgrading.
 

flare

Administrator
Dash Core Team
Moderator
May 18, 2014
2,286
2,404
1,183
Germany
84% - Not Bad !

not sure what to do about these other 16% - they seem to be officially MIA and do NOT care ??

The problem is that as long as there are 350+ nodes with closed ports, dashninja is unable to determine their current version

upload_2016-9-6_15-1-24.png
 

tungfa

Grizzled Member
Dash Core Team
Moderator
Foundation Member
Masternode Owner/Operator
Apr 9, 2014
8,902
6,735
1,283
Masternode versions (only compatible protocols)0.12.0.58 (70103): 84.2 %
;)

PLEASE check your Port settings !! (as flare pointed out above)

Call to all Masternode owners hosting at Amazon AWS, please check your firewall rules as your ports are closed !!
We are trying to get the last 16% of not updated MN’s onto 0.12.0.58 - you are a big part of this (as your ports are closed)
PLEASE check your Ports - or change providers !
https://www.dashninja.pl/?mnregexp=(Closed)#mnlistdetail
 

tungfa

Grizzled Member
Dash Core Team
Moderator
Foundation Member
Masternode Owner/Operator
Apr 9, 2014
8,902
6,735
1,283
85% - making progress :rolleyes:

MN numbers vary a bit these days - there are some OTC trades going on and MN's are changing hands - moving to new owners/hosting/....
(wait a bit - and they will all be back on)

 

camosoul

Grizzled Member
Sep 19, 2014
2,261
1,130
1,183
84% - Not Bad !

not sure what to do about these other 16% - they seem to be officially MIA and do NOT care ??

If they had to show up for votes or get kicked, by implementing an active abstain, so that abstain could be separated from null, this wouldn't happen. Voting needs to be part of the PoService, and this is how to add it. They would be kicked off and we wouldn't have to worry about them mucking up the network.

Between active abstain and protocol update, no more of this crap would happen.
 

nmarley

Moderator
Dash Core Team
Moderator
Jun 28, 2014
366
424
133
If they had to show up for votes or get kicked, by implementing an active abstain, so that abstain could be separated from null, this wouldn't happen. Voting needs to be part of the PoService, and this is how to add it. They would be kicked off and we wouldn't have to worry about them mucking up the network.

Between active abstain and protocol update, no more of this crap would happen.
Votes or not, I don't see how they're still on the network if their ports are closed. If ports are closed then they're not able to service the network, right? How are other MNs able to connect with them? What am I missing here?
 

tungfa

Grizzled Member
Dash Core Team
Moderator
Foundation Member
Masternode Owner/Operator
Apr 9, 2014
8,902
6,735
1,283