My tLP running v0.12.1.0-f81ea67 got stuck on block 138287. Updated it to v0.12.1.0-4b7bd6b and created a fresh working directory. Will provide additional debugging data if it gets stuck again.
missing masternode entry: 6aba20dac2a630319bf396b3be6fa2d0c6c9623767e5392b8543d15dd5773870-1
2017-01-20 11:52:50 CreateTxLockCandidate -- New Transaction Lock Candidate! txid=801ad48bfa986f7d1bb638a6b38e6dc3c2ef3b04f6e1f9e657718edb4e0637e7
2017-01-20 11:52:50 TXLOCKREQUEST -- Transaction Lock Request: 23.22.36.5:19999 /Dash Core:0.12.1/ : accepted 801ad48bfa986f7d1bb638a6b38e6dc3c2ef3b04f6e1f9e657718edb4e0637e7
2017-01-20 11:52:50 CreateTxLockCandidate -- New Transaction Lock Candidate! txid=74a3318e20aa2a2d0e1047a0272d7420b2f20129b4557fbbbd254914cc83ce4a
2017-01-20 11:52:50 TXLOCKREQUEST -- Transaction Lock Request: [2604:a880:1:20::ecf:2000]:19999 /Dash Core:0.12.1/ : accepted 74a3318e20aa2a2d0e1047a0272d7420b2f20129b4557fbbbd254914cc83ce4a
2017-01-20 11:52:50 tor: Thread interrupt
2017-01-20 11:52:50 torcontrol thread exit
2017-01-20 11:52:50 opencon thread interrupt
2017-01-20 11:52:50 addcon thread interrupt
2017-01-20 11:52:50 msghand thread interrupt
2017-01-20 11:52:50 net thread interrupt
2017-01-20 11:52:50 scheduler thread interrupt
2017-01-20 11:52:50 PrepareShutdown: In progress...
2017-01-20 11:52:51 StopNode()
2017-01-20 11:52:51 Verifying mncache.dat format...
2017-01-20 11:52:51 Loaded info from mncache.dat 44ms
2017-01-20 11:52:51 Masternodes: 114, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 0, nDsqCount: 8370
2017-01-20 11:52:51 Writting info to mncache.dat...
2017-01-20 11:52:51 Written info to mncache.dat 37ms
2017-01-20 11:52:51 Masternodes: 114, peers who asked us for Masternode list: 0, peers we asked for Masternode list: 0, entries in Masternode list we asked for: 62, nDsqCount: 8564
2017-01-20 11:52:51 mncache.dat dump finished 83ms
2017-01-20 11:52:51 Verifying mnpayments.dat format...
2017-01-20 11:52:52 Loaded info from mnpayments.dat 274ms
2017-01-20 11:52:52 Votes: 51459, Blocks: 4924
2017-01-20 11:52:52 Writting info to mnpayments.dat...
2017-01-20 11:52:52 Written info to mnpayments.dat 111ms
2017-01-20 11:52:52 Votes: 50106, Blocks: 4776
2017-01-20 11:52:52 mnpayments.dat dump finished 395ms
2017-01-20 11:52:52 Verifying governance.dat format...
2017-01-20 11:52:55 Loaded info from governance.dat 2757ms
2017-01-20 11:52:55 Governance Objects: 1328 (Proposals: 1258, Triggers: 53, Watchdogs: 17, Other: 0; Seen: 2375), Votes: 0
2017-01-20 11:52:55 Writting info to governance.dat...
2017-01-20 11:52:56 Written info to governance.dat 1049ms
2017-01-20 11:52:56 Governance Objects: 1304 (Proposals: 1258, Triggers: 43, Watchdogs: 3, Other: 0; Seen: 2401), Votes: 225223
2017-01-20 11:52:56 governance.dat dump finished 3860ms
2017-01-20 11:52:56 Verifying netfulfilled.dat format...
2017-01-20 11:52:56 Loaded info from netfulfilled.dat 0ms
2017-01-20 11:52:56 Nodes with fulfilled requests: 12
2017-01-20 11:52:56 Writting info to netfulfilled.dat...
2017-01-20 11:52:56 Written info to netfulfilled.dat 1ms
2017-01-20 11:52:56 Nodes with fulfilled requests: 9
2017-01-20 11:52:56 netfulfilled.dat dump finished 2ms
2017-01-20 11:52:58 Shutdown: done
2017-01-20 05:17:55 CActiveMasternode::ManageStateLocal -- Update Masternode List
2017-01-20 05:17:55 CMasternodeMan::UpdateMasternodeList -- masternode=06069e73139bbcfabb9371c8683190566988bf019deaeb090413f2bee0786613-1 addr=83.1.99.1:19999
2017-01-20 05:17:55 CActiveMasternode::ManageStateRemote -- NOT_CAPABLE: Masternode in NEW_START_REQUIRED state
2017-01-20 12:41:35 CActiveMasternode::ManageStateInitial -- Checking inbound connection to '83.1.99.2:19999'
2017-01-20 12:41:35 CActiveMasternode::ManageStateRemote -- NOT_CAPABLE: Masternode not in masternode list
2017-01-20 12:41:35 CActiveMasternode::ManageStateLocal -- Update Masternode List
2017-01-20 12:41:35 CMasternodeMan::UpdateMasternodeList -- masternode=06069e73139bbcfabb9371c8683190566988bf019deaeb090413f2bee0786613-1 addr=83.1.99.2:19999
2017-01-20 12:41:35 CreateTxLockCandidate -- New Transaction Lock Candidate! txid=23a238b7c787789830681e3f41e2f902f2aa45f4169a6598d2bf89c1bdbdd40e
2017-01-20 12:41:35 TXLOCKREQUEST -- Transaction Lock Request: 91.121.86.30:54144 /Dash Core:0.12.1/ : accepted 23a238b7c787789830681e3f41e2f902f2aa45f4169a6598d2bf89c1bdbdd40e
2017-01-20 11:28:05 CreateTxLockCandidate -- New Transaction Lock Candidate! txid=8dd5e1a37cc3bcafa62f44631fa6ea5bca64b1b99407220c28a1d9de21a49cf4
2017-01-20 11:28:05 TXLOCKREQUEST -- Transaction Lock Request: 82.196.0.241:19999 /Dash Core:0.12.1(bitcore)/ : accepted 8dd5e1a37cc3bcafa62f44631fa6ea5bca64b1b99407220c28a1d9de21a49cf4
2017-01-20 11:28:05 CreateTxLockCandidate -- New Transaction Lock Candidate! txid=af57aaf548e22039905fbdf74723aeb3fc35654d6580c1b0339b24103123bc00
2017-01-20 11:28:05 TXLOCKREQUEST -- Transaction Lock Request: 176.31.145.16:33238 /Dash Core:0.12.1/ : accepted af57aaf548e22039905fbdf74723aeb3fc35654d6580c1b0339b24103123bc00
61729 dash-wallet RET _umtx_op -1 errno 60 Operation timed out
61729 dash-wallet CALL clock_gettime(0,0x7fffdcfe5808)
61729 dash-wallet RET clock_gettime 0
61729 dash-wallet CALL _umtx_op(0x80237e248,UMTX_OP_WAIT_UINT_PRIVATE,0,0x18,0x7fffdcfe5688)
61729 dashd RET nanosleep 0
61729 dashd CALL nanosleep(0x7fffffffdeb8,0)
61729 dashd RET nanosleep 0
61729 dashd CALL nanosleep(0x7fffffffdeb8,0)
61729 dashd RET nanosleep 0
61729 dashd CALL nanosleep(0x7fffffffdeb8,0)
61729 dash-wallet RET _umtx_op -1 errno 60 Operation timed out
61729 dash-wallet CALL clock_gettime(0,0x7fffdcfe5808)
61729 dash-wallet RET clock_gettime 0
61729 dash-wallet CALL _umtx_op(0x80237e248,UMTX_OP_WAIT_UINT_PRIVATE,0,0x18,0x7fffdcfe5688)
Thanks for reporting - just wanted to says that we've hammered testnet with tx/is today at a load of 3x current Bitcoin tx. So your nodes getting stuck is more likely a problem of high cpu load...Latest build of 12.1 (yesterday - no new changes since today) from sources hungs. Three masternodes on testnet. Same behaviour. Here some more info.
1. Last log lines:
testmn01:
testmn02:Code:2017-01-20 05:17:55 CActiveMasternode::ManageStateLocal -- Update Masternode List 2017-01-20 05:17:55 CMasternodeMan::UpdateMasternodeList -- masternode=06069e73139bbcfabb9371c8683190566988bf019deaeb090413f2bee0786613-1 addr=83.1.99.1:19999 2017-01-20 05:17:55 CActiveMasternode::ManageStateRemote -- NOT_CAPABLE: Masternode in NEW_START_REQUIRED state
testmn03:Code:2017-01-20 12:41:35 CActiveMasternode::ManageStateInitial -- Checking inbound connection to '83.1.99.2:19999' 2017-01-20 12:41:35 CActiveMasternode::ManageStateRemote -- NOT_CAPABLE: Masternode not in masternode list 2017-01-20 12:41:35 CActiveMasternode::ManageStateLocal -- Update Masternode List 2017-01-20 12:41:35 CMasternodeMan::UpdateMasternodeList -- masternode=06069e73139bbcfabb9371c8683190566988bf019deaeb090413f2bee0786613-1 addr=83.1.99.2:19999 2017-01-20 12:41:35 CreateTxLockCandidate -- New Transaction Lock Candidate! txid=23a238b7c787789830681e3f41e2f902f2aa45f4169a6598d2bf89c1bdbdd40e 2017-01-20 12:41:35 TXLOCKREQUEST -- Transaction Lock Request: 91.121.86.30:54144 /Dash Core:0.12.1/ : accepted 23a238b7c787789830681e3f41e2f902f2aa45f4169a6598d2bf89c1bdbdd40e
So it looks that it can have something to do with ManageState or Transaction Locks.Code:2017-01-20 11:28:05 CreateTxLockCandidate -- New Transaction Lock Candidate! txid=8dd5e1a37cc3bcafa62f44631fa6ea5bca64b1b99407220c28a1d9de21a49cf4 2017-01-20 11:28:05 TXLOCKREQUEST -- Transaction Lock Request: 82.196.0.241:19999 /Dash Core:0.12.1(bitcore)/ : accepted 8dd5e1a37cc3bcafa62f44631fa6ea5bca64b1b99407220c28a1d9de21a49cf4 2017-01-20 11:28:05 CreateTxLockCandidate -- New Transaction Lock Candidate! txid=af57aaf548e22039905fbdf74723aeb3fc35654d6580c1b0339b24103123bc00 2017-01-20 11:28:05 TXLOCKREQUEST -- Transaction Lock Request: 176.31.145.16:33238 /Dash Core:0.12.1/ : accepted af57aaf548e22039905fbdf74723aeb3fc35654d6580c1b0339b24103123bc00
2. ktrace output (on all mn that same). Looks as some kind of loop.
Code:61729 dash-wallet RET _umtx_op -1 errno 60 Operation timed out 61729 dash-wallet CALL clock_gettime(0,0x7fffdcfe5808) 61729 dash-wallet RET clock_gettime 0 61729 dash-wallet CALL _umtx_op(0x80237e248,UMTX_OP_WAIT_UINT_PRIVATE,0,0x18,0x7fffdcfe5688) 61729 dashd RET nanosleep 0 61729 dashd CALL nanosleep(0x7fffffffdeb8,0) 61729 dashd RET nanosleep 0 61729 dashd CALL nanosleep(0x7fffffffdeb8,0) 61729 dashd RET nanosleep 0 61729 dashd CALL nanosleep(0x7fffffffdeb8,0) 61729 dash-wallet RET _umtx_op -1 errno 60 Operation timed out 61729 dash-wallet CALL clock_gettime(0,0x7fffdcfe5808) 61729 dash-wallet RET clock_gettime 0 61729 dash-wallet CALL _umtx_op(0x80237e248,UMTX_OP_WAIT_UINT_PRIVATE,0,0x18,0x7fffdcfe5688)
And... it looks that 12.1 still have some stability issues :-/
No it isn't. CPU usage is normal. Dashd looping for hours and not responding to rpc commands, etc. Only kill -9 helps ;-)Thanks for reporting - just wanted to says that we've hammered testnet with tx/is today at a load of 3x current Bitcoin tx. So your nodes getting stuck is more likely a problem of high cpu load...
Mind attaching a gdb instance to one of the stuck nodes and get a stacktrace? Maybe it is a locking issue.No it isn't. CPU usage is normal. Dashd looping for hours and not responding to rpc commands, etc. Only kill -9 helps ;-)
Ok. I just killed all three dashd processes. So probably i need to wait couple hours...Mind attaching a gdb instance to one of the stuck nodes and get a stacktrace? Maybe it is a locking issue.
TrySome at block 140443
mnsync reset
reconsiderblock 0000006341894d9a305f5ae5827740f0a865ba1d27b73bf06f4c3cee7e72207d
Tryand some (including windows wallet) at 140485.
mnsync reset
reconsiderblock 000000806c87fb3196de7c8ab05fc4094b2b741b3dbdfb232dbe4e5aa2d75e08
It is a issue which is special to testnet: We are running testnet on a networkThank you - those commands are working - MNs syncing again.
Is this just a testnet issue ?
The issue is not your node, but the other 95% of the network.Well, thanks for the detailed explanation. Although my MNs should have enough ressorces, it's hard for them to keep up with testnet these days.
Will try to start a couple of more powerfull servers next.
@flare can you please comment on this?? Could this be a problem? I would like to know other opinions because it sure sounds like a good point (depending on how much Data will be incomingI'm still a bit concerned about a bandwidth waste attack vector... With DAPI we're going to be moving so much data that a true DDoS wouldn't be needed. Just suck so much bandwidth that the quotas are tripped, and large swaths of the MN network go offline because the host shuts it down... Defeating storage integrity with this is entirely feasible. Even temporary disruption could cause permanent issues.
I think they're a little busy right now. I'm not demanding immediate response or anything... This is the sort of thing that would/should be on every MNOs mind, were they smart enough to comprehend it...again, @eduffield @flare @UdjinM6 @moocowmoo @t0dd Can anyone speak about what Camo is saying above?
I don't know how many api calls you're planning on servicing per second, etc... But have you compared what we're doing with other services like this? Are there any other services like our DAPI? LOL How does NASDAQ do it?
Anyway, if we don't hear from the boys, @camosoul, I think, whatever it will require will be affordable for the MN to implement because heavy use ought to equal higher price, no? But it is a good question and I'd like to know the guys are thinking on this, and that they believe it's do-able (not that they never thought on this!)
We'll probably have to see how it goes as we build this system and test it out and add functionality....?