11.2 - Dash Release

ErrorId

Member
Mar 9, 2014
158
41
88
Canada
I couldn't sleep and kept messing around with things.

On six low end servers running the daemon with the addnode removed from the conf file one out of the six hung a couple minutes after it synched the blockchain the other 5 are still ok, these are not mn just daemons running.

Below is where I think things started going wrong with the one that hung.

Code:
2015-04-02 03:14:55 ProcessBlock: ACCEPTED
2015-04-02 03:14:55 keypool reserve 2
2015-04-02 03:14:55 keypool return 2
2015-04-02 03:14:55 ProcessMessages(dsee, 253 bytes): Exception 'CDataStream::r$
2015-04-02 03:14:55 ProcessMessage(dsee, 253 bytes) FAILED
<snip>
2015-04-02 03:15:04 dseep - Asking source node for missing entry CTxIn(COutPoin$
2015-04-02 03:15:05 dseep - Asking source node for missing entry CTxIn(COutPoin$
2015-04-02 03:15:05 dseep - Asking source node for missing entry CTxIn(COutPoin$
2015-04-02 03:15:05 ProcessMessages(dsee, 253 bytes): Exception 'CDataStream::r$
2015-04-02 03:15:05 ProcessMessage(dsee, 253 bytes) FAILED
2015-04-02 03:15:05 ProcessMessages(dsee, 253 bytes): Exception 'CDataStream::r$
2015-04-02 03:15:05 ProcessMessage(dsee, 253 bytes) FAILED
2015-04-02 03:15:07 dseep - Asking source node for missing entry CTxIn(COutPoin$
2015-04-02 03:15:13 CDarksendPool::UpdateState() - Can't set state to ERROR or $
2015-04-02 03:15:13 dseep - Asking source node for missing entry CTxIn(COutPoin$
2015-04-02 03:15:13 ProcessMessages(dsee, 253 bytes): Exception 'CDataStream::r$
<snip>
2015-04-02 03:15:16 ProcessMessage(dsee, 253 bytes) FAILED
2015-04-02 03:15:19 dseep - Asking source node for missing entry CTxIn(COutPoin$
2015-04-02 03:15:25 dseep - Asking source node for missing entry CTxIn(COutPoin$
2015-04-02 03:15:25 ProcessMessages(dsee, 253 bytes): Exception 'CDataStream::r$
2015-04-02 03:15:25 ProcessMessage(dsee, 253 bytes) FAILED
The beefier server is still humming along 11.5 hours later, but it will hit its ram limit in the next couple of hours.
 
Last edited by a moderator:

kadrek

New Member
Apr 8, 2014
5
0
3
I see drk.mn says that 11.2.17 is the lastest version but it only makes up 0.1% of the network. I'm guessing it's a test node and the site just pulls the highest version that it sees and says it's the latest?
 

RenegadeMan

Member
Aug 6, 2014
61
92
58
Hey Everyone

I upgraded my single MN shortly after the release and haven't had a single hiccup or issue. In fact, I was around 3.5 days into a payment cycle and just assumed I'd be back at the start of the cycle as a result of the migration to the dashd daemon, but the dashninja page still reported my MN had been running for the same period (i.e. it 'remembered') and I received not one but two payments within an hour of each other shortly after the migration was completed! I was pleased with that given I'd missed out on payments a few times previously during other periods where enforcement had been turned off.

I just thought I'd put the server monitoring graphs from my Vultr VPS instance on here out of interest. You can see quite clearly when the MN was converted to using dashd and the reduction is CPU Usage as a result, so something's working properly.

My report then is that it's all good for me and my MN hasn't dropped off once.

20150402 Vultr VPS monitoring graphs for MN after migration to dashd.jpg
 

flare

Administrator
Dash Core Team
Moderator
May 18, 2014
2,286
2,404
1,183
Germany
I see drk.mn says that 11.2.17 is the lastest version but it only makes up 0.1% of the network. I'm guessing it's a test node and the site just pulls the highest version that it sees and says it's the latest?
drk.mn pulls version from latest git tag to determine latest version as far as i know
 

darkred

Active Member
Feb 6, 2015
235
262
123
Oh Scheisse!
Can you start them up again and run "top" to see how much cpu/ram it is using? Does either keep climbing before it dies? If it takes a long time to die, you could run "top -b -n 10000 >> /tmp/top.txt

I am curious if you're running out of memory or something. Is there a core file? We might need to run "ulimit -c unlimited" to have it drop a core file that could be reviewed.
 

GermanRed+

Active Member
Aug 28, 2014
299
109
113
I couldn't sleep and kept messing around with things.

On six low end servers running the daemon with the addnode removed from the conf file one out of the six hung a couple minutes after it synched the blockchain the other 5 are still ok, these are not mn just daemons running.

Below is where I think things started going wrong with the one that hung.

Code:
2015-04-02 03:14:55 ProcessBlock: ACCEPTED
2015-04-02 03:14:55 keypool reserve 2
2015-04-02 03:14:55 keypool return 2
2015-04-02 03:14:55 ProcessMessages(dsee, 253 bytes): Exception 'CDataStream::r$
2015-04-02 03:14:55 ProcessMessage(dsee, 253 bytes) FAILED
<snip>
2015-04-02 03:15:04 dseep - Asking source node for missing entry CTxIn(COutPoin$
2015-04-02 03:15:05 dseep - Asking source node for missing entry CTxIn(COutPoin$
2015-04-02 03:15:05 dseep - Asking source node for missing entry CTxIn(COutPoin$
2015-04-02 03:15:05 ProcessMessages(dsee, 253 bytes): Exception 'CDataStream::r$
2015-04-02 03:15:05 ProcessMessage(dsee, 253 bytes) FAILED
2015-04-02 03:15:05 ProcessMessages(dsee, 253 bytes): Exception 'CDataStream::r$
2015-04-02 03:15:05 ProcessMessage(dsee, 253 bytes) FAILED
2015-04-02 03:15:07 dseep - Asking source node for missing entry CTxIn(COutPoin$
2015-04-02 03:15:13 CDarksendPool::UpdateState() - Can't set state to ERROR or $
2015-04-02 03:15:13 dseep - Asking source node for missing entry CTxIn(COutPoin$
2015-04-02 03:15:13 ProcessMessages(dsee, 253 bytes): Exception 'CDataStream::r$
<snip>
2015-04-02 03:15:16 ProcessMessage(dsee, 253 bytes) FAILED
2015-04-02 03:15:19 dseep - Asking source node for missing entry CTxIn(COutPoin$
2015-04-02 03:15:25 dseep - Asking source node for missing entry CTxIn(COutPoin$
2015-04-02 03:15:25 ProcessMessages(dsee, 253 bytes): Exception 'CDataStream::r$
2015-04-02 03:15:25 ProcessMessage(dsee, 253 bytes) FAILED
The beefier server is still humming along 11.5 hours later, but it will hit its ram limit in the next couple of hours.
My MNs with addnode removed stay up after almost 10 hours now. The memory consumption increased by ~10 MB. So, I believe there is something wrong with MNs peers connected together for an extended period of time. Notice that when you have two MNs connected with addnode as peers, they usually stayed connected for a long long time (i.e. they don't disconnect from each other). I'm not saying that addnode is the problem. I'm saying that addnode can create the conditions for the problem. That may help the developers to recreate the problem for debugging purpose.
 

balu

Well-known Member
Dash Core Team
Foundation Member
Oct 9, 2014
117
239
213
Hoping for a fix, couldn't get my mns working with the new release. They were rock stable with the previous version (1.25)
 

GermanRed+

Active Member
Aug 28, 2014
299
109
113
Hey Everyone

I upgraded my single MN shortly after the release and haven't had a single hiccup or issue. In fact, I was around 3.5 days into a payment cycle and just assumed I'd be back at the start of the cycle as a result of the migration to the dashd daemon, but the dashninja page still reported my MN had been running for the same period (i.e. it 'remembered') and I received not one but two payments within an hour of each other shortly after the migration was completed! I was pleased with that given I'd missed out on payments a few times previously during other periods where enforcement had been turned off.

I just thought I'd put the server monitoring graphs from my Vultr VPS instance on here out of interest. You can see quite clearly when the MN was converted to using dashd and the reduction is CPU Usage as a result, so something's working properly.

My report then is that it's all good for me and my MN hasn't dropped off once.

View attachment 1260
I am running this version before the version bump. My MNs still dropped off until I removed the addnode lines in dash.conf. So, I think the bug hasn't been completely resolved yet. But, let's see how 0.11.2.17 works for others.
 

Bridgewater

Well-known Member
Foundation Member
Dec 14, 2014
183
164
203
Still running 0.11.2.16 on mine.

Update: With a little TLC, it seems I am able to coax the non-working daemons into a working state by killing the process and removing .dash/.lock (thanks UdjinM6), plus babysitting with tail -f .dash/debug.log to make sure it is alive, and if it stops, repeat the process until it finally stays alive and synced.
 

crowning

Well-known Member
May 29, 2014
1,414
1,997
183
Alpha Centauri Bc
Still running 0.11.2.16 on mine.

Update: With a little TLC, it seems I am able to coax the non-working daemons into a working state by killing the process and removing .dash/.lock (thanks UdjinM6), plus babysitting with tail -f .dash/debug.log to make sure it is alive, and if it stops, repeat the process until it finally stays alive and synced.
I'm testing a self-build v0.11.2.17 for a while now and it seems the Masternodes don't drop any more, so just wait until Evan announces this new version.
 

GermanRed+

Active Member
Aug 28, 2014
299
109
113
I'm testing a self-build v0.11.2.17 for a while now and it seems the Masternodes don't drop any more, so just wait until Evan announces this new version.
How long has it been running without being kicked off? Do you have addnode to other MNs in dash.conf?
 

GermanRed+

Active Member
Aug 28, 2014
299
109
113
Some about 12 hours now, the rest about 5 hours. Did not change dash.conf, just swapped dashd and restarted.
Then, the only other changes I had is a script that run dashd getblocktemplate. I stopped running getblocktemplate from it. Or, the patches in 0.11.2.17 completely fixed the problem. I did get kicked off before removing addnode in dash.conf and getblocktemplate in the script. Perhaps, I will addnode from the command prompt if more people reports no issue with 0.11.2.17.

The memory consumption stays constant at 366MB now if anyone is interested in this.
 

spinner

New Member
Oct 17, 2014
32
3
8
East Coast USA
I start-many'd 6 MNs yesterday, and they have all been solid ever since...no payments though. Just wanted to share a success story for the lurkers! ;)
I'll 2nd that - I have 1 MN running on AWS West coast (in case anyone is trying to verify providers who aren't having issues)

Updated yesterday evening, received first DASH mn payment a few hours later. Haven't hit a single snag.
 

GermanRed+

Active Member
Aug 28, 2014
299
109
113
I can confirm that "addnode" to another MN will create the high CPU load and high memory usage problem. Eventually, the MN will be kicked off with this high CPU load and memory consumption. Something is still wrong in v0.11.2.17.

EDIT: What I did is "addnode another_MN_IP_address_on_the_same_ISP:9999 add". Then, wait until the two MNs are connected as peers. I use "lsof -i -n" to check that or you can use "getpeerinfo" instead. The time to get these two nodes connected can take some time. When they are connected, you will see the CPU load increases and memory usage starting to rise. I just tested it on a node that ran without being kicked off for 15 hours. Right after the MNs are connected as peers, the CPU load started to ramp up and the memory usage gradually increased.

I haven't tried this on MN hosted by another ISP as I do not want to kick someone off. Could someone with MNs on two different ISPs try this? You can restart the nodes once you see CPU load over 100% and memory usage more than 366 MB. That will confirm the problem without being kicked off because a simple restart of the nodes will prevent that.
 
Last edited by a moderator:

oblox

Well-known Member
Aug 6, 2014
1,032
537
183
************ Dash Release Update : 11.2.17 ************************

- Fixed the "locking up" issue that would cause masternodes to go offline and general instability
- Masternode list changes - Udjin
- Better icon images - Crowning
- Improved Chinese Translation

https://www.dashpay.io/downloads/
What was the cause of the lock up issue? Memleak? Then again, it doesn't explain the sporadic-ness of it.
 

tungfa

Grizzled Member
Foundation Member
Masternode Owner/Operator
Apr 9, 2014
8,902
6,739
1,283

darkwing

Active Member
Apr 8, 2014
149
110
103
Great work.

Neither here nor there but.. Um icons still the same? I got all excited thinking my icons were in.
 

eduffield

Core Developer
Mar 9, 2014
1,084
5,320
183

jimbit

Well-known Member
Foundation Member
May 23, 2014
229
103
203
am i looking in the wrong place?
Code:
 wget https://github.com/darkcoinproject/darkcoin-binaries/raw/master/dash-0.11.2.17-linux.tar.gz
--2015-04-02 14:14:10--  https://github.com/darkcoinproject/darkcoin-binaries/raw/master/dash-0.11.2.17-linux.tar.gz
Resolving github.com (github.com)... 192.30.252.129
Connecting to github.com (github.com)|192.30.252.129|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2015-04-02 14:14:10 ERROR 404: Not Found.
 

orangecycle

Well-known Member
Foundation Member
Oct 2, 2014
169
239
203
node40.com
am i looking in the wrong place?
Code:
 wget https://github.com/darkcoinproject/darkcoin-binaries/raw/master/dash-0.11.2.17-linux.tar.gz
--2015-04-02 14:14:10--  https://github.com/darkcoinproject/darkcoin-binaries/raw/master/dash-0.11.2.17-linux.tar.gz
Resolving github.com (github.com)... 192.30.252.129
Connecting to github.com (github.com)|192.30.252.129|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2015-04-02 14:14:10 ERROR 404: Not Found.
https://www.dashpay.io/binaries/dash-0.11.2.17-linux.tar.gz
 

Icebucket

Active Member
Apr 11, 2014
268
129
103
Can someone please explain how to do this manually step by step for linux terminal nooobs, I got quite a few MN that I cant launch becourse im an Idiot that only uses linux to operate my MN

;/ edit
If you don't put out there a complete guide on how idiots like me can update then there is a unfair advantage that follows, IT guys will reap the benefits of fewer MNs while we nooobs try to scramble our shit together to try to update.. Dont assume that everyone that has a MN is a IT pro, rather assume everyone is an idiot.
 
Last edited by a moderator:

romario

New Member
Apr 2, 2015
31
5
8
Guys, this is my story

OS Debian 7

1. I did the darkcoin->dash upgrade on my encrypted wallet
2. After the restart it showed me Warning: wallet.dat corrupt, data salvaged! Original wallet.dat saved as wallet.{timestamp}.bak in .dash; if your balance or transactions are incorrect you should restore from a backup.
3. Shortly after that Error loading wallet.dat: Wallet corrupted (EXIT)

I managed to get it working after several restarts and -rescan -reindex options. However, now I cannot unlock the wallet with my passphrase, which was working just fine before the migration.

Could anyone advise on that please?