Keep your MN up and running after crash! (ubuntu > 1 users, one installed dashd)

TanteStefana

Grizzled Member
Foundation Member
Mar 9, 2014
2,861
1,854
1,283
AT the end of this tutorial, I'll make it quick and sweet for those who don't care about the whys ;) and just want the hows!

What will you learn?

You will learn here how to set up a couple of cron jobs per user, so that if the server is restarted or the dashd daemon crashes, it will automatically restart.

Why?
People have discussed this on the forums, but when I wanted to find the information, I simply couldn't or else it didn't work for me. Many thanks to everyone who has ever written tutorials on such things, or on the BCT forums because that's how I got the idea, how I learned how this works, and finally got it to work for me. Special thanks to GermanRed+ who gave me the final script which I just changed minorly (I simply put in the full path to my dashd program as it wasn't working otherwise)

Anyway, what we're going to do here is add two items to your cron jobs, per user.

Why per user?
You see, I have two users on one server (and two IP address if you're wondering) I like the one server setup because I only have to update dashd once for as many users I can fit on my machine (I highly suggest if you do this that you have a minimum 1 core, better 2- per masternode plus a minimum ~ 350 mb of ram - better 512.)

Another thing you need to know about my setup is that I put my dashd and dash-cli etc... into /usr/local/bin. By doing so, I can execute dashd from any user and it's run in a separate instance (that is, run it locally for each user)

Even if you only have one user per server, you can still use this bash script to restart your masternodes in case of a reboot/restart or crash.

So here we go!

First log on to a user account, lets say user1
Next we're going to open user1's crontab. Please type in:
crontab -e

the -e stands for edit. The first time you use crontab, you will be asked which editor you would like to use, I use nano, others like vi. Either one will do, and probably others that I'm not familiar with. Instructions here will be for nano. Please simply look up how to save in vi if you're using that editor. I think I remember nano was choice 2, so just enter 2 in the window and you will get a screen that looks like this:



use your arrow keys to get to the bottom of the file, and add these two lines:

@reboot /usr/local/bin/dashd
*/10 * * * * /home/user1/mn_watch.sh >/dev/null 2>&1


of course, replacing user1 with your user's name.



Finally,
"write" to file ctrl o
"exit and reset" ctrl x

Explaination:

The first line is pretty obvious. @reboot tells the computer at startup, that it is to start dashd, which is in the location /usr/local/bin/ The -shrinkdebugfile hasn't worked for me, but it's supposed to clean out the debug.log file which tends to get huge over time and cause issues. If anyone knows what isn't right there, please let me know and I'll fix it in this tutorial. -shrinkdebugfile is unnecessary and happens automatically for logs over 10mb as noted below by UdjinM6 :) Thanks!

The second line is saying that once every ten minutes */10 ( = 60/10 and could also be written with commas between them but still in first position, separated by a space when finished like so: 0,10,20,30,40,50 ) the stated action should be performed. The way to read the 5 spots separated by a space is minutes, hours, weekdays, months and years. You can read more on how this works here: https://help.ubuntu.com/community/CronHowto .

In the illustration above, I had it set to 1 minute intervals so I could test it without waiting so long.

There is one issue:

When updating your daemon to a new version, your daemon may start up after you've shut it down.

However, the 10 minute intervals should be fine since we have 70 minutes before we're knocked down to the bottom of the masternode winners list. Also, when updating from a previous version, if you stop your dashd, you should have more than enough time to update before it's automatically turned back on. Just start your work after a 10th of the hour. This is set to the server's clock. So, start your work at say 12:10 or 1:20, or something like that and you shouldn't find your daemon started while you're working on updating it.

Finally, for non-protocol updates, which is the majority of updates, I don't even bother turning dashd off until I'm ready to restart it. It has no effect (that is, my new dashd isn't overwritten at shutdown). So I just do my updates and then restart dashd when I'm ready, never bothering to stop it before hand.

If you should need more time, you can always comment out the line in your crontab and put it back afterwords. Just comment it out with a #

And the rest:
The rest of the restart line is simply telling the computer to run the script located at /home/user1/mn_watch.sh
And finally, this >/dev/null 2>&1 is weird code that says "don't email any notifications, just dump it in this hole of no return". Because cronjobs want to send out email notifications, and frankly, I don't need no stinkin' notifications, I get enough spam.

Finally, after pasting in the two lines with the correct user's name, hit CTRL o to write it to the file and CTRL x to save it. Both vi and nano have the keyboard shortcuts written at the bottom of the screen.
 
Last edited by a moderator:

TanteStefana

Grizzled Member
Foundation Member
Mar 9, 2014
2,861
1,854
1,283
OK, now we simply have to create the bash script mn_watch.sh. I put it in my home folder for the user, ie /home/user1. Replace user1 with your user's name. I open up nano again via:

nano mn_watch.sh


This opens the editor with a file named mn_watch.sh if it doesn't already exist

copy and paste the following into it, making sure there is no space before the first line:

Code:
#!/bin/bash
#!/bin/bash
DASHD_RUNNING=$( ps aux | grep $user1 | grep dashd | grep -v wc -l )
if [ "$DASHD_RUNNING -lt 1" ] ; then
  /usr/local/bin/dashd 2>&1 >/dev/null
fi
"write" to file ctrl o
"exit and reset" ctrl x

Explanation:
Moocowmoo gave me this script as it is better. Often, my script couldn't find dashd.pid, and would restart the daemon whether it needed to restart or not. So please use this updated script. I don't know how it all works, so I'm sorry for not explaining it line by line.

Now make sure your user can execute this script by setting permissions just in case:

chmod +x mn_watch.sh


So to sum it all up, Your cron job will restart dashd if the computer should be restarted and it will check every 10 minutes to see if dashd is running. If it's not, it will restart dashd. This will be true even if another user on the same machine is running dashd without problems. Because this script only checks for the dashd instance under the user in question.
 
Last edited by a moderator:
  • Like
Reactions: UdjinM6

TanteStefana

Grizzled Member
Foundation Member
Mar 9, 2014
2,861
1,854
1,283
OK, as promised, here is the quick version who just want the how to!

log in as the user

In putty or your terminal type
crontab -e

chose option 2 for nano (or maybe you'll automatically open up to your editor.

Use arrow keys to get to the bottom then add these two lines:

@reboot /usr/local/bin/dashd
*/10 * * * * /home/user1/mn_watch.sh >/dev/null 2>&1


replacing user1 with your user's name

"write" to file ctrl o
"exit and reset" ctrl x

Now make sure you're in your home directory for your user $ cd

create a bash script:

type: $ nano mn_watch.sh

again nano the editor opens, and you copy paste the following into it, changing user1 to your user's name:

#!/bin/bash
DASHD_RUNNING=$( ps aux | grep $user1 | grep dashd | grep -v wc -l )
if [ "$DASHD_RUNNING -lt 1" ] ; then
/usr/local/bin/dashd 2>&1 >/dev/null
fi


"write" ctrl o
"exit and reset" ctrl x


make sure you can execute this bash script:

chmod +x mn_watch.sh
 
Last edited by a moderator:

TanteStefana

Grizzled Member
Foundation Member
Mar 9, 2014
2,861
1,854
1,283
HA! No good deed goes unpunished. In all my testing, I lost my place with one of my masternodes, LOL.

Please let me know if this helped you so I can feel better about ^^^ this, LOL
 

UdjinM6

Official Dash Dev
Dash Core Team
Moderator
May 20, 2014
3,638
3,538
1,183
Awesome guide!

One note though: "-shrinkdebugfile" cut out old strings from debug.log file when its size is more than ~10Mb. Also this option is enabled right out of the box (but it will be turned off automatically if you run in debug mode i.e. specified "-debug") so there is no need to specify it explicitly.
 

TanteStefana

Grizzled Member
Foundation Member
Mar 9, 2014
2,861
1,854
1,283
Thanks guys, remember, this works just fine as a single user as well, so it can be used on a single server/single user.
 

grzem

Member
Nov 22, 2014
49
7
48
If you use cron, there is no need to use loop (while...) in mn_watch.
I'm affraid that cron runs many instances of mn_watch which has endless loop.
 
  • Like
Reactions: TanteStefana

TanteStefana

Grizzled Member
Foundation Member
Mar 9, 2014
2,861
1,854
1,283
If you use cron, there is no need to use loop (while...) in mn_watch.
I'm affraid that cron runs many instances of mn_watch which has endless loop.
Oh dear, can you explain how I should fix this please? Thank you so much!

OK, so it seems this works without the while statement in the first line, thanks!
 
Last edited by a moderator:

TanteStefana

Grizzled Member
Foundation Member
Mar 9, 2014
2,861
1,854
1,283
Thanks a lot for your efforts Tante !
the script is working like a charm !

Just a little correction about the lock file
the file name is dashd.pid and no dash.pid as mentioned

Thanks !
Please use moocowmoo s bash script, I updated the tutorial with it. It works far better. Thanks!
 
Last edited by a moderator:

Dash4Ever

Active Member
Sep 24, 2015
105
105
93
Sweden
Dash Address
XybaxnhtFBih2g4M2F71rWKBt5USzo8R
Hey thanks for the info, is there anyway to see if it works ? When Im runing ./mn_watch.sh I get error "/usr/local/bin/dashd: No such file or directory"
Any suggestions ?
 

TanteStefana

Grizzled Member
Foundation Member
Mar 9, 2014
2,861
1,854
1,283
Did you install dashd at that location? It should actually be where you installed it. So if you install under your User, like many do, it would be /home/YourUserName/DashFolder/dashd

It's been a while, I can't remember the default dash folder name. I put it in /usr/local/bin or /usr/bin because it doesn't require the ./ to launch, and I can start it from different users, so I only have to update once. Then I got used to it, LOL. But it actually causes issues with moocowmoo's scripts, so probably not worth it.


Oh, and to see if it is working, you can just stop dashd : IE ./dashd stop Then wait a bit, and then see if it is running again with ./dash-cli getinfo. If it is running again, it'll give you a status print out, or if not, it'll say something like not running.

Also, moocowmoo has done a lot of work on Dashman, if you're just setting up, you may want to use it to run your nodes: https://github.com/moocowmoo/dashman I just haven't gotten around to it yet. My mind is just too full ATM. Too many personal things to do, i can't concentrate, LOL.

He's usually around to help out, just pm him :)
 
Last edited by a moderator:

Dash4Ever

Active Member
Sep 24, 2015
105
105
93
Sweden
Dash Address
XybaxnhtFBih2g4M2F71rWKBt5USzo8R
Did you install dashd at that location? It should actually be where you installed it. So if you install under your User, like many do, it would be /home/YourUserName/DashFolder/dashd

It's been a while, I can't remember the default dash folder name. I put it in /usr/local/bin or /usr/bin because it doesn't require the ./ to launch, and I can start it from different users, so I only have to update once. Then I got used to it, LOL. But it actually causes issues with moocowmoo's scripts, so probably not worth it.


Oh, and to see if it is working, you can just stop dashd : IE ./dashd stop Then wait a bit, and then see if it is running again with ./dash-cli getinfo. If it is running again, it'll give you a status print out, or if not, it'll say something like not running.

Also, moocowmoo has done a lot of work on Dashman, if you're just setting up, you may want to use it to run your nodes: https://github.com/moocowmoo/dashman I just haven't gotten around to it yet. My mind is just too full ATM. Too many personal things to do, i can't concentrate, LOL.

He's usually around to help out, just pm him :)
Thanks for the reply! :) gonna check the filepath for dashd and update the script. Do you get any reply / message when you start mn_watch.sh ?
 

Dash4Ever

Active Member
Sep 24, 2015
105
105
93
Sweden
Dash Address
XybaxnhtFBih2g4M2F71rWKBt5USzo8R
Think I have it working now, seems like my dashd is installed under home/user/

Change the scripts to this:

@reboot /home/myusername/dashd
*/10 * * * * /home/myusername/mn_watch.sh >/dev/null 2>&1


and

#!/bin/bash
DASHD_RUNNING=$( ps aux | grep $myusername | grep dashd | grep -v wc -l )
if [ "$DASHD_RUNNING -lt 1" ];then
/home/myusername/dashd 2>&1 >/dev/null
fi


When I ran ./mn_watch.sh I recieved :

Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.
[email protected]:~$ Error: Cannot obtain a lock on data directory /home/myusername/.dash. Dash Core is probably already running.

So is everything fine ?
 

splawik21

Grizzled Member
Dash Core Team
Moderator
Foundation Member
Dash Support Group
Apr 8, 2014
1,917
1,274
1,283
do "ps aux" please and ask your self if you see the dashd running :)?
btw. simply ./dash-cli getinfo will answer this question too ;)

you can check yourself as well by:

Code:
./dash-cli masternode status
 

Dash4Ever

Active Member
Sep 24, 2015
105
105
93
Sweden
Dash Address
XybaxnhtFBih2g4M2F71rWKBt5USzo8R
do "ps aux" please and ask your self if you see the dashd running :)?
btw. simply ./dash-cli getinfo will answer this question too ;)

you can check yourself as well by:

Code:
./dash-cli masternode status
Yeah I can see it running :p also used command "top" for this.
Tried "pkill dashd" and watched all the processes in "top" and after a few minutes dashd started again :) now I can relax. Thanks for all help
 
  • Like
Reactions: splawik21

splawik21

Grizzled Member
Dash Core Team
Moderator
Foundation Member
Dash Support Group
Apr 8, 2014
1,917
1,274
1,283
yep, now you can relax with a cold beer in a hand :cool:
 
  • Like
Reactions: Dash4Ever

TanteStefana

Grizzled Member
Foundation Member
Mar 9, 2014
2,861
1,854
1,283
Thanks for the reply! :) gonna check the filepath for dashd and update the script. Do you get any reply / message when you start mn_watch.sh ?
no, it doesn't echo anything, though I suppose it's possible to do. Unfortunately, I got a lot of help with it, and don't know where or how to put a print out message. All I can say is I've been using it and my nodes haven't stopped in 2 months. Still, this is all kind of old, and if you don't run more than 1 masternode on one machine (I have one machine, 2 users and 2 ip addresses) then it's undoubtedly easier to have moocowmoo's script run them for you. Especially since it's made to help you install, update, monitor,etc... All I'd have to do is have 2 installations of Dash instead of one. And I plan on doing this some day, LOL. But I'm lazy :)

I leave this here for nothing else than to show myself what I once learned, LOL. My memory disappears so easily :p
 
  • Like
Reactions: Dash4Ever

Sammy90

New Member
Jun 25, 2018
2
0
1
47
Hello Everyone!

I wanted to try this here but I fear that it does not work. What Dash4Ever found is actually an error. Running the script

ps aux | grep $myusername | grep dashd | grep -v wc -l

will give an error of grep.

Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.


So the thing why nobody noticed is that the script simply starts dashd again. But since it is already running the process gets canceled. Therefore it really restarts dashd when it crashed. However, this is not a really nice solution (it is basically trying to restart dashd every 10 minutes). Furthermore, I want to add a command into the script that, in case of a crash, I want to receive a mail via ssmtp. This does not seem to work (probably because of the problem of grep).

EDIT: Okay I had the $ in front of my username. However, the IF command is even true if the dashd is already running.

Can we fix this issue ? I did not come up with a good solution.



Thanks guys!
 
Last edited:

Sammy90

New Member
Jun 25, 2018
2
0
1
47
Okay I managed to make it work with a new code:

#!/bin/bash
#!/bin/bash
if ! pgrep -fu USER pivxd >/dev/null; then
~/pivx-3.1.0/bin/pivxd -daemon 2>&1 >/dev/null
fi

This is for pivx but for dash it is the same I guess!