Welcome to the Dash Forum!

Please sign up to discuss the most innovative cryptocurrency!

Keep your MN up and running after crash! (ubuntu > 1 users, one installed dashd)

Discussion in 'Masternode Guides' started by TanteStefana, Sep 3, 2015.

  1. TanteStefana

    TanteStefana Moderator
    Linguistic Foundation Member

    Joined:
    Mar 9, 2014
    Messages:
    2,859
    Likes Received:
    1,854
    Trophy Points:
    1,283
    AT the end of this tutorial, I'll make it quick and sweet for those who don't care about the whys ;) and just want the hows!

    What will you learn?

    You will learn here how to set up a couple of cron jobs per user, so that if the server is restarted or the dashd daemon crashes, it will automatically restart.

    Why?
    People have discussed this on the forums, but when I wanted to find the information, I simply couldn't or else it didn't work for me. Many thanks to everyone who has ever written tutorials on such things, or on the BCT forums because that's how I got the idea, how I learned how this works, and finally got it to work for me. Special thanks to GermanRed+ who gave me the final script which I just changed minorly (I simply put in the full path to my dashd program as it wasn't working otherwise)

    Anyway, what we're going to do here is add two items to your cron jobs, per user.

    Why per user?
    You see, I have two users on one server (and two IP address if you're wondering) I like the one server setup because I only have to update dashd once for as many users I can fit on my machine (I highly suggest if you do this that you have a minimum 1 core, better 2- per masternode plus a minimum ~ 350 mb of ram - better 512.)

    Another thing you need to know about my setup is that I put my dashd and dash-cli etc... into /usr/local/bin. By doing so, I can execute dashd from any user and it's run in a separate instance (that is, run it locally for each user)

    Even if you only have one user per server, you can still use this bash script to restart your masternodes in case of a reboot/restart or crash.

    So here we go!

    First log on to a user account, lets say user1
    Next we're going to open user1's crontab. Please type in:
    crontab -e

    the -e stands for edit. The first time you use crontab, you will be asked which editor you would like to use, I use nano, others like vi. Either one will do, and probably others that I'm not familiar with. Instructions here will be for nano. Please simply look up how to save in vi if you're using that editor. I think I remember nano was choice 2, so just enter 2 in the window and you will get a screen that looks like this:

    [​IMG]

    use your arrow keys to get to the bottom of the file, and add these two lines:

    @reboot /usr/local/bin/dashd
    */10 * * * * /home/user1/mn_watch.sh >/dev/null 2>&1


    of course, replacing user1 with your user's name.

    [​IMG]

    Finally,
    "write" to file ctrl o
    "exit and reset" ctrl x

    Explaination:

    The first line is pretty obvious. @reboot tells the computer at startup, that it is to start dashd, which is in the location /usr/local/bin/ The -shrinkdebugfile hasn't worked for me, but it's supposed to clean out the debug.log file which tends to get huge over time and cause issues. If anyone knows what isn't right there, please let me know and I'll fix it in this tutorial. -shrinkdebugfile is unnecessary and happens automatically for logs over 10mb as noted below by UdjinM6 :) Thanks!

    The second line is saying that once every ten minutes */10 ( = 60/10 and could also be written with commas between them but still in first position, separated by a space when finished like so: 0,10,20,30,40,50 ) the stated action should be performed. The way to read the 5 spots separated by a space is minutes, hours, weekdays, months and years. You can read more on how this works here: https://help.ubuntu.com/community/CronHowto .

    In the illustration above, I had it set to 1 minute intervals so I could test it without waiting so long.

    There is one issue:

    When updating your daemon to a new version, your daemon may start up after you've shut it down.

    However, the 10 minute intervals should be fine since we have 70 minutes before we're knocked down to the bottom of the masternode winners list. Also, when updating from a previous version, if you stop your dashd, you should have more than enough time to update before it's automatically turned back on. Just start your work after a 10th of the hour. This is set to the server's clock. So, start your work at say 12:10 or 1:20, or something like that and you shouldn't find your daemon started while you're working on updating it.

    Finally, for non-protocol updates, which is the majority of updates, I don't even bother turning dashd off until I'm ready to restart it. It has no effect (that is, my new dashd isn't overwritten at shutdown). So I just do my updates and then restart dashd when I'm ready, never bothering to stop it before hand.

    If you should need more time, you can always comment out the line in your crontab and put it back afterwords. Just comment it out with a #

    And the rest:
    The rest of the restart line is simply telling the computer to run the script located at /home/user1/mn_watch.sh
    And finally, this >/dev/null 2>&1 is weird code that says "don't email any notifications, just dump it in this hole of no return". Because cronjobs want to send out email notifications, and frankly, I don't need no stinkin' notifications, I get enough spam.

    Finally, after pasting in the two lines with the correct user's name, hit CTRL o to write it to the file and CTRL x to save it. Both vi and nano have the keyboard shortcuts written at the bottom of the screen.
     
    #1 TanteStefana, Sep 3, 2015
    Last edited by a moderator: Sep 3, 2015
    • Like Like x 3
  2. TanteStefana

    TanteStefana Moderator
    Linguistic Foundation Member

    Joined:
    Mar 9, 2014
    Messages:
    2,859
    Likes Received:
    1,854
    Trophy Points:
    1,283
    OK, now we simply have to create the bash script mn_watch.sh. I put it in my home folder for the user, ie /home/user1. Replace user1 with your user's name. I open up nano again via:

    nano mn_watch.sh


    This opens the editor with a file named mn_watch.sh if it doesn't already exist

    copy and paste the following into it, making sure there is no space before the first line:

    Code:
    #!/bin/bash
    #!/bin/bash
    DASHD_RUNNING=$( ps aux | grep $user1 | grep dashd | grep -v wc -l )
    if [ "$DASHD_RUNNING -lt 1" ] ; then
      /usr/local/bin/dashd 2>&1 >/dev/null
    fi
    
    "write" to file ctrl o
    "exit and reset" ctrl x

    Explanation:
    Moocowmoo gave me this script as it is better. Often, my script couldn't find dashd.pid, and would restart the daemon whether it needed to restart or not. So please use this updated script. I don't know how it all works, so I'm sorry for not explaining it line by line.

    Now make sure your user can execute this script by setting permissions just in case:

    chmod +x mn_watch.sh


    So to sum it all up, Your cron job will restart dashd if the computer should be restarted and it will check every 10 minutes to see if dashd is running. If it's not, it will restart dashd. This will be true even if another user on the same machine is running dashd without problems. Because this script only checks for the dashd instance under the user in question.
     
    #2 TanteStefana, Sep 3, 2015
    Last edited by a moderator: Sep 13, 2015
    • Like Like x 1
  3. TanteStefana

    TanteStefana Moderator
    Linguistic Foundation Member

    Joined:
    Mar 9, 2014
    Messages:
    2,859
    Likes Received:
    1,854
    Trophy Points:
    1,283
    OK, as promised, here is the quick version who just want the how to!

    log in as the user

    In putty or your terminal type
    crontab -e

    chose option 2 for nano (or maybe you'll automatically open up to your editor.

    Use arrow keys to get to the bottom then add these two lines:

    @reboot /usr/local/bin/dashd
    */10 * * * * /home/user1/mn_watch.sh >/dev/null 2>&1


    replacing user1 with your user's name

    "write" to file ctrl o
    "exit and reset" ctrl x

    Now make sure you're in your home directory for your user $ cd

    create a bash script:

    type: $ nano mn_watch.sh

    again nano the editor opens, and you copy paste the following into it, changing user1 to your user's name:

    #!/bin/bash
    DASHD_RUNNING=$( ps aux | grep $user1 | grep dashd | grep -v wc -l )
    if [ "$DASHD_RUNNING -lt 1" ] ; then
    /usr/local/bin/dashd 2>&1 >/dev/null
    fi


    "write" ctrl o
    "exit and reset" ctrl x


    make sure you can execute this bash script:

    chmod +x mn_watch.sh
     
    #3 TanteStefana, Sep 3, 2015
    Last edited by a moderator: Sep 13, 2015
    • Like Like x 2
  4. TanteStefana

    TanteStefana Moderator
    Linguistic Foundation Member

    Joined:
    Mar 9, 2014
    Messages:
    2,859
    Likes Received:
    1,854
    Trophy Points:
    1,283
    HA! No good deed goes unpunished. In all my testing, I lost my place with one of my masternodes, LOL.

    Please let me know if this helped you so I can feel better about ^^^ this, LOL
     
  5. UdjinM6

    UdjinM6 Official Dash Dev
    Dash Core Team Moderator

    Joined:
    May 20, 2014
    Messages:
    3,631
    Likes Received:
    3,532
    Trophy Points:
    1,183
    Awesome guide!

    One note though: "-shrinkdebugfile" cut out old strings from debug.log file when its size is more than ~10Mb. Also this option is enabled right out of the box (but it will be turned off automatically if you run in debug mode i.e. specified "-debug") so there is no need to specify it explicitly.
     
  6. TanteStefana

    TanteStefana Moderator
    Linguistic Foundation Member

    Joined:
    Mar 9, 2014
    Messages:
    2,859
    Likes Received:
    1,854
    Trophy Points:
    1,283
    Ah, OK, thank you for that! I'll fix it :)
     
  7. bertlebbert

    bertlebbert Active Member

    Joined:
    Jul 17, 2014
    Messages:
    669
    Likes Received:
    289
    Trophy Points:
    133
    Nice, straight forward and right to the point... Very well done!
     
  8. TanteStefana

    TanteStefana Moderator
    Linguistic Foundation Member

    Joined:
    Mar 9, 2014
    Messages:
    2,859
    Likes Received:
    1,854
    Trophy Points:
    1,283
    Thanks guys, remember, this works just fine as a single user as well, so it can be used on a single server/single user.
     
  9. grzem

    grzem Member

    Joined:
    Nov 22, 2014
    Messages:
    49
    Likes Received:
    7
    Trophy Points:
    48
    If you use cron, there is no need to use loop (while...) in mn_watch.
    I'm affraid that cron runs many instances of mn_watch which has endless loop.
     
    • Like Like x 1
  10. TanteStefana

    TanteStefana Moderator
    Linguistic Foundation Member

    Joined:
    Mar 9, 2014
    Messages:
    2,859
    Likes Received:
    1,854
    Trophy Points:
    1,283
    Oh dear, can you explain how I should fix this please? Thank you so much!

    OK, so it seems this works without the while statement in the first line, thanks!
     
    #10 TanteStefana, Sep 4, 2015
    Last edited by a moderator: Sep 4, 2015
  11. TanteStefana

    TanteStefana Moderator
    Linguistic Foundation Member

    Joined:
    Mar 9, 2014
    Messages:
    2,859
    Likes Received:
    1,854
    Trophy Points:
    1,283
    Ooops, thank you! I'll fix that!
     
  12. TanteStefana

    TanteStefana Moderator
    Linguistic Foundation Member

    Joined:
    Mar 9, 2014
    Messages:
    2,859
    Likes Received:
    1,854
    Trophy Points:
    1,283
    Please use moocowmoo s bash script, I updated the tutorial with it. It works far better. Thanks!
     
    #12 TanteStefana, Sep 13, 2015
    Last edited by a moderator: Sep 13, 2015
  13. Dash4Ever

    Dash4Ever Active Member

    Joined:
    Sep 24, 2015
    Messages:
    106
    Likes Received:
    105
    Trophy Points:
    93
    Dash Address:
    XybaxnhtFBih2g4M2F71rWKBt5USzo8R
    Hey thanks for the info, is there anyway to see if it works ? When Im runing ./mn_watch.sh I get error "/usr/local/bin/dashd: No such file or directory"
    Any suggestions ?
     
  14. TanteStefana

    TanteStefana Moderator
    Linguistic Foundation Member

    Joined:
    Mar 9, 2014
    Messages:
    2,859
    Likes Received:
    1,854
    Trophy Points:
    1,283
    Did you install dashd at that location? It should actually be where you installed it. So if you install under your User, like many do, it would be /home/YourUserName/DashFolder/dashd

    It's been a while, I can't remember the default dash folder name. I put it in /usr/local/bin or /usr/bin because it doesn't require the ./ to launch, and I can start it from different users, so I only have to update once. Then I got used to it, LOL. But it actually causes issues with moocowmoo's scripts, so probably not worth it.


    Oh, and to see if it is working, you can just stop dashd : IE ./dashd stop Then wait a bit, and then see if it is running again with ./dash-cli getinfo. If it is running again, it'll give you a status print out, or if not, it'll say something like not running.

    Also, moocowmoo has done a lot of work on Dashman, if you're just setting up, you may want to use it to run your nodes: https://github.com/moocowmoo/dashman I just haven't gotten around to it yet. My mind is just too full ATM. Too many personal things to do, i can't concentrate, LOL.

    He's usually around to help out, just pm him :)
     
    #14 TanteStefana, Nov 13, 2015
    Last edited by a moderator: Nov 13, 2015
  15. Dash4Ever

    Dash4Ever Active Member

    Joined:
    Sep 24, 2015
    Messages:
    106
    Likes Received:
    105
    Trophy Points:
    93
    Dash Address:
    XybaxnhtFBih2g4M2F71rWKBt5USzo8R
    Thanks for the reply! :) gonna check the filepath for dashd and update the script. Do you get any reply / message when you start mn_watch.sh ?
     
  16. Dash4Ever

    Dash4Ever Active Member

    Joined:
    Sep 24, 2015
    Messages:
    106
    Likes Received:
    105
    Trophy Points:
    93
    Dash Address:
    XybaxnhtFBih2g4M2F71rWKBt5USzo8R
    Think I have it working now, seems like my dashd is installed under home/user/

    Change the scripts to this:

    @reboot /home/myusername/dashd
    */10 * * * * /home/myusername/mn_watch.sh >/dev/null 2>&1


    and

    #!/bin/bash
    DASHD_RUNNING=$( ps aux | grep $myusername | grep dashd | grep -v wc -l )
    if [ "$DASHD_RUNNING -lt 1" ];then
    /home/myusername/dashd 2>&1 >/dev/null
    fi


    When I ran ./mn_watch.sh I recieved :

    Usage: grep [OPTION]... PATTERN [FILE]...
    Try 'grep --help' for more information.
    [email protected]:~$ Error: Cannot obtain a lock on data directory /home/myusername/.dash. Dash Core is probably already running.

    So is everything fine ?
     
  17. splawik21

    splawik21 Grizzled Member
    Dash Core Team Foundation Member Dash Support Group

    Joined:
    Apr 8, 2014
    Messages:
    1,898
    Likes Received:
    1,277
    Trophy Points:
    1,283
    do "ps aux" please and ask your self if you see the dashd running :)?
    btw. simply ./dash-cli getinfo will answer this question too ;)

    you can check yourself as well by:

    Code:
    ./dash-cli masternode status
     
  18. Dash4Ever

    Dash4Ever Active Member

    Joined:
    Sep 24, 2015
    Messages:
    106
    Likes Received:
    105
    Trophy Points:
    93
    Dash Address:
    XybaxnhtFBih2g4M2F71rWKBt5USzo8R
    Yeah I can see it running :p also used command "top" for this.
    Tried "pkill dashd" and watched all the processes in "top" and after a few minutes dashd started again :) now I can relax. Thanks for all help
     
    • Like Like x 1
  19. splawik21

    splawik21 Grizzled Member
    Dash Core Team Foundation Member Dash Support Group

    Joined:
    Apr 8, 2014
    Messages:
    1,898
    Likes Received:
    1,277
    Trophy Points:
    1,283
    yep, now you can relax with a cold beer in a hand :cool:
     
    • Like Like x 1
  20. TanteStefana

    TanteStefana Moderator
    Linguistic Foundation Member

    Joined:
    Mar 9, 2014
    Messages:
    2,859
    Likes Received:
    1,854
    Trophy Points:
    1,283
    no, it doesn't echo anything, though I suppose it's possible to do. Unfortunately, I got a lot of help with it, and don't know where or how to put a print out message. All I can say is I've been using it and my nodes haven't stopped in 2 months. Still, this is all kind of old, and if you don't run more than 1 masternode on one machine (I have one machine, 2 users and 2 ip addresses) then it's undoubtedly easier to have moocowmoo's script run them for you. Especially since it's made to help you install, update, monitor,etc... All I'd have to do is have 2 installations of Dash instead of one. And I plan on doing this some day, LOL. But I'm lazy :)

    I leave this here for nothing else than to show myself what I once learned, LOL. My memory disappears so easily :p
     
    • Like Like x 1
  21. Sammy90

    Sammy90 New Member

    Joined:
    Jun 25, 2018
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    Hello Everyone!

    I wanted to try this here but I fear that it does not work. What Dash4Ever found is actually an error. Running the script

    ps aux | grep $myusername | grep dashd | grep -v wc -l

    will give an error of grep.

    Usage: grep [OPTION]... PATTERN [FILE]...
    Try 'grep --help' for more information.


    So the thing why nobody noticed is that the script simply starts dashd again. But since it is already running the process gets canceled. Therefore it really restarts dashd when it crashed. However, this is not a really nice solution (it is basically trying to restart dashd every 10 minutes). Furthermore, I want to add a command into the script that, in case of a crash, I want to receive a mail via ssmtp. This does not seem to work (probably because of the problem of grep).

    EDIT: Okay I had the $ in front of my username. However, the IF command is even true if the dashd is already running.

    Can we fix this issue ? I did not come up with a good solution.



    Thanks guys!
     
    #21 Sammy90, Jun 25, 2018
    Last edited: Jun 25, 2018
  22. Sammy90

    Sammy90 New Member

    Joined:
    Jun 25, 2018
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    Okay I managed to make it work with a new code:

    #!/bin/bash
    #!/bin/bash
    if ! pgrep -fu USER pivxd >/dev/null; then
    ~/pivx-3.1.0/bin/pivxd -daemon 2>&1 >/dev/null
    fi

    This is for pivx but for dash it is the same I guess!
     

Share This Page