Multiplexing masternodes on single VPS

xkcd

Active Member
Masternode Owner/Operator
Feb 19, 2017
229
193
103
australia
mnowatch.org
Dash Address
XpoZXRfr2iFxWhfRSAK3j1jww9xd4tJVez
This guide will attempt to show you how you can run more than one Masternode (MN) on a single VPS a technique I call multiplexing. To save space due to data redundancy the masternodes will share a common blockchain. The guide assumes you have setup your masternode as per guide https://www.dash.org/forum/threads/...e-setup-with-systemd-auto-re-start-rfc.39460/ and uses the VULTR hosting provider, because they offer additonal IPs for your hosting.

Starting with your existing running masternode created from my guide, goto your admin panel on VULTR and add another IPv4 IP.


upload_2020-3-14_13-46-33.png


Next as the dashadmin user click on 'networking configuration' on vultr and copy and paste the text for the latest Ubuntu build as described on thatpage for your OS.

upload_2020-3-14_15-30-1.png

After updating /etc/netplan/10-ens3.yaml with your new IP apply the changes.

Code:
sudo netplan apply
To enable the new IP now restart the VPS from the Vultr admin panel, sudo reboot won't do it.

ssh back into the machine and test the new network by trying to ssh on it, eg.

Code:
ssh [email protected]<new IP>
You should see a prompt asking you to confirm the security fingerprint of the server (yes/no). If you see that, then your IP is working correctly, otherwise you need to troubleshoot your VPS before continuing with this guide....

Next we create some new users, one for the second DASH MN and the other for the common files (blocks) that both MNs will share.

As dashadmin,

Code:
sudo useradd -m -c dash02 dash02 -s /bin/bash
sudo useradd -m -c dash-common dash-common -s /usr/sbin/nologin
sudo usermod dashadmin -a -G dash02,dash-common
sudo usermod dash -a -G dash-common
sudo usermod dash02 -a -G dash-common
Now we set a hard password for the new users, you do not have to write these down, you will never need them.

Code:
< /dev/urandom tr -dc A-Za-z0-9 | head -c${1:-32};echo
sudo passwd dash02
< /dev/urandom tr -dc A-Za-z0-9 | head -c${1:-32};echo
sudo passwd dash-common
Next, we create directories and set permissions.

Code:
sudo mkdir -p /home/dash-common/.dashcore/blocks
sudo chown -v -R dash-common.dash-common /home/dash-common/
sudo chmod -v -R g+wxr /home/dash-common/
Next, shutdown the running dashd and move the common files to the shared user, create a few links back to these files and restart the node...

Run the below as the dashadmin user
Code:
sudo systemctl stop dashd
Sudo to the dash user and run the rest.

Code:
sudo su - dash
Code:
# Creates a variable of the list of files (blocks) we need to copy over, excluding the current block, which is still being written into.
files=$(ls /home/dash/.dashcore/blocks/blk*dat|head -$(($(ls /home/dash/.dashcore/blocks/blk*dat|wc -l)-1)))

# Append the list of rev files.
files+=$(echo;ls /home/dash/.dashcore/blocks/rev*dat|head -$(($(ls /home/dash/.dashcore/blocks/rev*dat|wc -l)-1)))

# Move the blocks over to the common location
for f in $files;do mv -v $f /home/dash-common/.dashcore/blocks/;done

# Now the dash user will create symlinks back to those files to replace the moved ones.

cd ~/.dashcore/blocks/
for f in /home/dash-common/.dashcore/blocks/*;do ln -vs $f $(basename $f);done
Now, while the dashd is still down, we need to change the dash.conf file slightly.
Ensure the externalip is set like normal, but also set a new parameter called
bind= to the same IP, this is your orginal node. Then add a new parameter like below
rpcport=9998

Code:
nano ~/.dashcore/dash.conf
As the dashadmin user, update the file permisssions once more.

Code:
sudo chown -v -R dash-common.dash-common /home/dash-common/
sudo chmod -v -R g+wrx /home/dash-common/.dashcore/blocks/
Restart the dashd as the dashadmin user....

Code:
sudo systemctl start dashd
Verify that the dashd has restarted successfully before moving on, otherwise troubleshoot your changes.

Initialise, the second masternode, this will be a clone of the other masternode, but we will update the details in the dash.conf shortly.

Stop the dashd daemon once more and copy the files over. :)
Remove some stale cache files that would conflict with the original node, these will be rebuilt. note the below rm command will remove wallets, you should not have any dash stored on a MN anyway, but make sure that is the case before proceeding. o_O

Code:
sudo systemctl stop dashd
sudo cp -va /home/dash/.dashcore /home/dash02
sudo chown -v -R dash02.dash02 /home/dash02/
sudo rm -fr /home/dash02/.dashcore/{.lock,d*.log,*.dat} /home/dash02/.dashcore/backups/

Now that the data has been copied over, you can again restart the orginal node as the dashadmin user....
Code:
sudo systemctl start dashd
Since, the dash02 is a clone of the first node, we need to enter the specifics for this node, edit the dash.conf file like so.

Code:
sudo nano /home/dash02/.dashcore/dash.conf
and the things you need to change are listed below.

rpcuser
rpcpassword
externalip
bind
masternodeblsprivkey
rpcport


Change the RPC port to 9997, although you can choose another number larger than 1024 that the system is not currently using... Make sure the bind and externalip are both set to your new (second) IP and that rpcuser/password are set to something different than the original node, also place the bls key for the new masternode too.


Now we setup up a systemd unit file for this second node so it starts and shuts down automatically.
The below should be run as the dashadmin user.

Code:
sudo mkdir -p /etc/systemd/system&&\
sudo bash -c "cat >/etc/systemd/system/dashd02.service<<\"EOF\"
[Unit]
Description=Dash Core Daemon (2)
After=syslog.target network-online.target


[Service]
Type=forking
User=dash02
Group=dash02

OOMScoreAdjust=-1000

ExecStart=/opt/dash/bin/dashd -pid=/home/dash02/.dashcore/dashd.pid
TimeoutStartSec=10m

ExecStop=/opt/dash/bin/dash-cli stop

TimeoutStopSec=120

Restart=on-failure
RestartSec=120

StartLimitInterval=300
StartLimitBurst=3

[Install]
WantedBy=multi-user.target

EOF"
I have removed the comments from the above file to condense it, but it is very similar to the one used to start/stop the other dashd.

Next, we register the file with systemd and start the daemon.

Code:
sudo systemctl daemon-reload &&\
sudo systemctl enable dashd02 &&\
sudo systemctl start dashd02 &&\
echo "Dash02 is now installed as a system service and initializing..."
 
Last edited:
  • Like
Reactions: bittersea

xkcd

Active Member
Masternode Owner/Operator
Feb 19, 2017
229
193
103
australia
mnowatch.org
Dash Address
XpoZXRfr2iFxWhfRSAK3j1jww9xd4tJVez
Make sure you install sentinel :mad:

run the below as the dash02 user !

sudo su - dash02

Code:
cd &&\
git clone https://github.com/dashpay/sentinel &&\
cd sentinel &&\
virtualenv venv &&\
venv/bin/pip install -r requirements.txt &&\
venv/bin/py.test test &&\
venv/bin/python bin/sentinel.py

# Add a crontab entry.
echo "*/10 * * * * { test -f ~/.dashcore/dashd.pid&&cd ~/sentinel && venv/bin/python bin/sentinel.py;} >> \
 ~/sentinel/sentinel-cron.log 2>&1" \
|crontab -&&echo "Successfully installed cron job."



Verify the daemons are running in top, also at this time, make sure you have executed the protx 3-part shuffle to register your new MN on the DASH network.

upload_2020-3-14_20-12-58.png

Let's check the nodes are bound to the correct IPs.
As the dashadmin user...
Code:
echo "dash....."
sudo grep "Bound\|AddLocal" /home/dash/.dashcore/debug.log
echo "dash02....."
sudo grep "Bound\|AddLocal" /home/dash02/.dashcore/debug.log
What you should see is the second node is bound to a different (secondary) IP.

On my system I have the folowing disk space being used.

upload_2020-3-14_20-15-56.png

Notice that there is 14GB which is common to both installations, so I effectively saved 14GB of disk space, if I were to spin up a third node, I would only need another 5 GB to do so.

upload_2020-3-14_20-18-2.png

I still have 45 GB to go, plenty for growth and another node. o_O

RAM usage is looking fine too, if I run low, I would increase the swap space rather than adding costly RAM. :cool:

upload_2020-3-14_20-19-40.png

DASH is generally not a CPU bound app, it tends to be memory hungry and disk hungry.

I hope you've enjoyed the guide!
Leave a like and a comment. :D
 
Last edited:

xkcd

Active Member
Masternode Owner/Operator
Feb 19, 2017
229
193
103
australia
mnowatch.org
Dash Address
XpoZXRfr2iFxWhfRSAK3j1jww9xd4tJVez
Update to this guide, I have been running without issues for weeks now with three instances of dashd on the VPS above, but a couple of days ago the VPS ran out of memory, both RAM and SWAP were full (6GB) a hard restart was required to sort it out. To figure out what happened, I now have a cron running that records the free RAM available and the RAM used by each of the dashd instances running every minute, I aim to see if the escalation in RAM usage is gradual or sudden. I have also increased my swap to 4GB, so I now have 4+4=8GB available on the VPS. I have also pre-emptively written a script that will monitor the RAM usage of the system and restart the dashd using the most RAM before it crashes the system, but I rather avoid such workarounds if at all possible.

If you face out of memory issues, let us know, it could help the devs figure out if this normal behaviour or some kind on memory leak.
 

xkcd

Active Member
Masternode Owner/Operator
Feb 19, 2017
229
193
103
australia
mnowatch.org
Dash Address
XpoZXRfr2iFxWhfRSAK3j1jww9xd4tJVez
I have been running multiplexed masternodes for many months now without any issues at all, however, over time, any new blocks created will be stored by each node resulting in redundancy. The below script will save disk space by moving duplicate blocks into the dash-common user's directory and adjust each node accordingly.

To do this, run the below block as root and be careful to not make errors and do it all in one sitting without logining out and back in!

To become root, as the dashadmin user issue sudo su -

Code:
# Shutdown all the nodes!  This is important otherwise the data will not be consistent!
systemctl stop dashd dashd02 dashd03

files=$(find /home/dash/.dashcore/blocks -type f -name "blk*"|sort|head -$(($(find /home/dash/.dashcore/blocks -type f -name "blk*"|wc -l)-1)))
files+=$(echo;find /home/dash/.dashcore/blocks -type f -name "rev*"|sort|head -$(($(find /home/dash/.dashcore/blocks -type f -name "rev*"|wc -l)-1)))
# Only do the below if the $files variable contains elements.
for f in $files;do mv -v $f /home/dash-common/.dashcore/blocks/;done
chmod -v -R g+wrx /home/dash-common/.dashcore/blocks/
chown -vR dash-common:dash-common /home/dash-common/.dashcore/blocks/
cd /home/dash/.dashcore/blocks/
for f in $files;do ln -vs "/home/dash-common/.dashcore/blocks/$(basename $f)" $(basename $f);done
chown -Rv dash:dash /home/dash/.dashcore/blocks/

cp -v /home/dash02/.dashcore/dash.conf /tmp/
rm -fr /home/dash02/.dashcore/
cp -va /home/dash/.dashcore /home/dash02
rm -fr /home/dash02/.dashcore/{.lock,.walletlock,d*.log,*.dat} /home/dash02/.dashcore/backups/
cp -v /tmp/dash.conf /home/dash02/.dashcore/
chown -v -R dash02:dash02 /home/dash02/

# This block is the same as the above block duplicated for this node.
cp -v /home/dash03/.dashcore/dash.conf /tmp/
rm -fr /home/dash03/.dashcore/
cp -va /home/dash/.dashcore /home/dash03
rm -fr /home/dash03/.dashcore/{.lock,.walletlock,d*.log,*.dat} /home/dash03/.dashcore/backups/
cp -v /tmp/dash.conf /home/dash03/.dashcore/
chown -v -R dash03:dash03 /home/dash03/

# Just reboot and all the nodes will come back online themselves.
reboot
 
Last edited:
  • Like
Reactions: Sven and bittersea

xkcd

Active Member
Masternode Owner/Operator
Feb 19, 2017
229
193
103
australia
mnowatch.org
Dash Address
XpoZXRfr2iFxWhfRSAK3j1jww9xd4tJVez
Dear MultiDashers! I have updated the OP and the maintenance post just above this one to now also archive the rev*dat files, which saves us an additional 3GB disk space per instance !!! o_O

1626012599026.png

I have tested a full reindex and the dashd was perfectly happy with it, so I consider this stable. To take advantage of this change simply follow the steps in the above post and the rev files will be moved over and replaced with links. :cool: