MasternodeMonitor : SMTP alerts for your VPS

[chatgpt420]

Hello MNOs,

I've put together a python script to assist in monitoring, and hopefully rescuing, your VPS before incurring pose penalties.

This is achieved by monitoring CPU, memory and disk usage, and sending out an early warning email before you run out of resources.
In the event that memory usage climbs to 96%, the script will send out an email alert and immediately reboot the server to prevent a crash - not very elegant I know, but desperate times call for desperate measures!

Now the exciting feature for specific to MNOs - regular automatic port checks for all IPs connected to the VPS - no additional configuration required!
The script will gather all connected IP addresses and check them against the port test the owners of https://mnowatch.org have built into their site - thanks guys
Ports will be checked every 10 minutes and an email alert will be sent listing all of the offending IP addresses.

Setup of the script is quite straight forward so long as you;
- have root access to the system
- are running a linux vps with some swap space configured (this requirement can be removed in future iterations)
- have some SMTP credentials handy

Installation
---
Log into your VPS as root
Download the code into a suitable directory --- git clone https://github.com/chatgpt420/nodemonitor.git
Run the 'firstrun.py script in python --- python3 firstrun.py
Follow the prompts until you see some system stats being printed at 10 second intervals, then quit the process (Ctrl-c)

During the installation a service file is installed and enabled to ensure the script is always running.
To make sure the email alerts will be working when you need them, stop one of your dashd instances and wait 10 minutes

I'm on discord as 'BitterSeaTrading' if you have any questions, though I'll check in here too
Feedback and suggestions welcome

 

[qwizzie]

See also : https://github.com/dashpay/platform/issues/1648

A Masternode / Evonode can also be PoSe Penalty scored or PoSe banned during a manual dashd stop / manual dashmate stop from its owner / operator (for example to update Core or to perform server maintance or to perform a reboot), as that Masternode / Evonode could already be involved in a llmq quorum, at which point they will receive a high PoSe Penalty score (66%). If they are involved with two llmq quorums, that will get that Masternode/Evonode an instant PoSe ban.

The llmq quorums in red (and perhaps also the one in brown) are the llmq quorums that owners / operators should not interrupt with a manual dashd stop or a manual dashmate stop or a server reboot, untill it is clear their Masternode / Evonode is currently not involved with them.

Owners / operators can find information about these llmq quorums in their Masternode debug.log or in their Evonode core.log
Owners / operators can download these logs, open them with Notepad++, press CTRL + end, press CTRL + f and search backwards for

quorum initialization OK for llmq_60_75

Return :

CDKGSessionManager::InitNewQuorum -- height[1998463] quorum initialization OK for llmq_60_75 qi[31]

If returned height (1998463 in this case) is older then current block height (https://chainz.cryptoid.info/dash/) & it indeed returns [31], then there is currently no involvement with this specific quorum.

Note : this llmq quorum runs from 0-31, so if it returns anything other then 31 then wait with stopping dashd / dashmate untill it reached [31] and current block height is higher then quorum block height

quorum initialization OK for llmq_100_67

Return :

CDKGSessionManager::InitNewQuorum -- height[1998648] quorum initialization OK for llmq_100_67 qi[0]

If returned height (1998648 in this case) is older then current block height (https://chainz.cryptoid.info/dash/), then there is currently no involvement with this specific quorum.

quorum initialization OK for llmq_400_60

Return :

CDKGSessionManager::InitNewQuorum -- height[1998432] quorum initialization OK for llmq_400_60 qi[0]

If returned height (1998432 in this case) is older then current block height (https://chainz.cryptoid.info/dash/), then there is currently no involvement with this specific quorum.

It would be nice to have a program residing on the VPS that could optionally (when owner / operator wants to) look-up these llmq quorums directly in the live logs of our VPS, check them against current block height and then give a 'clear to stop' signal to the owner / operator. Instead of having to download and search through the logs ourselves, as described above.

Beside measuring CPU, memory and disk usage, it would (on request of owner / operator) also be checking llmq quorum initialization through the log, by using a tail -f command on the debug.log or the core.log.

 

[chatgpt420]

Hi qwizzie,

Thanks for the info. I could possibly scan the home folder for all debug.log files, scan through them and add the quorum participation statuses for each node to a list and output that to a text file. You would then just need to check the text file before stopping the node.

Does that sound like what you're after?

 

[qwizzie]

Hi qwizzie,

Thanks for the info. I could possibly scan the home folder for all debug.log files, scan through them and add the quorum participation statuses for each node to a list and output that to a text file. You would then just need to check the text file before stopping the node.

Does that sound like what you're after?

I think that would be very handy for Masternode and Evonode owners.

Note :

For Evonodes the log is called 'core.log' and can be found at location : /home/user/.dashmate/logs/mainnet/core.log
For Masternodes the log is called 'debug.log' and can usually be found at location : /home/user/.dashcore/debug.log (not totally sure, pls verify)

The logs output are node specific. So if your script can manage to locate them (you will either need to fetch the specific 'home/user' or just search within the home directory in order to locate them) and search them through (starting at last), that script should be able to fetch the quorums participation status for that specific Masternode or Evonode. Which the script can then return in a text format to the user or maybe even print-to-screen for users to read on the spot ?

 

[qwizzie]

@chatgpt420

Found some working commands :

tac ~/.dashmate/logs/mainnet/core.log | grep "height=" | head -n 1 (to get latest block height)

tac ~/.dashmate/logs/mainnet/core.log | grep "quorum initialization OK for llmq_60_75" | head -n 1
tac ~/.dashmate/logs/mainnet/core.log | grep "quorum initialization OK for llmq_100_67" | head -n 1
tac ~/.dashmate/logs/mainnet/core.log | grep "quorum initialization OK for llmq_400_60" | head -n 1

For more details and for a script shell : https://www.dash.org/forum/index.ph...ode-setup-through-dashmate.54445/#post-237453

 

[chatgpt420]

Hi Qwizzie,

I've updated the repo with a new file called 'quorums.py' - running that with 'python3 quorums.py' will print out a list for each of the 3 mentioned quorum types every 10 seconds until cancelled and list which nodes are participating.

I've also included some minor updates to other scripts. ie error handling and logging and improved the combined memory usage calc