Hey qwizzie, I had not seriously considered that Docker/Dashmate could be a factor in the memory over run,
I hope you are collecting per process statistics this time for the dashd, so we can analyze it together.
Here I have a cluster of three evonodes after running for 5 days on v20.1 and the VM size is stable at around 12GB.
View attachment 12366
Source :
https://github.com/dashpay/dash/issues/5872#issuecomment-1952300731
I made some changes a few days ago to the above, so that dashd Process Memory Reserve (a rename of process_memory_usage) is more clearly split from Memory Usage and shown as a percentage of total system RAM as well, together with a trigger to alert when exceeding 85% reserve....
# Function to get memory usage
get_memory_usage() {
memory_info=$(free -m || true)
if [ -n "$memory_info" ]; then
memory_usage=$(echo "$memory_info" | awk '/Mem/ {print $3}')
memory_usage_gb=$(echo "scale=2; $memory_usage / 1024" | bc)
echo "Memory Usage: ${memory_usage_gb}GB"
else
echo "Error: Unable to retrieve memory usage."
fi
}
# Function to get process memory reserve (renamed from process_memory_usage to process_memory_reserve)
get_process_memory_reserve() {
process_memory_reserve=$(ps -opid,vsz,cmd -C dashd | awk 'NR>1 {sum += $2} END {print sum}')
process_memory_reserve_gb=$(echo "scale=2; $process_memory_reserve / (1024 * 1024)" | bc)
echo "Process Memory Reserve: ${process_memory_reserve_gb}GB"
# Calculate the percentage of process memory reserve
total_ram_gb=24 # Total RAM in GB
process_memory_reserve_percentage=$(echo "scale=2; $process_memory_reserve_gb / $total_ram_gb * 100" | bc)
echo "Percentage: ${process_memory_reserve_percentage}%"
# Check if dashd Process Memory Reserve exceeds 85%
if (( $(echo "$process_memory_reserve_percentage > 85" | bc -l) )); then
echo -e "\e[1;31mAlert: dashd Process Memory Reserve exceeds 85%!\e[0m"
echo -e "\e[1;31mAlert: dashd Process Memory Reserve exceeds 85%!\e[0m" >> hardware.log
# You can add additional actions here, such as sending an email or triggering a notification.
fi
}
So yes, i am collecting per process statistics for dashd since 19th of February 2024.
Evonode 1 (dashd.pid startdate 18th of January 2024)
24GB RAM
OS : Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-169-generic x86_64)
Dashmate : 0.25.21
Dash Core : 20.0.4
Evonode 2 (dashd.pid startdate 21st of January 2024) :
24GB RAM
OS : Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-169-generic x86_64)
Dashmate : 0.25.21
Dash Core : 20.0.4
github.com
