Evonode Ubuntu & Docker daily logging

[qwizzie]

You will need to create three seperate shell scripts : hardware.sh, docker.sh (first post) and monitor.sh (second post)
With regards to the first two :

hardware.sh and docker.sh will run every 24 hours and log their output in log files.
hardware.sh also contains a trigger alert, it will visually warn you when dashd Process Memory Reserve exceeds 85%

These two files will not be reboot resistant, they will stop daily logging after a reboot. You can also manually stop them by issuing
ps -aux | grep yourusername and giving a kill proces command. Or just use dashmate stop and sudo reboot to kill them and then start dashmate again.
These two shell scripts need to be run only once and they will then perform daily log operations on the time your started them.

monitor.sh (second post) will show the content of those log files on screen in a nice colorfull overview and can be used as many times as needed.
Afterwards you can also make a selection to show a specific container by pressing y, or just end the script by pressing n.
./monitor.sh is the command that you will keep using after having set this up. Make sure Putty is running as a full window.

Step 1 : nano hardware.sh
Step 2 : add following code for Ubuntu logging (adjust total_ram_gb=32 under section ' # Calculate the percentage of process memory reserve' if you have lower RAM installed)

#!/bin/bash

# Set script to exit immediately if any command fails
set -e

# Function to get CPU usage
get_cpu_usage() {
  cpu_info=$(top -bn1 | grep "Cpu(s)" || true)
  if [ -n "$cpu_info" ]; then
    cpu_usage=$(echo "$cpu_info" | awk '{print $2 + $4}')
    echo "CPU Usage: ${cpu_usage}%"
  else
    echo "Error: Unable to retrieve CPU usage."
  fi
}

# Function to get memory usage
get_memory_usage() {
  memory_info=$(free -m || true)
  if [ -n "$memory_info" ]; then
    memory_usage=$(echo "$memory_info" | awk '/Mem/ {print $3}')
    memory_usage_gb=$(echo "scale=2; $memory_usage / 1024" | bc)
    echo "Memory Usage: ${memory_usage_gb}GB"
  else
    echo "Error: Unable to retrieve memory usage."
  fi
}

# Function to get process memory reserve
get_process_memory_reserve() {
  process_memory_reserve=$(ps -opid,vsz,cmd -C dashd | awk 'NR>1 {sum += $2} END {print sum}')
  process_memory_reserve_gb=$(echo "scale=2; $process_memory_reserve / (1024 * 1024)" | bc)
  echo "Process Memory Reserve: ${process_memory_reserve_gb}GB"

  # Calculate the percentage of process memory reserve
  total_ram_gb=32  # Total RAM in GB, adjust if necessary in case you have lower RAM
  process_memory_reserve_percentage=$(echo "scale=2; $process_memory_reserve_gb / $total_ram_gb * 100" | bc)
  echo "Percentage: ${process_memory_reserve_percentage}%"

  # Check if dashd Process Memory Reserve exceeds 85%
  if (( $(echo "$process_memory_reserve_percentage > 85" | bc -l) )); then
    echo -e "\e[1;31mAlert: dashd Process Memory Reserve exceeds 85%!\e[0m"
    echo -e "\e[1;31mAlert: dashd Process Memory Reserve exceeds 85%!\e[0m" >> hardware.log
    # You can add additional actions here, such as sending an email or triggering a notification.
  fi
}

# Function to get disk usage
get_disk_usage() {
  disk_info=$(df -h / || true)
  if [ -n "$disk_info" ]; then
    disk_usage=$(echo "$disk_info" | awk '/\// {print $5}')
    echo "Disk Usage: ${disk_usage}"
  else
    echo "Error: Unable to retrieve disk usage."
  fi
}

# Function to log data
log_data() {
  echo "$(date) - $(get_cpu_usage) - $(get_memory_usage) - $(get_process_memory_reserve) - $(get_disk_usage)"
}

# Main function
main() {
  while true; do
    log_data >> hardware.log
    sleep 24h
  done
}

# Execute the main function in the background
main &

# Notify user that the script is running in the background
echo "Script is running in the background. You can close the terminal."

Ctrl + X (Save? y enter)
Step 3 : chmod +x hardware.sh
Step 4 : ./hardware.sh

Step 5 : nano docker.sh
Step 6 : add following code for Docker containers logging

#!/bin/bash

# Set script to exit immediately if any command fails
set -e

# Function to log Docker stats for all containers
log_docker_stats() {
  docker_stats=$(docker stats --no-stream | sed '1d')
  if [ -n "$docker_stats" ]; then
    echo "$docker_stats"
  else
    echo "Error: Unable to retrieve Docker stats for the containers."
  fi
}

# Function to log data
log_data() {
  # Prepare data with timestamp and Docker stats
  echo "$(date)" >> dockerinfo.log  # Log the date on the first line
  log_docker_stats >> dockerinfo.log  # Append Docker stats on the next line
}

# Main function
main() {
  while true; do
    log_data  # Call log_data function
    sleep 24h  # Sleep for 24 hours
  done
}

# Execute the main function in the background
main &

# Notify user that the script is running in the background
echo "Script is running in the background. You can close the terminal."

Ctrl + X (Save? y enter)
Step 7 : chmod +x docker.sh
Step 8 : ./docker.sh

 

[qwizzie]

Step 9 : nano monitor.sh
Step 10 : add following code for viewing all the logging on screen

#!/bin/bash

# Fetch the username's home directory
home_dir=$HOME

# Specify the log file locations using the dynamic home directory
hardware_log_file="$home_dir/hardware.log"
dockerinfo_log_file="$home_dir/dockerinfo.log"

# Function to process the hardware log
process_hardware_log() {
    if [ -f "$hardware_log_file" ]; then
        header="\033[1;32m Date\t\t\t\t\033[1;33mCPU Usage\t\033[1;34mMemory Usage\t\033[1;35mProcess Memory Reserve\t\033[1;36mPercentage\t\033[1;37mDisk Usage\033[0m"
        echo
        echo -e "\033[1;37m Ubuntu logging\033[0m"
        echo
        echo -e "$header"

        while IFS= read -r line1 && IFS= read -r line2; do
            date_part=$(echo "$line1" | cut -d ' ' -f 1-6)
            cpu=$(echo "$line1" | grep -oP 'CPU Usage: \K[0-9.]+%' || echo "-")
            memory=$(echo "$line1" | grep -oP 'Memory Usage: \K[0-9.]+[BbGg]+' || echo "-")
            disk_usage=$(echo "$line2" | grep -oP 'Disk Usage: \K[0-9]+%' || echo "-")
            process_memory_reserve=$(echo "$line1" | grep -oP '(?<=Process Memory Reserve: )[0-9.]+[BbGg]+' || echo "-")
            percentage=$(echo "$line2" | grep -oP 'Percentage: \K[0-9.]+%' || echo "-")

            output="\033[1;32m $date_part\t\033[1;33m$cpu\t\t\033[1;34m$memory\t\t\033[1;35m$process_memory_reserve\t\t\t\033[1;36m$percentage\t\t\033[1;37m$disk_usage\033[0m"
            echo -e "$output"
        done < "$hardware_log_file"
    else
        echo "Error: Log file $hardware_log_file not found."
    fi
}

# Helper function to convert GiB to GB
convert_gib_to_gb() {
    echo "scale=2; $1 * 1.073741824" | bc
}

# Function to process the Docker log
process_dockerinfo_log() {
    if [ -f "$dockerinfo_log_file" ]; then
        # Header for Docker logs
        header="\033[1;32mContainer Name\033[0m$(printf '%*s' 59 '')\033[1;33mCPU\033[0m$(printf '%*s' 13 '')\033[1;34mMemory Usage\033[0m$(printf '%*s' 6 '')\033[1;35mLimit\033[0m$(printf '%*s' 14 '')\033[1;36mPercentage\033[0m"
        echo
        echo -e "\033[1;37m Docker Containers logging\033[0m"
        echo
        echo -e "$header"
        echo "----------------------------------------------------------------------------------------------------------------------------------------"

        # Process the log in blocks of 10 lines
        current_date=""
        line_counter=0

        while IFS= read -r line; do
            if (( line_counter % 10 == 0 )); then
                # Extract and store the date
                echo
                current_date=$(echo "$line" | cut -d ' ' -f 1-8)
                echo -e "\033[1;37mDate: $current_date\033[0m"
            else
                # Extract values from the log
                name=$(echo "$line" | awk '{print $2}')
                cpu_usage=$(echo "$line" | awk '{print $3}')
                mem_usage=$(echo "$line" | awk '{print $4}')
                mem_limit=$(echo "$line" | awk '{print $6}')
                percentage=$(echo "$line" | awk '{print $7}')
               
                # Convert memory usage and limit to GB if needed
                mem_usage_value=$(echo "$mem_usage" | sed 's/MiB//; s/GiB//')
                mem_usage_unit=$(echo "$mem_usage" | grep -oP 'MiB|GiB')
                if [[ "$mem_usage_unit" == "MiB" ]]; then
                    mem_usage_gb=$(echo "scale=2; $mem_usage_value / 1024" | bc)
                else
                    mem_usage_gb=$mem_usage_value
                fi
                mem_usage_formatted=$(printf "%.2f GB" "$mem_usage_gb")

                mem_limit_value=$(echo "$mem_limit" | sed 's/MiB//; s/GiB//')
                mem_limit_unit=$(echo "$mem_limit" | grep -oP 'MiB|GiB')
                if [[ "$mem_limit_unit" == "MiB" ]]; then
                    mem_limit_gb=$(echo "scale=2; $mem_limit_value / 1024" | bc)
                else
                    mem_limit_gb=$mem_limit_value
                fi
                mem_limit_formatted=$(printf "%.2f GB" "$mem_limit_gb")

                # Output the container data
                echo -e "\033[32m$name\033[0m$(printf '%*s' $((72 - ${#name})) '') \033[33m$cpu_usage\033[0m$(printf '%*s' 10 '') \033[34m$mem_usage_formatted\033[0m$(printf '%*s' 10 '') \033[35m$mem_limit_formatted\033[0m$(printf '%*s' 10 '') \033[36m$percentage\033[0m"
            fi
            ((line_counter++))
        done < "$dockerinfo_log_file"
    else
        echo "Error: Log file $dockerinfo_log_file not found."
    fi
}

# Function to focus on a specific Docker container
focus_on_specific_container() {
    echo -e "\033[1;36mDo you want to focus on a specific Docker container? (Y/N): \033[0m\c"
    read -r response
    if [[ "$response" == "Yes" || "$response" == "yes" || "$response" == "Y" || "$response" == "y" ]]; then
        clear
        echo -e "\033[1;32mSelect a container (1-9) to focus on:\033[0m"
        echo -e "\033[1;32m1.\033[0m \033[1;36mdashmate_6693980b_mainnet-gateway-1\033[0m"
        echo -e "\033[1;32m2.\033[0m \033[1;36mdashmate_6693980b_mainnet-dapi_api-1\033[0m"
        echo -e "\033[1;32m3.\033[0m \033[1;36mdashmate_6693980b_mainnet-drive_tenderdash-1\033[0m"
        echo -e "\033[1;32m4.\033[0m \033[1;36mdashmate_6693980b_mainnet-gateway_rate_limiter-1\033[0m"
        echo -e "\033[1;32m5.\033[0m \033[1;36mdashmate_6693980b_mainnet-dashmate_helper-1\033[0m"
        echo -e "\033[1;32m6.\033[0m \033[1;36mdashmate_6693980b_mainnet-dapi_core_streams-1\033[0m"
        echo -e "\033[1;32m7.\033[0m \033[1;36mdashmate_6693980b_mainnet-gateway_rate_limiter_redis-1\033[0m"
        echo -e "\033[1;32m8.\033[0m \033[1;36mdashmate_6693980b_mainnet-core-1\033[0m"
        echo -e "\033[1;32m9.\033[0m \033[1;36mdashmate_6693980b_mainnet-drive_abci-1\033[0m"
        echo -e "\033[1;32mEnter the number (1-9): \033[0m\c"
        read -r container_number

        case $container_number in
            1) container_name="dashmate_6693980b_mainnet-gateway-1" ;;
            2) container_name="dashmate_6693980b_mainnet-dapi_api-1" ;;
            3) container_name="dashmate_6693980b_mainnet-drive_tenderdash-1" ;;
            4) container_name="dashmate_6693980b_mainnet-gateway_rate_limiter-1" ;;
            5) container_name="dashmate_6693980b_mainnet-dashmate_helper-1" ;;
            6) container_name="dashmate_6693980b_mainnet-dapi_core_streams-1" ;;
            7) container_name="dashmate_6693980b_mainnet-gateway_rate_limiter_redis-1" ;;
            8) container_name="dashmate_6693980b_mainnet-core-1" ;;
            9) container_name="dashmate_6693980b_mainnet-drive_abci-1" ;;
            *) echo -e "\033[1;31mInvalid choice. Exiting.\033[0m"; exit 1 ;;
        esac

        clear
        echo -e "\033[1;32mFiltering dockerinfo.log for container: $container_name\033[0m"
        echo
        
        # Header for the output
        header="\033[1;32mContainer Name\033[0m$(printf '%*s' 26 '')\033[1;33mCPU\033[0m$(printf '%*s' 13 '')\033[1;34mMemory Usage\033[0m$(printf '%*s' 12 '')\033[1;35mLimit\033[0m$(printf '%*s' 11 '')\033[1;36mPercentage\033[0m"
        echo -e "$header"
        echo "----------------------------------------------------------------------------------------------------------"

        # Filter and display the log entries, excluding the container ID
        grep -E "^[A-Za-z]{3} [A-Za-z]{3} [ 0-9]{1,2} [0-9:]{8} UTC [0-9]{4}|$container_name" "$dockerinfo_log_file" | awk -v container="$container_name" '{
            if ($0 ~ container) {
                # Output the formatted container data with color coding and alignment
                printf("\033[32m%-30s\033[0m\t\033[33m%-10s\033[0m\t\033[34m%-16s\033[0m\t\033[35m%-10s\033[0m\t\033[36m%-10s\033[0m\n", $2, $3, $4, $6, $7);
            } else {
                print;
            }
        }' | sed 's/KiB/ KB/g; s/MiB/ MB/g; s/GiB/ GB/g' | while read -r line; do
            echo -e "\033[1;37m$line\033[0m"
        done
        else
        echo -e "\033[1;31mExiting script.\033[0m"
        exit 0
        fi
        }

# Call the functions
process_hardware_log
process_dockerinfo_log
focus_on_specific_container

Ctrl + X (Save? y enter)
Step 11 : chmod +x monitor.sh
Step 12 : ./monitor.sh

 

[qwizzie]

Ubuntu 22.04.5 LTS
32GB RAM

As you can see from above screenshot i have limited the Docker containers RAM usage to max 24GB for the Core container and the rest of the containers i have limited to 16GB. The reason for this being that per default Docker has a max limit of 31.35GB on 32GB RAM for all Docker containers. I found that a bit too high. If a Docker container got to a point of really high RAM usage (perhaps because of memory leakage), it would interfer with OS RAM. Specially with no OOM protection currently on a Docker install per default (this will most likely get fixed at some point through adjustments in docker compose).

docker stats --no-stream : shows you the container ID's

docker update --memory="24g" --memory-swap="24g" containerID : to limit the Core to 24GB
docker update --memory="16g" --memory-swap="16g" containerID : to limit other containers to 16GB

Note : this is not reboot consistent, if you have a reboot it will reset your container limits to those defined in docker compose (?)
For me that limit will reset to 31.35GB for all my Docker containers.

 

[qwizzie]

Adjusted monitor.sh (now on second post) to also let users select a specific Docker container to view its logging.

Note : I had to move the adjusted monitor.sh code snippet to second post as first post contained too much code to save.

 

[qwizzie]

I had a bit of a problem trying to convert MiB to MB and GiB to GB after selecting a container, as it kept messing up other field values. So i am going to leave that as is. I just made the final change to the code snippet. If people copied monitor.sh code before, try copy and paste again from post 2 to get latest version.

 

[qwizzie]

Managed to have ChatGPT successfully change the monitor.sh code on post 2 to get KiB, MiB, GiB renamed to KB, MB and GB after selecting a specific container to focus on.