Ongoing DDoS attack on masternode network

[UdjinM6]

Dear community,

We have an ongoing attack on masternode network for about 3 hours now. The attack is a mix of SYN flood, UDP flood with empty payload and protocols like sFlow and GRE. So far the only result attacker was able to achieve is that he caused higher CPU and bandwidth usage for most of masternodes which in its turn cased ~100 of masternodes to go down. We believe these masternodes were hosted on extremely low end VPS, like $1/mo, so it's not that surprising to see some of them down now. We encourage all masternode owners whose nodes were affected to move to a better hardware or upgrade their hosting plan to ensure that your masternode doesn't fall out of payment queue during such events.

Meanwhile, you can mitigate the attack by following steps in iptables setup part of this guide https://www.dash.org/forum/threads/how-to-set-up-ec2-t1-micro-ubuntu-for-masternode-part-2-3.241/

UPDATE:
Incident report: https://www.dash.org/2017/03/08/DDoSReport.html

 

[halso]

Dear community,

We have an ongoing attack on masternode network for about 3 hours now. The attack is a mix of SYN flood, UDP flood with empty payload and protocols like sFlow and GRE. So far the only result attacker was able to achieve is that he caused higher CPU and bandwidth usage for most of masternodes which in its turn cased ~100 of masternodes to go down. We believe these masternodes were hosted on extremely low end VPS, like $1/mo, so it's not that surprising to see some of them down now. We encourage all masternode owners whose nodes were affected to move to a better hardware or upgrade their hosting plan to ensure that your masternode doesn't fall out of payment queue during such events.

Meanwhile, you can mitigate the attack by following steps in iptables setup part of this guide https://www.dash.org/forum/threads/how-to-set-up-ec2-t1-micro-ubuntu-for-masternode-part-2-3.241/

Interesting. I thought this might happen when the value increased. I wonder if it was an economic attack by other MNs or possibly short sellers.

Would the tails IP blinding solution be a londer term response?

 

[AjM]

I've been waiting for this to happen.
This makes the Dash network even stronger.

 

[AjM]

Interesting. I thought this might happen when the value increased. I wonder if it was an economic attack by other MNs or possibly short sellers.

Would the tails IP blinding solution be a londer term response?

Probably Moronsons trying their last effort to tackle Dash, us.

 

[AjM]

Attack is causing my mnodes cpu load going to up 70% now, no problem so far any of my nodes.

 

[qwizzie]

How costly is this ongoing DDoS attack for the attacker and how long can we exspect such an attack to continue ?

Looks like this attacks also causes some incorrect feedback on port checks in dashninja.pl
(closed ports instead of open ports). At least i hope its feedback is incorrect.
(Dash Central does not give any errors)

 

[UdjinM6]

How costly is this ongoing DDoS attack for the attacker and how long can we exspect such an attack to continue ?

Looks like this attacks also causes some incorrect feedback on port checks in dashninja.pl
(closed instead of open). At least i hope its feedback is incorrect.

No idea how costly it is.
Yep, port checker is confused because all ports are used by the attacker if there is no defense in place on that masternode.

 

[qwizzie]

No idea how costly it is.
Yep, port checker is confused because all ports are used by the attacker if there is no defense in place on that masternode.

Will we be safe if we set the firewall on remote as follows ?

apt-get update

You will see a lot of things happening, just wait for them to stop.

apt-get install ufw

Enter the following commands EXACTLY (in this order) to set up your firewall:
Please note: Make sure you enter the code in this order! If you do not, the program will not work! If need be you can disable your firewall by entering (as root): ufw disable.

ufw allow ssh/tcp

ufw limit ssh/tcp ---this command limits SSH connections to 6 every 30 seconds for greater security---

ufw allow 9999/tcp

ufw logging on

ufw enable


Check your firewall's status by entering the following command:

ufw status

Or do we still need to add those iptables ?

 

[UdjinM6]

Will we be safe if we set the firewall on remote as follows ? Or do we still need to add those iptables ?

I'm not sure about ufw capability to stop this, I do know that configuring iptables the way it's described in that guide helps.
Let's ping our network guru @chaeplin for more info

 

[qwizzie]

I'm not sure about ufw capability to stop this, I do know that configuring iptables the way it's described in that guide helps.
Let's ping our network guru @chaeplin for more info

Good idea, and if it is indeed needed maybe someone can provide an easy to use Linux 14.04 specific up to date guide to installing these iptables on remote.

 

[AjM]

From ubuntu wiki:

The Uncomplicated Firewall (ufw) is a frontend for iptables and is particularly well-suited for host-based firewalls. ufw provides a framework for managing netfilter, as well as a command-line interface for manipulating the firewall. ufw aims to provide an easy to use interface for people unfamiliar with firewall concepts, while at the same time simplifies complicated iptables commands to help an adminstrator who knows what he or she is doing.

 

[qwizzie]

I'm trying to find a graph where we can observ the attack, but i cant really seem to find it. This one looks normal but then again it is only focussing on transactions :

Are there other graphs networkwide that shows the attack more clearly ?

 

[AndyDark]

I'm trying to find a graph where we can observ the attack, but i cant really seem to find it. This one looks normal but then again it is only focussing on transactions :

Are there other graphs networkwide that shows the attack more clearly ?

Source: @chaeplin

You can see the MNs that dropped off center of the top row, at the attacks peak (t+5 hrs) the # of nodes offline was 114 or ~2.8% of the network, probably the nodes with the least performance / provision / cost. Now in hour 6 and the nodes are recovering, 2.1% are still down.

 

[Figlmüller]

I'm observing not only SYN-Floods, but valid connection attempts to exhaust the connection limit of the dashd. Except for limiting the connection rate per client, I think there is currently no way to stop a part of the attacks without some behaviour-analysis-based firewall. Luckily, the DDoS mitigation of the provider kicked in now, so everything is back to normal.

 

[AndyDark]

I'm observing not only SYN-Floods, but valid connection attempts to exhaust the connection limit of the dashd. Except for limiting the connection rate per client, I think there is currently no way to stop a part of the attacks without some behaviour-analysis-based firewall. Luckily, the DDoS mitigation of the provider kicked in now, so everything is back to normal.

Yes DDOS protection, the IP tables/hardening that @UdjinM6 posted above and making sure your node is high enough spec / on decent infrastructure is the best protection.

This kind of attack can be done on any Crypto p2p network and it's clearly not very effective against Dash considering only 2.8% of service was disrupted vs the cost of flooding 4,100 nodes with that much bandwidth for 6 hrs.

 

[Walter]

The fact that only 2.8% of nodes have been affected to the point of crashing is great news for the network... We're in good shape!

We now know that it would take a significant DDOS attack to cause any kind of statistically meaningful attack on the functions and operation of network, and is very unlikely to succeed in disrupting the network. For sure, some MN operators that took the low bid on their VPS provisioning are probably a bit pissed but that's the true cost of cheap.. Lesson learned hopefully!

Let's see if this is just an exploratory DDOS and whether we get more targetted and higher intensity attacks going forwards..

Get yer hard hats on MN operators!

Walter

 

[crowning]

@qwizzie : You can only estimate it from the CPU and network load.

If you want to monitor it, you would need to have the firewalling iptables-rules log into a file or so, something which would make the attack even worse.

In other words, I could do it for one of my nodes, but I won't

 

[tungfa]

good test to remind all the 1U$ vultr nodes to migrate to more solid services i guess

 

[chaeplin]

Will we be safe if we set the firewall on remote as follows ?



Or do we still need to add those iptables ?

I will learn ufw and test.
time to learn ufw



https://www.cyberciti.biz/faq/howto-limiting-ssh-connections-with-ufw-on-ubuntu-debian/

 

[ericsammons]

good test to remind all the 1U$ vultr nodes to migrate to more solid services i guess

Maybe it was vultr conducting the stress test to get everyone to upgrade!

 

[emmo]

Any solution for those of us which use UFW(Ubuntu)-are we must switch to iptables ?Thanks

 

[akhavr]

ufw does use iptables internally. So it's just matter of preference.

 

[emmo]

ufw does use iptables internally. So it's just matter of preference.

okay, but from UFW commands i don't see any custom rules options

 

[qwizzie]

okay, but from UFW commands i don't see any custom rules options

If you followed Tao's guide it should be included already during setup of your firewall :

ufw limit ssh/tcp ---this command limits SSH connections to 6 every 30 seconds for greater security---

Chaeplin refers to it here : https://www.cyberciti.biz/faq/howto-limiting-ssh-connections-with-ufw-on-ubuntu-debian/

 

[tungfa]

Update:

- 16 h by now
- Network holding up well
- only dropped ca. 300 weak Mn's
(good flushing out weak Nodes experiment)

 

[UdjinM6]

If you followed Tao's guide it should be included already during setup of your firewall :

ufw limit ssh/tcp ---this command limits SSH connections to 6 every 30 seconds for greater security---

Chaeplin refers to it here : https://www.cyberciti.biz/faq/howto-limiting-ssh-connections-with-ufw-on-ubuntu-debian/

Attack is executed on mainnet p2p port - 9999, so I guess you need to add:

ufw limit 9999/tcp

to protect it too.

 

[kointrend]

Hi all, no techi guy here!
Is this attack one of the strongest we can be affected by?
I mean, it's a serious test stress or just a basic one?
I see that it is "easy" to defend against.
Some easy narrative (when and if you have time) could be appreciated, thanks.

 

[tibolt]

is there any side effect for "ufw limit 9999/tcp" ?

 

[UdjinM6]

is there any side effect for "ufw limit 9999/tcp" ?

This should limit it to 6 connection attempts per 30 seconds from a single IP. There is no functionality in Dash Core which would require such high rate iirc, so imo should be fine BUT I haven't tried it myself (using iptables here).

 

[tibolt]

i tried this command, lets see whats gonna happen thx for reply.

update; after i tried this command my masternode status at dashninja turned to unknown masternode. i think ip tables good choise

update 2: my masternode is working about 1 day with" ufw limit 9999/tcp" command. seems good for now