Ongoing DDoS attack on masternode network

UdjinM6

Official Dash Dev
Dash Core Team
Moderator
May 20, 2014
3,638
3,538
1,183
Dear community,

We have an ongoing attack on masternode network for about 3 hours now. The attack is a mix of SYN flood, UDP flood with empty payload and protocols like sFlow and GRE. So far the only result attacker was able to achieve is that he caused higher CPU and bandwidth usage for most of masternodes which in its turn cased ~100 of masternodes to go down. We believe these masternodes were hosted on extremely low end VPS, like $1/mo, so it's not that surprising to see some of them down now. We encourage all masternode owners whose nodes were affected to move to a better hardware or upgrade their hosting plan to ensure that your masternode doesn't fall out of payment queue during such events.

Meanwhile, you can mitigate the attack by following steps in iptables setup part of this guide https://www.dash.org/forum/threads/how-to-set-up-ec2-t1-micro-ubuntu-for-masternode-part-2-3.241/

UPDATE:
Incident report: https://www.dash.org/2017/03/08/DDoSReport.html
 
Last edited:

halso

Active Member
Apr 27, 2016
439
235
113
Sydney, Australia
Dear community,

We have an ongoing attack on masternode network for about 3 hours now. The attack is a mix of SYN flood, UDP flood with empty payload and protocols like sFlow and GRE. So far the only result attacker was able to achieve is that he caused higher CPU and bandwidth usage for most of masternodes which in its turn cased ~100 of masternodes to go down. We believe these masternodes were hosted on extremely low end VPS, like $1/mo, so it's not that surprising to see some of them down now. We encourage all masternode owners whose nodes were affected to move to a better hardware or upgrade their hosting plan to ensure that your masternode doesn't fall out of payment queue during such events.

Meanwhile, you can mitigate the attack by following steps in iptables setup part of this guide https://www.dash.org/forum/threads/how-to-set-up-ec2-t1-micro-ubuntu-for-masternode-part-2-3.241/
Interesting. I thought this might happen when the value increased. I wonder if it was an economic attack by other MNs or possibly short sellers.

Would the tails IP blinding solution be a londer term response?
 

AjM

Well-known Member
Foundation Member
Jun 23, 2014
1,341
575
283
Finland
Interesting. I thought this might happen when the value increased. I wonder if it was an economic attack by other MNs or possibly short sellers.

Would the tails IP blinding solution be a londer term response?
Probably Moronsons trying their last effort to tackle Dash, us.:D
 
  • Like
Reactions: qwizzie and halso

qwizzie

Well-known Member
Aug 6, 2014
1,608
764
183
How costly is this ongoing DDoS attack for the attacker and how long can we exspect such an attack to continue ?

Looks like this attacks also causes some incorrect feedback on port checks in dashninja.pl
(closed ports instead of open ports). At least i hope its feedback is incorrect.
(Dash Central does not give any errors)
 
Last edited:

UdjinM6

Official Dash Dev
Dash Core Team
Moderator
May 20, 2014
3,638
3,538
1,183
How costly is this ongoing DDoS attack for the attacker and how long can we exspect such an attack to continue ?

Looks like this attacks also causes some incorrect feedback on port checks in dashninja.pl
(closed instead of open). At least i hope its feedback is incorrect.
No idea how costly it is.
Yep, port checker is confused because all ports are used by the attacker if there is no defense in place on that masternode.
 

qwizzie

Well-known Member
Aug 6, 2014
1,608
764
183
No idea how costly it is.
Yep, port checker is confused because all ports are used by the attacker if there is no defense in place on that masternode.
Will we be safe if we set the firewall on remote as follows ?

apt-get update

You will see a lot of things happening, just wait for them to stop.

apt-get install ufw

Enter the following commands EXACTLY (in this order) to set up your firewall:
Please note: Make sure you enter the code in this order! If you do not, the program will not work! If need be you can disable your firewall by entering (as root;)): ufw disable.

ufw allow ssh/tcp

ufw limit ssh/tcp ---this command limits SSH connections to 6 every 30 seconds for greater security---

ufw allow 9999/tcp

ufw logging on

ufw enable


Check your firewall's status by entering the following command:


ufw status
Or do we still need to add those iptables ?
 

UdjinM6

Official Dash Dev
Dash Core Team
Moderator
May 20, 2014
3,638
3,538
1,183
Will we be safe if we set the firewall on remote as follows ? Or do we still need to add those iptables ?
I'm not sure about ufw capability to stop this, I do know that configuring iptables the way it's described in that guide helps.
Let's ping our network guru @chaeplin for more info :)
 

qwizzie

Well-known Member
Aug 6, 2014
1,608
764
183
I'm not sure about ufw capability to stop this, I do know that configuring iptables the way it's described in that guide helps.
Let's ping our network guru @chaeplin for more info :)
Good idea, and if it is indeed needed maybe someone can provide an easy to use Linux 14.04 specific up to date guide to installing these iptables on remote.
 

AjM

Well-known Member
Foundation Member
Jun 23, 2014
1,341
575
283
Finland
From ubuntu wiki:

The Uncomplicated Firewall (ufw) is a frontend for iptables and is particularly well-suited for host-based firewalls. ufw provides a framework for managing netfilter, as well as a command-line interface for manipulating the firewall. ufw aims to provide an easy to use interface for people unfamiliar with firewall concepts, while at the same time simplifies complicated iptables commands to help an adminstrator who knows what he or she is doing.
 

qwizzie

Well-known Member
Aug 6, 2014
1,608
764
183
I'm trying to find a graph where we can observ the attack, but i cant really seem to find it. This one looks normal but then again it is only focussing on transactions :



Are there other graphs networkwide that shows the attack more clearly ?
 
Last edited:

AndyDark

Well-known Member
Sep 10, 2014
353
705
163
I'm trying to find a graph where we can observ the attack, but i cant really seem to find it. This one looks normal but then again it is only focussing on transactions :

Are there other graphs networkwide that shows the attack more clearly ?
Capture.PNG

Source: @chaeplin :)

You can see the MNs that dropped off center of the top row, at the attacks peak (t+5 hrs) the # of nodes offline was 114 or ~2.8% of the network, probably the nodes with the least performance / provision / cost. Now in hour 6 and the nodes are recovering, 2.1% are still down.
 
Last edited:

Figlmüller

Member
Sep 2, 2014
85
45
58
Vienna, Austria
I'm observing not only SYN-Floods, but valid connection attempts to exhaust the connection limit of the dashd. Except for limiting the connection rate per client, I think there is currently no way to stop a part of the attacks without some behaviour-analysis-based firewall. Luckily, the DDoS mitigation of the provider kicked in now, so everything is back to normal.
 

AndyDark

Well-known Member
Sep 10, 2014
353
705
163
I'm observing not only SYN-Floods, but valid connection attempts to exhaust the connection limit of the dashd. Except for limiting the connection rate per client, I think there is currently no way to stop a part of the attacks without some behaviour-analysis-based firewall. Luckily, the DDoS mitigation of the provider kicked in now, so everything is back to normal.
Yes DDOS protection, the IP tables/hardening that @UdjinM6 posted above and making sure your node is high enough spec / on decent infrastructure is the best protection.

This kind of attack can be done on any Crypto p2p network and it's clearly not very effective against Dash considering only 2.8% of service was disrupted vs the cost of flooding 4,100 nodes with that much bandwidth for 6 hrs.
 
  • Like
Reactions: stan.distortion

Walter

Active Member
Masternode Owner/Operator
Jul 17, 2014
231
201
103
The fact that only 2.8% of nodes have been affected to the point of crashing is great news for the network... We're in good shape! :)

We now know that it would take a significant DDOS attack to cause any kind of statistically meaningful attack on the functions and operation of network, and is very unlikely to succeed in disrupting the network. For sure, some MN operators that took the low bid on their VPS provisioning are probably a bit pissed but that's the true cost of cheap.. Lesson learned hopefully!

Let's see if this is just an exploratory DDOS and whether we get more targetted and higher intensity attacks going forwards..

Get yer hard hats on MN operators!

Walter
 

crowning

Well-known Member
May 29, 2014
1,415
1,997
183
Alpha Centauri Bc
@qwizzie : You can only estimate it from the CPU and network load.

If you want to monitor it, you would need to have the firewalling iptables-rules log into a file or so, something which would make the attack even worse.

In other words, I could do it for one of my nodes, but I won't :p
 

tungfa

Administrator
Dash Core Team
Moderator
Foundation Member
Masternode Owner/Operator
Apr 9, 2014
8,906
6,729
1,283
good test to remind all the 1U$ vultr nodes to migrate to more solid services i guess ;)
 
  • Like
Reactions: methusaleh

emmo

New Member
May 23, 2014
37
11
8
Any solution for those of us which use UFW(Ubuntu)-are we must switch to iptables ?Thanks :)
 

UdjinM6

Official Dash Dev
Dash Core Team
Moderator
May 20, 2014
3,638
3,538
1,183

kointrend

Member
Jan 22, 2015
45
55
58
Hi all, no techi guy here!
Is this attack one of the strongest we can be affected by?
I mean, it's a serious test stress or just a basic one?
I see that it is "easy" to defend against.
Some easy narrative (when and if you have time) could be appreciated, thanks.
 

UdjinM6

Official Dash Dev
Dash Core Team
Moderator
May 20, 2014
3,638
3,538
1,183
is there any side effect for "ufw limit 9999/tcp" ?
This should limit it to 6 connection attempts per 30 seconds from a single IP. There is no functionality in Dash Core which would require such high rate iirc, so imo should be fine BUT I haven't tried it myself (using iptables here).
 

tibolt

New Member
Dec 8, 2014
5
3
3
i tried this command, lets see whats gonna happen thx for reply.

update; after i tried this command my masternode status at dashninja turned to unknown masternode. i think ip tables good choise :)

update 2: my masternode is working about 1 day with" ufw limit 9999/tcp" command. seems good for now
 
Last edited: