Welcome to the Dash Forum!

Please sign up to discuss the most innovative cryptocurrency!

Ongoing DDoS attack on masternode network

Discussion in 'Official Announcements' started by UdjinM6, Mar 7, 2017.

  1. UdjinM6

    UdjinM6 Official Dash Dev
    Core Developer Moderator

    Joined:
    May 20, 2014
    Messages:
    3,157
    Likes Received:
    3,169
    Trophy Points:
    1,183
    Dear community,

    We have an ongoing attack on masternode network for about 3 hours now. The attack is a mix of SYN flood, UDP flood with empty payload and protocols like sFlow and GRE. So far the only result attacker was able to achieve is that he caused higher CPU and bandwidth usage for most of masternodes which in its turn cased ~100 of masternodes to go down. We believe these masternodes were hosted on extremely low end VPS, like $1/mo, so it's not that surprising to see some of them down now. We encourage all masternode owners whose nodes were affected to move to a better hardware or upgrade their hosting plan to ensure that your masternode doesn't fall out of payment queue during such events.

    Meanwhile, you can mitigate the attack by following steps in iptables setup part of this guide https://www.dash.org/forum/threads/how-to-set-up-ec2-t1-micro-ubuntu-for-masternode-part-2-3.241/

    UPDATE:
    Incident report: https://www.dash.org/2017/03/08/DDoSReport.html
     
    #1 UdjinM6, Mar 7, 2017
    Last edited: Mar 9, 2017
    • Informative Informative x 11
    • Like Like x 8
    • Friendly Friendly x 1
  2. halso

    halso Active Member

    Joined:
    Apr 27, 2016
    Messages:
    399
    Likes Received:
    204
    Trophy Points:
    113
    Interesting. I thought this might happen when the value increased. I wonder if it was an economic attack by other MNs or possibly short sellers.

    Would the tails IP blinding solution be a londer term response?
     
  3. AjM

    AjM Well-known Member
    Foundation Member Masternode Owner/Operator

    Joined:
    Jun 23, 2014
    Messages:
    1,161
    Likes Received:
    514
    Trophy Points:
    283
    I've been waiting for this to happen.
    This makes the Dash network even stronger.
     
    • Agree Agree x 8
    • Like Like x 7
  4. AjM

    AjM Well-known Member
    Foundation Member Masternode Owner/Operator

    Joined:
    Jun 23, 2014
    Messages:
    1,161
    Likes Received:
    514
    Trophy Points:
    283
    Probably Moronsons trying their last effort to tackle Dash, us.:D
     
    • Like Like x 2
    • Agree Agree x 1
  5. AjM

    AjM Well-known Member
    Foundation Member Masternode Owner/Operator

    Joined:
    Jun 23, 2014
    Messages:
    1,161
    Likes Received:
    514
    Trophy Points:
    283
    Attack is causing my mnodes cpu load going to up 70% now, no problem so far any of my nodes.

    Clipboard01.png
     
    • Like Like x 4
  6. qwizzie

    qwizzie Well-known Member

    Joined:
    Aug 6, 2014
    Messages:
    1,050
    Likes Received:
    591
    Trophy Points:
    183
    How costly is this ongoing DDoS attack for the attacker and how long can we exspect such an attack to continue ?

    Looks like this attacks also causes some incorrect feedback on port checks in dashninja.pl
    (closed ports instead of open ports). At least i hope its feedback is incorrect.
    (Dash Central does not give any errors)
     
    #6 qwizzie, Mar 7, 2017
    Last edited: Mar 7, 2017
    • Useful Useful x 2
  7. UdjinM6

    UdjinM6 Official Dash Dev
    Core Developer Moderator

    Joined:
    May 20, 2014
    Messages:
    3,157
    Likes Received:
    3,169
    Trophy Points:
    1,183
    No idea how costly it is.
    Yep, port checker is confused because all ports are used by the attacker if there is no defense in place on that masternode.
     
  8. qwizzie

    qwizzie Well-known Member

    Joined:
    Aug 6, 2014
    Messages:
    1,050
    Likes Received:
    591
    Trophy Points:
    183
    Will we be safe if we set the firewall on remote as follows ?

    Or do we still need to add those iptables ?
     
  9. UdjinM6

    UdjinM6 Official Dash Dev
    Core Developer Moderator

    Joined:
    May 20, 2014
    Messages:
    3,157
    Likes Received:
    3,169
    Trophy Points:
    1,183
    I'm not sure about ufw capability to stop this, I do know that configuring iptables the way it's described in that guide helps.
    Let's ping our network guru @chaeplin for more info :)
     
  10. qwizzie

    qwizzie Well-known Member

    Joined:
    Aug 6, 2014
    Messages:
    1,050
    Likes Received:
    591
    Trophy Points:
    183
    Good idea, and if it is indeed needed maybe someone can provide an easy to use Linux 14.04 specific up to date guide to installing these iptables on remote.
     
  11. AjM

    AjM Well-known Member
    Foundation Member Masternode Owner/Operator

    Joined:
    Jun 23, 2014
    Messages:
    1,161
    Likes Received:
    514
    Trophy Points:
    283
    From ubuntu wiki:

    The Uncomplicated Firewall (ufw) is a frontend for iptables and is particularly well-suited for host-based firewalls. ufw provides a framework for managing netfilter, as well as a command-line interface for manipulating the firewall. ufw aims to provide an easy to use interface for people unfamiliar with firewall concepts, while at the same time simplifies complicated iptables commands to help an adminstrator who knows what he or she is doing.
     
    • Informative Informative x 1
  12. qwizzie

    qwizzie Well-known Member

    Joined:
    Aug 6, 2014
    Messages:
    1,050
    Likes Received:
    591
    Trophy Points:
    183
    I'm trying to find a graph where we can observ the attack, but i cant really seem to find it. This one looks normal but then again it is only focussing on transactions :

    [​IMG]

    Are there other graphs networkwide that shows the attack more clearly ?
     
    #12 qwizzie, Mar 7, 2017
    Last edited: Mar 7, 2017
  13. AndyDark

    AndyDark Well-known Member

    Joined:
    Sep 10, 2014
    Messages:
    288
    Likes Received:
    564
    Trophy Points:
    153
    Capture.PNG

    Source: @chaeplin :)

    You can see the MNs that dropped off center of the top row, at the attacks peak (t+5 hrs) the # of nodes offline was 114 or ~2.8% of the network, probably the nodes with the least performance / provision / cost. Now in hour 6 and the nodes are recovering, 2.1% are still down.
     
    #13 AndyDark, Mar 7, 2017
    Last edited: Mar 7, 2017
    • Like Like x 6
  14. Figlmüller

    Figlmüller Member

    Joined:
    Sep 2, 2014
    Messages:
    49
    Likes Received:
    38
    Trophy Points:
    58
    I'm observing not only SYN-Floods, but valid connection attempts to exhaust the connection limit of the dashd. Except for limiting the connection rate per client, I think there is currently no way to stop a part of the attacks without some behaviour-analysis-based firewall. Luckily, the DDoS mitigation of the provider kicked in now, so everything is back to normal.
     
    • Like Like x 5
  15. AndyDark

    AndyDark Well-known Member

    Joined:
    Sep 10, 2014
    Messages:
    288
    Likes Received:
    564
    Trophy Points:
    153
    Yes DDOS protection, the IP tables/hardening that @UdjinM6 posted above and making sure your node is high enough spec / on decent infrastructure is the best protection.

    This kind of attack can be done on any Crypto p2p network and it's clearly not very effective against Dash considering only 2.8% of service was disrupted vs the cost of flooding 4,100 nodes with that much bandwidth for 6 hrs.
     
    • Like Like x 1
    • Informative Informative x 1
  16. Walter

    Walter Active Member
    Masternode Owner/Operator

    Joined:
    Jul 17, 2014
    Messages:
    202
    Likes Received:
    163
    Trophy Points:
    103
    The fact that only 2.8% of nodes have been affected to the point of crashing is great news for the network... We're in good shape! :)

    We now know that it would take a significant DDOS attack to cause any kind of statistically meaningful attack on the functions and operation of network, and is very unlikely to succeed in disrupting the network. For sure, some MN operators that took the low bid on their VPS provisioning are probably a bit pissed but that's the true cost of cheap.. Lesson learned hopefully!

    Let's see if this is just an exploratory DDOS and whether we get more targetted and higher intensity attacks going forwards..

    Get yer hard hats on MN operators!

    Walter
     
    • Like Like x 6
  17. crowning

    crowning Administrator
    Core Developer Moderator

    Joined:
    May 29, 2014
    Messages:
    1,417
    Likes Received:
    1,999
    Trophy Points:
    183
    @qwizzie : You can only estimate it from the CPU and network load.

    If you want to monitor it, you would need to have the firewalling iptables-rules log into a file or so, something which would make the attack even worse.

    In other words, I could do it for one of my nodes, but I won't :p
     
    • Like Like x 3
  18. tungfa

    tungfa Administrator
    Dash Core Group Foundation Member Moderator

    Joined:
    Apr 9, 2014
    Messages:
    7,316
    Likes Received:
    5,848
    Trophy Points:
    1,283
    good test to remind all the 1U$ vultr nodes to migrate to more solid services i guess ;)
     
    • Like Like x 1
  19. chaeplin

    chaeplin Active Member
    Dash Developer

    Joined:
    Mar 29, 2014
    Messages:
    689
    Likes Received:
    329
    Trophy Points:
    133
    Dash Address:
    XiDWe5fkVcrXBQApmCFQUxpue5iuWcbmcK
    • Like Like x 2
  20. ericsammons

    ericsammons Active Member
    Masternode Owner/Operator

    Joined:
    Jan 1, 2016
    Messages:
    134
    Likes Received:
    478
    Trophy Points:
    113
    Maybe it was vultr conducting the stress test to get everyone to upgrade! :D
     
    • Winner Winner x 4
    • Funny Funny x 1
  21. emmo

    emmo New Member

    Joined:
    May 23, 2014
    Messages:
    37
    Likes Received:
    12
    Trophy Points:
    8
    Any solution for those of us which use UFW(Ubuntu)-are we must switch to iptables ?Thanks :)
     
  22. akhavr

    akhavr Active Member
    Masternode Owner/Operator

    Joined:
    Oct 11, 2014
    Messages:
    380
    Likes Received:
    233
    Trophy Points:
    113
    ufw does use iptables internally. So it's just matter of preference.
     
  23. emmo

    emmo New Member

    Joined:
    May 23, 2014
    Messages:
    37
    Likes Received:
    12
    Trophy Points:
    8
    okay, but from UFW commands i don't see any custom rules options
     
  24. qwizzie

    qwizzie Well-known Member

    Joined:
    Aug 6, 2014
    Messages:
    1,050
    Likes Received:
    591
    Trophy Points:
    183
    • Like Like x 1
    • Agree Agree x 1
    • Informative Informative x 1
  25. tungfa

    tungfa Administrator
    Dash Core Group Foundation Member Moderator

    Joined:
    Apr 9, 2014
    Messages:
    7,316
    Likes Received:
    5,848
    Trophy Points:
    1,283
    Update:

    - 16 h by now
    - Network holding up well
    - only dropped ca. 300 weak Mn's
    (good flushing out weak Nodes experiment)

    [​IMG]
     
    • Like Like x 4
    • Informative Informative x 1
  26. UdjinM6

    UdjinM6 Official Dash Dev
    Core Developer Moderator

    Joined:
    May 20, 2014
    Messages:
    3,157
    Likes Received:
    3,169
    Trophy Points:
    1,183
    Attack is executed on mainnet p2p port - 9999, so I guess you need to add:
    Code:
    ufw limit 9999/tcp
    to protect it too.
     
  27. kointrend

    kointrend Member

    Joined:
    Jan 22, 2015
    Messages:
    48
    Likes Received:
    59
    Trophy Points:
    58
    Hi all, no techi guy here!
    Is this attack one of the strongest we can be affected by?
    I mean, it's a serious test stress or just a basic one?
    I see that it is "easy" to defend against.
    Some easy narrative (when and if you have time) could be appreciated, thanks.
     
  28. tibolt

    tibolt New Member

    Joined:
    Dec 8, 2014
    Messages:
    5
    Likes Received:
    3
    Trophy Points:
    3
    is there any side effect for "ufw limit 9999/tcp" ?
     
  29. UdjinM6

    UdjinM6 Official Dash Dev
    Core Developer Moderator

    Joined:
    May 20, 2014
    Messages:
    3,157
    Likes Received:
    3,169
    Trophy Points:
    1,183
    This should limit it to 6 connection attempts per 30 seconds from a single IP. There is no functionality in Dash Core which would require such high rate iirc, so imo should be fine BUT I haven't tried it myself (using iptables here).
     
    • Like Like x 2
  30. tibolt

    tibolt New Member

    Joined:
    Dec 8, 2014
    Messages:
    5
    Likes Received:
    3
    Trophy Points:
    3
    i tried this command, lets see whats gonna happen thx for reply.

    update; after i tried this command my masternode status at dashninja turned to unknown masternode. i think ip tables good choise :)

    update 2: my masternode is working about 1 day with" ufw limit 9999/tcp" command. seems good for now
     
    #30 tibolt, Mar 8, 2017
    Last edited: Mar 9, 2017
    • Like Like x 2
    • Informative Informative x 1

Share This Page