Earlier this week the server that hosts this blog and other sites that I run became unreachable. I know this because it is being monitored by New Relic and I got notifications emails. I couldn’t access the server either via HTTP or SSH so all I could do was reboot it and hope I could hop on. The issue had the feel of a DoS attack and so once I was back on the server I stopped Apache and inspected the logs. A while later, I restarted the service and all seemed ok.
Then I received this worrisome email from the company that hosts my server:
We received an abuse complaint from your server (below). Please respond within the next 48 hours with a resolution. Please let me know if you have any questions. A public NTP server on your network, running on IP address 17x.254.25x.9x and UDP port 123, participated in a very large-scale attack against a customer of ours, generating UDP responses to spoofed "monlist" requests that claimed to be from the attack target.
Clearly this is not what you want to receive but at least it did explain the issues. With it being so easy to host your own servers these days it is possible to get into these problems without realising. For me I turned off NTP Server as advised below. This seems to be a bit of standard text sent in the event of an issue such as this. I will now be applying to all my servers.
- If you run ntpd, upgrading to the latest version, which removes the “monlist” command that is used for these attacks; alternately, disabling the monitoring function by adding “disable monitor” to /etc/ntp.conf file.
- Setting the NTP installation to act as a client only. With ntpd, that can be done with “restrict default ignore” in /etc/ntp.conf; other daemons should have a similar configuration option. More information on configuring different devices can be found here: https://www.team-cymru.org/ReadingRoom/Templates/secure-ntp-template.html.
- Adjusting your firewall or NTP server configuration so that it only serves your users and does not respond to outside IP addresses.
If you don’t mean to run a public NTP server, we recommend #1 and #2. If you do mean to run a public NTP server, we recommend #1, and also that you rate-limit responses to individual source IP addresses — silently discarding those that exceed a low number, such as one request per IP address per second. Rate-limit functionality is built into many recently-released NTP daemons, including ntpd, but needs to be enabled; it would help with different types of attacks than this one.
Fixing open NTP servers is important; with the 1000x+ amplification factor of NTP DRDoS attacks — one 40-byte-long request can generate up to 46800 bytes worth of response traffic — it only takes one machine on an unfiltered 100 Mbps link to create a 100+ Gbps attack!
If you are an ISP, please also look at your network configuration and make sure that you do not allow spoofed traffic (that pretends to be from external IP addresses) to leave the network. Hosts that allow spoofed traffic make possible this type of attack.