Dev4Press server started experiencing occasional problems with overload in back in July. The problems started getting worse in August and September. And, the investigation lead to some interesting findings.
Since July, Dev4Press server had occasional downtime every few days. It was not long, few minutes at the time, in few cases up to an hour. Nothing too serious, but it was an annoying problem. With the help of Siteground support staff, analyzing the access logs, I had the complete picture of what was going on.
Downtime periods were caused by 3 contributing factors:
- Rogue search engine (geolocated in Russia). This search engine visited the website from time to time, but, unlike normal search engines, it doesn’t have throttle control to limit the number of requests sent over a period of time. So, it was sending thousands of requests in a period of few minutes, overloading the server, and the server was running out of memory.
- Spam bots. Many spam bots are attempting to send spam comments all the time. But, in this case, few spam bots (same IP’s or IP range) were doing so without limiting the number of requests, and they were attempting to send a large number of comments at the same time. This was happening every few weeks, resulting is overload similar to the previously found search engine.
- Use of outdated Dev4Press Updater by third party website. This was the very unusual problem, but a website was using very old, and very outdated version of our own Dev4Press Updater plugin. That particular version was over 3 years old, and it was not tested with WordPress since version 3.8. And, something is obviously wrong with that website, since it was sending requests for update information once every 2 to 3 seconds! This started 2 months ago, so it is most likely related to serious issues with the website using the plugin.
Identifying problems were only the first step, now, I had to solve these permanently and stop the downtimes. Well, the solution for all these was to finally start using the full power of our own GD Security Toolbox Pro plugin.
GD Security Toolbox Pro was released back in March, but I was only using some of its features since then. Now, the time has come to use this plugin the way it was intended to be used. Also, in the past 3 weeks I have expanded the scope of plugin features to include few more tools that are now also in use.
Here is the overview of the tools used:
- Antispam addon. All antispam tools are in use, including checks against 2 DNSBL databases, hidden field, and various antispam filters. Each IP that breaks antispam rules 3 times was put on the blacklist, and was not able to access the website for a period of 2 days.
- Content Security Policy addon. I have created full CSP rule and plugin added it to the .htaccess file. Any time CSP was broken, it was logged into the events log.
- DNSBL Blacklisting. Any URL attempting to access the website with high threat level was automatically blocked for the period of 14 days.
- Username Trap addon. Any time user attempts to log in with the username from the banned username list, the attempt is blocked and IP banned for a period of 2 days. These are usernames that don’t exist but they are commonly used by attackers – admin, administrator, webmaster.
- Login Honeypot addon. A simple security measure, any time it is triggered, it will ban IP for a period of 2 days.
- Login Limit addon. Too many failed logins get the IP banned, first for a few hours, and after that permanently.
- Registration control. All filters available for registration control are in use. IP caught in any of the filters is banned for a period of 7 days.
As soon as these measures were enacted, things started to change. In the first few hours, more than 20 IP’s were banned. And, it went on. For the past 2 weeks, Dev4Press had only 3 downtimes, 6 minutes total. But, the most important thing that happened is the improved performance of the server.
Security measures contributed to banning a great many attackers and spam bots, rogue search engines or spiders. As it turns, they make more than 50% of all visits to any website, and they take significant server resources to process and serve. By improving security, I managed to bring down CPU and memory usage and to make the server stable for legitimate visitors and users. All the measures also managed to bring down spam from 800 spam comments a day to 20.