This site has been seeing some downtime recently, and I’ve been learning a little about hardening a server.

The first symptom of the problem was an intermittent “Error Establishing Database Connection” message when trying to load a page. Restarting mysql would do the trick:

sudo service mysql start

But it would keep going down — sometimes after a month, sometimes after 20 minutes. Server Fault guided me to apache, and the possibility that apache was consuming so much memory that mysql was getting killed off by the OS. I found a script to help monitor the behavior of apache:

while true; do ps auxfww | grep apache | grep -v -e cronolog -e grep | awk '{ vsum+=$5; rsum+=$6 } END { print "VSZ:", vsum, "(", vsum/NR, ") RSS:", rsum, "(", rsum/NR, ") Procs:", NR }'; sleep 5; done;

This lead me to check out the apache logs (via sudo cat /var/log/apache2/access.log):

163.172.176.64 - - [28/Aug/2016:12:33:32 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.177.210 - - [28/Aug/2016:12:33:32 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.177.210 - - [28/Aug/2016:12:33:32 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.177.210 - - [28/Aug/2016:12:33:32 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.177.210 - - [28/Aug/2016:12:33:32 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.174.28 - - [28/Aug/2016:12:33:33 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.177.210 - - [28/Aug/2016:12:33:33 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.174.28 - - [28/Aug/2016:12:33:33 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.176.64 - - [28/Aug/2016:12:33:33 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.176.64 - - [28/Aug/2016:12:33:34 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.177.210 - - [28/Aug/2016:12:33:34 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.177.210 - - [28/Aug/2016:12:33:34 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.174.28 - - [28/Aug/2016:12:33:34 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.176.64 - - [28/Aug/2016:12:33:34 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.177.210 - - [28/Aug/2016:12:33:34 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.177.210 - - [28/Aug/2016:12:33:34 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.174.28 - - [28/Aug/2016:12:33:34 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.176.64 - - [28/Aug/2016:12:33:34 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
163.172.177.210 - - [28/Aug/2016:12:33:34 -0400] "POST /xmlrpc.php HTTP/1.1" 200 596 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"

It seemed odd that the Googlebot was hitting the server so often, and trying to POST to /xmlrpc.php.¹ Some searching led me to the use of iptables to just directly block these IP addresses, since they’re probably just trying to spam my database. So I checked my current iptables rules:

sudo iptables -L

But I received an access denied message. A little more searching led me to the possibility that when I had updated my VPS’ operating system, its kernal was possibly out of step with Digital Ocean’s settings. So I reinstalled the kernal:

sudo apt-get install --reinstall linux-image-4.2.0-25-generic

At that point I was able to review the iptables rules. To get a list of IP addresses to ban, I used this command:

sudo grep xmlrpc /var/log/apache2/access.log | cut -d' ' -f1 | sort | uniq -c | sort -rn | head -n26

And I got the following list:

73257 163.172.176.64
70151 163.172.174.28
53242 163.172.177.210
49839 163.172.180.159
35499 163.172.148.152
31605 163.172.167.37
29711 163.172.132.17
27755 163.172.167.174
26929 163.172.179.115
25984 212.47.248.17
25816 163.172.155.237
25536 163.172.132.42
24879 163.172.154.114
20752 163.172.158.253
15008 163.172.179.147
13471 212.47.243.235
13133 163.172.177.30
11662 163.172.179.127
  921 163.172.179.90
  903 163.172.171.182
  815 163.172.149.145
  765 163.172.169.151
  576 37.1.213.195
  541 163.172.141.248
  252 185.106.122.248
  126 163.172.159.56

To add those IPs to the list of rules for iptables, I ran this command:

for ip in $(sudo grep xmlrpc /var/log/apache2/access.log | cut -d' ' -f1 | sort | uniq -c | sort -rn | head -n26 | awk '{print $2}'); do sudo iptables -I INPUT 1 -s $ip -j DROP; done

And confirmed those were added to the rules with:

sudo iptables -S

To persist these rules across reboots of the machine, I used iptables-persistent:

sudo apt-get update
sudo apt-get install iptables-persistent

Subsequent rule changes can be persisted by reconfiguring iptables-persistent:

sudo dpkg-reconfigure iptables-persistent

I also stumbled onto fail2ban, and the wordpress plugin WP fail2ban:

The tool Fail2ban is useful in preventing unauthorized access to both your Droplet and your WordPress site. It notes suspicious or repeated login failures and proactively bans those IPs by modifying firewall rules for your Droplet.

Digital Ocean has a great guide to setting up fail2ban to protect your wordpress install. Since fail2ban dynamically modifies the iptables rules, you’ll want to persist your rules while fail2ban is disabled:

sudo service fail2ban stop
sudo dpkg-reconfigure iptables-persistent
sudo service fail2ban start

Hopefully establishing this firewall will keep the site up more regularly.