mersenneforum.org - Blocking malicious 'bots...

So, like most admins of web servers, I've grown tiered of my logs being filled will entries of bots "knocking on the door", sometimes making hundreds of requests per second...

I have fail2ban installed, but I wanted something which worked a bit faster, so I came up with a solution which has worked quite well. I'm posting this in the hopes it might be useful to others.

The environments are CentOS 7.3 servers of which I have root access, and mod_rewrite used under an Apache web server.

In .htaccess:[CODE]
# Let's just block stupid spiders...

RewriteCond %{HTTP_USER_AGENT} Jorgee
RewriteRule .* blockip.pl [L,QSA]

RewriteCond %{HTTP_USER_AGENT} Sogou
RewriteRule .* blockip.pl [L,QSA]

RewriteCond %{HTTP_USER_AGENT} nbertaupete95
RewriteRule .* blockip.pl [L,QSA]

RewriteCond %{HTTP_USER_AGENT} Companybook-Crawler
RewriteRule .* blockip.pl [L,QSA]

# And block exploit searchers...

RewriteRule cgi\-bin blockip.pl [L,QSA]
RewriteRule \.php blockip.pl [L,QSA]
RewriteRule \.cgi blockip.pl [L,QSA]
RewriteRule truts2 blockip.pl [L,QSA]
RewriteRule \.action blockip.pl [L,QSA]
RewriteRule wp-admin blockip.pl [L,QSA]
RewriteRule mysql blockip.pl [L,QSA]
RewriteRule ^status$ blockip.pl [L,QSA]
RewriteRule id_rsa blockip.pl [L,QSA]
RewriteRule id_dsa blockip.pl [L,QSA]
RewriteRule blog blockip.pl [L,QSA]
RewriteRule shop blockip.pl [L,QSA]
RewriteRule w00tw00t blockip.pl [L,QSA]
RewriteRule joomla blockip.pl [L,QSA]
RewriteRule manager blockip.pl [L,QSA]
RewriteRule editor blockip.pl [L,QSA]
RewriteRule uploadify blockip.pl [L,QSA]
RewriteRule phpmy blockip.pl [L,QSA]
RewriteRule wp-includes blockip.pl [L,QSA]
RewriteRule xmlrpc blockip.pl [L,QSA][/CODE]

In the web-root directory, a perl script named blockip.pl:[CODE]
#!/usr/bin/perl -w

my $IP = $ENV{REMOTE_ADDR};

my $URI = $ENV{REQUEST_URI};
$URI =~ s/"/[QUOTE]/g;

my $UA = $ENV{HTTP_USER_AGENT};
$UA =~ s/"/[QUOTE]/g;

print "Content-type: text/html\n\n";
print "Thanks ${IP}....\n";

$rslt = `sudo /usr/local/bin/blockip ${IP} "${URI}" "${UA}"`;[/CODE]

In /usr/local/bin/blockip:[CODE]
#!/usr/bin/perl -w

my $IP;
my $URI = "";
my $UA = "";
my $DS = RenderDateStamp();
my $cmd;

if (defined ($ARGV[0])) {

$IP = $ARGV[0];

if (defined ($ARGV[1])) { $URI = $ARGV[1]; }
if (defined ($ARGV[2])) { $UA = $ARGV[2]; }

open (LOG, ">>/var/log/blockip/blocked.log");
print LOG "${DS} : ${IP} -- ${URI} -- ${UA}\n";
close LOG;

`/sbin/iptables -I INPUT -s ${IP} -j DROP`;
}

sub RenderDateStamp {
my ($sec,$min,$hour,$mday,$mon,$year) = gmtime(time);

return sprintf("%04d.%02d.%02d %02d:%02d:%02d", $year+1900, $mon+1, $mday, $hour, $min, $sec);
}[/CODE]

And lastly, in /etc/sudoers:[CODE]
apache ALL = (root) NOPASSWD :/usr/local/bin/blockip[/CODE]

This results in these kind of results in /var/log/blockip/blocked.log:[CODE]
2017.10.26 09:19:20 : 177.142.82.124 -- http://74.208.74.21:80/mysql/admin/ -- Mozilla/5.0 Jorgee
2017.10.26 11:01:14 : 116.213.144.159 -- /manager/html -- Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)
2017.10.26 14:05:47 : 106.38.241.157 -- / -- Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
2017.10.26 15:53:26 : 120.148.205.179 -- /wp-login.php -- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
2017.10.26 16:55:53 : 119.152.135.48 -- /wp-login.php -- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
2017.10.26 19:05:49 : 196.217.241.68 -- /wp-login.php -- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
2017.10.26 20:51:36 : 36.81.13.218 -- /wp-login.php -- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
2017.10.26 20:53:17 : 185.172.110.214 -- /w00tw00t.at.blackhats.romanian.anti-sec:) -- ZmEu
2017.10.26 22:05:53 : 12.27.214.2 -- /wp-login.php -- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
2017.10.26 23:16:03 : 41.104.255.33 -- /wp-login.php -- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
2017.10.26 23:51:05 : 65.94.25.223 -- /wp-login.php -- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
2017.10.27 02:04:39 : 173.208.244.34 -- /wp-login.php -- Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; 125LA; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022)
2017.10.27 05:00:51 : 91.200.12.65 -- /wp-login.php -- Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
2017.10.27 06:27:10 : 171.25.193.77 -- /wp-login.php -- Mozilla/5.0 (compatible; MSIE 6.01; Windows NT 5.1)
2017.10.27 06:44:35 : 71.213.246.116 -- /wp-login.php -- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
2017.10.27 07:02:14 : 84.245.121.55 -- /wp-login.php -- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
2017.10.27 07:11:51 : 178.114.186.58 -- /wp-login.php -- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
2017.10.27 08:11:09 : 185.172.110.216 -- /w00tw00t.at.blackhats.romanian.anti-sec:) -- ZmEu
2017.10.27 09:03:21 : 175.141.20.214 -- /wp-login.php -- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
2017.10.27 09:57:57 : 178.159.37.114 -- /tmp/sfx.php --
2017.10.27 10:22:30 : 79.119.87.176 -- /wp-login.php -- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
2017.10.27 10:56:18 : 80.25.75.140 -- /cgi-bin/test-cgi -- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
2017.10.27 11:10:48 : 187.58.222.213 -- /wp-login.php -- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
2017.10.27 13:18:31 : 78.156.111.171 -- /wp-login.php -- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1[/CODE]

And these kinds of results when running "iptables -L -nv | head -50":[CODE]
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
21 2370 DROP all -- * * 78.156.111.171 0.0.0.0/0
16 1834 DROP all -- * * 187.58.222.213 0.0.0.0/0
8 999 DROP all -- * * 80.25.75.140 0.0.0.0/0
19 2128 DROP all -- * * 79.119.87.176 0.0.0.0/0
20 860 DROP all -- * * 178.159.37.114 0.0.0.0/0
18 2088 DROP all -- * * 175.141.20.214 0.0.0.0/0
53 3008 DROP all -- * * 185.172.110.216 0.0.0.0/0
16 2008 DROP all -- * * 178.114.186.58 0.0.0.0/0
14 1754 DROP all -- * * 84.245.121.55 0.0.0.0/0
15 1446 DROP all -- * * 71.213.246.116 0.0.0.0/0
23 1196 DROP all -- * * 171.25.193.77 0.0.0.0/0
20 2464 DROP all -- * * 91.200.12.65 0.0.0.0/0
24 1108 DROP all -- * * 173.208.244.34 0.0.0.0/0
21 2382 DROP all -- * * 65.94.25.223 0.0.0.0/0
18 2088 DROP all -- * * 41.104.255.33 0.0.0.0/0
21 2598 DROP all -- * * 12.27.214.2 0.0.0.0/0
53 3008 DROP all -- * * 185.172.110.214 0.0.0.0/0
13 532 DROP all -- * * 36.81.13.218 0.0.0.0/0
14 1754 DROP all -- * * 196.217.241.68 0.0.0.0/0
14 1754 DROP all -- * * 119.152.135.48 0.0.0.0/0
17 1886 DROP all -- * * 120.148.205.179 0.0.0.0/0
29 1160 DROP all -- * * 106.38.241.157 0.0.0.0/0
9 360 DROP all -- * * 116.213.144.159 0.0.0.0/0
115 7836 DROP all -- * * 177.142.82.124 0.0.0.0/0
20 2180 DROP all -- * * 73.93.143.154 0.0.0.0/0
20 2168 DROP all -- * * 41.97.193.5 0.0.0.0/0
23 1324 DROP all -- * * 187.105.38.66 0.0.0.0/0
19 856 DROP all -- * * 192.36.27.7 0.0.0.0/0
18 804 DROP all -- * * 199.87.154.255 0.0.0.0/0
23 4632 DROP all -- * * 85.248.227.164 0.0.0.0/0
19 4342 DROP all -- * * 104.236.141.156 0.0.0.0/0
18 4228 DROP all -- * * 96.64.149.101 0.0.0.0/0
16 3868 DROP all -- * * 145.239.82.79 0.0.0.0/0
21 4139 DROP all -- * * 204.8.156.142 0.0.0.0/0
16 880 DROP all -- * * 163.172.223.200 0.0.0.0/0
15 3738 DROP all -- * * 62.210.37.82 0.0.0.0/0
29 5713 DROP all -- * * 197.231.221.211 0.0.0.0/0
16 3871 DROP all -- * * 91.221.57.129 0.0.0.0/0
22 4508 DROP all -- * * 51.15.8.100 0.0.0.0/0
16 2200 DROP all -- * * 122.55.20.18 0.0.0.0/0
17 2222 DROP all -- * * 189.232.99.32 0.0.0.0/0
17 2048 DROP all -- * * 186.62.147.225 0.0.0.0/0
19 2140 DROP all -- * * 156.197.109.231 0.0.0.0/0
18 1801 DROP all -- * * 124.122.190.106 0.0.0.0/0
12 1500 DROP all -- * * 78.97.129.112 0.0.0.0/0
14 800 DROP all -- * * 179.155.95.22 0.0.0.0/0
21 984 DROP all -- * * 60.12.233.164 0.0.0.0/0
28 1312 DROP all -- * * 119.15.136.243 0.0.0.0/0[/CODE]

As new vector attempts are made I add them to the .htaccess file. But overall things have quieted down significantly.