Solution: Bots with no user agent eating up bandwidth

Last night, i faced a server issue, that a robot with empty user agent was eating my CPU (i guess) and bandwidth.

Similar issue was discussed on this thread

So, i found a solution to block such request on apache level.


Blocking empty user agents is simple in lighttpd. Edit your lighttpd.conf as follows:

Ensure that mod_access is loaded:

server.modules = (    
 … other modules …

Add the following line:

$HTTP["useragent"] == "" 
{ url.access-deny = ( "" )}

Reload the lighttpd configuration and you’re done.


Enable mod_rewrite and add the following to your configuration:

RewriteEngine on 
RewriteCond %{HTTP_USER_AGENT} ^$ 
RewriteRule ^.* - [F]

more details here …