htaccess: Block bad bots for security purposes

Please double check any bots you wish to allow are not in the block list. This is a list of known bots, as well as default user agents.

I have had a scenario where a website was getting scraped by a number of bots, causing excessive resource usage on the server and disrupting sales and bad experience for real customers.

Here is a basic technique to stop those pesky bots, simply add the following to your .htaccess file. You may need to modify the exceptions at the top if you do require certain files to be accessed by bots. Currently it will allow access to robots.txt and acme-challenge for LetsEncrypt.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.