Internet have lots of unwanted traffic, which causes high load on your dedicated or virtual private server. Traffic can be from bot networks, A Web crawler or normal web traffic different sources. This tutorial we show how you can block Majestic search engine access to your server with modsecurity.
What is MJ12bot?
Majestic is a UK based specialist search engine used by hundreds of thousands of businesses in 13 languages and over 60 countries to paint a map of the Internet independent of the consumer based search engines. Majestic also powers other legitimate technologies that help to understand the continually changing fabric of the web.
Analyse server Access log
First, we need to identify traffic from server access log, so below example GET request on the server.
127.0.0.1 - - [06/Jul/2017:00:48:57 +0300] "GET / HTTP/1.1" 200 16492 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)"
As we see User-Agent header, there is mj12bot string, which Majestic search engine adds when it’s crawling web sites on the server. With this User-Agent string we can block traffic to server with ModSecurity.
Block MJ12bot based on User-Agent string with ModSecurity
Following the steps below you can block Majestic from being able to access your server. We need generate ModSecurity rule for that. We wanna search string in User-Agent header and block all requests to the server.
SecRule REQUEST_HEADERS:User-Agent "@contains mj12bot" "id:1000000,t:none,t:lowercase,deny,nolog,msg:'BAD BOT - Detected and Blocked. '"
We using nolog directive to not fill up the server error log with these requests. Also we can do lowercase search, if there are different version on search string.
Final words
Read more about Malware Expert – ModSecurity rules and protect your web server vulnerabilities with Web Application Firewall.