Many of you might have gone through the horrifying experience of finding your websites being removed from the search engines one fine morning. This must have left you with a question mark on your faces as to why it happened when you had done nothing to offend or displease the search engine spiders. This is exactly what happened with some webmasters whose websites were hosted by a compay that used SonicWALL Inc.'s special firewall software.
As a result of using this software, search engine spiders couldn't access their websites because the Firewall software didn't allow them to do so. The standard security settings of this software state that:
"An attacker could retrieve robots.txt from the server, then use the contents of this file to discover the path of an unprotected administration interface for the server. The attacker may gain control of the webserver using this interface.The information gathered from robots.txt could be used for system compromise and control of the web server." (reference)
In simple and un-technical terms, it means that the search engines will not spider your website if this software is used without customizing it. In case the firewall has these settings, the connection to anyone asking for the robots.txt file will not be given so that it appears as if the site is offline.
The bad news is that this is not good from SEO point of view because the robots.txt file is the one which all search engines request and ask for before they index your website. So, the bottom line is that if you too are finding your site being removed from the search engines, regardless of good incoming links and well optimized web pages , you need to talk to your web host whether their firewall is the culprit, blocking search engines to request for robots.txt file. Chances are that your web host is not aware of this problem.
As a result of using this software, search engine spiders couldn't access their websites because the Firewall software didn't allow them to do so. The standard security settings of this software state that:
"An attacker could retrieve robots.txt from the server, then use the contents of this file to discover the path of an unprotected administration interface for the server. The attacker may gain control of the webserver using this interface.The information gathered from robots.txt could be used for system compromise and control of the web server." (reference)
In simple and un-technical terms, it means that the search engines will not spider your website if this software is used without customizing it. In case the firewall has these settings, the connection to anyone asking for the robots.txt file will not be given so that it appears as if the site is offline.
The bad news is that this is not good from SEO point of view because the robots.txt file is the one which all search engines request and ask for before they index your website. So, the bottom line is that if you too are finding your site being removed from the search engines, regardless of good incoming links and well optimized web pages , you need to talk to your web host whether their firewall is the culprit, blocking search engines to request for robots.txt file. Chances are that your web host is not aware of this problem.
No comments:
Post a Comment