Bing has added a new feature to the new Bing Webmaster Tools  : a robots.txt tester. This is a feature Bing had in the tool prior to 2009 but dropped supporting it for the past ten-years or so. Now it is back and better.
What does it look like? Below is a screenshot of the tool. Enter a URL, test it with BingBot and Bing’s Ad bot and also see how the robots.txt file differs from HTTP to HTTPS:
More details. The Bing help document  says that this “Robots.txt Tester helps Webmasters to analyze their robots.txt file and highlight the issues that would prevent them from getting optimally crawled by Bing and other robots.”
Fabrice Canal from the Bing team told us  , “We resurrected and enhanced our old robots.txt tester.” Canal explained that this was a feature they had many years ago that the company brought back to life and made better.
Why we care. Now you can see how BingBot sees your content. If you need to troubleshoot an indexing or ranking issue with Bing, this is one of the tools you can add to your toolkit to see how Bing can access your pages.
About The Author
Barry Schwartz a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick  , a NY based web consulting firm. He also runs Search Engine Roundtable  , a popular search blog on very advanced SEM topics. Barry’s personal blog is named Cartoon Barry  and he can be followed on Twitter here.
Powered by WPeMatico