How to Delay Bing & MSNBot Crawling for Your Website with Robots.txt

ADVERTISEMENTS

Crawling is process of accessing your website by search bot from search engines like Google, Bing, live search, yahoo etc. Crawling rate is frequency at which the search bot visites your website. From the start of search engine era there has been no control over crawling rateby webmasters but lately many webmaster has felt the need of control crawling rate.

The main reason is with increase crawling frequency your server load may increase specially for large websites. As most of the search engines are attempting to have real time search results and improved infrastructure has given the potential of increase crawling for every website by many folds.

Google Webmaster does provides the option to control Crawling rate and now Bing has started offering delayed crawling via directives of the Robots Exclusion Protocol (REP) as listed in a site’s robots.txt file.

How to Set Crawl Delay Parameter for Bing and MSNbot Crawler

In the robots.txt file, within the generic user agent section, add the crawl-delay directive as shown in the example below:

User-agent: *

Crawl-delay: 1

The crawl-delay directive accepts only positive, whole numbers as values. Consider the value listed after the colon as a relative amount of throttling down you want to apply to MSNBot from its default crawl rate. The higher the value, the more throttled down the crawl rate will be.

Here is illustration of effect on crawling if you set following crawl-delay parameter.

Crawl-delay setting	Index refresh speed
No crawl delay set	Normal
1	Slow
5	Very slow
10	Extremely slow

Source: Bing Webmaster Blog