Live Search introduced some improvements to their search bot, mainly concerning bandwidth usage and crawling issues. Two major changes have been introduced:

1. HTTP Compression - static files and applications responses are now being packed using HTTP compression which reduces network load
2. Conditional Get - the bot will not download a page again unless it has been modified since the last crawl session; this is regulated according to RFC 2616 specifications

The user-agent signature has been updated to show msnbot/1.1.

Share This