Hi all,
We remarked in last months that many crawling bots from Microsoft, Google or other similar, were pretty agressive when crawling websites, even causing high resource usage of accounts, whereas it should not be the case.
The tip, is to insert a delay in your robots.txt file, add "Crawl-delay: 5", just below existing line : "User-agent: * ", this will tell robots to crawl your site with a 5 seconds delay between each page :
It should looks as follow :
User-agent: *
Crawl-delay: 5
Best regards,
YOORshop Team
