in order to avoid unwanted content material within the search indexes, webmasters can instruct spiders not to crawl selected information or directories throughout the standard robots.txt file in the basis Listing from https://www.seoninjas.co.za