Add a robots.txt File to Your Root Directory
By default, search engines crawl and index everything in a website. You can block some files or folders that aren’t relevant or don’t want to get indexed by placing a robots.txt in your root folder. An example of a robots.txt could be:
User-agent: *
Disallow: /images/
Disallow: /js/
Advertisement
