The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt. it is a type of “bot” that visits websites. Such as search engine crawlers. These types of bots “crawl” are helping search engines like Google index and rank the billions of pages.
Robots.txt is allowing website owners to control how the bots interact with their site. They can block bots entirely, restrict their access to certain areas of their site, and more.
If the website owners want to stop some pages from being included in search engine results, then they have to use a meta noindex tag or password protection.