The Role of robots.txt
The robots.txt file sits at the root of your domain and tells crawlers which parts of your site they're allowed to access. It's not a security measure because it's publicly readable and crawlers can choose to ignore it. But well-behaved crawlers, including Google's, respect it. A misconfigured robots.txt can accidentally block your entire site from being indexed. It can also specify your sitemap location and set crawl delays to prevent your server from being overwhelmed.