These files must be served at the root of each host, or alternatively you can redirect the root /robots.txt path to a destination URL and most bots will follow.
You might want to protect certain areas from your website from being crawled, and therefore indexed, such as your CMS or admin, user accounts in your e-commerce, or some API routes, to name a few. The robots.txt file is a web standard file that most good bots consume before requesting anything from a specific domain. A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site.
