What is the difference between robots.txt and noindex?

robots.txt provides crawl rules for cooperating crawlers. noindex is an instruction in a page or response header asking supporting search engines not to index that resource. Neither is access control.

A crawler must be able to retrieve the resource to see its noindex instruction. Blocking the fetch in robots.txt can prevent that. Do not rely on sitemap omission, crawl rules or indexing instructions to protect private content; use authentication.