In the early 2000s, webmasters learned to use a simple text file, robots.txt, to tell search engines which pages they could ...