Fbook does it but I don't know about other sites.
[Here's a page that explains it.](https://moz.com/learn/seo/robotstxt)
Hey thanks! I understand about half of it but get the gist. Is that how scrapers work? I have zero coding experience but wanted to try and create a scraper (I looked into trying to learn how to write a python or a java based scraper script) but it seems too complicated for a beginner. My interest is scraping high quality images of old hollywood film stars.
Hey thanks! I understand about half of it but get the gist. Is that how scrapers work? I have zero coding experience but wanted to try and create a scraper (I looked into trying to learn how to write a python or a java based scraper script) but it seems too complicated for a beginner.
My interest is scraping high quality images of old hollywood film stars.
I'm pretty sure that's what robots.txt is for, but there's no way to enforce such a thing.
You could always place Phuks on an .onion (tor) but that has its own issues.