Hacker News new | past | comments | ask | show | jobs | submit login

To nitpick: FYI robots.txt is not enough. In fact, to make sure stuff doesn't show up at all in Google, you must not use robots.txt blocking, but allow everything, and use noindex meta tag.

Otherwise you might have uncrawled, but indexed entries in search results, if a URL is popular for some reason (linked somewhere else etc.)

See https://youtu.be/KBdEwpRQRD0




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: