Hacker News new | past | comments | ask | show | jobs | submit login

It sounds like you care more about google traffic than you care about real users, if you want to do this without having more than a few entries in your robots.txt and your sitemap then you could simply remove the other pages until you're ready to have them spidered, alternatively have them behind a login and hand out invitations.

In a nutshell, if you put up millions of pages and tell google about it it will index you, if you don't want that you'll have to make choices about the quantity and/or switch to a different kind of host.

Also, this kind of 'bot trap' tends to attract penalties so if this is not some ploy to get traffic out of google you may want to re-consider how you've laid things out, the difference between a legitimate site with a lot of generated pages and a page-spammer is hard to determine and google tends to err on the side of caution.




Consider applying for YC's W25 batch! Applications are open till Nov 12.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: