On: Lessons learned scraping 100B product pages

[afandian]

Can I piggy-back off this submission to ask HN if you're running a scraper, have the recent wave of GDPR splash-screens caused you issues? How are you dealing with them? https://news.ycombinator.com/item?id=17471599

[moltar → afandian]

You can just remove them from DOM or hide with CSS.

[blattimwind → moltar]

Except when the backend redirects you to consent.someco.example unless you're logged in or send a consent cookie along.

[afandian → moltar]

What about e.g. http://discourseontheotter.tumblr.com/ ?

Edit: In the UK I see this: https://imgur.com/a/zlWOByh

[tomjen3 → afandian]

In that case I would just accept everything and forward the cookie.

Of course if I was actually trying to read the link I would have to give up, because there appears to be no way to navigate through and opt out.

[jklein11 → afandian]

I don't see a GDPR challenge on this page?

[afandian → jklein11]

That's probably because it's being inconsistently applied and you're not in Europe. If that's the case it's all the more insidious! Or you accepted the Tumblr terms in the past.

This is what I get in the UK: https://imgur.com/a/zlWOByh

[YouKnowBetter → afandian]

:s/Europe/EU/

I'm in Switzerland (Europe), don't get the https://imgur.com/a/zlWOByh