This is highly domain (and sometimes User-Agent) dependent and in my experience ... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

missblit on Aug 12, 2021 | parent | context | favorite | on: Mastering Web Scraping in Python: Crawling from Sc...

This is highly domain (and sometimes User-Agent) dependent and in my experience JS is required more and more.

e.g. good luck trying to get much out of youtube.com (or any other video site) without executing JS.

maltz on Aug 12, 2021 [–]

YouTube has "var ytInitialData" & "var ytInitialPlayerResponse" params hardcoded in HTML. No need to run JS!

Join us for AI Startup School this June 16-17 in San Francisco!
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact