OP here with some background info. Headless Recorder started life as “Puppeteer Recorder” which was posted here earlier. However, we just added Playwright support and renamed the project!
Looks amazing, it is generally very difficult to find well-maintained, open source web scraping/robot process automation tools. Thank you for building this and congratulations on the successful execution!
Do you ever run into issues with puppeteer scripts where you do waitUntil: networkIdle2 and every 10-15 page loads it just sits there forever? There's a timeout parameter that you can pass too, but it seems to do nothing.
I seem to have this problem often and I'm wondering how common it is.
I remember this problem. Different framework though. It’s probably something buggy deep in whatever driver library we are still using. Our solution was to not use that function.
So it looks like this keys off of class selectors. Anyone know of a solution for cases where you have dynamic selector names? Frameworks like next.js (using styled-jsx) add random strings to class names at build to create classname uniqueness [0].
I imagine in the future we'll have ML-backed tools to give instructions like "Find the submit button", and that they'll be able to account for A/B testing and other things that get in the way of automation.
We use https://taiko.dev/ to do E2E testing, and it's basically that. It's very far from perfect but dramatically cuts down needing to use selectors or IDs
Yes it lets you script chrome, usually headless, but not necessarily. I don't know whether it can use existing cookies in an open browser, but I'd assume so. I use it with headless chrome, and set cookies manually.
But Puppeteer works on headless chrome. So I'm not sure the benefit of recording in Firefox. Unless this other platform they now support works with headless Firefox?
