On: ScrapeGraphAI: Web scraping using LLM and direct g...

[KhoomeiK]

This is essentially what we're building at https://reworkd.ai (YC S23). We had thousands of users try using AgentGPT (our previous product) for scraping and we learned that using LLMs for web data extraction fundamentally does not work unless you generate code.

[nodoodles → KhoomeiK]

Awesome to hear! Looking forward to a launch -- the Waitlist form was too long to complete, need to take another LLM to fill that :)

[KhoomeiK → nodoodles]

1 month away ;)

[spxneo → KhoomeiK]

all around automation sucks with LLM thrown on top of it

the statistics are not in its favour

[visarga → spxneo]

Code is also hard. You got to generate code that accounts for all possible exceptions or errors. If you want to automate an UI for example, pushing a button can cause all sorts of feedback, errors, consequences that need to be known to write the code.

[KhoomeiK → spxneo]

Yep, until you generate code—it's harder from a technical POV but you can get way higher performance & reliability.