On: Upton: A Web Scraping Framework

[Shorel]

What I need is something that can scrape .NET sites with lots of weird and signed AJAX stuff just to populate a select control.

I'm using node.js with several libraries and nothing has worked so far.

[gee_totes → Shorel]

You might want to look at one of the headless web browsers, like phantom or casper,

http://phantomjs.org/ http://casperjs.org/

[Shorel → gee_totes]

Already tried that, and already failed.

Some JS in the page makes webkit die.

I think a headless Firefox is my only hope.