Hacker News new | past | comments | ask | show | jobs | submit login

Your app looks really good.

What tech did you use if you don't mind answering?




Thanks!

My Shop Data is all PHP & MySQL, with Slim, Twig & Bootstrap. The web scraping aspect is another product of mine (forgive the clunky homepage, I'm going to turn this into an API platform) - https://grabnotify.com

GrabNotify is Node.js, Mongo, PHP, Bootstrap and PhantomJS. The undocumented API allows you to create a web crawler but define a JavaScript algorithm to extract the data off the page. Some retailers have dropdowns which update stock, images, etc, so this crawler can simulate mouse events, etc. My Shop Data will supply a custom crawler algorithm for each e-commerce web site through the API.

And finally, I've written a HTML to Markdown translator to extract page descriptions but keep some formatting while being transferable to other systems that don't support HTML.

The whole legality issue of web scraping is an interesting one. I'm planning to position GrabNotify as a web crawler, page monitor and HTML -> data tool, but only if you own or have permission to scrape the original content but need a simple way to grab and monitor the HTML into data. I'm not really interested in building a business of scraping other people's content without their permission.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: