It's not clear enough what you were doing before that led them to a C&D. Were you doing what Scrapinghub was doing? A web scraping tool vendor and service provider? It sounds like you were doing something shady enough for them to not even email you but C&D you directly.
There's a distinct line between what you do with the data you scrape vs. writing the tools and code to build a script that will get you that data.
You don't arrest the kitchen knife company's CEO because someone used it to stab someone. And the fact that the web scraping/crawler vendors have saturated the market is testament to the fact that you are overreacting.
I'd imagine the fear of facing a devastating legal battle and how it might have permanently shifted your view on web scraping but I see no valid basis for all web scraping services and vendors to shut down.
I also find it puzzling you would be acting against your interests to continue to openly talk in details about a legal situation like this because that padmapper guy pretty much just shut up as soon as the details were involved.
but feel free to provide more details that shows that you were running a web scraping service or software company.
We weren't doing anything remotely nefarious with the non-copyrightable data we gathered. Some details are intentionally unclear. You can continue to make your own inferences on these.
Big companies don't send polite emails asking you to pretty please stop. They just let their lawyers deal with the whole kit and kaboodle.
It is illegal, or close enough to illegal, to scrape from practically any company in the U.S., because "unauthorized access" is a floating definition; as soon as that company makes a decision that they don't want you doing that thing you do anymore, you're doing something illegal; their change of heart can make your previously fine action a crime. The Terms of Use for almost all companies state as much. The statute does not state any required notice period or method, so you'd have to argue to the relevant magistrate that you didn't have reasonable knowledge that your scrape was unauthorized. This is the crux on which all scraping cases have hung, and the results are usually not favorable at all to the scrapers, although 1 or 2 recent decisions are sort of hopeful. Also note that this is only the CFAA portion; these suits usually allege a bunch of other torts too, which have proven similarly difficult to beat.
Scrapinghub's existence depends essentially on luck; first, that they won't get sued, and second, if they do get sued, that they'll get a sympathetic judge who will find that no contract was entered due to insufficient notice. That is not likely due to the nature of scrapinghub's operations (see Register.com v. Verio). The fact that some people are able to scrape and get away without being sued doesn't change the legal reality or the dubiousness of investing in a company with such a large risk profile.
The knife CEO analogy fails because CFAA claims are NOT about how the data is used. They are about the method used to obtain the data. The entity exceeding authorized computer access or accessing a computer without authorization -- in this case, that is scrapinghub, kimono, et al -- is the entity that has committed the violation of the CFAA. In your knife analogy, if the knife company had illegally acquired the metals used to manufacture the knife, it would be the culpable party, not the end user that bought its knives. The data that scrapinghub goes out, obtains, crafts and packages according to customer specifications ("make this page on craigslist a CSV file that auto-updates every 5 minutes") is the metal that the knife company goes out, obtains, crafts and packages according to customer specifications ("make this metal a sharp cutting utensil").
The person using the data that results from CFAA violations may be doing other illegal things, but in almost all of these types of cases, they're not violating the CFAA if they're not the ones accessing the computer that supplies the data.
I'm really not sure what you're arguing about anymore. The CFAA isn't a real law because the person gathering the data isn't necessarily the one putting it to use? I don't understand.
There's a distinct line between what you do with the data you scrape vs. writing the tools and code to build a script that will get you that data.
You don't arrest the kitchen knife company's CEO because someone used it to stab someone. And the fact that the web scraping/crawler vendors have saturated the market is testament to the fact that you are overreacting.
I'd imagine the fear of facing a devastating legal battle and how it might have permanently shifted your view on web scraping but I see no valid basis for all web scraping services and vendors to shut down.
I also find it puzzling you would be acting against your interests to continue to openly talk in details about a legal situation like this because that padmapper guy pretty much just shut up as soon as the details were involved.
but feel free to provide more details that shows that you were running a web scraping service or software company.