Bot hunters is a full time job! With our proprietary SaaS cybersecurity technology, we detect and manage non-human traffic in real-time to protect our clients’ websites and APIs.
DataDome is a growing French startup pioneering in bot detection and protection for content publishers, eCommerce websites and advertisers across the world.
Our stack:
• Stream data processing: Kafka / Flink / Elasticsearch
• Real time detection engine: Java
• Webservers modules (nginx,haproxy…): C++ / C#, nodeJS, lua
• Infrastructure: 100VM (AWS/Azure/baremetal)
• Dashboard: Angular5 / symfony3
Achievement: 500 million hits daily protected below 3ms (99p)
Basically, we are using two sort of technics : technical and behavior.
Technical : if the UserAgent claim to be a regular browser (let say Chrome 43) we will check on network level if the client implement http protocol like Chrome 43 usually do and on the JS side if the Javascript render is correct for Chrome.
In case it's a real Chrome, we will check if the Browser is controlled by automation Tool.
Behavior : we will check if the path of requests is regular according to the website usage.
This module seems to introduce nginx stats close to haproxy native monitoring webpage.
I'm wondering what others features are missing in nginx compare to haproxy?
Interesting for unprotected websites but it's easy to detect and to block: no valid js, no valid meta header, no valid cookie, suspect behavior...
Selenium is a much "elaborated" solution, but still, can be detected most of the time.
Disclosure: I'm DataDome co-founder.
If you want to detect bad bots and scrapers on your website, don't hesitate to try out for free and to share your feedback with us https://datadome.co
I realize you have reasons not to answer this question, but out of curiosity, what sorts of thing can tip off the fact that a site is getting scraped by a real browser and selenium?
https://datadome.co https://docs.datadome.co/docs
Bot hunters is a full time job! With our proprietary SaaS cybersecurity technology, we detect and manage non-human traffic in real-time to protect our clients’ websites and APIs.
DataDome is a growing French startup pioneering in bot detection and protection for content publishers, eCommerce websites and advertisers across the world.
Our stack:
• Stream data processing: Kafka / Flink / Elasticsearch
• Real time detection engine: Java
• Webservers modules (nginx,haproxy…): C++ / C#, nodeJS, lua
• Infrastructure: 100VM (AWS/Azure/baremetal)
• Dashboard: Angular5 / symfony3
Achievement: 500 million hits daily protected below 3ms (99p)
We’re expanding our team and are looking to hire:
• FullStack wEB Engineer : Angular5 / symfony3
• Senior Software Engineer : Java, Scala, C++
• Data Engineer : Scala, Java, Python
• VP Engineer
Join us: apply@datadome.co