ES is not "bloated"; it does require decent hardware, but so does anything else that's taking in gigabytes of data an hour and making it all searchable and indexed. Why bite off working on some weird, less-supported mechanism because of your lack of confidence in Elasticsearch?
> I've been tempted to just ship straight to a dB and skip all these crazy shippers and parsers and all the other middle men in the equation.
You need the parsers. You want to find a needle in a haystack? Good luck without having things broken down into proper fields with metadata.
You need the shippers. Elastic Beats has full backpressure support so that when your cluster is busy, it can intelligently back off. Otherwise, you'll drop logs, or overwhelm the system to the point of uselessness....
> why has no product unified monitoring and logging?
Metricbeat from Elasticsearch + Grafana aimed against Elasticsearch to get you better dashboards and alerting.
Please don't reinvent the wheel on this one. Deploying ELK + Beats + Grafana is not that hard, there's tons of documentation, and it is a very stable product.
> I've been tempted to just ship straight to a dB and skip all these crazy shippers and parsers and all the other middle men in the equation.
You need the parsers. You want to find a needle in a haystack? Good luck without having things broken down into proper fields with metadata.
You need the shippers. Elastic Beats has full backpressure support so that when your cluster is busy, it can intelligently back off. Otherwise, you'll drop logs, or overwhelm the system to the point of uselessness....
> why has no product unified monitoring and logging?
Metricbeat from Elasticsearch + Grafana aimed against Elasticsearch to get you better dashboards and alerting.
Please don't reinvent the wheel on this one. Deploying ELK + Beats + Grafana is not that hard, there's tons of documentation, and it is a very stable product.