Ask HN: Splunk Alternatives?

[bhattchaitanya]

At our company we love Splunk for its amazing query capabilities, dashboards, rich set of APIs, speed etc. Are there any credible competitors for this tool? We are open to both proprietary or opensource. Splunk is too expensive and is not cost-effective for our business and we are desperately looking for an alternative. any guidance would help.

[luizfelberti]

I'll give a non-orthodox suggestion: ClickHouse

You'll need to manage some stuff yourself, and assemble your own dashboards and stuff, so there will be some labor involved. That being said, I doubt it will be more painful than managing an ELK stack: there are just too many ways you can destabilize a cluster with it.

ClickHouse clusters from my experience are ridiculously scalable, fast, and stable. There are several other accounts to back that up, and a good case study is Cloudflare, which uses it to store and query all of their logs and metrics from all data centers (that's quite a few PB of data).

There are some projects on GitHub you can use to get inspired, but what you need is pretty much a ClickHouse cluster, Grafana, and a Log Shipper.

[kodyo]

Elasticsearch (well, the whole ELK stack)

https://www.elastic.co/

Depending on your volume and situation, there are hosted options or you can roll your own on-prem.

[sporkland → kodyo]

+1 on elastic search. We swapped splunk out for es + some inhouse pipelines and it's been a decent replacement. It's not 100% the same. We can't throw random garbage logs at it and create structure, we pushed structured logging onto service owners, but it's covered a lot of our common use cases e.g. what happened with this request?

We haven't moved off of splunk for all of our logs, but have reduced the volume going there significantly.

[bhattchaitanya → kodyo]

The query capabilities of elastic is subpar compared to Splunk and not feasible for us. Elasticsearxh does not support JOINS which is a huge trade off for us

[ram_rar]

At my company, we used to pay 7 figures to splunk, until we realized how much data do we actually need. We cut down a lot of unnecessary logging and sent only the things that were needed.

Most of the data that was needed by business folks was sent to segment in terms of events and for genuine errors were sent to sentry with stack traces and converted most of the logs to metrics.

This helped us cut down a lot of $$ for logging and monitoring. Elastic search is a great alternative, but it takes an entire team to maintain it. I'm not sure, if its worth the time and effort.

[pych]

It'll really depend on your use-case. If you're not doing _too much_ volume (less than 1-2TB) and you're willing to put in the work, ELK stack (as others mentioned) will be more cost-effective from a software cost perspective. You'll end up making up those costs in person-hours though, as you'll lose the benefits of managed/SaaS.

If you'd rather not spend the time managing an ELK stack, there's a lot of logging options (disclaimer: I work at one of them, https://logdna.com). It'd be helpful to unpack what you mean by "credible" competitors however. Our product, and most others in our space frankly, can't match Splunk in terms of feature-set today, so knowing what is most important to you would be helpful. For example, if you're looking for basic log storage and search, you'll have tons of options. If you need compliance, that would narrow the field a bit.

I'm happy to chat if you're interested in giving us (LogDNA) a shot or if you have other general questions, feel free to shoot me an email, I'm just peter@[ourcompanydomain]. I'm not in sales so I don't really have any incentive to push you towards us unless it makes sense, and a lot of our competitors have great products as well so happy to try and point you in the right direction.

[pych → pych]

Quick follow-up on my last comment, if you're interested in trying us out, a few highlights: - don't need stress about structuring logs beforehand, we'll parse common log types automatically, parse JSON automatically, and you can create custom parsers after the fact - it takes two kubectl command to dump all your kubernetes logs onto us, and we'll add metadata after the fact like pod name, container name, namespace, etc (we also have a few dozen other integrations/ingestion options, of varying levels of quality/support) - we're responsive with customer feedback, and love to talk to customers about how we can make our product better

Hopefully that's helpful information!

[mindcrash]

Graylog: https://www.graylog.org/

It can also integrate in your existing Splunk setup if you want: https://www.graylog.org/post/graylog-splunk-integration-is-n...

(which allows you to ship data to Graylog and/or Splunk and setup analytics in both as needed)

[mamcx]

I work in the past on a big virtual ecommerce company and their solution was to move all their nosql(primary store)/log data to postgresql for reporting and analysis.

I think that is smart.

P.D: Better to not use nosql but well marketing of nosql back in the days...

p.D2: Most logs are pure NOISE. What I wish to have in some ways to reduce them when incoming and the distilled put in a rdbms. I think this is the best for most but not see much info about this.?

[castillar76]

It's important to consider whether your requirements include long-term retention and search of data, or only recent search and dashboards. Many of the open-source solutions such as Graylog-free or ELK don't retain information long-term by default, so if long-term data retention is your goal, you should take that into account (Graylog offers it, but only for $$$).

On the other hand, if your primary use-case is near-term searches and reviews of data (e.g. "we just need to see the last 90 days of information for troubleshooting and stats"), you'll be pleasantly surprised with the capabilities in both Graylog and ELK without the additional overhead of Splunk. I'll say, too, that I found working with Filebeat for ingestion to be a lot easier and automation-friendly than working with Splunk forwarders—so much so that we're basically adopting an "only use Splunk for indexing" approach to ingestion and using everything else to get data into it.

[stuntkite]

I had to go through this last year with my previous company. Splunk's licensing is outrageous. They cap the data transfer on a PAID license arbitrarily. Even getting coordinated with their sales to just pay them was a chore. They do nothing to foster open development on their platform. IMO, the open offerings actually seem designed to make it impossible to improve/contribute/extend in a cost effective way. It is also ill equipped to deal with Kubernetes even though they did release a reasonable if Windows Guy bloated example of a k8s deployment at the beginning of this year. I was completely baffled when I found out it was a Django project. It feels like something from the depths of enterprise Java.

Elasticsearch (or ELK/EFK) is really eating their lunch but if you have any experience managing an Elasticsearch cluster at scale you might have some reservations about it. If you'd like to audit it I might suggest using a Helm chart like this one[0] to deploy a full stack to a Kubernetes cluster. Even if you aren't using k8s for production it's a fast way to get a handle on it. You can deploy a test cluster easy on GKE quick for nearly no money. If that seems too inaccessible, there are a few good docker-compose[1] implementations too that can get you going right away.

I don't think it's quite ready for production, but we were already using Prometheus and Grafana so I was auditing Loki[2] and was pleasantly surprised. Though at the time I determined it would not be a substitute at scale for Splunk or ELK but could be viable for many people and the project is moving fast to do cool stuff.

[0] https://github.com/helm/charts/tree/master/stable/elastic-st...

[1] https://github.com/deviantony/docker-elk

[2] https://grafana.com/oss/loki

[trm42 → stuntkite]

In my previous job Splunk payments were ok enough not to warrant change but I spent couple of days trying to figure how to renew the license and pay for it. Their license portal was so horroble and full of funnel loops that it was next to impossible to renew the license.

[YuriGrinshteyn]

I work at Google Cloud, and this is a common request. If you're on GCP, you can build your own using logging exports and BigQuery. Otherwise, I typically recommend Elastic.

[swengw]

Scalyr is great, but I'm not sure if it's any cheaper. https://www.scalyr.com/

[pranaygp]

Depending on what you're looking for, you might not want to spend a lot of time maintaining your own stack for dashboard.

For really good user/account dashboards, which is the most common first tool companies build, you check try windsor.io which does most things out of the box with no setup

If you're looking to build custom dashboard and are okay with spending time actually coding and maintaining it (just code snippets, not a full ELK stack) try retool.com

[bhattchaitanya → pranaygp]

Thanks I will checkout windsor and retool

[serinjune]

No one mentioned GrayLog. They're a relatively new player in the game, but they show great promise. I should mention that their dashboards are probably not as mature as Splunk and some of the other platforms, but I've met a few people who use it and they seem happy! https://www.graylog.org/

[hwj → serinjune]

I was using Graylog but I wasn't happy because it took ~8GB RAM and ~20% CPU (IIRC).

[garganshum]

I work at Datadog and we've been pretty successful at attracting customers over from Splunk depending upon the use case. Others in industry are SumoLogic, ELK etc. You can check us out here if you want https://www.datadoghq.com/

[bhattchaitanya → garganshum]

Thanks ! Do you guys sell logging separately or should we buy both metrics and logs to use your platform ?

[solatic]

I work at Coralogix https://coralogix.com

We're a relatively new, relatively small player in this space, differentiating by real-time anomaly detection and tooling to help our customers keep costs low (by filtering out logs known to be irrelevant).

[zug_zug]

I work DevOps at a 60 person company. I wanted Splunk very badly, but due to pricing had to compromise.

We got SumoLogic. I'm very happy with it relative to ELK.

It's almost as good and much cheaper. The limitations would be that dashboards aren't as customizable and that some advance searches are harder.

[bhattchaitanya → zug_zug]

Thanks for your input!! Is sumo Logic pure Saas or hosted ? How do they scale at load ?

[zug_zug → bhattchaitanya]

They host it for you. My experience has been that they scale fine.

I don't know about your use case specifically, but for our case (~100GB of searchable data at any given moment), searching takes a few seconds for basic queries.

[jonahbenton]

Elastic has been getting a lot of business from priced out Splunk users.

[Fnoord]

AlienVault, or its FOSS version, OSSIM. [1]

[1] https://www.alienvault.com/products/ossim

[twunde → Fnoord]

Alienvault (now AT&T Cybersecurity) is really focused on the SIEM space and is only ok at that. While it has a lot of features, it's not very flexible. For example, neither the vulnerability scanner nor the compliance scanner can send out alerts upon newly found vulnerabilities/compliance issues. My feeling is that if you're used to using most of Splunk's capabilities, Alienvault will be disappointing

[aspleenic]

Full disclosure: I work at Humio. Great, cost-efficient monitoring, sub-second ingest latency, on-prem or in the cloud. Feel free to check it out - humio.com

Runs smoother than the ELK stack.

[somedanishguy]

I’d have to mention humio.com They’re mainly just for logs, but I really love their query style.

[rafaelgarrido → somedanishguy]

We recently migrated from splunk to humio at work and not only their query is easier to work with but creating dashboards and reports is simpler and faster. Also, their UI is extremely responsive. Kudos for humio.

[not_a_cop75]

Elastic which used to be called Elasticsearch. There is a free community tier of this, though according to recent drama, not sure how much longer it will remain open as such.

[not_a_cop75 → not_a_cop75]

Wow....generous person. Thanks for the downvote. I have an identical answer to like 1/5 of the people here. I guess even popularity is a great reason to censor!