This is one of the best businesses I have heard of. Even 6-7 years back, Sidekiq was grossing $80k/month [1]. I imagine it must be much more now. No servers to maintain, no employees, minimal support work and Mike has the complete freedom to work on updates whenever he prefers. Almost entire revenue minus payment processing costs must be profit. Imagine making $1M / yr working only 10-20hrs per week (Don't wish to presume here, but that's my estimate of the man-hours it costs to run).
> I still have 0 employees and don't plan to hire. I tune my business processes to run as lean as possible: Most of my customers are on credit card so their payments are automatic. The gem servers take about one day of maintenance per year. I can't really outsource much of my support work because it is so technical and specialized.
> My gem server is a $6 droplet on DigitalOcean. Because gems are just static files, that little droplet can handle millions of requests per day with just Apache. Oh, and I run three servers in parallel for failover purposes for a grand total of $18/mo.
I wonder what makes Sidekiq special? I don't imagine any equivalent product (background job processing) in the other languages is raking in that much, if making any revenue in the first place.
Hangfire and Oban are two other background job systems which have a commercial aspect. AFAIK Hangfire has been around almost as long as Sidekiq and is still actively supported.
I'm closer to $10m than $1m in annual revenue now.
My take on Sidekiq's secret sauce: a job system is a distributed system. Most of Sidekiq's commercial features are available as OSS gems but the complexity sneaks up on you as you integrate 3-6 of those features together. Building your own almost always leads to a worse system than the mature, well-debugged system which I have curated.
That is absolutely absurd that you have built something many have tried and failed to do with millions of dollars of venture capital behind them, all on your own.
Genuine kudos to you, you should be an inspiration to any indie hacker.
Sidekiq is special because it's crazy performant, and it _works_.
Mike has built a system that's rock solid, and he doesn't twiddle around with it willy nilly. He keeps it super stable.
The people paying for it (I'm one of those people) are paying for the reliability, in addition to the features.
Mike's also managed to find a real sweet-spot with the features he holds out for Pro/Enterprise. The stratification is just right to nudge you up, but it always feels like you're getting a good deal when you pay.
Oban is a roughly equivalent elixir library. They claim to have hundreds of customers and their cheapest plan is $500/yr, so maybe they make an order of magnitude less? Considering the relative language popularity that seems pretty good.
Time, know-how, and long term dedication are the key ingredients from my perspective. Maybe he's working 10-20 hours now, but there's support and development every day for the past 11 years.
> I don't imagine any equivalent product (background job processing) in the other languages is raking in that much, if making any revenue in the first place.
I guarantee, there are background job processors in other languages making revenue.
> I guarantee, there are background job processors in other languages making revenue.
I had no idea until someone just hinted me that this was self-referential... LOL, apologies for the now-embarrassing other comment I made LOL (I only know you as "sorentwo", whoops!)
Just like the underlying platform it was built for (Rails) - it is so stupidly simple to deploy Sidekiq. Performance is great and a lot of complexity is well hidden. This makes it a great product overall.
Mike Perham's Sidekiq was what I modeled my own business, EmailEngine, after. I took my decade's worth of expertise in email protocol implementations (I also run the open-source Nodemailer library) and built an email gateway app my customers can download and run on their machines or servers. Just like Sidekiq Pro, the customers need an active subscription to run it, but the app runs on their premises, and they maintain it. I only run the key generation and validation server, which costs me around $50 per month, including all the backups and so on.
I quit my job to go full in with EmailEngine exactly one year ago, when MRR for EmailEngine was $500, now it is $3,900 and steadily growing. I do work full time on it, but the result is async - the work I put into EmailEngine today brings me income sometime later, and the recurring income I receive today is unrelated to any effort I put into the company right now.
Oh, yeah, I forgot my pitch. The link is https://emailengine.app - EmailEngine acts as a mail client, basically the same way Thunderbird runs on desktop, or the iPhone Mail on the phone, but instead of a GUI it has REST API and instead of desktop notifications it sends JSON webhooks. And instead of a single email account, it can manage thousands of accounts.
How does it compare to eg Nylas? I was looking into that for an email app I wanted to make, but they were enterprise only it seems. I also found your business but I didn't want to self host just to try it out.
TL;DR Nylas is hosted service that does way more than just emails. Also, it syncs all emails on monitored accounts to their DB, so when you make an API request, it hits their DB. EmailEngine is self-hosted and runs directly against the mail servers, so when you run an API request, the request ends up in the mail server. Both approaches have their pros and cons. Other than that, there aren't any other good solutions than building everything yourself by using any IMAP library.
I appreciate the answer. I believe Nylas also has a self hostable option, just to let you, as when I was doing my research they mentioned that many secure industries like finance or healthcare wouldn't allow storage of emails on Nylas' database, so they allow self hosting. Of course, that is probably much more expensive than yours. Thanks for your last sentence as well, it clears up a bit more about the business, I asked in another thread but was just curious what types of customers need to integrate with email providers, as I'm making an email client so it's core functionality, but I assume most other industries simply use APIs from Gmail or whatever, rather than raw IMAP these days.
Edit, looks like you've answered in the other thread, thanks.
You have a bullet point about compliance / hosting concerns. I'm curious, do you have any idea how many of your customers care about that?
Do you have any concern that a customer is going to look under the covers and re-implement your product? I would guess the overwhelming majority of customers are going to be happy to just consume the API and go about their own business, but some products attract copycats, and handing out code / build artifacts makes that easier.
The tradeoff you've made makes a ton of sense if you're not concerned about copycats.
Compliance issues are a real concern. For example, I have some customers who switched from Nylas to EmailEngine because Nylas closed their Canada DC, even though EmailEngine is an inferior product. I'm not really worried about copycats, as it's a niche and complex project. There are literally hundreds of email RFCs, some as long as a book, and you would need to have at least an overview of these. You can already use IMAP client libraries in any language, but what EmailEngine does is cover all the weird edge cases, like handling sequence numbers as identifiers. Those who have the skill likely have better things to do, and those who don't probably find easier things to copy.
Who are your main customer segments? I ask because I'm implementing Microsoft, Google and Apple email support for my app and Microsoft and Google have an API now, while Apple is still a pain. Beyond that, for custom email providers, I'm not sure who else is major enough to implement for. I guess I'm asking, couldn't your customer do something like I'm doing, to cover 80% of the use cases by using the Microsoft and Google APIs and spending some time using an IMAP client for Apple iCloud, and discarding the other smaller percentage of other email providers?
Granted, it still does kind of suck to implement even these three, but I'm sure it would likely suck less if I were part of a bigger enterprise who could afford Nylas for example, which I looked into but seemed too enterprisey for my needs.
Most EmailEngine customers are smaller SaaS services like niche CRM providers etc. EmailEngine can also work with Gmail and MS365, as they support IMAP in addition to their proprietary APIs. So you get quite a large coverage by using EmailEngine out of the box (all smaller providers + most Gmail/MS365 accounts). Additionally, in some cases going with IMAP is the only way - for Gmail email API access, you need a very expensive security assessment that you might also fail for various reasons, like when your use case is not allowed by Google's terms. In comparison, there are no conditions for using app-password-based IMAP for Gmail. EmailEngine does not work with email accounts where their organization has disabled IMAP/SMTP access entirely (common with on-prem Exchange servers, a lot of MS365 accounts, etc.)
Makes sense, I saw the security assessment stuff for Gmail which I suppose I'll have to contend with as well, haha. I'll take another look into EmailEngine then. How do you charge, per connected user (and how much, generally)? Is it a monthly usage based subscription if so?
Edit, I just took a look at your FAQ and saw the price was 695 Euros a month. It is strange to not put this pricing on a prominent pricing tab I believe, rather than making me click a link in the FAQ to figure it out. Any reason why it's set up like that?
My main customer base is smaller SaaS services (less than 10 people), and this is the price that works for them or what they would expect - if they compare the costs of building in-house vs buying EmailEngine. Very small companies (single-person ones) and very large ones (one of my customers is a major bank in Europe) are exceptions that I do not target directly, but if they decide to become paying customers, then I don't object and gladly take their money.
Thanks. Yeah it would help future customers looking for pricing as that's the expected paradigm on landing pages these days, to have a dedicated pricing page or section.
There are no limits on how many EmailEngine instances you can run or how many email accounts you connect. So, I don't differentiate between customers at all. I have single-person startups and large enterprises paying the exact same amount. Maybe not be the wisest option potential-revenue-wise, but this approach has worked well for me so far.
If you're worried about copying the code, I could see having the self hosted artifact be compiled or otherwise obfuscated would probably deter most people who wanted to reverse engineer it, since it would take significantly more time and effort to open up a compiled app than something just made in JS for example.
If you're worried about copying the API design, well, the implementation is the hard part, not necessarily the API design, which the Google vs Oracle lawsuit also showed, interestingly enough.
EmailEngine has all the code public [1], even though it is not open-source but is source-available. Some core parts I even published under the MIT license, like the IMAP client library I built from scratch to serve the special requirements EmailEngine has for IMAP access [2]
My thinking has always been that those who try to hack the license validation stuff and replace the missing build pipeline were never going to be my customers in the first place, so every second I would spend on them is a wasted effort.
Yeah, it is so similar because I blatantly copied it. I had the old Changelog podcast episode with Mike Perham pretty much on repeat when I tried to figure out how exatly to set up my business.
There seems to be an ever-present high-level of defensiveness from the Ruby community over JS/React. Rails has had a profound impact on web development, promoting fast starts and shipping quickly. Companies like Shopify and Twitter benefited greatly from this. But the framework is 18 years old. The web that it was created for has changed significantly.
2. For people who had to ship a Rails API + JS SPA, their workflow felt slow, brittle, and cumbersome. It wasn’t their imagination either—testing the stack required integration tests, which are always slow. Maintaining an HTTP API to talk to the SPA is additional effort that’s not needed, which Hotwired has demonstrated clearly.
I still think Rails has a lot of fuel left in its tank, thanks to Hotwired and the big companies behind it, but I agree the “Rails is the most productive framework” gets way overplayed. It was def “most productive” 18 years ago, but most other modern frameworks took notice, caught up, and have even surpassed Rails in a lot of ways.
The Ruby runtime leaves a lot to desire when you compare it to runtimes like Elixir/BEAM, Go, etc. I also think Rails has a terrible view layer, but most folks don’t quite understand that that means yet. This is something I’m working on at the moment.
The Rails view layer used to be quite nice at its peak. Way better than many of the alternatives. That you could execute actual code inside a view was just magical.
The framework was first conceived 18 years ago, but continues to evolve. The framework has also changed significantly to deal with functionality that modern web applications need, in much the same way the Javascript community has evolved to better suit the needs of the modern web application.
Matz is nice but that culture's definitely not coming back, anymore than the kind of fun and curiosity that _why brought to us. Maybe it's inevitable as communities grow and people taste success?
Regardless, it's probably better if we leave room for little jokes with each other.
As someone who quite likes js & our wild teaming ecosystem a lot, I can also say: we definitely deserve some/mamy jokes too. No one's wrong on that. (jokes are good!)
I wish they felt like they had some punchlines though. No one ever bothers to make the JS world laugh along. We know it's wild here (Come play! So much fun freedom!). It comes off more like a beat down.
Context also matters. Conversationally Mike's words could be an amusing wink & grin quip. I can see that. On paper, & seeing it repeated with the same reckless unnuanced antipathy, it lacks the personal connection & feels indicative of a general attitude situation that is quite prevalent.
Again I think there's plenty of valid negativities in js, but looking at the distribution of where folks fall on the alignment chart when they talk about JS issues, it concerns me how lopsidedly & with what casual acceptance folks tend towards the scariest boxes of the chart. To me we are all in the challenge of making computing better together, & we can help ourselves by helping others.
See like, this isn't constructive maligning at all, it's more undirected blanket unniceness.
It costs nothing to actually put down a reason for being upset, versus having totally generic downputs. Very few things are truly rotten to the core, most bad things have some bad aspects but could be much better if ___. If something is rotten to the core the central articles of faith for why that's so should be declared.
Plenty of Ruby folks have gone on to do great friendly Rails ish JS works. Ember.js is very batteries included, tries to show that yes all the potential is there to pave a nice well integrated happy-path system. To assume misery seems unreasonable; many have done fine.
I think we have a real obligation to do better to ourselves & one another than to foster shallow prejudices. Trump in his April 11th Tucker Carlson interview was asked why he thought Dems weren't worried about nuclear weapons. "That's because they don't understand life. That's because they don't understand what it is that you have to understand." This is an irrefutable claim. There's no point to start discussing here, no possibility that the other side might change or have some nuances. Let's not do this. Let's rise to higher places where we take real appraising concerned looks at things, rather than just dismiss stuff out of hand. We don't even have to be nice, but let's at least strive to be somewhat useful.
Both agree in this case with Mike's right to fun, but also, I feel like the idea of lighthearted jabs has become a steady stream of blows. I said elsewhere:
> Context also matters. Conversationally Mike's words could be an amusing wink & grin quip. I can see that. On paper, & seeing it repeated with the same reckless unnuanced antipathy, it lacks the personal connection & feels indicative of a general attitude situation that is quite prevalent.
"Don't get so upset it's just a joke" is sometimes indeed a legit statement, sometimes we do take things too seriously & need to be reminded. But when the light jabs keep coming, when there's so rarely an attempt to ever bring both parties into the laughing, when it's always jokes at expense, the action whatever it's intent becomes part of a larger behavior that is more than a lighthearted jab.
How many folks would stand up and say, no, there is no pattern of behavior about JS receiving short shift put downs? Do folks think there is no issue? How above board Vs how trashy do we think we've gotten with how programmer culture especially those parts that don't like it treats JS? What other parts of programmer culture endure repeated joke making at their expense today?
> How many folks would stand up and say, no, there is no pattern of behavior about JS receiving short shift put downs? Do folks think there is no issue?
If you identify with JavaScript so strongly that you'll go over to message boards and rend your garments when you see people making fun of it, and you consider it a big problem that you have to do this so often... maybe consider whether there are some faults in JavaScript that lead it to attract a steady stream of mockery?
> What other parts of programmer culture endure repeated joke making at their expense today?
This is a callous take that refuses to pause for even the briefest millisecond of reflection. Personal attacks like this & shaming me may make you feel good & smart but as far as I can tell there's still a real topic here & it still deserves some discussion.
I don't go out of my way to start this discussion every time. But sometimes I do. Because it's super notable to me how asymmetric it is. If there was more evidence of programmers dropping little nasty grams about any old topic, I could see it as cultural. That's why I tried to survey the room. It's remarkable to how persistently nasty people are to the most popular language on the planet. It feels decidedly unhacker like to bully me into silence over this search for meaning & inquiry.
I'd like an example of programmers persistently being mean & denigrating to other programmer spaces or languages. That was the context of the rest of my line of questioning, and implied by this question too. The popular/mainstream perception of programmers is unimportant & distracting, disinformation that misleads the discussion; why do some programmers enjoy dropping little vacuous nasty gram jabs at JS, and is this pattern repeated broadly against any other targets?
Are we concluding that JS is the only bad programming thing, thus no one else gets mocked? Maybe JS is a magnet given how popular it is? Or do we think the casual barbs really do go every way & I'm just missing it?
> It costs nothing to actually put down a reason for being upset, versus having totally generic downputs
Actually, it does cost something- time and effort- but thanks to the latest AI, I can cut some of that out right now:
As an Elixir enthusiast, I'd like to point out some technical criticisms of JavaScript and its ecosystem, particularly in comparison to Elixir's approach to dependency management and community.
A) Dependency bloat: The JavaScript ecosystem, especially when working with Node.js, often suffers from excessive dependencies. Even when you require just a few dependencies for your project, you may end up pulling in a large number of indirect dependencies. This can lead to increased complexity, slower build times, a larger attack surface for potential security vulnerabilities, and an ever-growing workload spent applying security fixes that have nothing to directly do with the purpose of the web app.
In contrast, Elixir's dependency management system, powered by Mix and Hex, encourages a more conservative approach. Elixir libraries are often designed with minimal dependencies, and the community values self-contained, focused libraries.
B) Security concerns: The extensive use of third-party packages in JavaScript projects can expose your application to security risks. When you pull in a large number of packages, you become responsible for ensuring that all of those dependencies are up-to-date and secure.
Elixir's ecosystem tends to have fewer dependencies, reducing the potential attack surface. Additionally, Elixir's focus on immutability and the actor model can help minimize the impact of security vulnerabilities.
C) Lack of native concurrency and parallelism: JavaScript is single-threaded by design, which can limit its ability to take full advantage of multi-core systems. While Node.js introduced asynchronous I/O to help mitigate this issue, it can still be challenging to build highly concurrent applications in JavaScript.
Elixir, built on the Erlang VM (BEAM), provides excellent support for concurrency and parallelism through lightweight processes, message-passing, and the actor model. This allows Elixir applications to scale across multiple cores and handle a large number of simultaneous connections more efficiently than most JavaScript applications.
D) Callback hell and async/await complexity: JavaScript has traditionally relied on callbacks for asynchronous programming, which can lead to a phenomenon known as "callback hell" when dealing with nested callbacks. While the introduction of async/await has improved this situation, it can still be cumbersome and confusing for developers.
Elixir offers a cleaner approach to concurrency with its process-based model and message-passing mechanism, enabling easier-to-read and maintainable code.
E) Mutable data- One of the key differences between Elixir and JavaScript is the way they handle data. Elixir enforces immutability, meaning that once a data structure is created, it cannot be modified. Instead, operations on data structures in Elixir return new versions of the data, leaving the original untouched. This approach offers several advantages compared to JavaScript's mutable data structures:
1) Predictability and easier reasoning: Immutability in Elixir makes it easier to reason about and understand your code. Since data structures cannot be changed once created, you don't have to worry about unintended side effects or data being altered in unexpected ways. This results in more predictable and maintainable code.
2) Concurrency safety: Elixir's immutable data structures simplify concurrent programming by eliminating the need for locks, mutexes, or other synchronization primitives. This is because multiple processes can safely read and share the same data without the risk of data corruption due to concurrent modifications. In JavaScript, managing shared mutable data in concurrent scenarios can be error-prone and challenging.
3) Functional programming: Immutability is a core principle of functional programming, and Elixir is heavily influenced by this paradigm. Functional programming promotes the use of pure functions that do not produce side effects, which can make code more modular, reusable, and easier to test. While JavaScript supports functional programming concepts, its mutable data structures can make it more challenging to adhere to functional programming principles.
4) Reduced cognitive load: Immutable data in Elixir means that developers don't have to keep as much track of changing state as they read or write code. This can make it easier to understand the flow of data and logic in your application, leading to a more pleasant and efficient development experience and ultimately, fewer bugs generated per LOC.
F) Community values: The JavaScript community is known for its rapid pace of change and the constant introduction of new libraries, frameworks, and tools. While this can drive innovation, it can also lead to fragmentation and inconsistency across projects.
The Elixir community tends to be more focused on stability, long-term support, and collaboration. This can result in a more consistent and cohesive ecosystem, where tools and libraries are more likely to work together seamlessly.
In conclusion, while JavaScript has its strengths and has undoubtedly played a vital role in the growth of web development, its dependency management and ecosystem can be seen as less than ideal from an Elixir enthusiast's perspective. Elixir's dependency management system, focus on concurrency, and community values offer a more robust, sane and secure alternative.
(How's that for "actually putting down a reason" as to why Javascript is terrible? Hey, you asked!)
Works for me! Very much matches the MO I've been using as "JS is wild, ain't it great!," so I tend to agree with the shown perspective.
I also think a huge amount of these tend to be pretty typical conservative-developer overconcern (see Yegge's Notes From The Mystery Machine Bus https://gist.github.com/cornchz/3313150), governing yourself on fear, that is probably not really as fully warranted or deserved.
But I'm also a huge fan of wild & fun & diversity. I also love that we try & do other things, that programming languages are not a Last Man Standing situation, & that we have great technical ecosystems with caring trying folks finding new ways to move forward & progress like Elixer. I look forward to seeing what lessons there prove really valuable (and perhaps aping them)! It's a choice in JS world but I do use immutable.js, for example, and it's great & as described as an advantage above. Ditto for some fp tools, but often I will mutate & use effectual/imperative styles, I often find there are significant performance or understandabity wins.
I would say, yes, the cost of talking vs snarking may be real, but it doesn't have to be this large & well examined a list as you've madd. I don't like JS, the amount of dependencies terrifies me. I don't like JS, it's too fragmented. I'd rather use Elixer, I like having a good actor model underpinning my ecosystem. Sharing some sort of tidbit can help calibrate where a person is & turn destructive contagious negativity into healthy discourse.
I forgot to reply to this comment but I did like it.
I don't actually hate JS. It's possible to write great JS. (And honestly, things like Elm do exist, and Svelte looks pretty amazing, but these also have their cost.) The failing for me is that there's nothing stopping anyone, especially a library I might decide to depend on, from writing bad JS. This was the same problem I had with Ruby. We spent all this time aiming for thread safety in our app (a whole team spent a lot of time on it) and it turns out the 2 culprits were 2 libs we depended on, because Ruby simply makes no guarantees about thread safety (and cannot, because it's mutable, thus making forking either expensive or dangerous). That and probably a few other things burned me enough to make me say "no more" and I suddenly loved the idea of a language where "thread safety" is completely a given and impossible to violate and you don't have to mutex-lock everything.
I don't think FP will ever get popular, unfortunately, unless we NEVER teach ANYONE ANYTHING but FP, because the people that pay the cost of learning the FP way are paying it because of all the ways they got burned by the non-FP (OOP, imperative, mutable) ways...
If you ever get there yourself, you have my hat-tip. And truly, it is worth the journey.
Sidekiq is one of my favorite things about using Ruby, I include it in my short list every time a "Why use Ruby?" question gets asked. I miss it desperately when I work in other languages. I plan on giving Faktory a shot ASAP.
Mike has always struck me as a great guy, really the kind of person who you're happy to know is building one of your favorite tools. In a world of starving open source contributors and mega corps throwing around weight, his success with Sidekiq stands out. It makes me very happy to see.
> I was Director of Technical Operations at The Clymb, which was outdoor-focused e-commerce vendor (think REI without the stores). I had a team of four and we were responsible for site devops and infrastructure. This was perfect because they were my alpha customer and I was in charge of the tech stack. The Clymb switched to Sidekiq immediately, of course, and every Sidekiq Pro feature I wrote was running in production before it was released.
This makes me curious about either
(a) conflict-of-interest rules at The Clymb that may or may not have governed the DTO directing the company to use his own separately-owned (commercial?) software.
(b) negotiating private ownership of software written for The Clymb.
I started work on Sidekiq right before I joined so I made sure my employment contract had a IP carve out for anything related to Sidekiq.
I never charged The Clymb to use the software. Previously they had been using delayed_job and were suffering badly from scale issues due to the use of MySQL as their queue store.
Same. He's always been a role model for my business as well, especially when I was just starting out. My business uses Sidekiq every day. It really is a great piece of software. He's also one of the reasons I think open source software can be highly profitable if people just figure out a business model that works.
I know tons of other companies will share my view that we would never stake anything remotely important on a stolen library to save a minuscule amount of money. It would be absolutely idiotic.
Instead, we shell over the negligible monthly fees, spend no time thinking about it, and move on to solve real problems.
There are enough of us that it probably simply doesn’t matter.
It's always interesting to find out about these niche subcultures. Having worked with batch schedulers of all varieties for a couple decades, I've never run across Sidekiq but, then again, I didn't realize anyone had actually built anything in Ruby in the past decade.
I guess Matsumoto should have charged for Ruby. I'm sure Mike Perham and Derek Kraan would have been glad to pay for that and that the Ruby community would be in a strong and healthy shape.
People need to eat man, doing something for free is a luxury not many possess. We can see how messed up the financial situation is for a lot of Open Source devs and their lives would be better if they were charging money instead of subsisting off grants and donations.
I don't even necessarily disagree with your point that without being free those things wouldn't have taken off but we need to find a way to strike a balance in the developer community.
Sidekiq having a free version and an enterprise version walks an okay middle line imo.
I personally try to spend money or use ad supported anything that is open-source just to help someone else eat. It really hit home for me about five years ago when the author of a WoW addon[0] couldn't develop anymore because of his financial and life situation.
So many communities across the web rely on people putting in their spare hours for free just to enjoy things. Whether it's spreadsheets in Eve, Addons and Weak Auras in WoW, forum analysis posts, or whatever goes on in the depths of pvpoke, so much free labor underpins massive parts of the world today.
I would love something that I could donate x money to per month and then based on usage, have it dole out to all the content providers with perhaps a minimum per month. It just seems daunting to do that as a) not a crypto scheme and b.) across all the various creator landscapes.
Well Sidekiq is free to use. It's only the pro version that he charges and the free version code is open source.
I don't see the problem in having that kind of business model, it still allows the community to thrive and offers entreprises a way to have premium support.
Plus it allows him to invest more time in maintaining the free version.
I have no problem paying for the Pro version, but one if its marketing pitches is "enhanced reliability", which is a wild marketing spin on "the free version will lose jobs in fairly common scenarios".
In sidekiq without super_fetch (a paid feature), any jobs in progress when a worker crashes are lost forever. If a worker merely encounters an exception the job will be put back on the queue and retried but a crash means the job is lost.
Again, no problem paying for Pro, but I would prefer a little more transparency on how big a gap that is.
I wish this was prominently documented. Most people new to Sidekiq have no idea that the job will be lost forever if you simply hard kill the worker. I have seen a couple of instances where the team had Sidekiq Pro, but they had not enabled reliable fetch because they were unaware of this problem
The free version acts exactly like Resque, the previous market leader in Ruby background jobs. If it was good enough reliability for GitHub and Shopify to use for years, it was good enough for Sidekiq OSS too.
Here's Resque literally using `lpop` which is destructive and will lose jobs.
> If it was good enough reliability for GitHub and Shopify to use for years, it was good enough for Sidekiq OSS too.
Great point, and thanks for chiming in. I wonder if containerization has made this more painful (due to cgroups and OOMs). The comments here are basically some people saying it's never been a problem for them and some people saying they encounter it a lot (in containerized environments) and have had to add mitigations.
Either way, my observation is a lot of people not paying for Sidekiq Pro should. I hope you can agree with that.
When we used Sidekiq in production, not only did I never see crashes that lost us jobs, but there are also ways to protect yourself from that. I highly recommend writing your jobs to be idempotent.
Idempotence doesn't solve this problem. The jobs are all idempotent. The problem is that jobs will never be retried if a crash occurs.
This doesn't happen at a high rate, but it happens more than zero times per week for us. We pay for Sidekiq Pro and have superfetch enabled so we are protected. If we didn't do so we'd need to create some additional infra to detect jobs that were never properly run and re-run them.
I'm still confused about what you're saying though. You're saying that the language of "enhanced reliability" doesn't reflect losing 2 jobs over about 50*7 million (from your other comment)?
And that if you didn't pay for the service, you'd have to add some checks to make up for this?
Crashes are under your control though. They’re not caused by sidekiq. And you could always add your own crash recovery logic, as you say. To me that makes it a reasonable candidate for a pro feature.
It’s hard to get this right though. No matter where the line gets drawn, free users will complain that they don’t get everything for free.
How are crashes under your control? Again they aren't talking about uncaught exceptions, but crashes. So maybe the server gets unplugged, the network disconnects, etc.
To me 'crash' means any unexpected termination, whether it's caused by an uncaught exception, OOM, or hardware/network issues.
I guess you can say that hardware issues on your host aren't under your control, but it's under your control to find a host that doesn't have these issues. And not even a full-on ACID database is going to be 100% reliable if you yank the power cord at the wrong moment.
I hope my tone doesn't come across as rude or too argumentative, but I think your understanding is a bit inaccurate.
> it's under your control to find a host that doesn't have these issues
All hosts will have these issues, the only question is how often. If you need 100% consistency, then you can't use the free Sidekiq. Personally, I've never needed Sidekiq pro (as these kinds of crashes are extremely rare). But this will depend on your scale and use case.
> And not even a full-on ACID database is going to be 100% reliable if you yank the power cord at the wrong moment
This is only true if there's bugs in the DB, or some underlying disk corruption happens. The whole point of an ACID database is that they're atomic, durable, and consistent, even in the worst case scenario. If a power failure corrupted my SQL database I would feel very betrayed by the database.
It wouldn’t be corrupted, but in-flight transactions could fail to commit, just like queued jobs can be lost with sidekiq. The failure modes are similar.
I take your point that at a certain scale, hardware failure is inevitable, but if you’re running that many servers, you can afford sidekiq’s enterprise plan. It’s not something that will realistically happen if you’re just running like 20 instances on AWS. It’s perfectly reasonable to charge extra for something only large organizations with huge infrastructure budgets need.
I would say that queued jobs being lost is different from an in-flight transaction being auto-rolled-back, but it's not a super important distinction. Like others have said, I think Sidekiq really nailed the free vs premium features and its success is evidence of that.
Jobs may crash due to VM issues or OOM problems. The more common cause of "orphans" is when the VM restarts and jobs can't finish during the shutdown period.
We process around 50M sidekiq jobs a day across a few hundred workers on a heavily autoscaled infrastructure.
Over the past week there were 2 jobs that would have been lost if not for superfetch.
It's not a ton, but it's not zero. And when it comes to data durability the difference between zero and not zero is usually all that matters.
Edit for additional color: One of the most common crashes we'll see is OutOfMemory. We run in a containerized environment and if a rogue job uses too much memory (or a deploy drastically changes our memory footprint) the container will be killed. In that scenario, the job is not placed back into the queue. SuperFetch is able to recover them, albeit with really lose guarantees around "when".
This isn't about 2 in 350,000,000 jobs failing. It's about 2 jobs disappearing entirely.
It's not reliability we're talking about, it's about durability. For reference, S3 has eleven 9s of durability.
Every major queuing system solves this problem. RabbitMQ uses unacknowledged messages which are pinned to a tcp connection, so when that connection drops before acknowledging them they get picked up by another worker. SQS uses visibility timeouts, where if the message hasn't been successfully processed within a time frame it's made available to other workers. Sidekiq free edition chooses not to solve it. And that's a fine stance for a free product, but just one I wish was made clearer.
If you want to focus on durability then I think your complaint makes even less sense. Somehow I doubt S3 is primarily backed by Redis.
I think it's fair to assume that something backed by Redis is not durable by default because that's not what Redis is known for, whereas the other options you listed are known for their resiliency and durability. I wouldn't view Sidekiq as a similar product to RabbitMQ and SQS.
it’s not uncommon to lose jobs in sidekiq if you heavily rely on it and have a lot of jobs running. If using the free version for mission critical jobs, I usually run that task as a cron job to ensure that it will re-try if the job is lost.
I have in the past monitored how many jobs were lost and, although a small percentage, it was still recurring thing.
In containerized environments it may happen more often due to OOM kills or if you leverage autoscalers and have long running sidekiq jobs that have a runtime that exceeds your configured grace period for shutting down a container during a downscale and the process is eventually terminated without prejudice.
OOM kills are particularly pernicious as they can get into a vicious cycle of retry-killed-retry loops. The individual job causing the OOM isn't that important (we will identify it, log it and noop it), it's the blast radius effect on other sidekiq threads (we use up to 20 threads on some of our workers), so you want to be able to recover and re-run any jobs that are innocent victims of a misbehaving job.
This is a very bad take. From an OSS perspective languages can attract large communities of contributors and corporate sponsors because of their broad appeal and utility. Specialized libraries will have more trouble doing both and may need alternate models to sustain themselves. From a business perspective, Mike offers not only a free version but the paid enterprise version comes with support from Mike and his team, which is something you can't get from a language owner unless you outright hire them or they run a consultancy.
Anyone who has a free thing, sell something that produces an expendable invoice or receipt. Makes it easy to use company funds to pay for company work. “Buy me a coffee” buttons don’t really cut it.
"If I had asked people what they wanted, they would have said faster horses." or whatever the quote is.
There are tons of businesses selling products that are ideas brought to life from scratching their own itch or simply a desire to make something and put it out there.
Personally, I am one of those people who started a business based on an idea with no validation before launching it.
I built it in its entirety, then went to places and people who I expected would want it and low-and-behold, I am making a living doing it.
> There are tons of businesses selling products that are ideas brought to life from scratching their own itch or simply a desire to make something and put it out there.
I'd bet money that the number of businesses which fail because they boil down to "a solution in search of a problem" is vastly larger than the number of businesses that succeeded despite performing "no validation".
That said, "making something" and "starting a business" are two different things. I would challenge you to point out where in the post he argues against making something, especially for the reasons you mention.
> I am one of those people who started a business based on an idea with no validation before launching it. I built it in its entirety, then went to places and people who I expected would want it and low-and-behold, I am making a living doing it.
Consider the possibility that you're in the minority there, and that you succeeded despite performing no validation.
> Sorry, this is bogus.
So if it doesn't apply to you personally, or in all cases, then you dismiss it as "bogus", full stop? Are you in the habit of doing this often?
Also, for future reference, it's "lo-and-behold", not "low-and-behold". [1]
> That said, "making something" and "starting a business" are two different things. I would challenge you to point out where in the post he argues against making something, especially for the reasons you mention.
The author states up directly, "Don’t make a website or an app. Don’t build a system".
Starting a business doesn't have to follow any real "formula". It can be a step in the beginning, middle, or end of a process. Yes, you should have the ingredients for a cake before you bake it, but you don't need to find someone who will eat your cake before you bake one or offer it for sale and you sure as shit don't need to "find real people whose problem you can solve. You listen deeply to find their dream scenario."
> Consider the possibility that you're in the minority there, and that you succeeded despite performing no validation, rather than because of it.
While I never claimed it was due to not performing validation, we can clarify that yes, it is not due to this, it is despite not doing it. Validation itself can be found in successful sales or other means, it does not need to be done pre-market and there are many examples of this, aside from the horses quote I provided.
And yes, I, in the same way the author declares it, declare it as bogus because these rules do not need to be followed in the way the author claims.
The post is riddled with questionable content, IMO, made to hit the wannapreneur market.
I don't dare tell someone how to start a business. It's their business, it's their journey, they should do it how they want.
> I still have 0 employees and don't plan to hire. I tune my business processes to run as lean as possible: Most of my customers are on credit card so their payments are automatic. The gem servers take about one day of maintenance per year. I can't really outsource much of my support work because it is so technical and specialized.
> My gem server is a $6 droplet on DigitalOcean. Because gems are just static files, that little droplet can handle millions of requests per day with just Apache. Oh, and I run three servers in parallel for failover purposes for a grand total of $18/mo.
I wonder what makes Sidekiq special? I don't imagine any equivalent product (background job processing) in the other languages is raking in that much, if making any revenue in the first place.
[1]: https://www.indiehackers.com/product/sidekiq