Others have already mentioned several reasons that would justify their choices. I completely understand why a company of Netflix's size has half a dozen different databases.
Recently, I ran into a somewhat similar situation that left me asking "why the hell are there five different databases running on this ONE virtual machine?".
It was one of VMware's virtual appliances running one of their products. I want to say it was either vRealize Network Insight or vRealize Operations Manager but I'm not 100% sure. It was likely one of those or something along those lines.
To stand up the smallest possible deployment of this virtual appliance, the "minimum recommendations" were something like 8 or 10 vCPUs, 32 GB of RAM, and 800 GB of storage -- and, if memory serves, this was for a single node!
Anyways, I got looking into it a bit and discovered that there were a total of five (5) separate database instances running on this one single virtual machine. I can't recall the specifics now but I'm pretty sure there were two PostgreSQL instances and one MongoDB instance. The other two have escaped my memory but I believe #4 was Cassandra. NFI now what #5 was.
On a positive note, afterwards I completely understood why the "minimum recommendations" were what they were. I would have (naively?) thought it'd be better to spread those database instances across a few VMs but I suppose (from VMware's perspective) when your customers call up screaming because their appliance they just paid six figures for is running like crap before they've even really put it into production, 1) you've got just one VM you've got to deal with and 2) the easy "fix" is to simply tell the customer to give it more resources ("another 10 vCPUs and an additional 64 GB of RAM and it should be fine!").
What you're seeing is a collective action problem. Each team makes a rational decision, but only does so locally. Solving a collective action problem requires widely distributed efforts that are valuable collectively, but which don't make sense individually.
I collected a list of these starting at my previous employer and got up to 91 examples. Stuff like "consistent configuration names" (ssl_verify vs verify_ssl), "consistent log format", "consistent Kubernetes CRD naming scheme" etc are all examples.
It is not an easy thing to solve. Especially since folks don't usually think of it in economics terms.
Disclosure: I work for VMware, but not on the products you mentioned.
Others have already mentioned several reasons that would justify their choices. I completely understand why a company of Netflix's size has half a dozen different databases.
Recently, I ran into a somewhat similar situation that left me asking "why the hell are there five different databases running on this ONE virtual machine?".
It was one of VMware's virtual appliances running one of their products. I want to say it was either vRealize Network Insight or vRealize Operations Manager but I'm not 100% sure. It was likely one of those or something along those lines.
To stand up the smallest possible deployment of this virtual appliance, the "minimum recommendations" were something like 8 or 10 vCPUs, 32 GB of RAM, and 800 GB of storage -- and, if memory serves, this was for a single node!
Anyways, I got looking into it a bit and discovered that there were a total of five (5) separate database instances running on this one single virtual machine. I can't recall the specifics now but I'm pretty sure there were two PostgreSQL instances and one MongoDB instance. The other two have escaped my memory but I believe #4 was Cassandra. NFI now what #5 was.
On a positive note, afterwards I completely understood why the "minimum recommendations" were what they were. I would have (naively?) thought it'd be better to spread those database instances across a few VMs but I suppose (from VMware's perspective) when your customers call up screaming because their appliance they just paid six figures for is running like crap before they've even really put it into production, 1) you've got just one VM you've got to deal with and 2) the easy "fix" is to simply tell the customer to give it more resources ("another 10 vCPUs and an additional 64 GB of RAM and it should be fine!").
</rant>