Story time! Back around 2008 when we were developing the first version of AWS CloudWatch (which was then code-named "Granite") we had two problems to solve:
1. What should the API for PutMetricData look like?
2. Where should we store that data?
(you know, minor issues)
At the time, we decided that #1 was more critical than #2 (and of course #2 depends deeply on the answer to #1), and that the best way to solve it would be to do some learning-by-prototyping - stand up an internal version of CloudWatch to dogfood with some other Amazon services and be able to go to prospective internal customers with a "working" service that they could kick the tires on, not just some API docs and paper prototypes. So I quickly hacked up an implementation of "GraniteDataStore" called "InMemoryDataStore", which would store every PutMetricData request in a HashMap and nowhere else.
When I sent the code out for peer review, my reviewer was aghast at the idea that we would stand up a service that would knowingly store different copies of the data on every server, and lose it all on process restart. I responded by renaming InMemoryDataStore to FakeInMemoryDataStore, but that wasn't enough. Finally, we settled on a name of (if memory serves) FakeFakeFakeDataStoreThatLosesAllYourData and that was enough to get my merge approved.
I (also at Amazon) once wrote a script that fetched data from a whole bunch of different sources and aggregated them into a single static snapshotted overview Wiki page (no, they weren't live-dashboardable, for...reasons). One of those sources was an oncall-diary, that had to be manually updated because it was tracking fluffy human things like sentiment and thoughts rather than ticket count. So, naturally, I put a `--yes-I-promise-that-I-have-updated-the-oncall-diary-page` flag on the script, and made it error out with a very clear error message (telling you what needed to be updated, and why) if that was not set.
I went off to do something else for a few months, came back, and found that script being run in a Cron job, with that flag hard-coded set. Sometimes even your clearest "watch out, do this thing!" naming won't do any good.
(I guess the takeaway from this is that the manual oncall diary wasn't all that important. Which was something I questioned at the initial requirements gathering, but ¯\_(ツ)_/¯ )
---
btw, _thank you_ (and your team) for CloudWatch. It's great, and I am continually frustrated by teams that ingest events, process them, and emit calculated metrics to PMET, rather than just logging and letting Log Insights figure it out. The extra overhead to change PMET-emission code every time we want to investigate some new measure is such a waste.
> my reviewer was aghast at the idea that we would stand up a service that would knowingly store different copies of the data on every server, and lose it all on process restart
> InMemoryDataStore
This...is just how in-memory data stores work, right? That's what "in-memory* means? Or am I missing something obvious?
In-memory data stores still typically have a concept of "durability" which you can think of as "the same transaction is committed on all servers in {some definition of a replica group}." So like any disk backed store, all servers see transactions consistently, and all have a copy of the data.
Obviously, there's no replication/transaction processing occurring when you're just storing stuff in an ordinary hashmap. Hence the Fake* naming.
1. What should the API for PutMetricData look like?
2. Where should we store that data?
(you know, minor issues)
At the time, we decided that #1 was more critical than #2 (and of course #2 depends deeply on the answer to #1), and that the best way to solve it would be to do some learning-by-prototyping - stand up an internal version of CloudWatch to dogfood with some other Amazon services and be able to go to prospective internal customers with a "working" service that they could kick the tires on, not just some API docs and paper prototypes. So I quickly hacked up an implementation of "GraniteDataStore" called "InMemoryDataStore", which would store every PutMetricData request in a HashMap and nowhere else.
When I sent the code out for peer review, my reviewer was aghast at the idea that we would stand up a service that would knowingly store different copies of the data on every server, and lose it all on process restart. I responded by renaming InMemoryDataStore to FakeInMemoryDataStore, but that wasn't enough. Finally, we settled on a name of (if memory serves) FakeFakeFakeDataStoreThatLosesAllYourData and that was enough to get my merge approved.