>Rails developers at the time who were largely encouraged to treat their database as a dumb data store by convention and to let the application do the work
I've worked in a big enterprise once (hated it for all sorts of reasons) where the business logic was entirely in the database, I get that that could work too. But in the sort of environments where I've worked most, there's a lot of knowledge about the application programming language, and less about the database. No dedicated DBAs. And we all know our SQL, we build basic NOT NULL and foreign key contraints, maybe a simple uniqueness constraint in SQL, but otherwise concentrate all the application logic in... you know, the application. For better or worse, I guess it's a coping mechanism for application developers. (And then it becomes a habit, maybe for some, like the article said, dogma.)
The logic of the app I currently work on lives 99% in the database... As a C++ plugin to the DB.
Not stored procedures, or triggers, or anything like that... Just a giant plugin loaded when the DB starts that provides business specific functions that can be called by the client instead of SQL queries (think "CreateAccount" or "GetFooDetails").
Oh, and probably 50% of the columns are giant JSON objects.
I was expecting OP to keep blaming database choice given that their opinion on decision making is critical. But they seem to give lots of praise to the databases full text search features. Appearantly he was judging Heroku's choice of database, which I guess would not be a random one. Can not grasp the incentives of why it is put like that in the article. Though dramatizing is a nice contour to have but gives away prejudice bias, as this comment does for me.
(not a rebuttal on your comment, just a continuation of the conversation)
I found it an interesting criticism, and from what we know about the project I don't think I agree with the author.
In the space of what database to use, there are really few differences between Postgres and Postgres' peers, that "using what's available" is a perfectly sound reasoning to use Postgresql over, say, MySQL. I think around that time MongoDB was a possible alternative as well, however it was experimental and often mocked. Postgres and Mysql are pretty much feature parity, even in 2011, except for very niche use cases, which this project did not sound like it would have.
Factoring in that Heroku's PG UI is great, and to use an alternative would mean to have to set things up yourself (in 2011 the Heroku marketplace was not yet as expansive iirc), or at least have another account at another SaaS company with dubious maturity, the decision to use PG seemed to have been given the correct amount of critical thinking ("will it be able to support all that we need? most likely yes").
The author may also not have been aware that for RoR applications the code is pretty DB-agnostic, an unawareness he hints at in other parts of the story as well. Meaning that in the span of the year that they're working on rebuilding the site in RoR, if they discover a use case that another DB would be extremely helpful with, they could relatively easily switch over to this other DB without having to rewrite all of the code. At most it'd take rewriting of some migrations for FK's and possibly some (hopefully few) custom SQL queries. Remember that they had a one-time-switch-over when the site was done being built, so during this year of development time, the database did not accrue live user data that would have made such a change more difficult.
You’re not wrong in your assessment but I would still expect a company architecting a total replacement for an existing system to know why they are choosing critical components. In this case it was an extensive lack of database consideration in many decisions.
Much of this and the supporting articles were focused on the security and fraud topics. Since those details were linked out from this article, most of what's left is around the technical bits that didn't get their own posts.
For an audience of programmers on HN, I'm not surprised that this topic got more attention than the fraud but it definitely wasn't intended to be the focus of the post.
Since fraud is a broad problem to tackle with and it requires individual awareness. It challenges densely education of the users rather than technical enhancement which ends up on OTP and 2FA. Honesty of the point of view of OP is the most attractive part of the article which leads one to complete reading. Then focus on the domains varies on readers interests to discuss.
A couple years ago, the company I was working at had a dedicated team just for JIRA (and other workflow mgmt systems).
They were looking for some people to fill out the team so I asked my SRE friends if they knew any good JIRA people.
I was surprised that the #1 answer was "Dedicated JIRA people? Why don't you just have someone do it from your team. JIRA isn't that hard."
This immediately made me think of the "Why do we even need DBAs? It's all ORM anyway." Which of course leads to "Why is the DB so slow?" etc etc
I thought that was a bad scenario till a friend was telling me at their current employer (which uses a lot of databases) they routinely ask "Is anyone here the DBA?" and the answer is "What's a DBA??"
I think like 95% of a DBA's value is a willingness to touch the db. It's not that hard to find slow running queries or missing indexes. Just that most devs don't know how and/or never check. I'm guilty of it too. Then some DBA comes along, takes five minutes to run a query, and sends the devs 10 easy performance fixes.
I think the OP is referring to the time before Rails natively supported foreign keys.
While I agree there's a place for business rules (the app), properly enforcing referential integrity and uniqueness can now easily be done in both - and I trust PostgreSQL more.
> In order to load just 100 paginated items on his account history page, over 50,000 queries were triggered using so much RAM that it crashed the server.
Issuing 50k queries to get 100 items is just wrong, no matter how you look at it. I can see the attractiveness of writing all your logic in one language; and in a way it's unfortunate there's not a simple way to write your filter in Ruby, pass it to the database, and have it executed there. But you have to face reality and deal with the fact that such complicated logic needs to be close to the data.
Edit: Initially misunderstood to mean 50k items, not 50k queries, which is even worse.
Since this was written Rails has become much better about this kind of thing.
I think Arel and some of the other work that really helped was happening more than 10 years ago now (how time flies!) but it was a non-trivial upgrade so a lot of production sites didn't switch over for a few more years.
It's pretty common for Rails developers to know very little about databases (sql specifically).
It's very frustrating because right out of the gate, Rails pushes thinking of the db as a graph, leading to highly inefficient queries
My comment was based on Java environments, and it's the same. It's unacceptable to build poorly-designed schemas in any environment. But good schema design is something else than what I thought the article might have been referring to, which is building the entire business logic in SQL, using triggers and what not. I've seen enterprises with a database team doing that, for application layer people to come in and built their application on top of this "fait accompli" database. And this means you need two highly-skilled teams: the database and the application team. (Aside from a third, the infrastructure team.)
It’s common for JR developers fresh out of college or code camps. It’s also good reason not to let the ratio of Junior to seniors get too far out of whack.
I’ve had several jobs now, where I came in to clean up after they hired a bunch of juniors with no experience.
Usually they were quite good at making things pretty.
But fundamentals like N+1, foreign key, insane / broken build times, desperation of concerns, etc abounded.
It’s sort of nice because you come in and be a hero for fixing everything up.
Can really sucks because there are often good reasons things got to be that bad.
My biggest issue on this particular project was an almost intentional neglect of database specifics, such as not enforcing uniqueness at the database level even for usernames. There were about 50 users that had 2 records which led to all sorts of crazy issues to track down.
Hours to days of data cleanup work that could have been bypassed by setting a simple unique index.
> I've worked in a big enterprise once (hated it for all sorts of reasons) where the business logic was entirely in the database, I get that that could work too.
This used to be quite common back in the day. In a sense the database was thought of as the application platform so the reasoning was to move business logic as close to the data as possible.
Performance was also a big driver since the DB was often the most high-powered server / cluster in the datacenter. An argument I heard was that by pushing logic into the DB via stored procedures and the like, you would get better performance that way vs. pulling a result set into application memory and then dealing with it there.
Note that I am not rendering a qualitative judgment on any of this, just sharing my experience. :)
"I went from thinking this was the job to realizing it wasn't going to work and trying to figure out how to tell my wife that I was going to have to start the job hunt all over again."
I just wanted to say - I have been there, and I feel you. When I read that, it was visceral - took me right back to the couple times I've had those moments. Good on you for keeping on, and using what the experience offered to make things better afterwards.
> It showed us a chart of the verified and caught spammers, as well as how much time they were spending manually typing in messages while completing Captchas...that were never delivered. Our worst offender was doing this for 15 hours a day straight.
Whenever I read something like "Increased revenues by 93% by balancing user and business priorities against available resources" in CVs I immediately disregard it.
Maybe I've read it too much but doing the actual analysis of "what have I done and how was the impact" is a lot more than simply taking revenue beforehand and revenue afterwards and calling it a success. Also if I solve a bug that was discovered by QA, specified by PM that was hogging a lot of resources, did I really increase revenue? This trend in CVs is really annoying me as most of the time it's complete BS and only there because people watched a YT video on how to build your resume.
Sometimes thats the only real number you have though? Maybe you can just ask candidates more about that?
I'm on a team that rebuild a sales/onboarding funnel. I was told it resulted in 5 million in additional revenue generated this year. I'll use that number on my resume at some point. But mostly because I don't have a more useful number to use.
If you ask me I can share all about how and why we build it which is probably what you actually care about, right? 5 mill might be a lot at a small company or almost nothing at a large company so $ and % are pretty much irrelevant. It's the process of building that you would actually care about, no?
I agree that is particularly vague however resumes are expected to be one pagers, in contrast to CVs, so those statements are really meant as teasers for further discussion after passing through the first HR gate (which wants these sorts of statements.)
At a guess, they think claiming that an individual increased revenue by 93% using a simple balancing method is not very plausible, and as it is most likely just and outright lie can thus be discarded as hot air (which is actually pretty charitable).
One objection is of course that 93% is very achievable if the numbers involved are very small, but in that case the statement might just be code for "selling the office sofa" which is often seen as less impressive.
Neither really makes sense from a statistical perspective: your number is effectively the true value, and there is no repeatable measurement process you’re evaluating.
Your use of the non-statistical meaning of “accurate” as “free from error” is spot on.
No, the number is given in a causal statement, it is not simply the reporting of a measurement as you claim. So the distinction between "accurate" and "precise" is definitely relevant.
No one really doubts that the revenue increased. The question is whether the claim that it was by aligning the blah-de-blahs with the foobars is an accurate one and, related, what the individual contribution/leadership was. (Could another peer make the exact same claim with similar credibility?)
In this situation I was actively evaluating support issues, customer reported problems and new feature requests to triage what was worked and when. I’m comfortable suggesting that if somebody else was making those decisions at the time, the entire company would have folded. It’s hard to expand further on that without going into a lot more detail though.
> While this was a shock in the moment, I would later learn that this attitude was fairly common among Rails developers at the time who were largely encouraged to treat their database as a dumb data store by convention and to let the application do the work.
This is not a bad attitude depending on specifically what levels of "dumbness" you mean. IMHO, the golden rules for database setup are:
- Spend a lot of time designing your schema. Then spend some more time.
- Do not skimp on data integrity measures like proper primary/foreign keys
- Pay attention to potential usage patterns and build indices for performance; this is not a one-time activity, keep observing the usage and look out for hot-spots
Some other tangential points that are often overlooked:
- Do not push all your business logic into a hairball of nested stored procs and triggers; debugging/testing all that is a bitch compared to writing unit/integration tests or dropping into debugger in an IDE. You can be a million times more expressive in a full-featured modern language like Java or C# etc. The main exception to this is when you need to push/transform tons of data around and all of it is on the database - a well tuned stored proc would be orders of magnitude faster. This is also not something is needed frequently and could be symptomatic of a bigger issue.
- Tons of database trips are expensive; an efficient join instead can do wonders for the performance. If you are doing that all the time, maybe its time to put that behind a view. Also, cache what you can and be conservative about it.
- If you are almost always including a filter clause in all your joins, maybe you can explore partitioning.
Basically imagine the database like an appliance that does only a few things but those things are FUNDAMENTAL so we ensure that it does them very very well. Don't make it make your coffee too (though PostgreSQL probably can do even that efficiently).
> There were other factors too, such as depending on Rails to validate uniqueness rather than the database. It resulted in a lot of messy data cleanup that I had to deal with which were causing hard to diagnose issues because race conditions will always happen.
> Second, the database is the only place that can make guarantees about data integrity so use them.
Typically, these rules are not what people mean when they say they use their DB as a dumb data store. What they mean is:
- your "schema" is not that important. Sure, you spend a lot of time modeling data in such a way that your application always handles valid data. Once that's done, your DB is "just" a store for that data.
- data integrity is enforced before data reaches the DB. pks/fks are mostly there to ensure fast queries.
The idea is that data in your system should be in a sane state way before it reaches the database, and once it's sane, making more controls at DB level is just a layer of administrative checks done in a language that is not really well-suited for that.
> One of the unfortunate things that happened is I was put in a situation where I had to choose between doing what was needed to save the company and the businesses who relied on it or listen to the daily change in priorities from inexperienced non-technical leadership.
Learning to navigate this has been a skill in and of itself. Glad the intense effort worked out for the author
>Even though this story took place 10 years ago, I won’t be naming the company or anyone involved.
I have never heard of Audiogon, but reading the article I looked forward to my time in the comments section and seeing the mask pulled off. It seems such a naive thing to write for a guy who is clearly not an idiot. So much so that I guess that this line was more like a mandatory disclaimer kind of thing, while not intent at all on keeping things a secret? The company and timing of his employment are on their LinkedIn, which is linked from the article.
> When things eventually stabilized, the leadership was happy with the result but displeased with my obedience. During my annual review, I was told I wasn't working hard enough. [...] Over this month, I really was not productive for the first time in my entire career. [...] When emotions had settled, the owner told me I saved the company and thanked me for it.
This is why I laugh when HN commenters tell me that the companies make reasonable choices when it comes to their employees.
It was one of the worst experiences of my life. I didn't appreciate what depression felt like prior to this.
For a month or so, it honestly felt like I was on some type of drug that made me constantly tired, distracted and stressed all at the same time. You keep telling yourself to get moving and just...can't. It sucked.
Hindsight is big on this project though. I got so much experience from this work that it truly did level up my career. I learned more as a side effect of getting deep into these issues than I ever expected too.
The cynic in me wants to really say, typically the sacrifices never pay off.
My example time:
Company I was working for planned a promo on July 4th. This promo was in collaboration with a GIANT sports brand. Think of the most known sports brand (not sportswear) in the world. No real testing in our QA env due to (some reason I can't remember).
Day of promo (July 4th) I get a frantic call "IT'S NOT WORKING."
I spend my July 4th, stopped in my tracks right before I'm about to head out the door to the lake with some friends fixing a bug in this promo.
My reward for doing frantic work to save probably 3 peoples assess above me even when I wasn't on call? "thanks"
Not even the basic 2.5x daily rate that is mentioned in our company handbook.
All this to say: The sacrifices hardly ever pay off in my mind.
It's a general database access anti-pattern. It happens when you make one simple query to get a list of stuff. Then for every row, you fire off another query to get more details about that item. Instead of this, a single properly-designed query could have retrieved everything you need, and nothing you didn't. This can be highly detrimental to both application and database performance, especially if the number of initial results gets long or you have another layer of query loops after the first one.
ORMs are notorious for making it easy to write this anti-pattern without noticing, though you can do it in raw SQL too. ORMs usually also have various helpers for eager loading of associations to try and prevent it. In raw SQL, it's on you to notice that you're doing it and write the correct query.
You're using an ORM to query a record and as you're looping through the results you need to reference something in an association.
You get one query that returns 50 results, then loop through those 50 results and trigger a one off query for every result in the loop. Now you're executing 51 queries instead of the original one.
Now imagine you go another level deep or have multiple results from each of those sub queries to loop through. It can get out of control. In Rails and a lot of ORM's you can typically solve these by eager loading the nested associations with very little effort. The rest of the code will work the same but behind the scenes only 1 additional query will be triggered to fetch the records from the association.
I once took a Python API with SQL Alchemy and took 30% of the speed by adding an eager load instruction to a query. Python profiling showed a lot of time in the ORM & database, then turning on SQL logging made the issue obvious.
Still feels like it should be called the "1 + N" problem tho. The 1 query happens first, then the N queries afterwards.
It remind me of my first professional experience: the ship was sinking, it was hard, but I never ever again learned so much.
Sometimes I'm nostalgic if the intensity I lived in, but I think I'm too old for this.
I know the feeling. I couldn't imagine inserting that 24/7 type commitment into my life right now.
It was crazy driving home from work, getting a call that a phishing email went out when they clearly waited until we wouldn't be around to take it down. Then getting home, opening up the computer, making Skype calls to an ISP in South Africa to notify their abuse teams for the take down.
I think it's a difference between European English and American English. American English would spell it "combating". Which I kinda learned was wrong in an American school.
"Swimming", "running", "batting" come from "swim", "run", and "bat", respectively. The present participle form doubles the consonant at the end of the word "as a rule" I was told. A single consonant would mean that there is a silent "e" at the end of the word which suggests a preceding long vowel (e.g., cure -> curing, compete -> competing, combate -> combating).
Anyway, it's because nobody follows rules when writing and speaking English and as long as the other person understood your meaning, your grammar was correct.
I actually had to look it up after publishing. I've always used "combatting" because it seems like it should maintain the proper "a" sound...but my spell checker flags it now.
-----
Jenny: https://darknetdiaries.com/episode/90/
The Beirut Bank Job: https://darknetdiaries.com/episode/6/
Black Duck Eggs: https://darknetdiaries.com/episode/21/
John and Brian's Big Adventure: https://darknetdiaries.com/episode/95/
-----
And this is just scratching the surface. There's some GREAT stuff.