>That's ok because we have compilers, the OS, and other layers to handle that for us.
This is the essence of my argument. Why can't the database API occupy a layer similar to the OS or compilers?
If I don't understand the underlying implementation of compilers and OSs' why do I need to understand the underlying implementation of my database? Why is the OS abstraction different from the DB sql abstraction? Can't we write abstractions that are zero cost?
The answer is we can, however we haven't and therefore the only solutions available to us is to use SQL and understand the lower level details of the database.
>However, even for your bog typical web dev, a database is right up there in the list of systems you need to understand in order to do your job.
I'm not talking about your responsibilities on the job. I'm questioning the current state of tech. A web developer uses an API exposed by an OS and doesn't have to understand the OS. A web developer uses an API exposed by the compiler and doesn't have to understand Assembly. A web developer uses an API exposed by a database but he MUST understand the underlying features of a database. See the difference? I'm pointing out an issue that could be improved not what you need to know for your job.
>A carpenter doesn't have to be a blacksmith, but they do need to know how a saw works.
SQL is a saw that breaks in half on every use. In order to use such a saw a carpenter must have the knowledge of a blacksmith to reforge the metal and combine the two halves of the saw into a single metal blade. A leaky abstraction. That is a proper analogy for SQL.
>This is the essence of my argument. Why can't the database API occupy a layer similar to the OS or compilers?
>If I don't understand the underlying implementation of compilers and OSs' why do I need to understand the underlying implementation of my database? Why is the OS abstraction different from the DB sql abstraction? Can't we write abstractions that are zero cost?
Because it's closer to your application. An OS doesn't need to know how e.g. a Customer relates to a Business in order to read a file. Some of us do need to know the ins and outs of the CPU and OS because our application requires it. Yours doesn't, but it does require knowledge of how the DB works.
>I'm not talking about your responsibilities on the job. I'm questioning the current state of tech. A web developer uses an API exposed by an OS and doesn't have to understand the OS. A web developer uses an API exposed by the compiler and doesn't have to understand Assembly. A web developer uses an API exposed by a database but he MUST understand the underlying features of a database. See the difference?
Not really, no. As a web dev you are about 100 layers away from the OS most of the time, but your application logic and architecture is very much dependent upon the information store. I spent years working in digital imaging and automation and had to know what my compiler was doing and how my OS implemented all sorts of things because my application depended on it.
>SQL is a saw that breaks in half on every use. In order to use such a saw a carpenter must have the knowledge of a blacksmith to reforge the metal and combine the two halves of the saw into a single metal blade. A leaky abstraction. That is a proper analogy for SQL.
This is what irks me. I hear stuff like this all of the time from people who solve relatively trivial problems day to day, but for whatever reason think things should be better than they are. They, of course, have no idea what that would look like, but damn it if they don't have a strong opinion on the matter.
I'm not saying that's you necessarily, but if you hadn't noticed, these things are hard. This is the best we have come up with so far. Planes still crash, cars don't drive themselves, and my laptop crashed literally 30 minutes ago.
In my experience, the people who complain most loudly about poor abstractions, those who harp on you for not following "modern design patterns", and generally complain about everything write the worst code. I eagerly await your proposal for a better data store and application protocol that solves all of the problems SQL currently does without any of the warts.
>Because it's closer to your application. An OS doesn't need to know how e.g. a Customer relates to a Business in order to read a file. Some of us do need to know the ins and outs of the CPU and OS because our application requires it. Yours doesn't, but it does require knowledge of how the DB works.
SQL is 100% trying to achieve the goal opposite of what you say. Look at the query language. The literal intention of SQL is to hide everything from the user. EXPLAIN ANALYZE is tacked on later when people started using SQL in the wrong way causing the godawful and impossible to remove mess we have today.
I'm arguing for a language that's one to one with the implementation. Zero cost abstractions like C, C++ and Rust.
>Not really, no. As a web dev you are about 100 layers away from the OS most of the time, but your application logic and architecture is very much dependent upon the information store. I spent years working in digital imaging and automation and had to know what my compiler was doing and how my OS implemented all sorts of things because my application depended on it.
No to you dude. You're not 100 layers away from the OS. All the languages that you use to develop a web app have system calls, you are essentially 1 layer away from the OS. The reason why we can use high level languages like ruby or python is because the the database is a bottleneck and it doesn't really matter how slow the app is... But even these languages expose OS level system calls, so it's essentially the same experience as a "low level language" sans memory management.
Also the fact that you needed to understand what you're compiler was doing is important. Did you have to do SQL level hacks in order to get your compiler to do what you wanted it to do? I'm sure your compiler was DESIGNED to be low level so that you didn't see BS like UNION vs. OR.
>Because it's closer to your application. An OS doesn't need to know how e.g. a Customer relates to a Business in order to read a file.
You realize the entire initial purpose of SQL was for business questions by laymen. The things we do with it a lot nowadays are optimizations via hacks and query string construction via meta programming. I'm not saying don't understand your toolset implementation (even though in the ideal world you shouldn't need to) I'm saying there's a problem when an optimization involves a high level api hack. Literally a week ago I wrote a SQL optimization hack that improved speed by at least 3x, it involved serializing a data-structure into a string and doing string manipulation to change the data structure into something easier to deal with... before casting it back down... What other API in the universe requires you to do this? A bad one.
>In my experience, the people who complain most loudly about poor abstractions, those who harp on you for not following "modern design patterns", and generally complain about everything write the worst code. I eagerly await your proposal for a better data store and application protocol that solves all of the problems SQL currently does without any of the warts.
I have no proposal, I'm not trying to design a solution I'm saying there's a problem. I'm not a database developer. You trying to imply that I shouldn't talk unless I'm a database developer with a solution in mind? Are you a database developer?
Also I hate design patterns, I avoid design patterns like the plague.
>This is what irks me. I hear stuff like this all of the time from people who solve relatively trivial problems day to day, but for whatever reason think things should be better than they are. They, of course, have no idea what that would look like, but damn it if they don't have a strong opinion on the matter.
What's your opinion on "trivial problem"? I'm literally doing that everyday. I'm working on building SQL views for a database littered with technical debt and JSON typed columns. Literally all I'm doing is writing SQL views to cover up all this crap up so customers can query our database as if everything was properly sequestered into tables rather then JSON blobs. Day in and day out I try to poke at a query trying to coax more speed out of it using non-intuitive and illogical hacks. All I do is SQL (currently). Trivial as hell.
What do you do? Must be some super crazy dream project for solving some ultra hard problem. You talk as if you're elite.
>What do you do? Must be some super crazy dream project for solving some ultra hard problem. You talk as if you're elite.
I don't have any more time for this at the moment, but I promise you that your views are not nearly as complex as designing and maintaining a database engine. I'm sorry if that hurts your feelings. Remember; I'm not the one bemoaning "poor abstractions" and offering exactly zero ideas as to how it could be improved.
>I don't have any more time for this at the moment, but I promise you that your views are not nearly as complex as designing and maintaining a database engine. I'm sorry if that hurts your feelings. Remember; I'm not the one bemoaning "poor abstractions" and offering exactly zero ideas as to how it could be improved.
No time or more like nothing left to say?
Why be sorry? My feelings aren't hurt when you're completely and utterly wrong. You should be the one with your feelings hurt not that you've been shown how off-base you are.
Additionally who said I'm talking about a database engine?? I'm talking about the API man. I never mentioned anything about the engine itself. I'm talking about how a USER accesses lower level calls that conduct JOINS and SCANS. We don't have explicit control over the individual commands you see in an EXPLAIN, we can manipulate those calls via a SQL query. I'm literally only referring to SQL.
Man when you started speculating about what I do with "non-trivial problems," that's when you took this argument into no mans land.
The very idea you have to be an expert in something to comment on a problems is literal mental stupidity. I don't restrict myself and neither do you because clearly YOU aren't a database developer either.
Think about it this way, I'm not an expert but am I allowed to comment on the environment? About global warming? Am I not allowed to support either of these causes without a PHD in environmental science? Of course I'm allowed. As are you.
You're even allowed to speculate on my coding practices, patterns and career... "trivial problems" as you've already implied. But it's a dirty low blow when you do stuff like that, I tend to avoid it.
This is the essence of my argument. Why can't the database API occupy a layer similar to the OS or compilers?
If I don't understand the underlying implementation of compilers and OSs' why do I need to understand the underlying implementation of my database? Why is the OS abstraction different from the DB sql abstraction? Can't we write abstractions that are zero cost?
The answer is we can, however we haven't and therefore the only solutions available to us is to use SQL and understand the lower level details of the database.
>However, even for your bog typical web dev, a database is right up there in the list of systems you need to understand in order to do your job.
I'm not talking about your responsibilities on the job. I'm questioning the current state of tech. A web developer uses an API exposed by an OS and doesn't have to understand the OS. A web developer uses an API exposed by the compiler and doesn't have to understand Assembly. A web developer uses an API exposed by a database but he MUST understand the underlying features of a database. See the difference? I'm pointing out an issue that could be improved not what you need to know for your job.
>A carpenter doesn't have to be a blacksmith, but they do need to know how a saw works.
SQL is a saw that breaks in half on every use. In order to use such a saw a carpenter must have the knowledge of a blacksmith to reforge the metal and combine the two halves of the saw into a single metal blade. A leaky abstraction. That is a proper analogy for SQL.