Yes. But people use "you have to hire people to manage it" as an argument against on-perm. When in practice, most companies hire "devOps engineers" or "cloud engineers" to manage their cloud infrastructure too. And those salaries seem higher than the infrastructure/ops guys of yesteryear. So resource cost isn't really an advantage.
don't forget the "cloud cost engineer", just to figure out how much it will cost to run the 20 services that should have been part of your app with random ways of billing the usage of it
What nerdix said, basically. People assume most devops work when you own hardware is hardware related, but for a typical setup, if your hardware related devops work make up more than a few percent of your total devops cost something is wrong.
As soon as the machine boots over PXE and is on the network, it becomes software. Hardware actually becomes easier to manage as you get more identical machines as you can do differential analysis to figure out which components could possibly be out of spec.
I have seen multiple thousand host clusters managed by 1-2 people, and day to day ops looked like absolutely nothing was happening. This included messing with hardware racking and networking. The most resource intensive was just uncrating, moving and routing cables.
We're an Ad Tech startup - cross device attribution measurement aimed at CTV specifically... and we have very few clients, just enough to pay the server bills and expenses but not enough to actually pay ourselves. Been like this for 3 years now.
It's hard, because the job is fairly demanding and it takes effort to keep this kind of big data apparatus going, but at the same time it's never quite dead and almost just barely always on the cusp of getting big enough to support ourselves with it.
We tried (and failed) to raise last year, so still entirely bootstrapped. With funding down right now another attempt at a raise doesn't seem like it's in the cards.
Merging as a one-keyword feature for all flavors of SQL is almost impossible.
It gets hairy when you have columns with composite types. E.g. depending on database, records can be JSON objects, protobufs, structs, or other composite types, like tuples.
It is possible to define semantics to merge each of these, but they vary and the merge operation becomes verbose to allow to handle each weirdness of each of these underlying types.
Merging is also sensitive to NULL vs 0 vs a DEFAULT, plus in some databases, NULL is still allowed even when a column is marked NOT NULL.
You'd almost need a sub-language for the merging operation, specifying how to handle each corner case.
> I don't find that it becomes a mess at all, because properly crafted views and procedures strongly encourage forced indexing and checking execution plans when you refactor them; as opposed to in-code, ad-hoc queries where it's hard to know if someone ever ran tests to optimize them.
I have to provide caution here, when you create views and stored procedures, I've always found it a mess to maintain:
1) VCS for stored procedures and views is usually non existent. Good luck understanding how these change over time and who changed them.
2) Deploying application logic alongside these systems is very painful, because unless you've versioned your views/stored procedures, when you make a breaking change you need to stop the entire application, make the change in the DB, and restart the entire application again on the new version. Zero-downtime deploys would require versioned views/stored procedures.
3) It quickly becomes a challenge to answer the question "what happens when I do X?" reliably, where X is insert, delete, or even select a row. Once you have complex enough stored procedures, modifying the data changes things in unexpected or unintended ways. A dev unaware of how the system works might assume that inserting a row simply inserts a row, but instead due to stored procedures hidden from application logic, it might cascade into inserting rows elsewhere, deleting rows, or modifying other rows in other tables. Discovering these issues without knowing that they could exist is often done around midnight during a production outage, because a new feature was released that did something in the DB that was presumed safe, but wasn't. If the code for the business logic were in one place, the application, it would been much easier to see what the change would actually do.
I understand entirely that performance gains from good DB use are astronomical, but in my humble experience, I've found them to be more trouble in the long run than they are worth.
> VCS for stored procedures and views is usually non existent
This is a mindset problem, not a technology problem. Treat your stored procedures/functions/views like code, not like data. Keep a separate "code" schema, put your procedures/functions/views definitions in files, and store your files in Git, just like Java or Ruby code. Deployment then becomes an automated drop and recreate of the code schema in a single transaction with zero downtime.
> Deploying application logic alongside these systems is very painful
This is not my experience at all. The stored procedures form an API. If changing the API breaks your application, you are doing something fundamentally wrong. API versioning should only be necessary if third parties depend on your API, but I wouldn't recommend giving third parties direct access to the database anyway.
> Once you have complex enough stored procedures, modifying the data changes things in unexpected or unintended ways
I assume you mean triggers because stored procedures don't happen as side effects, you have to call them explicitly. Regarding triggers, I agree with everything you say.
Re: VCS, I’ve found that with a small amount of setup, tools like Liquibase[0] allow you to maintain functions and views (with “CREATE OR REPLACE” or equivalent) as SQL files in a file system with full VCS, diffing, etc. just like any other code.
You still need to sequentially define your changes (migrations) which isn't exactly the VCS experience people are used to. If all changes to your db require new file entries, rarely do you need to diff let alone bisect to find out the history. It does make it harder to find out the current shape of your app though with all the ALTER statements to a single table strewn about multiple files. I wonder if it's feasible to write a tool that allows you to generate sequential migrations from DDL organized like tradition code with modules and items dealing with the same domain in the same file after diffing it with the sequential migrations already in the codebase.
No you don't have to structure your project that way. You can have all your logic (views/functions) stay in a single file per, and use git like usual with the standard diff you are used to. Your changelog xml just needs to be setup for it.
So, after digging about in the docs, I found the Best Practices[0] page which lays out what you describe. I was disappointed to find it is just using idempotent creation statements for procedures and views and putting them all in files that are re-run on each migration step-along with each new roll forward migration-so that new versions of these logic objects replace any old ones. This is not exactly something that liquidbase provides, should be easily replicatable, and I was hoping it'd do diffs to provide similar experience on tables as well.
After some web searching, I came across a project[1] that's trying to do what I describe but it appears to be dead. I'm surprised that migration tools (that I looked through) don't already support such a basic approach that I suspect I'm missing something obvious. Some fundamental block in how SQL works or maybe it doesn't work that well in practice for, in concept, it sounds easy to script using what they call a shadow db and `pg_diff`.
> The problem is letting other people use it; of course it's nice to help people, and it's altruistic to do so for free, but some of those people might actually need this homework to learn, and you may have deprived them of that.
This, when the scope is limited to yourself, it's very different from when it impacts others.
Back when AOL Instant Messenger (AIM) was super popular, I was in university and had read about ARP poisoning. Our school was pretty cheap, so all the dorms had hubs instead of switches. This meant that it would be, theoretically, possible to ARP poison an entire dorm, MITM attack and read all the text being sent on AIM since it was sent in the clear. I had a bit of a cyber security passion lab in my dorm room, so I wrote a PoC and ran it on a LAN air-gapped from the rest of the network. I proved that it should work for myself, having confirmed that similar cleartext messages would get passed to the machine intending to listen in between two other machines.
I told my classmate of my project and he expressed interest, so I gave him a copy. Fortunately, I didn't add any authorship info, mostly because I forgot to. I did caution him that ARP poisoning is a pretty "noisy" attack, and someone who was paying attention would notice it. He foolish ran it on the university network, and confirmed he was able to see AIM messages flying back and forth for all the dorm, as well as all the other traffic. It didn't take long for our school's IT to notice that one dorm was funneling all traffic through one machine. A week later he was banned from having a computer in his dorm room for a school year. Thankfully he never gave me up, admitting it was his stupidity that brought it on himself, but nevertheless it was a lesson learned - if you're going to play in the grey space between ethical and not, do so responsibility and don't share the exploits with others.
> you're going to play in the grey space between ethical and not, do so responsibility and don't share the exploits with others
Aka "don't get caught".
One of the times I got in bother at the first university I attended was because I kept logging into their production servers as the root user every morning.
Their admins had left a few glaring holes open that I'd patched (and evicted some fellow travellers), but I kept their SSH keys to explore a bit.
One morning one of them happened to peruse the SSH logs, and spotted a pattern where someone on the student network was logging in every morning.
Didn't take them long to work out something was deeply fucked, and they cut my network access before pulling up the contact info they had on file for me and summoning me to their office for a bollocking.
Luckily for me they figured it would be better for their job security if they kept it purely informal as opposed to notifying the university proper and having me face a disciplinary committee.
They never rotated those ssh keys, and I learned the "don't get caught" lesson as opposed to the "don't do this" lesson.
Look, it's obvious when it's an aged degree and you can easily see that the granting of that degree has little chance to a highly paid job. What do you do when it isn't obvious? When I got my student loans for my comp sci degree, it was right at the bottom of the dot-com bust. Everyone was convinced all tech was going to be outsourced and that it was a dead end degree, foolish to pursue.
If the banks had any say in if I would have gotten loans that year, it would have been no. And I wouldn't have gone on to a lucrative career in tech.
I'm not convinced that letting loan underwriters pick the future labor force allocation is the solution.
Having used Pheonix for one of my small failed startups in the past (failed because of the business, not because of the tech) - I loved every second of it. It was super easy to use and extremely quick to be productive. Ecto is a fantastic ORM, the OTP system is great to build on, and LiveView is magic. As a solo dev, it is a true isomorphic web dev experience with a nice clean language and structure to build in. Even tests work.
> if your service succeeds, you will definitely want to move some components out of aws, but not all of it. aws can always remain the control plane.
Do you factor in the costs of dev time in a cross cloud environment at all? Nevermind all the AWS specific tooling and code you write, which needs to be thrown away and re-done to accomplish this?
Don't get me wrong, I love AWS when I'm not paying the bill, it makes my job of shipping features quickly so much easier. It's good and it works well. But it's a little crazy to think it's "cheap" and to ignore the costs of vendor lock in.
Do you factor programming language and library and operating system and cpu architecture into your vendor locking and specific tooling you need to write?
vendor lockin is fine. it’s like having a great employee, but not leveraging them fully because bus factor and cogs. you should be sad if a great employee leaves, or your vendor shuts down. choose both carefully, and trust them.
my only experience expanding out of aws is to cloudflare for egress. i’m sure it can be rough, but so can anything solved with dev time.
expanding out of aws with individual tightly scoped components should be easier than general cloud migration. just bandwidth egress. just cheaper cpus for background processing. just whatever.
keep the complicated stuff on main cloud where it belongs.
Just so I understand, you decided to protest this pricing not by boycotting the service, or making a public statement about it, but instead by targeting an employee's use of the platform with an attack? I'm sorry, I really don't get why you did this or why you thought it would be effective.
Don't you also need them in a colo?