So true. At my old job, higher-ups insisted we use Spark on a data-set my iPhone could handle. Blows my mind to this day.
On a personal note, I loathe having to do interviews (and I also dislike interviewing people), but if none of my projects take off, I'll have to head back into the maw of that soul-crushing corporate machine all over again.
> At my old job, higher-ups insisted we use Spark...
Saw the same thing, but with Kubernetes. They spent so much making the platform infinitely scalable, that they ran out of budget to build the actual product that was supposed to sit on top of that infrastructure (╯°□°）╯︵ ┻━┻
What's really screwed up is that several people that made that god-awful decision have now moved on to different companies (with pay raises I'm sure). So not only did they make an objectively terrible decision, but they also left the mess to whomever it ended up falling in the lap of, and not only that, but they got rewarded for it, too!
Always jump on the cool stuff even if it makes no sense for the business. In most places as SW engineer you are not rewarded for creating business value or saving money. You are being rewarded for knowing technologies.
For lower positions/companies, I would probably agree.
If you want to make a lot of stable liquid money (400,000$+ for principal engineers at big companies and 7 figures for the top individual contributors), then you need to join a big company and deliver high value results consistently to get those offers/promotions (probably a little luck too).
That's all well and good, but those are highly competitive positions. Even joining the companies where this is possible in the first place is competitive. It isn't something you can "just" do.
Also, this only applies to an elite subset of "big companies". Few, if any, individual contributors at Oracle or IBM are anywhere near that.
Getting in is definitely competitive but study enough data structures and algorithms and you should be able to get into at least 1.
Making principal is even harder but I think it is pretty attainable if you put in extra work where the extra work is learning about your field and applying it to the business problems for your org.
I work at one of these companies and most people are 9-5'ers who work as cog's, don't spend extra time learning about their field, don't think big, and don't apply them to the job (aka, aren't leaders which is what principal engineers are).
True, but if you're not one of those principle engineers already then you likely have no ability influence what tech choices are being made and you should just learn the new tech so you can jump ship more easily in the future
Really? I propose tons of ideas to my team/org about technical decisions I think we should move towards and while a good portion of them get shot down, lots get positive feedback/management backing from my team. This combined with me owning the projects/proposals I made got me promoted very quickly.
What I've learned from the past few days on hackernews is that apparently I have worked at some shitty jobs. Even when I have owned a project and led it to massive success as far as the company was concerned, like under half the budget and half the timeline, its just led to someone higher up taking notice. Those higher ups then either come in and start directing things themselves because they think we cant possibly keep succeeding, or they hire someone from outside the company to come in and take over.
A decade is less than 1/4th of an employees career at the current retirement age of 66/67 and there are people that get this compensation in a decade (a rare few even less, granted it is not trivial).
Realistically most of us will never be in the position to make 400k or more. Once you are at that level you are part of the "in" crowd but most of us will always be replaceable cogs in the wheel.
Wanna make that kind of money? Go found a unicorn and get the silly money VCs to flood your company with cash, and then take your platinum parachute out.
In other words, go be Elon Musk or Steve Jobs.
If you’re not already Elon Musk or Steve Jobs, then that might be a little harder for you to do.
Same in my compnay. We have a dataset of 1 Terabyte that needs to be queried once a week. IT has talked management into installing a Hadoop cluster for this and we have to pay an analytics team. I told my boss that we can probably run this on a SQL database on a USB drive attached to his laptop without problems.
Suggest to your boss to do a test. If your off-the-shelf solution performs equally well, axe the cluster and analytics team. Maybe go to him with an estimated cost comparison.
We already have done tests for in-house testing but nobody wants to stick his neck out and tell IT to go away. They have convinced upper management that we are doing "big data" and upper management likes the sound of this. It's the good old problem that when you do something that's "cool" at the moment you get more credibility than using tried and true tech like SQL databases. I'll probably also jump on the Hadoop bandwagon because it's good for the resume and more exposure inside the company. Total waste of money and we lose a lot of flexibility but who gives a f..k these days. Gotta go with the flow.
> We already have done tests for in-house testing but nobody wants to stick his neck out and tell IT to go away.
So true, but in my experience, once you spread the words, other business owners will jump in like an infectious disease and ask for a ticket to use your system. Now your USB won't work anymore. So people "higher-up" want to bet on "fail big" than "fail short", very counter-intuitive, but that's exactly how some enterprise IT spend their budget.
I would advice you to jump into Hadoop and also spark. It's a bloody mean world out there who wouldn't care what you have done or how you have done but just wanted to know if you can configure the most popular tool in the market.. you may explain them you didn't get a chance etc, but most of the time interviewers are like your current management.
On a personal note, I loathe having to do interviews (and I also dislike interviewing people), but if none of my projects take off, I'll have to head back into the maw of that soul-crushing corporate machine all over again.