Same in my compnay. We have a dataset of 1 Terabyte that needs to be queried once a week. IT has talked management into installing a Hadoop cluster for this and we have to pay an analytics team. I told my boss that we can probably run this on a SQL database on a USB drive attached to his laptop without problems.
Suggest to your boss to do a test. If your off-the-shelf solution performs equally well, axe the cluster and analytics team. Maybe go to him with an estimated cost comparison.
We already have done tests for in-house testing but nobody wants to stick his neck out and tell IT to go away. They have convinced upper management that we are doing "big data" and upper management likes the sound of this. It's the good old problem that when you do something that's "cool" at the moment you get more credibility than using tried and true tech like SQL databases. I'll probably also jump on the Hadoop bandwagon because it's good for the resume and more exposure inside the company. Total waste of money and we lose a lot of flexibility but who gives a f..k these days. Gotta go with the flow.
> We already have done tests for in-house testing but nobody wants to stick his neck out and tell IT to go away.
So true, but in my experience, once you spread the words, other business owners will jump in like an infectious disease and ask for a ticket to use your system. Now your USB won't work anymore. So people "higher-up" want to bet on "fail big" than "fail short", very counter-intuitive, but that's exactly how some enterprise IT spend their budget.
I would advice you to jump into Hadoop and also spark. It's a bloody mean world out there who wouldn't care what you have done or how you have done but just wanted to know if you can configure the most popular tool in the market.. you may explain them you didn't get a chance etc, but most of the time interviewers are like your current management.
