I really appreciate your sharing the data with us and I like your service. But, this is a poorly done and a far from subtle plug of your business at the cost of LinkedIn.
1) Technical ability vs # of endorsements
Jesus. Hiding stats that you don't like through aggregations? And please read up on Simspons Paradox, which is clearly the case here just by looking at your plot. Try a basic t-test, or rather some statistical rigor, the next time you try to make conclusions from data.
2) Most endorsed vs Language of Choice
As pointed out, this is not the way to frame your problem. By obfuscating what's happening in your histogram (which isn't technically constructued right either) you are again hiding what you dont like through aggregation. By the way, language matters greatly here, and you'd have benefitted by standardization.
3) Your conclusion
"After running some significance testing, though" and not posting your results or methodology, which is at best questionable after reading your analysis.
Again, I enjoy your service, but blog posts on technical ability that are ironically lacking in technical ability don't really make me want to come back.
PS: A little birdie told me that endorsements are quite strong in predictive power for jobs :)
> It turns out that people’s interview language of choice matched their most endorsed language on LinkedIn just under 50% of the time, so, you know, just slightly worse than flipping a coin.
A coin has 2 sides. How many programming languages are there, again?
I read that as flipping a coin to predict if the most-endorsed language will be the language of choice in an interview. I make no claim to the statistical usefulness of such a statement.
language of choice == linkedin endorsed lang -> {TRUE,FALSE}
Yet if there was no relationship between the two entities and there were N languages, one would expect the random probability of TRUE to be N/N^2 = 1/N.
Although, the writer doesn't seem to allege that he's comparing to random or anything like that.
And the variables are measuring different things - you'd expect the most endorsed language to mirror the language that you have the most experience in during your (possibly very long) career, and there's no reason to suppose that it should match your current language of choice (which quite likely is more modern than what you used 10 years ago) more than 50% of the time; no matter how (un)reliable endorsements are, this isn't a valid argument against them.
Author here. I appreciate the notes and am happy to revisit and make corrections when needed. To respond to your points:
1. As a sanity check, I did do a t-test of technical ability vs. # of endorsements before publishing. There is no statistically significant relationship between the 2. (P < 0.335)
2. What do you mean by "language matters here" (re the histogram)?
After reading the way in which your work is being criticized (regardless of the validity of the criticisms), I'm very impressed to see you here making lemonade of it all. I'd be crying in the corner if it were me. Good work.
Do you mean that you fit a simple linear model, of the form below?
ability = b0 + b1*endorsements + error
And when you say t-test, are you saying you did a t-test for the parameter b1?
Usually when people refer to a t-test, without more information, they are saying they tested the difference of means between two groups. (or one mean against a number).
> Do you mean that you fit a simple linear model, of the form below?
That would be the form of the best-fit line in the scatterplot. (and it would make sense to assume that the t-test refers to b1 != 0, as there is only one group)
Edit: on second thought, you're probably right. I think I was too off the cuff in responding. Left original response below.
If by best fit you mean minimizing sum squared error, that's fair.
But to be sure, if someone said t-test, and they only had one group, I would first guess they were doing a one-sample t-test.
Even with two dependent variables and one group, I would think over whether they did a dependent t-test.
I figured it was a simple linear model (in this case a correlation) because they mentioned that they tested the relationship, and it makes sense, but it seems important to sanity check the use of the term t-test, which can be highly ambiguous (and I have seen used in very surprising ways).
It's unclear what you t-tested here. Ideally, you would test for difference between groups of "Is there a difference in number of endorsements between people who got a "yes" in advancing to the next round or not". As a followup, is there a difference between those who's preferred was most endorsed or not?
I'm a bit stunned that you didn't recognize Language as programming language...... :(
As an example, people probably get endorsed for SQL or CSS far more than their programming language of choice that is tested in an interview.
IMO, abusing endorsements like this isn't very useful. At best, it's sort of replaces some attribute of a reference. Endorsement is really feedback to validate the field that somebody works in and is probably not a bozo.
If you're interviewing a mid-career programmer, and her endorsements are all for accounting and financial related things, you may have a fit issue to explore. Likewise, if you have a candidate with 200 connections and 3 endorsements, and that's atypical for the industry, you may want to focus more heavily on real reference checking.
Seriously, I'm not sure how they can even pretend to have reached that conclusion. It would require interviewing random LinkedIn users without endorsements (including ones with no technical ability at all) and then rating them.
I certainly think that LinkedIn endorsements aren't exactly a meaningful metric. But they're not meant to be. They're just a simple tool for finding people who can even nominally program.
If you breakup the plot by any number of categories of technical ability, then there are trends. But to your question, what I was suggesting was that the aggregations done, especially with categorical data that is averaged, are very susceptible to this. And those clusters are reminiscent of situations like this:
I'd more so like to see this analyzed against who got to the next round (their binary signal), or yes against preferred language, which I suspect will be much more telling.
The takeaway from that plot is, there is more to the story.
World class engineers (especially data eng) who built some amazing open source things, but internally was a mess.
The reason LinkedIn is a terrible product is because it was mired internally by chronically poor engineering leadership and product politics. Nothing ever got shipped. This new rollout is nice but 5 years too late.
I'm really hoping the MSFT buy will change things, as I really believe in the product and there are some great engineers there.
Honestly, the paper wasn't bad, but I'm shocked they let this article get out.
Combine the author's incredibly poor understanding,shit link bait headline, and baseless article with ZERO results or novelty and this is 10 minutes of my life I want back.
No, you just had a bad point that you'll come to realize if you ever have to manage a team or company.
More often than not, you outsource items that are not the highest value projects your resources could be spent on. Or, you simply cannot afford to hire a specialist for a specific need. Try to find me experts for some of the pattern recognition algorithms offered who will work at your shitty startup, your shitty big-co with minimal tech wing, or your random mid-sized in the middle of nowhere. I just described 95% of companies on the planet.
And to second relix's point, try to justify implementing something that may have taken years to optimize, even if you think you can do it better. Also, do you want to maintain it? What happens if/when the company loses you? Would they rather have some poorly documented, poorly optimized in house version with nobody around who knows how it works or have a contract with another party they can count on for maintenance and support?
Is this always the case? No. Sometimes it's a core function you want to develop in house. But, it's usually considered.
As a developer, a big step in growth is realizing how to best spend your time and resources, and realizing that contracting implementation of X to further speed up building Y is a choice you'll often have to make.
There's a balance to be had though with every project between IP/licensing concerns, the implementation of the algorithm itself may not be appropriate for all hardware / software combinations, and cost vs. implementation time and complexity of algorithm.
Component reuse is generally preferable to re-implementation, but it's not a blind creed to follow.
Every model's output quality is dependent on the quantity of data it ingests.
Statistics developed as a science because of the need to overcome the weakness of large samples being expensive. Machine learning has taken off as a direct result of the field's ability to take advantage of and get serious performance gains from the massive amounts of data being generated and leveraged recently.
Here is the best summation I can reference, and I can tell you from personal experience it is very true:
"The accuracy & nature of answers you get on large data sets can be completely different from what you see on small samples. Big data provides a competitive advantage. For the web data sets you describe, it turns out that having 10x the amount of data allows you to automatically discover patterns that would be impossible with smaller samples (think Signal to Noise). The deeper into demographic slices you want to dive, the more data you will need to get the same accuracy."
I completely agree with this and, as I said, it's obvious that more data produces better predictions, even with simple models. My point is that it looks backwards to me putting effort into finding more and better data (creating a corpus for a given subject is a challenge in itself) instead of trying to come up with a model that infers more and produces better predictions with less data. Once you have such a model then you can surely collect and feed it a lot of data to improve the output, but until then, why even bother?
It's not like they aren't trying to improve the model as well, all the time. It's just saying that right now the benefit of getting more data for existing (already very sophisticated) models is greater than the incremental benefits of model improvements given existing data.
They do [1], but not as often as you expect. Most games don't actually use AI, as the goal is to make in game characters appear intelligent, rather than actually being so.
I really appreciate your sharing the data with us and I like your service. But, this is a poorly done and a far from subtle plug of your business at the cost of LinkedIn.
1) Technical ability vs # of endorsements
Jesus. Hiding stats that you don't like through aggregations? And please read up on Simspons Paradox, which is clearly the case here just by looking at your plot. Try a basic t-test, or rather some statistical rigor, the next time you try to make conclusions from data.
2) Most endorsed vs Language of Choice
As pointed out, this is not the way to frame your problem. By obfuscating what's happening in your histogram (which isn't technically constructued right either) you are again hiding what you dont like through aggregation. By the way, language matters greatly here, and you'd have benefitted by standardization.
3) Your conclusion
"After running some significance testing, though" and not posting your results or methodology, which is at best questionable after reading your analysis.
Again, I enjoy your service, but blog posts on technical ability that are ironically lacking in technical ability don't really make me want to come back.
PS: A little birdie told me that endorsements are quite strong in predictive power for jobs :)