Several flaws immediately jump out, all potentially fatal to the conclusion and title:
1. "We will do so by removing outliers in the 10th and 90th percentiles." you can't remove outliers when investing unless you have a time machine
2. Their "starting price" includes an average of 10 days, 5 of which are before the analyst releases the recommendation. Again, to buy before the release requires a time machine (or inside information). Suppose a stock is at $50 for 5 days, an analyst releases a $75 target, stock jumps to $60, then hits $70 over the next 12 months. The gain based on the average of $55 would be $70/$55= 27%. But the average based on what price you could actually buy it at is $70/$60= 16%.
3. They narrow down analysts apparently a second time to pick out only the top 10, which seems to be distinct from the first outlier removal (not entirely clear what's being removed in either case.)
I see no statistical tests relating to removing outliers, significance, etc. This is all besides the considerable degrees of freedom (cutoff price, cutoff marketcap, 100 rating minimum for analysts which seems to have been implemented after collecting data, etc).
This is nothing like the sharpshooter fallacy. The analysis determined the average performance of the analysts with >100 stocks rated and 10 analysts out of 16 did better than the rest.
To be fair, looking at real data if you look at the 20 best analysts _one year ago_ they made a pretty decent margin (the article doesn't cover this). I can see if I can dig up some old charts if you're interested.
There are a couple of services which collect analyst ratings and at least in the US and Europe, they have gone to great lengths to be nice to analysts so analysts will enter their ratings into their systems or banks will integrate with them. One of the things that have been built into most systems is the ability to "correct" so called data-entry errors. This functionality is heavily abused with analysts revising bad calls to make themselves look better than they actually are.
One will find fairly large differences between data sets that collect analyst ratings and "corrections" as they happen (which you are going to have to collect yourself) and analyst ratings as reported by the various services and the differences almost always make the analysts seem more accurate.
It is possible things are different in Canada, but if you look at bias-free data for the US or Europe, analyst recommendations are mostly random. Analyst recommendation data is mostly useful so you can bet against the short term spike or drop caused by people reacting to a change in analyst recommendation. This was very effective up to the 2000s but doing this with a reasonable Sharpe Ratio today is very hard (otherwise I wouldn't write about it) though it may be useful as an add on to an otherwise profitable strategy.
TipRanks was founded (I'm no longer affiliated) to solve that. Analyst opinions are extracted from real world news rather than paid for.
The point was to hold analysts accountable for their actions by objectively collecting them. TipRanks sells this data and I'd love to run a study on it but unfortunately I no longer has access.
The Marketbeat data set used here is tiny in comparison and only consists of a third of analyst recommendations if I remember it correctly.
Bloomberg has a much better dataset which is nearly as good as TipRanks'.
Again, I can't really make claims on their behalf (I'm employed at Peer5 (YC@W17) at the moment and am no longer affiliated).
Also, can confirm that TipRanks never "correct"ed any errors - which often resulted in "cease and desist"s and legal threats. No analyst who loses money likes a site saying it like it is.
The legal threats held to merit - but to be fair they were very scary as a young startup before partnerships with E*Trade, Nasdaq and others started.
Yep this is like coming up with a stock algorithm that says "every tech stock goes up on average 20% in the first week of September unless its an even year and the stock starts with the letter T"
Obviously, predicting company earnings is not the same as predicting stock prices, but if sell-side analysts have such a terrible track record at the former , one would expect them to have a terrible track record at the latter too.
Outliers were removed to get a better measure of the "accuracy" of the price targets.
10 day windows were used to reduce the amount of volatility/noise in a time frame
Return horizons for 1 years was used because price targets are for one year.
Theres only 15 or so analysts I looked at.
I was doing this as an exploratory data analysis and didn't want to pull out my old stats textbook.
Cutoffs were chosen to reduce volatility of measurements since I was looking at percentages. A stock going from $1.5 to $2.0 is a 33% increase whereas the movement of $100 to $133 is significantly more impactful. Stock with lower market cap have more volatility. The minimum analyst rating was chosen to eliminate analysts with very small number of ratings as they would be unreliable.
Even though there may be some flaws in the way stock price basis etc.., the Pandas and python code and the methodology of calculating 'returns based on Analysts 12 month price forecasts' is very valuable .
One can easily tweak the code to suit their needs, I appreciate CODE Author for the effort . The Pandas and Python code can be used as spring board to further one's reserach
I've never seen a backtest that wasnt amazing. Problem is that there are so many errors, mistakes, statistical biases that creep into a study. Without a proper review by multiple people before publishing, chances or you will be plain wrong. I subscribe to the strong EMH. Only way to make money in the markets is to either insider trade (which will certainly land you in jail) or be a service provider.
1. Removing outliers was for making the data easier to analyze as some outliers were skewing the average. Of course when investing you cannot ignore outliers, but you could possibly curb them with stop losses/stop limits.
2. A more in-depth analysis could be done on analyst releases' effect on prices but assuming that this does occur, then the performances are understated and provides further evidence to the conclusion that outperform ratings can do better than the market.
3. Not sure how narrowing down the top analysts is a flaw here.
This blogpost is probably not as mathematically rigorous as it could be as I just wrote it as an exploratory analysis for fun and out of curiosity.
The problem is "making the data easier to analyze" may make the analysis invalid. Your response is not increasing my faith that you carefully considered what removing the outliers would do to validity of analysis.
> Not sure how narrowing down the top analysts is a flaw here.
Potentially, because how did you decide what "top analysts" were? If it's using the same methods you used to determine they were successful, it just means analysts that come out of your math come out of your math.
If a 50K people flip 10 coins, one of them might flip 10 heads. It doesn't mean that person is better at flipping heads. We could in fact calculate the chances of one of 50K people flipping ten heads. If I decided it meant that some people really were better at flipping heads, I'd probably be wrong. (Although if I calculated the chances and discovered it was like a one in bazillion chance that even one of 50K people would flip ten heads... I'd probably at least consider that they might be better at flipping heads! But I'd probably run the experiment again. :) )
If I pick the top 100 heads-flippers from my 50K coin flippers, and show that they really are better at flipping heads because they flipped more heads in the same dataset that I used to pick them as the top 100 heads-flippers in the first place --- I haven't really shown that at all. By "narrowing down top analysts", depending on how you did it, it's possible you simply found the analysts who got lucky, while ignoring the ones who didn't.
If 50,000 people each flip 10 coins, it's actually overwhelmingly likely that someone will get 10 heads. The chance that it doesn't happen is about one in a sextillion (10^21).
It's a completely different matter.
As far as I know analysts are doing their analysis when pricing stocks.
They may not be so good or discount all the factors, but analysing the products of some companies, their revenues, returns and other parameters seems extremely different than flipping coins to me.
So your analysis has no whatsoever basis given that you are comparing a completely random outcome of some well known physical action to a chaotic system (the market) in which at least the basics influence factors on his constituents are well understood.
Or are you suggesting for example that warren buffet is just being lucky for endless decades and you and everyone else know at least how to equate his performance?
If is that what you think then please, I would be rather amused to see your performance as an investor compared to him in the course of several decades.
Well, see, that's the whole deal, investigating _how_ different it is than flipping coins. That's the whole question, really. Starting with the assumption that they _must_ be doing better than chance is not the right place to start in order to analyze if they are or not.
Most statistical analysis is about trying to distinguish meaningful results (implying a repeatable correlation of some kind that means something), from random chance with no meaning. The whole point is you _don't_ start out knowing if the thing you are investigating is random chance or not, if you did, you wouldn't need to analyze it. That's what statistical analysis is for. In part because we humans are really really good at finding patterns and assuming a meaningful correlation when in fact it's just random chance.
The coin example is useful because we all know (or define for the sake of the discussion) that it must be random chance, so any analysis that appeared to say it wasn't is probably in error. And using the same sort of analysis on something where you don't know how much of the effect is due to random chance--is not going to answer the question.
I mention him because apparently for the parent message he is only a coin thrower and he will give us only insights on the percentage of people that can get 10 heads in a row.
If someone tried to use Buffet as an example of market beating but did zero statistical analysis to determine how likely it is to be actual skill they would also deserve to be dismissed.
I probably should have included the outliers when analyzing overall performance but if I recall correctly they did not have a significant effect.
The top analysts were determined by the average performance from one year after their ratings have been made. This isn't the top analysts out of 50,000 it's the top out of 50 or so analyst-rating pairs. There were only 16 or so analysts in total that I looked at. This isn't an instance of survivor bias as your example states. If I were to be more rigorous I could give a statistical test for this.
Looking for top performers is always invoking survivor bias. It's a classic data snooping issue where the common sense approach is exactly wrong, but it'll sell a lot of books and it'll convince people who you know what you're doing as a stock analyst when it's just random luck.
If you're ultimately making a claim that strategy X beats the market, you need to make sure strategy X is something you can implement without time travel.
For 2, it's not just the effect of the release on prices, but could also be the effect of other things on both releases and prices - imagine great news came out which bumped up the price and also caused analysts to upgrade their ratings.
You should use price data from the day after release.
For top analysts, if you only know who's top after looking at their performance, then it's again not a repeatable strategy. Compare: "you can beat the market just by buying the top 10 stocks!"
It's nice to play with data, I'm just laying out some of the reasons these won't work in the "real world", and pointing towards where a future analysis could be improved.
You're correct that I should have accounted for outliers when measuring performance of this strategy. If I recall correctly, even with the outliers, they did not significantly affect the average performance of analyst ratings. I would have to rerun the numbers though.
The analysis was more about measuring the performance of analysts which is why the price data for before and after the recommendation. For practical purposes of using this strategy, you are right that the price data from days after release would be better.
If the top 10 stocks you picked beat the market and have consistent earnings and dividends over a period of time, would this not be a repeatable strategy?
May or may not be related, but a number of years ago (< 10) Bloomberg started ranking analysts based on performance and that info is available on the ANR screen. Prior to that you had to do itself. So at some point that reference info was made easily available.
> Sadly earnings announcements have gotten a lot harder to trade on.
They're absolutely more difficult for reversion strategies, but if you're armed with more granular data of a high enough signal specific to the company you can do targeted trades very successfully.
Accurately forecasting the earnings results ahead of time on a per-equity basis requires more setup and data processing, but it is close to infallible when it comes to profiting on outperforms. Collect ultra-specified "alternative" data on individual companies with optimistic or pessimistic analyst forecasts, analyze it, then take a contrarian position if the data predicts an unexpected earnings result. This fails is in the case of extremely uninformed sentiment in the opposite direction of your own position (likely by unsophisticated investors not interpreting the results "correctly" en masse and opting for hype instead). But you can establish a win rate with a positive cushion.
One of the interesting things I've also done is collecting this data and using it beyond discretionary earnings trades. For example, by selling option premium for equities with very high theta, conversely stable revenue as forecasted by the data and low overall sentiment activity in the market. This is harder to pull off but it allows you to conduct a greater number of trades on a rolling basis through the quarter.
Scanning the article, it appears that the author is using two years of data in order to reach conclusions and this is because only two years of data was available to the author. If these same criteria were measured with two years worth of data ending on December 31, 2008, the results would look quite a bit different.
Unfortunately, the website I used to scrape the data only had 2 years of data. But each datapoint is looking at each individual analyst rating and price target for a stock and then comparing it to the price in 1 year. However, you are correct that datapoints during a downturn would be quite different.
Founder of MarketBeat (website mentioned) here. We do have more than two years of data (more like 7), but profile pages get too large when we publish it all in one shot. We can make the full history available as a giant CSV file if there's a researcher that wants to do a similar analysis over a long period of time.
Hey, I'm no longer affiliated with any site. If you're looking for data - I'd compare the TipRanks data set as well. I've found it to be a lot bigger than MarketBeat's and fairly more accurate. Data collection is automated through natural language processing and cross-validated with machine learning techniques that guarantees a much higher accuracy.
I'm not sure who you should reach out to there - but I think support [at] tipranks.com should work.
Things might have changed since I parted ways with TipRanks though - but you probably want both data sets for comparison.
So, until recently I was employee #1 at TipRanks who was founded to answer the question "Are Analysts bullshitting us?" - https://www.tipranks.com
TipRanks has loads of great and interesting information - literally what _any_ analyst ranks on _any_ stock since 2009 - huge amounts of verified data from multiple objective sources.
> You could possibly beat the market by only buying stocks with sector outperforms or buy ratings and selling in one year.
This is true in theory (and very easily benchmarkable with TipRanks). If you follow the top 25 analysts you can beat the market pretty regularly (easily confirmed with a regression test).
You might be surprised, but you won't beat it by that much and your beta will be higher.
Also, Marketbeat has maybe 1/3rd of all the ratings and a study based on their data would be worthless anyway. Basing a study on their bought data would not be indicative of a whole market trend. The data source is simply not a reliable one.
In general - be very wary of recommendations - I've not found analyst opinions a good indicator of market performance over time (and I worked in a company that ranked analysts for 5 years). I have found however that combining that with several other signals (news sentiment, bloggers, insider opinions etc) can create strategies that outperform the market (TipRanks sells such strategies to big organizations).
I've sent my old employer an email to see if they can open some data regarding this - and reproducing the above study with real market data.
One of the important properties of the stock market is that it is a chaotic system which changes based on observations people make of the system itself. This has fascinating implications.
To understand why this is important, compare the following: If everyone looks up at the sky and sees rainclouds, this does nothing to effect the likelihood of rain. The weather doesn't depend on what people think about it.
With the stock market things are different. If Tesla stock goes to $400, you might have a pool of people looking at charts, comparing Tesla's performance to it's 200 day moving average and deciding that this movement is "too quick" - so they start selling Tesla and thus change the stock's behavior. It is a self-reflective variety of chaos.
The same is true of stock analyst opinions: A publicly voiced opinion of a stock directly effects the stock because people look to analysts for guidance. If 20 analysts from important firms like Goldman Sachs appear on CNN / Bloomberg swearing that Tesla is an outrageous BUY stock, it will cause more people to buy the stock because analysts opinions effect the price behavior.
Long story short, what the author may actually be observing to some degree is that analyst opinions may CAUSE stock outperformance.
My dad had an old book on technical analysis, published in the early 60s. The intro talked about how prices move, usually with top execs telling all their friends and family to buy, and then those people telling their network, and so on, ending with a press release. I'm sure there's still plenty of that going on, but I found it funny to see it just spelled out as the default mechanism 50+ years ago.
The dynamics of the stock market are even more complicated. Say TSLA does hit the $400 then you'll see a bunch of sell orders hitting month old targets. Certain dates when a lot of short/long contracts are resolved (Sept 15th is the next I think? and there is a bunch of call options with a $400 base price) can somewhat have an influence. And of course with a stock like TSLA there might be a lot of people being in the short doing everything they can to get the stock down to make their profit. Analysts are such a tiny part in this game.
Right now the whole tech industry is overheating, just like the dot-com bubble back in the days.
In my analysis, I do hypothesize that analyst opinions become a self-fulfilling prophecy as you described. However, I would like to believe that analysts do some sort of sophisticated breakdown and analysis of a company's financial statements and market outlook when releasing a justifiable rating.
I suspect the model comes from overfitting the training data, that is, you would not see expect to see 15-20% outperformance from the same group of analysts next year. Or BigBankCo would immediately hire them to invest their cash.
There are a number of other flaws that require time traveling to make this scheme work, and if you have a time traveling machine, beating the stock market is even easier than this.
They beat the market until they don't. Patterns that generate better returns don't last. In this case, everyone sees that outperform rated stocks (OSR):
- OSR yield better returns
- OSR demand increases
- price of OSR goes up
- higher prices yield lower returns
- in near future, non-OSR have more attractive returns
Careful going from programming to investing. In programming you have a confidence & that makes sense because the results are often up to your own skills. With investing.. there is so much that is beyond what you know. Things go against you all the time. There is all the finance stuff. But then there are mechanics to the market that can surprise you. Triple witching? (etc etc) It's often not about the fundamentals of a company but what the perception or the trend is.
If I could give myself advice a year ago it would be to start with a paper account, & seriously try to make "money" that way. It requires a ton of research & time. Everyone else is trying the same thing so not only must you figure out which company is under or over valued, you must figure it out before others do. People with entire staffs doing just that.
As programmers, it can be a good idea to stick to software & service companies, because we have a leg up on the understanding of how big those things can get. A lot of investors fail to realize how software companies can grow.
Check out Square.
Square has some of the best programming talent around. They have a Point of Sale system but are also expanding into small business services (like running payroll). I imagine they will tackle inventory, maybe wholesale ordering, do they have a loyalty system? They are using AI to choose who gets small business loans. They offer loans that are smaller than a typical bank would, thus carving out a new niche. As long as they don't hand out too much money to the wrong people, I see this business growing massively. No other software companies seem to combine a great UI, with a mobile / tablet focus. The moat is just once you get set up with it, it'd be a pain to switch out. Curious what others think. (hijack the thread!)
It's my top pick at the moment. I saw it at 9 a share last summer & did not invest enough in it. Even with a market cap of 7 billion, it would seem that it could fit those shoes & then some.
If I could give myself advice 20 years ago it'd be 70/30 (world) index/(world) bonds and keep doing so. Go a little heavier (80/20) on stocks in times of crisis (>20% correction). That's all. You can just get lucky over a year and confuse that with skill.
This is what I've basically gone with and I've had good results so far. I basically wanted the most return with the least effort, so I just went for index funds with the lowest expense ratios.
"""
There are some potential advantages to the share class structure of the Vanguard S&P 500 ETF. Although VOO is only a fraction of the size of IVV and SPY, VOO offers the lowest expense ratio of the group, charging just five basis points.
"""
http://etfdb.com/equity-etfs/closer-look-at-sp-500-options/
I get that SPY is better if you need to liquidate millions on a moments notice, but I'm definitely not there, so the nearly-double expense ratio didn't make sense IMO.
The tough part will be staying the course if the market crashes, which it will at some point. It'll look like the world is ending and you should be investing in gun powder, canned foods and gold. You need to be comfortable with the idea that it may go down (even 40%-50%) and take a long time to come back up but you can keep buying through that and also get dividends...
It's not easy to tell whether the market is overvalued or not. Right now if you believe interest rates and inflation will go up within the next few years to anywhere close to historic norms then the market is probably on the hot side. But if rates stay low for very long durations then the market isn't hot. Also international markets have lower valuations than the US so you get something out of diversifying there. You could consider a little less stock in your mix these days but shorting the market would be an extremely risky proposition - more of a gamble. In the long term even if the market is hot and corrects you're still going to do OK.
Would be nice to correlate analysts ratings that are similar to rare earth or chemical correlations to public companies here "Profiting from Python & Machine Learning in the Financial Markets" https://hackernoon.com/unsupervised-machine-learning-for-fun...
This is a great exploration of something many folks wonder about. Kudos to the author, especially in realizing that target prices aren't a great variable to use when optimizing investments.
1. "We will do so by removing outliers in the 10th and 90th percentiles." you can't remove outliers when investing unless you have a time machine
2. Their "starting price" includes an average of 10 days, 5 of which are before the analyst releases the recommendation. Again, to buy before the release requires a time machine (or inside information). Suppose a stock is at $50 for 5 days, an analyst releases a $75 target, stock jumps to $60, then hits $70 over the next 12 months. The gain based on the average of $55 would be $70/$55= 27%. But the average based on what price you could actually buy it at is $70/$60= 16%.
3. They narrow down analysts apparently a second time to pick out only the top 10, which seems to be distinct from the first outlier removal (not entirely clear what's being removed in either case.)
I see no statistical tests relating to removing outliers, significance, etc. This is all besides the considerable degrees of freedom (cutoff price, cutoff marketcap, 100 rating minimum for analysts which seems to have been implemented after collecting data, etc).