sigh i got really excited about this because I was considering the interpretation of the word "hackers" like we use it in "Hacker News" (relevant: http://paulgraham.com/gba.html)
I know that the startup community at large doesn't largely respect short term trading, particularly HFT. I understand it, though I honestly think its misguided and the outrage is disproportionately large, but thats another story.
The reason why this article TITLE excited me was because I think its totally true, or least close to true. I think awesome HTFs and algo traders have a very similar skillset to top notch startup hackers.
Small prop shops ARE startups. You:
1. wear a lot of hats
2. release early, release often
3. write really fast code, really fast
4. validate theories against the market (or consumers)
5. if you're right, keep going, if not, pivot
6. success is dependent on understanding lots of different sub-specialties
7. lots of overlapping skills: big data, distributed computing, machine learning
At some point, I'd love to get a bunch of hackers together and start a small prop shop. The jobs are different, but in so many ways it takes the same kind of person to succeed at either.
Oddly enough, I was originally expecting that same thing.
Still, I think it's reasonable to ask the question: where does the boundary between good, useful hacking and evil, insidious hacking lie? And I think it's reasonable to not assume that the boundary lies at what is legal or that the boundary lies at what is "interesting".
I mean, how you produce and profit from spam is a hard, interesting problem. Flame and Stuxnet might, in some courts, be legal. In fact, a lot of "Black hats" satisfy many if not all of your list items.
The argument that HFT is working with a nearly zero-sum situation needs to be addressed. A "normal" startup is usually trying to create "new usefulness". Where do "props" fit in to this question?
Trading is a zero-sum game, but investing is not, and trading exists to make markets and snipe inefficiencies. The amount of money that can be made here can be surprising, until you consider that they're oiling the gears for manifold trillions of dollars in commerce and wealth. Some of that is gambling, but most of it isn't. Stuff like institutional hedgers and investment funds are huge sources of market activity.
It's is very hard to see how predicting short term market position unrolling by individual institutions is making a market or sniping inefficiency. That is an arms race that creates a minimum cost to enter to compete.
When success at sniping depends upon privileged network access, both human and technical, it's more a matter of profiting on an inefficiency you enforce upon everyone else.
I want to make an important note here: trading is not a zero-sum game. It generates value through the exact mechanisms that you noted, market making (liquidity) and inefficiency elimination (accuracy). Via the efficient market hypothesis, the value created by adding this asset liquidity and price accuracy finds its way back into the hands of the traders, often through real value increases in the assets going to traders with call positions.
This functions to keep demand for trading moving by making the average market return stay close to or above the market discount rate. So stock trading doesn't end up as zero-sum, but rather benefits parties at the market discount rate. If it didn't, no one would trade but those seeking to gamble, and there would be no notion of fundamental analysis in the market.
In reality, outside of an ideally efficient market, it's easy to intuitively suspect that the average return associated with trading is above the discount rate, and that more savvy investors obviously earn higher returns. This is more or less correct: the S&P averaged around 8.5% annual returns over the last few decades [1], and that only increases the farther you go back and lower the more recent your range's early bound is, corroborating the association between unideal market efficiency and high returns. Both "investing" and "trading" are less distinct in the stock market than in, say, VC, and so that 8.5% applies equally in aggregate.
But trading is most certainly not a zero-sum game. Like all market activity, it converges to producing value at the market discount rate in an efficient market. What that rate is is incredibly hard to say, but it can vaguely be associated with S&P returns, which have hovered around 5-6% in recent years on time-weighted rolling averages.
I think the zero sum nature of HFT is already providing an answer.
There is so much competition in the U.S. that HFT firms are having to expand to other markets like London and beyond that don't have the same level of saturation.
There are only so many markets in the world, so in about 10 years (probably much less) when they are all saturated with very low margin HFT, equilibrium will be reached firms will simply evaporate and/or glom together.
I find HFT is a lot similar to SEO, and the perception is similar as well.
People just don't like either because they feel it's not bringing any extra value to society, and it's all about gaming something for your own personal wealth. But most people aren't looking out for your personal interest anyway, so you should ignore them.
The thing is that "good" SEO isn't gaming the system but rather working to make sure that your site is a "notable authority" in whatever niche you're targeting. If the article is the counter-argument, what is the argument that HFT provides some similar kind of usefulness.
I disagree. SEO serves a purpose. It makes web sites more accessible to users, by correctly using HTML (title, h1, alt="", etc.) and it makes the web more machine readable and thus easier to find useful things. But I guess you mainly refer to the spammer SEOs that try to trick Google.
HTF otoh does not serve any purpose whatsoever. On the contrary, it brings instability into the market and makes the market less useful. But my guess it that it will need a major market crash before HFT gets banned.
No. The dominate strategies of HFT firms are market making and related arbitrage such as latency arbitrage between venues. There is nothing nefarious about having a computer do this in general as it is much more efficient than a human doing this. These strategies are not very complicated and instead require a lot of infrastructure to run. It used to be you needed to buy or rent a floor seat at the exchange to be able to run these types of strategies. Today that money goes to servers, hosting and engineers.
Automated trading, such as statistical arbitrage would be lower frequency, although often still high frequency by most peoples standards. This algorithms often are liquidity takers from HFT.
The problem with HFT algorithms in the eyes of most hackers is they are almost entirely a zero sum game. Assuming minimal volatility and free trades the absolutely dumbest algorithm* will make money.
*Buy at X sell at X + 1cent with a simple ratchet of buy orders from 1 cent up to market price + a small number of shares to be sold if the market price increases.
PS: Random walk that averages +/- one cent 10 times a second = 5 cent's profit a second assuming ~2,000 trading hours a year = 360k profit. Cost to make that for a stock worth ~20$ = ~30,000$ of capital + whatever it takes to get on the HFT floor.
Everything matters if you're doing HFT (or back in the day, high speed algorithmic trading). In Australia, the support for algorithmic trading means you're required to house your server in ASX itself (ASX has recently introduced their HFT platform in 2011). And that meant ball-shrinking dollar values from place rental alone, not to mention the hardware and software times.
And that meant ball-shrinking dollar values from place rental alone, not to mention the hardware and software times.
I got the first part of that, but what do you mean by "hardware and software times"? Yes, HFT relies on fast hardware/networks/software. Was that your point, or is there something I'm missing?
FWIW, despite a strong entrepreneurial bent, I would (possibly even will) look to join an existing HFT operation, rather than start my own.
(Unless you want to do it in the Bitcoin world - there is a Bitcoin stock exchange now and it's exciting times, my friend!)
Exactly my thought - I guessed that MC was praising us. Sigh, you'd think we HFT practitioners were, I dunno, Goldman Sachs or something like that. ;) (jk, GS, you know we love you ...)
Good list, also. I work with some really smart programmers and tech people and HFT is a great laboratory to work on tough technical problems in the real-world, with real results.
For a corporation, the psycopathic behaviour is that if it makes a profit and is not illegal (yet), then do it.
The real problem is that large financial institutions have somehow ingrained themselves in society in such a way that they bring us all down when they fail, and so society would _have_ to bail them out. It is an ingenious stragegy - by aligning failure with the downfall of many others, these large financial institutions gets the protection they don't deserve. I wish it was me who was reaping the benefits, but unfortunately, being a "commoner", that aint so.
It's interesting how many other areas of the economy have pulled off the same hat trick, but largely go unnoticed. Regulatory capture is a good marker because it's as conspicuous as it is essencial.
I have no problem with HFT, but only because it is turning machines into things that are doing what humans did before, only at a speed that we can't keep up with as humans with our inherently physical interfaces.
How is HFT different than traditional arbitrage except for the fact that it is faster and potentially at a scale that normal traders would never be able to keep up with? How is buying something one second and selling the next inherently different than buying one day and selling the next? Speed?
Does computers being faster make it any more unfair than when one trader has access to millions or billions of dollars when I only have access to hundreds or thousands? I don't think so.
Now, one could argue that HFT disrupts the way that Wall Street has worked for decades, but really, it is shifting the profit from traders, bankers, and fund managers to HFT firms. Sure, Wall Street probably doesn't like it, but does anyone like it when someone disrupts their market and takes their profit?
You could also argue that HFT doesn't add value because HFT firms inherently don't do anything other than trade millions of stocks only for the sake of trading them, but how is this worse than day trading or running any other fund other than the turnover rate?
Perhaps I'm blind to the real issues, but I just don't see a huge difference between HFT and traditional Wall St. trading, except maybe that it magnifies the problems inherent in the system due to speed and scale.
The real issue is that in order to get an edge, HFT traders are engaging in electronic warfare in order to slow down their opponents.
One of the worst things they are doing is "quote stuffing", which means submitting bids and cancelling them tens of thousands of times per second. They stuff the exchanges with quotes and eventually cause a huge latency with respect to the quotes that are shown by the exchange, such as NYSE, NASDAQ, etc. However, they aren't affected because they can see the prices because they have a raw data feed. This gives them an edge relative to the other traders that don't have their own raw feed straight from the exchanges, and they use that to basically shoot fish in a barrel.
This was one of the main causes for the flash crash in 2010. The excessive quotes
The thing is that they could stop this by implementing a minimum time for a quote to exist, something innocuous like 100 ms, but they don't because this would be a detriment to the HFT shops.
The other thing is that HFT shops behave like market makers, and they claim to provide liquidity, however, they have no obligation to provide liquidity, like real market makers. So just like during the flash crash, they turned off their machines, and the markets went haywire. So they want their cake and eat it too. If they are going to trade like market makers, they need to have the obligation to continue to provide liquidity.
If I recall correctly, wasn't there, like, an extremely fast recovery from the flash crash? Why does the flash crash even matter? (I'm not being snarky at all, BTW.)
A lot of people lost a lot of money during the flash crash, through margin calls, stop loss hits, etc. It also called into question the stability of our markets. If people lose confidence in our stock markets, then people stop trading on them.
If we have another flash crash, you can bet that the already-low volumes on the markets will be met with even lower volumes, which could damage our economic recovery. Most peoples' 401ks are tied to the stock market, and the last thing we need is a generation of un-retireable people because they lost a lot of their retirement on the stock market.
A lot of people lost a lot of money during the flash crash, through margin calls, stop loss hits, etc.
Well, if day traders lost money, who cares? That's the business they're in.
It also called into question the stability of our markets. If people lose confidence in our stock markets, then people stop trading on them.
True, but there was a fast recovery, and people are still trading. You mentioned that volume is low. I wasn't aware of that (just don't follow things that closely). Does that correlate to when the flash crash was? I'd be surprised if the flash crash rattled people so much that it still has an effect on market confidence.
Most peoples' 401ks are tied to the stock market, and the last thing we need is a generation of un-retireable people because they lost a lot of their retirement on the stock market.
Well, investment managers who control people's 401ks should not be trading during a flash crash. I mean, I'm just not so sure that it affected everyday people much.
Also, there are a LOT of things to blame the current economic scewing-over of people on, and the market would be near the bottom of my list.
It's not the day trader losses we're worried about. It's the average investor who had trailing loss orders in on P&G (just an example) who had to sell their positions at steep losses only to see the market rebound immediately leaving them with zilch. This attitude of "If day traders lost money, who cares" is missing the forest for the trees. Lots and lots of people got screwed by the flash crash and that starts to cause credibility issues.
Serious question: Aren't stop-loss just another form of algorithmic trade? I understand their purpose (in principle), but perhaps its not an intelligent move for a long-term investor...? (Or to put it another way, perhaps the risk mitigation aspects of stop-loss trades are not properly valued.)
That's what a stop loss order does, it gets you out of your position regardless of the price. When P&G was dropping like a stone there was no way to know if the market was going to go straight back up or straight down. If it hadn't been a 'flash' crash but a straight up crash, if at that moment North Korea had invaded someone those average investors would be thanking their lucky stars they got out at only a 20% loss.
You can't have it both ways, either you set hard limits for your position and take the hit when you guess wrong or you let your position ride and risk losing everything. The crazy prices were busted after the fact anyway so if you had a stop loss further out than 60% of the pre-nonsense price your trade never happened.
I know the exchange can automatically put a halt to the trading of an asset, and that seems (to me) like a good tool in this case. e.g. if there is a certain pre-defined amount of volatility in a certain time. Do you think that would address this concern?
Definitely an avenue for exploration. I'd argue that the exchanges have been doing this over time, but that market manipulators will always move faster than the exchanges can tune circuit breakers and other mechanisms to prevent manipulation. Folks who stand to benefit from the type of volatility embodied in the flash crash will always be able to outrun these sorts of changes.
Oh, also don't forget that the exchanges are subject to regulatory capture from the firms who benefit from real investors losing money to market manipulators. It's not like Goldman Sachs is going to be pounding the table for the NYSE to make it harder for HFT desks to manipulate markets.
Oh come on. Anyone who understands even the basics of equity trading should know that trailing loss orders are no real protection. That's been well understood since at least Black Monday 1987.
This is something people miss entirely in this argument, the fact that lots of investors have trailing stop trades in and in a flash crash, those people get their asses handed to them. Then we immediately start having a credibility problem. Once you have a credibility problem, the end game is getting close. When you combine that with things like the MF Global bankruptcy where client accounts were stolen from, you start to think maybe the game is rigged. Or at least I do.
Exactly. $1.6 billion lost from customer accounts that were not supposed to be touched, and no one is arrested? It just evaporated into thin air? The entire thing is disgusting, and I pulled out most of my money from my futures trading account exactly because of this, because I didn't have confidence that I would see my money again. Hedging through futures is something that many commodities producers do in order to have predictable revenues, and they just lost their shirts because of criminal fraud, and nothing is going to be done.
I don't really understand how you could abuse an exchange through quote stuffing - I've heard(and it seems reasonable to me) that they would use round robin queues to protect from exactly this sort of thing.
If you have a 401K or own shares in a mutual share, you should care about it.
The HFT algos siphon off billions every year from the big, slow "dumb money" large institutional investors (e.g. pension funds, index funds, insurance companies).
For example, if an insurance company needs to liquidate its portfolio to pay a claim, the HFT will sense the insurance company placing an order in the market and try to skim 5-10 cents per share by manipulating the share price during the trade. Its a zero-sum game.
The insurance company lost and the algo won. Market markets exist to provide liquidity (match buyers and sellers), not to front-run and bid up orders from customers.
First, HFT is very difficult. It requires the talents of extremely intelligent individuals. These individuals are a scarce resource. Now, being as they are near the generation and handling of money, HFT firms are best placed to outbid all others for these people. So R&D and other such lose out on top minds. Additionally, if much of the money of high net people is being invested in funds without finding its way into risky high tech research, this too can be a drag on the economy in the long term. http://sirc.rbi.org.in/downloads/4Cecchetti.pdf
The second is a problem of misaligned priorities. It's one thing to suck in all the best minds and it's another to have them all expending a massive amount of effort trying to shave down microseconds so their order goes in first. Chris Stucchio here http://www.chrisstucchio.com/blog/2012/hft_whats_broken.html argues very well that the so called sub-penny rule (abs(changeinprice) >= 0.01) is antiquated and from a time when humans reigned. This rule is a source of friction for the bots and is very likely a key player in the current misallocation of talent. It's removal would make HFTing cheap enough that not so many people would be needed. Some would leave to cure cancer or mine asteroids. The rest could then work on more sophisticated pricing techniques - which is a net boon, maybe those techniques could make prices more stable or be reused for optimal donor matching or something.
It's at least conceivable that the correct valuation of a company changes from one day to the next due to human-scale events, and that speculators are providing a useful pricing signal. Changes from one millisecond to the next are amplifying pure noise produced by flaws in the mechanics of trades reaching the market and clearing.
To increase speed, you must decrease computation and memory access. The less computation and memory access possible, the smaller the variety of strategies possible. In regular timescales, the strategy space is much larger than the number of machines / agents competing. The ratio of the number of agents to the number of possible strategies approaches zero. As more machines compete in nanosecond time scales, the ratio of the number of machines to the number of strategies in this restricted space becomes much larger.
As this ratio increases, the system dynamics undergo a phase shift. The probability of dramatic movements becomes much greater. When too many agents are crowded onto the same strategy, they all try to take the same side of a trade -- liquidity disappears. This was the cause of the flash crash of 2010, and will be the cause of many more.
What exactly is a flash crash and why should individuals care? Isn't it just bank's computers temporarily offering stock at a severe discount and then the price going back to normal. It doesn't hurt anyone but the people with poorly programmed algorithms. No trades should ever be broken just because someone can't control their trading bots.
You're describing the stop-loss orders that non-professionals use to protect themselves from losing money when the fundamentals of their companies deteriorate suddenly. A stop loss is a very simple algorithm that is widely available in retail brokerage platforms. (Example stop-loss order: "Sell when the last trade is 3% below my purchase price.")
For instance, you own 100 shares of Pepsi and plan to hold for a while (years, not days). Tomorrow while you are in a meeting, a large-scale accounting fraud at Pepsi is announced, sending the stock down some large %age before the circuit breakers halt the stock. Prudently, you had a stop-loss order limiting your losses to (say) 3%. In this case, the stock probably won't recover when the stock starts trading again.
If the stock had dropped and suddenly recovered due to a non-news flash crash, you would have just lost 3% of your investment for no fundamental reason. The flash crash in essence caused you sell at a localized bottom.
Except that in the real world your stop loss order for PEP probably wouldn't work under that scenario. The price is likely to instantly gap down further than -3%. Or trading may be halted immediately before your order is executed at all. When everyone rushes for the exit at once no market is going to be orderly.
> The price is likely to instantly gap down further than -3%
Right, but in a crash due to news, it is likely to go down and stay down, or at least take some time to recover so you can manually make a decision. In a flash crash it can go down 50% and recover within minutes.
They are different scenarios, and they behave differently.
>>According to Information Week Magazine¹: “A one (1) millisecond advantage in trading applications can be worth $100 million a year to a major brokerage firm”. Currently electronic trading makes between 60% and 70% of daily volume of NYSE¹. Tabb Group, a research firm, estimated that High-frequency traders generated about $21 billion in 2008.²
From a site promoting microwave communication towers for low latency packet transfer.
Why do we need day traders? Well, they can only exist because the investors exist, and they provide the function of pricing the market accurately. And provide liquidity to investors. Nobody is arguing that we should ban day trading.
Why do we need market makers? Well, they definitely benefit the day traders, because they do lower the bid/ask spread, once again making pricing more accurate. And provide liquidity to day traders.
So what I'm proposing is that we only "really need" investors - but day traders are good for investors, and market makers are good for day traders. Thus, market makers are good for investors.
I'd be legitemitely interested to hear criticism of the above argument. I've been learning about market making recently, and I'm certainly sure that there's still more for me to learn.
I don't disagree in broad terms, but I think it's also quite possible that some configurations of trading will result in the "support" tail wagging the investment dog. At least, that would be the expected result if you think of markets like a dynamical system; in non-trivial dynamical systems (which I'd assume the economy is), the usual case is that there are significant internal dynamics (feedback loops, attractors, etc.) that in a number of cases will modulate, and may even dominate, the movements caused by external inputs. If that transfers to trading, there may well be pathological market dynamics that have little to do with the fundamentals of providing services to (ultimately) investors. Whether the dis-value of those pathologies produced outweighs the value of the services provided would then be the relevant question. Viewed as a machine, does the machine that includes the HFT lubricants run better in general, or is the reduced friction offset by a undesired effects such as greater risk of malfunction?
"As far as narrowing spreads, that’s absolutely true, but in absolute terms what does it translate into? For the individual investor it might save them a quarter a month."
In a properly designed, information age stock market there should not be a spread. All stocks should trade via a programmed, black box auction that runs on an interval. The HFT practice of creating phony orders that are immediately canceled, just to gain visibility into the current bids and asks, would be eliminated. No seeing other people's bids, and then front running them, no canceling orders. You put in a limit order for the value of the stock, and the stock goes to the highest bidder. All the extra pennies and quarters go to the shareholder, nothing to the HFT algorithms (unless they provide actually value such as market making or smoothing irrational volatility). It's the most efficient design for stock market trading possible.
In theory, if this system were to be adopted, the Stock Exchange Operator could charge a small tariff so that some of the money that now goes to HFT's goes to the stock exchange, with the rest going to the shareholders. So in theory, adopting this more optimal system would be financially beneficially to both the corporations and the Stock Exchange (and in theory, those two are the only two players who's agreement is needed to change the system). So I wonder why the optimal, auction based system does not get adopted - is there some reason for the status quo other than the normal inertia and stupidity of large institutions?
Not a good theory. All the players in this game believe they are above average among players in this game.
They all want the system to be exploitable because they all believe they will exploit it better than their peers.
At the very least better than the investors (the real suckers here).
You are falling into the game theory economics trap. People are highly irrational actors who don't even act in their own self interest most of the time, nevermind some form of optimal rational behavior.
You are falling into the behavioral economics trap. In general, the big movers and shakers in financial markets act in quite economically rational ways. There is a natural selection process at work where the people who are good at using their brain to accumulate money, accumulate money, and thus have more impact in the market. If you look at many cases where wealthy market players appear to be acting irrationally, what you'll often see is they are acting in a personally rational manner, but are operating in a context with very perverse incentives. So I am curious if anyone in the HN commentariat knows the specific perverse incentives involved that prevent this more optimal system from happening. ( Joe Retail investor can be quite irrational. But I'm wondering why the Stock Exchanges and big corporations accept the status quo with regards to HFT.)
"In general, the big movers and shakers in financial markets act in quite economically rational ways"
They are certainly more intelligent, knowledgeable and invest more rationally but I've seen nothing to make me believe that they lack or compensate for the types of positivity biases that makes everyone think they are above average and can beat their peers.
We weren't talking about the contexts of the investment choices they make (where I mostly agree with you) but about changing to a system where the winners can't exploit the system to extract far more money than the value they put in.
If they are A-type human beings they have a positivity bias if they have a human brain. You don't need perverse incentives to keep a system that over-rewards winners if everyone thinks they are, or soon will be, the winners.
That said, I would also be interested in knowing about any perverse incentives here, I just don't think they are necessary.
How would this be better than what we have now? Right now stocks already go to the highest bidder - if there is a seller willing to sell at that price. how would you run an auction if there are 40 firms trying to sell the same stock?
The ad market is well suited towards auctions because you have one seller for many buyers, but in finance you have a symmetric relationship between buyers and sellers.
Google "HFT frontrunning". The issue is that HFT algorithms can use their superior speed and the ability to cancel orders to essentially figure out what institutional buyers are bidding, and then front run the market to buy up shares in front of them. This would be prevented in my system by the black box auction and trading on an interval.
There are many opportunities for institutional investors to trade in exactly the manner you suggest. NASDAQ runs the opening and closing cross as well as other intra-day crosses. Other platforms allow private crosses, dark pools allow crossing not in public and today 1/3 of all shares are crossed internally on brokerage platforms. Institutional investors have many opportunities to trade in the dark. There is a price to be paid for instantly accessing liquidity in the market, and it is much cheaper than it used to be.
This is not efficient. If you use interval bidding, there will frequently be gaps in market prices. Remember that at any instant, the order book (bids and asks) represents only a small portion of market sentiment (real demand and supply). The market prices move smoothly when people are able to react to order changes and price spikes. If it's a black box, it's very easy to manipulate the market in a massive scale.
For example, AAPL is $600 now. Under your system, $600 is the price that generated the highest volume in previous bidding event. (There will be sellers placing orders under $600, and buyers over $600 - they'll all be settled at $600 and the exchange obviously wants to maximize this overlap, hence the price). If a large institutional investor wants to manipulate the price, he can place huge buy and sell orders at $700. And very likely in the next bidding event the price of AAPL will move to close to $700, because that will be the price that maximizes the overlap. This is almost impossible in current market system, because you have to continuously buy from all the sellers who are willing to sell at $700 to get the price. Meanwhile, traders (especially day traders) will add to the order book to take advantage of this irrational move. In a black box, irrational moves can't be detected and corrected by other market participants. So you can't really get the idea of the actual matching price in a black box system.
When you place a buy order, if there's overlap with any sell orders, it should be matched and executed instantly. Otherwise it will stay in the book to wait for sell orders. The market equilibrium is reached when the spread is minimal and no trade occurs. (Yes, this is in theory.) However, in your "efficient" system, there's no market equilibrium, because buyers and sellers are discouraged from making their real demand and supply invisible. The exchange has the incentive to maximize trade volume, so the only way to result in zero volume is no overlapping orders in the entire interval, even when the buyers are sellers have no knowledge of any orders. This only happens when people have no confidence or interest to trade at all. Again, it makes market vulnerable to be manipulated.
In China, stock exchanges partially adopted your idea. In Shanghai Securities Exchange, the first 10 minutes (9.15am to 9.25am) of the day is set to be non-continuous bidding period. In Shenzhen Securities Exchange, the first 10 mins and last 3 mins are the same. This was introduced to reduce market manipulation because the order book is empty at the beginning of the day. However, after the period, trade starts continuously and all remainder orders in the bidding event will automatically enter the order book. This way, the price movement of the whole day is smooth with minimal spikes, while the opening and closing prices reflect market sentiment more accurately.
Generally, the more the visible orders, the more smoothed the price. Interval bidding discourages order placement because the next price is absolutely unpredictable. It can be adopted in extremely illiquid market situations though.
This is why we have Game Theory and Mechanism Design. Mathematicians and computer scientists study the different types of auction methods for their properties. Let's take a look at your example:
AAPL is $600 now. (...) If a large institutional investor wants to manipulate the price, he can place huge buy and sell orders at $700.
All your example shows is that a good auction mechanism should avoid this possibility of manipulation. But that's easy enough: the investor's buy at $700 will be matched with sells that are closer to $600, the same as in the current system. The sell at $700 doesn't affect anything.
So in fact the price will move the same as it would in continuous buying.
Not exactly. In a black box, there will be less orders because market participants are discouraged to be market makers, because the point their orders have been triggered is also likely the point the market deviates significantly and they have no chance to cancel or stop loss immediately in a non-continuous market. Market makers are resistance of such manipulation. And most market makers place orders to fill the "gaps" in the order book. Without a visible order book, it's much more risky to provide liquidity without any actual demand or supply for the shares. I think it's okay to assume that less than 1% of traders place orders based on real supply and demand. You know you would sell AAPL if it spikes to $700 today, but you won't place an order until that happens. (The financial market is an incomplete coverage of demand and supply anyway, so the more liquidity, the better.)
AAPL is $600 now. (...) If a large institutional investor wants to manipulate the price, he can place huge buy and sell orders at $700.
To what profit? The investor would lose their shirt. Let's say period A had 20 shares sold and 20 shares bought at around ~$600. Period B the investor enters with 50 shares being sold at min price $700.00 and 50 shares being bid on at max price $700.01. In period B there are also another 20 shares from other people being offered at min price $600.10 and 20 being bid on at $600.00. The auction runs and the most seller advantageous clearing price is $700. The institutional investor gets 20 shares from the $600.10 but at the clearing price of $700. The investor exchanges 30 shares between his right and left hands. So overall, the investor has on net bought 20 shares at a ridiculously inflated price. This was a stupid, money losing strategy for the investor.
You know you would sell AAPL if it spikes to $700 today, but you won't place an order until that happens.
Why not? Under my system, at every interval you could simply put a limit order in for selling at a minimum $700. In fact, I suspect a few hedge funds would pop up that specialize in figuring out a true and accurate price of a stock according to the fund's analysis, and then placing, constant, across the board limit orders that would automatically snap up shares in the case of irrationality. If the market was as jump and irrational as you think it would be hedge funds would make a killing by being smart and rational, until enough entered the market with standing limit orders that the price smoothed out.
Without a visible order book, it's much more risky to provide liquidity without any actual demand or supply for the shares.
And yeah, under my system market making actually requires work/risk, not just riskless front running. The free lunch is gone. That is the point. There would be less volume. Buyers and sellers would trade slightly slower trade execution (waiting for the auction interval) for the benefit of not paying any tax to market makers.
> And yeah, under my system market making actually requires work/risk, not just riskless front running. The free lunch is gone. That is the point. There would be less volume. Buyers and sellers would trade slightly slower trade execution (waiting for the auction interval) for the benefit of not paying any tax to market makers.
The spread earned by market makers is definitely not a tax. They earn the money from (increased volume * reduced spread). Suppose you anticipate AAPL will rise from $600 to $601 based on your fundamental/technical analysis. If there're no market makers, the wouldn't be enough liquidity to inspire any confidence to take the trade. The order book would be like:
601.00 3
600.50 1
5 600.00
2 599.00
There's no point to make that $1. It's simply too risky. If there are a buyer and a seller they both want to trade 1 share instantly. They pay $0.50 in total to liquidity providers in the market (compared to private settling).
If there are lots of market makers, the order book would look like this:
600.10 950
600.05 401
1200 600.00
450 599.95
Of course, you will choose to trade and make the $1 if market moves as expected (and a lot of people will make similar decisions). At this time, if a buyer and a seller comes, they will pay $0.05 in total to market makers compared to private settling.
I know that your original intent is to make the "private settling" option available in the market by aggregating all orders in an interval and execute at once. If that really happens, the order book (which is hidden from public) will look like this:
600.50 2
600.00 3
5 601.00
2 600.00
Execution price: $600.60 (Weighted average of overlapping orders). Volume: 5 shares.
In this case, it seems that both the buyer and the seller received a benefit. But actually it's not true. If your aim is to make profit by selling at $601, rationally you will keep buying until the price reaches $601. The maximum price you're willing to pay is actually $601. However, when you see a selling order at $600, you will place a $600 buy order instead (and you pretend to have a demand only at $600 or below because you know it will be fulfilled anyway). In a non-continuous system, you're forced to signal your true demand at $601 so that you are able to take advantage of favourable prices when you're lucky (and orders get executed only when you're lucky).
> If the market was as jump and irrational as you think it would be hedge funds would make a killing by being smart and rational, until enough entered the market with standing limit orders that the price smoothed out.
They are exactly market makers. The market will be "jump and irrational" without these market makers, especially when everyone signals their true demand. That's why black-box auctions usually yield much higher prices than public auctions when people are rational (i.e. not counting the emotional effects of public bidding). It's just how market works. Similar concepts can apply to free rider problem as well (for public goods). You know that the national military can provide you security worth $1,000 a year, but obviously you pretend to be unwilling to pay anything when the service can only be provided for free. Market equilibrium price quickly reaches $0 with no guess work.
The way market works makes the prices very predictable. Even the flash crashes are smooth (with the market makers). The "riskless front running" is a symbol of market competition. Yes, some guys are going to offer you one cent better, they should have the priority in the queue.
I don't want to comment on the influence in economic activity. What I know is, more liquidity = less risk for holding shares. What market makers earn is not a tax. It comes from the money that bigger market makers will earn anyway ($0.01 spread with 50 shares traded vs $0.50 spread with 1 share traded).
You might have different auction interval times based on the volume of the particular market or stock. An AAPL auction happens every minute, but some low volume securities might trade once an hour or even once a day.
I keep hearing about HFT, and have read about it a little. But there's a surprising dearth of data.
Can someone come up with a data set? For example: here are N inputs to the program. If the program can make a decision D in time T, then it can make money. Or something to that effect. I have no idea right now what the inputs to these HFT programs are; what the expected actions under the time constraints are; and how the effectiveness is measured (in terms of the given data). Someone please enlighten me.
1. Data set - order book of your chosen stock exchange. Events being streamed in at rates up to 1Gb/s [1]. This is the entire set of actions affecting the order book - bids, offers, cancellations, adjustments, etc. - it's huge. If you want to get fancy - most do - you would typically pull in several of these feeds (or subscribe to a consolidated feed) containing several exchanges, and look to arb any price/book inefficiencies.
2. Actions - given the changes to the order book, figure out how you need to change your positioning to make money. Time is everything here - steps here are counted in tenths of milliseconds, and overall response times in low ms. HFT firms even try to minimise cable lengths within the colocated datacentres to shave the time to response down even further.
3. Effectiveness - if you didn't blow up, and if so, if you made money - typically you'd examine intraday PnL volatility, returns, etc. and assess yourself on those and other metrics.
How about releasing some data? Do some feature generation and come up with features, along with the desired action (the best possible action, given that you know how things are going to go in the future). Then us clueless types can see what all the hubbub is all about :-)
> Why should our best hackers and mathematical minds be wasted on something so shallow as gaming the market?
The reason why these shops can employ the "best hackers and mathematical minds" is because they're willing to pay for them.
Everyone wants to hire 10-to-1s but nobody wants to pay them even 1.5x the salary of a blub hack. These days people say "We pay market rate!" and wear it like a badge of honor.
One thing you will notice about certain topics is advocates of one side or the other will nitpick inconsequential portions of sentences rather than actually discussing the topic.
His assertion is that they are gaming the market, with no net benefit to society (it could be argued that their actions are detrimental). Furthermore, bright minds such as these could be doing things that are a potentially large net benefit to society.
Can someone explain this mindset to me? That there is somehow a fixed amount of mathematical talent in the world and if there are people whose preferences make taking a job in HFT optimal then society is necessarily worse off.
Well, speaking broadly about finance vs other more productive fields of human endeavor, there is a staggering amount of brain drain of the (very much finite) graduates of ivy league schools to the finance industry:
I don't think it assumes a fixed amount of talent.
Perhaps a hypothetical analogy can help: Imagine that a tax of 0.000001% is applied to every stockmarket transaction, and the proceeds are given to the top 10 people who can best memorise the digits of PI.
Most people would say that this scheme necessarily makes society worse off.
Would you oppose this hypothetical scheme? If so, what is the significant difference between it and HFT?
Uh, where do you think we can get more mathematical talent from? At the end of the day, there's only so many people at the right end of the bell curve.
I forget where I read this, but in an article by someone else I read something like:
"[There's something very sinister about a company that takes society's greatest engineer and science minds away from important tasks for humanity and diverts them to the task of optimizing ads]
A lot of people will say what you say about HFT, but have no qualms about Google and Facebook and Twitter. Why is that?
I for one would say this about HFT, Facebook, twitter, and google. Though to a lesser extent for google as they have lots of other impressive projects such as self driving cars that could have a good impact.
Indexing all of the worlds information isn't something for our best hackers?
Setting up a global communications network where I can easily share photos videos with friends and families (helping millions and millions stay in touch) isn't worthy of some great hackers?
> A lot of people will say what you say about HFT, but have no qualms about Google and Facebook and Twitter. Why is that?
Because Google and Facebook and Twitter provide legitimate value and services (some more valuable than others), and offer something that wasn't there before.
HFT does none of the above, unless you believe that lack of liquidity was somehow a big problem prior to them entering the market and generating 80% of the trades, and skimming off as much money as possible in the process.
Society does nothing about underpaying the educators of our children; so they should deal with the fact that great minds will be lured away to less critical tasks.
I don't think it's as simple as all that. There are legitimate parties at either end of a stock transaction, and the HFT's are pretty much just siphoning off small amounts of the proceeds. The reason this can occur is that our government ensures that public companies have to do business this way. And all of these rules are in place for good reasons and it works pretty well, but HFT is an aberration.
A good analogy is credit card companies; it's perfectly rational for a customer to use a rewards credit card and pay it off at the end of every month, and it's perfectly rational for a merchant to allow credit card transactions for convenience. Arguably, however, the credit card company is little else but a 3-5% drain on the economy, especially for brick-and-mortar stores.
That said, credit card companies provide some value in the form of convenience, so maybe the transaction fee is worth it. HFT's, however, provide no benefit to anyone but themselves. The transactions would still occur whether or not the HFT existed. The distinction Mark Cuban makes between "investors" and "traders" is an astute one.
Before HFT, to execute a block trade, your only option was to go to a specialised liquidity provider a.k.a your friendly local investment bank - the CC company in your analogy - , who would take a huge spread - the 3%-5% drain on the economy that you mentioned - in exchange for taking that liquidity risk.
Now, with HFT, the increased speed of markets has democratised liquidity provision. There is a corresponding larger supply of liquidity in the markets, and so spreads have come down - very significantly.
The aggregate affect - it is now cheaper to trade on stock exchanges. What does this mean? When your 401(k), pension fund, mutual fund, etc. rebalances, it doesn't lose 3% of your money to an investment bank. It loses ~0.01%-0.05% (1-5bp) of your money as a transaction cost (earned by HFT firms as compensation for taking on the liquidity risk), and thus you keep more of your money.
It's easy to say that HFT's provide 'no benefit to anyone but themselves' without any understanding of the historical context, but once you do, it's hard to see it in such black and white terms.
Full disclosure - I was an algo trader in a previous life.
Ok, this seems plausible, however, I am skeptical. Do you have any data or studies proving out these theories, including data that would give credence to the idea that the $ taken out of the market now due to HFT is less than that taken out by old fashioned liquidity providers?
Brick-and-mortar stores themselves don't create anything. They buy in bulk and sell at a markup, thus adding liquidity to the grocery market. Imagine if you had to buy carrots by the tonne.
This post is being used as somewhat of a stalking horse for any number of other agendas and soapboxes, which I guess isn't surprising, so I'll join the fray.
1. There is a misconception about market makers and the nature of a spread. In particularly illiquid markets, market makers allow you to buy and sell immediately. This is a service. The spread pays for this service. The more liquid a market is the lower the spread (eg it is essentially if not actually zero on US Treasuries);
2. For those questioning our brightest and best being "wasted" on HFT, I see comparatively better value on all the bright minds being wasted on social networks;
3. Finance is a constant arms race. The flavour of the month is HFT. In 5-10 years it'll be something else where low-latency transactions are just taken for granted. This kind of market is a gold mine for innovators as the existing players need a certain amount of innovation just to stand still (relative to the other players). There is classic "mine the miners" opportunity here;
4. There is a lot of highly-leveraged trading that goes on, automated and otherwise, beyond the world of HFT. (I believe) Warren Buffett describes this as "picking up nickels in front of a bulldozer" [1]. In extreme market situations, market models go out the window and the highly leveraged are the first to go to the flames. Just ask Bear Stearns [2] with its 35:1 leveraged ratio.
5. Financial institutions have access to way more data and cheaper (typically free) transactions compared to the average investor. It's common practice to do things like whipsaw the market to hit stop loss orders and the like. Honestly, unless you're a bank in the short term it is an insider's game;
6. A lot of the problems with the lack of transparency, assymmetry of information between parties and bad behaviour (like flooding exchanges) comes down to a failure of regulation. The SEC is either toothless or has been subverted by political interests. Much of what happens in the US just doesn't happen in Australia (to the same degree at least). Just take all the fake documentation in the US housing bubble. How bank executives, loan originators and the like didn't go to jail for this is really astounding;
7. Banks and market funds have limited to no downside risk. If things go horribly pear-shaped the US Federal government will--and has--bail them out. This breeds an adverse appetite for risk. I subscribe to the view that the Federal Reserve, the World Bank and the IMF are in large part welfare for investment bankers; and
8. Banks and funds are too large. If something is "too big to fail" then it's too big and needs to be broken up.
"""The SEC is either toothless or has been subverted by political interests."""
Or politics is subverted by financial interests? If you have to find your own financial sponsors in order to run for a government office, it is hardly surprising that finance runs the show.
'Is the market supposed to be a platform for companies to raise money for growth and to create liquidity and opportunity for shareholders as it has been in the past? Or is the stock market a laissez-faire platform that evolves however it evolves? The missing link in all the discussions is: What is the purpose of the stock market?'
If a person is successful in one business should we subject ourselves to his idiotic views on another business?
By definition they can’t go into an equity unless there already is liquidity.
HFTs who are usually market makers [1] maintain liquidity in the market by constantly having buy/sell orders open for many instruments/scrips. In their absence, a counter trade may not exist for matching against when you or I put out a sell/buy order. Being able to sell/buy a scrip/instrument within seconds is a valuable service that investors appreciate. In India, the securities exchange regulator mandates that all market makers make a minimum value of trades per day. I am not sure of the exact current values in each segment but it's in the order of millions of rupees. This is how regulators create conditions of liquidity so that markets are nicer to trade in especially for retail investors.
The missing link in all the discussions is: What is the purpose of the stock market?
The stock market can be divided into a primary market and a secondary market. The primary market exists to finance companies. When they IPO, cash is generated which they can use to fund business activities. The secondary market exists to encourage purchases in the primary market and to efficiently discover the price of the securities. Without a liquid secondary market, very few investors would be interested in a primary market.
HFT is only doing what was already done since many years ago manually. Humans make the same mistakes computers are being programmed to make today. In their defence, they execute exactly as asked - they feel no fear, doubt or greed - unlike a human trader . Should we treat them as being harmful because computers make mistakes at a significantly faster rate? HFT is a still nascent concept. With sufficient regulation, we will likely be able to get rid of the problems and retain the advantages.
>hould we treat them as being harmful because computers make mistakes at a significantly faster rate?
Yes. Significantly doesn't really cover it. When you increase the rate of an activity by many orders of magnitude and put an AI behind it, it becomes a totally different activity, even if the mechanics are notionally the same.
HFT is just another area where everybody knows that it is broken, it does not serve any useful purpose for society, and everybody knows that eventually it will lead to a huge crash. And nobody does anything, because there is just too much quick money to be made. Humans...
From what I have read, its all in house, the software (including the mathematical algorithms) is the most important center point of a HFT shop. No vender would sell a competitive software system because they would make more money using it themselves.
There's the sentiment that exchanges allow this because they are "businesses". What's to to stop exchanges from front running the HFT front runners without their knowledge and stealing all the marbles?
it seems to me that the exchange adding a small random delay (say 1ms) on message reception would dampen the benefits of speed and then there would be less work invested in saving a few microseconds here and there. you still have program trading but a little bit of silly dynamics is dampened.
HFT has some qualities which make it unattractive to me:
- strategically, it's an unbounded arms race, due to competing against lots of other HFT forces all trying to squeeze profits quickly out of micro opportunities in bid/ask numbers, news leak/announce gaps, etc.
- due to above, there are high technical and talent costs just to have the minimum setup needed to play in that game
- also following from the first thing, HFT becomes an additional profit motive for adding destructive, dishonest political vectors to society, such as corruption and propaganda, in order to time trades, do front running, insider trading, war profiteering, bankster conspiracies, etc.
- short-term thinking
- lastly, you're fundamentally not building anything real or positive for the rest of society outside HFT and your direct clients/owners; or making the world a better place; to others, you're just a parasite on the money flows, adding more noise and chaos, and arguably risk, to the economy
I like to compare it to the alternative, strategically-speaking: LFT (low f.t.)
- lower costs (your fixed & marginal costs "scale down" better, which is also better in the startup phase. Think lemonade stand, a small simple web SaaS, contract development -> software product company, etc.)
- actually building, investing, helping others, bringing new products/services to the market, solving problems for others for profit
LFT could be used to describe traditional stock investing, it could include personal relationship investing, personal health/skills investing, building lifestyle businesses, building startups to flip, doing R&D, invention & marketing, etc.
Both HFT and LFT can make you wealthy on the top end scenarios. But LFT scales down to the low end scenarios and early phases better, and you can tangibly and visibly see improvements to the world that you've made happen. Both can benefit from hacker thinking. But to me, it's clear which is the healthier choice for the individual and society, in the long run.
I know that the startup community at large doesn't largely respect short term trading, particularly HFT. I understand it, though I honestly think its misguided and the outrage is disproportionately large, but thats another story.
The reason why this article TITLE excited me was because I think its totally true, or least close to true. I think awesome HTFs and algo traders have a very similar skillset to top notch startup hackers.
Small prop shops ARE startups. You:
At some point, I'd love to get a bunch of hackers together and start a small prop shop. The jobs are different, but in so many ways it takes the same kind of person to succeed at either.