When I was in high school, my math teacher instilled me with the value of always labeling axes when plotting graphs. This GIF provides us with exactly no information on anything, but it sure seems scary enough. The line is getting higher and more erratic! This can’t be good!
Let’s piece apart some of the language here:
* “High-Speed Trading Bots Go Berserk” — doesn’t mean anything, they’re processes, they’re just following their algorithm.
* “activity becomes much, much more frenetic and erratic” — to a human observer. These algorithms don’t have concepts of ‘frenetic’ or ‘erratic’; neither does the universe.
* “…so complex that its potential systemic repercussions are literally unknowable” — suggests that systemic repercussions of anything were ever ‘knowable’. If they were, we wouldn’t have markets. c.f.: http://mises.org/daily/2608/
Yeah, but you have to admit that this is pretty skilled sensationalist journalism :P
How convenient Nanex didn't include labels in the graphics they provided! [1] In fact, they specifically say that high frequency trading doesn't concern them, and are talking about quote spamming. Not. The. Same. The animated graph used in the article combines metrics of both, as far as I can tell, but Nanex makes it clear that High Frequency Trading is stalled or declining while High Frequency Quoting is on the rise.
Furthermore, the bit saying "activity becomes much, much more frenetic and erratic" is just... totally silly. Partly because of what you mentioned, but also because... well I don't know what else to say, it's just so utterly, embarrassingly silly. I'd wager the graph's movement isn't even nearly as 'frenetic' as the stock market itself.
In any case, it's quite clear that the author didn't even peek at the page he got the graph from.
Based on the Nanex article (http://www.nanex.net/aqck/2804.HTML), it appears the chart is showing thousands of quotes per second on the left, vs time of day on the bottom.
Seriously? Market participants certainly have nerves, and it is perfectly clear to anyone without an ideological axe to grind that the term "market" is used here as shorthand for "market participants".
Lets stop and think for a moment about what we are discussing. Knight gave other people about $400M by making some bad trades. Also, some graphs wiggled.
The net impact on the market was negligible. If you buy on the timescale of days to weeks, you can completely ignore HFT. Here is a blog post with graphs demonstrating the market impact of various HFT events:
"If you buy on the timescale of days to weeks, you can completely ignore HFT"
Not if you have to deal with intra-day margin calls or stop losses.
The trouble with most HFT-based algos is that most of them rely on a set of common signals (regardless of how "diverse" their creators may claim them to be) and end up making the various markets and assets highly correlated.
So when stops trigger, they cause a massive cascade. This needs to be protected against at the exchange-level (edit: they do so via various mechanisms). Most bots usually don't have the sophistication to protect against tail-events.
If there are people making money off those HFT trades, then somebody else is losing money. It's a persistent tax on the entire market. Just because it's in the background where you can't see it doesn't mean it isn't affecting you.
Yes but the money losers are gaining hair style, cat pictures, and and food, not just intraday liquidity that they didn't know they were paying for and didn't want or need.
I'm confused - there is a bid/ask spread of $10.00/10.05. A person comes along and deliberately chooses to buy at $10.05, rather than placing a passive ALO buy order at $10.00 or $10.01.
Why do you think this person didn't want or need liquidity?
And why do you feel that it's worse for a machine to sell it to him at a low price than for a human to sell it to him at a high price?
Things like the Flash Crash[1] are what occur. Te scary thing in my mind is that error states are the hardest to test for and tend to be multiplicative in cascading severity. The HFT world is hiring a lot of really smart people. We have no choice but to trust that enough of that brain power is going towards addressing those error states.
They're sucking a few pennies out of every transaction on these exchanges, the long-term effect is that of a parasite in a larger host. Sure they may have some small benefit of improving liquidity to the market, but overall they're causing harm by reducing the returns of legitimate traders.
If they're adding a lot of liquidity, I'm happy with them reducing my returns a little. If they add a little liquidity, I'm not happy with them reducing my returns a lot. There are questions to be answered that we can't get at from an armchair (at least, the metaphorical armchair).
I think we need the government to step in and slow down google. They return search results unfairly fast, it prevents underfunded companies from competing.
As someone who has designed, built and ran systems that operated at a scale that made the software that HFT firms look like toys and built several web companies, I can honestly say that you have absolutely no idea what you are talking about.
Web development and low-level systems engineering are entirely different and neither one is particularly harder than the other. If anything, both are simply tedious.
And pubsub was old by the time wall street got a hold of it.
It's not as if many webpages couldn't be made to respond to faster.
Not to brag, but I've built trading engines in the past year that can process 20 years worth of data in less than a ms.
Now I'm a founder at your usually derrided photos-sharing start up. Sure we could make most requests come out of the server in less than 2ms, but what of it? It's still going to take the user upwards of 1.5 seconds to load all of the content, and at least 100ms spent on the wire. Working at a consumer level startup, there's no point in optimizing the server yet when network costs are so high, and there's no point in improving the network when there's no ROI in it.
----
Answering to his question:
> what HFT experience do you have that makes you think of those systems as toys?
I'd imagine that massive distributed systems are more complex to think through that HFT systems because the later are more monolithic due to speed constraints.
where i worked - a small shop with 10 engineers - we had 350 machines running in 5 different data centers in chicago, new jersey, and korea. each machine ran around 100 different processes, which were coordinated using a combination of several different pubsub systems:
- in house reliable tcp-based broker mediated (we called it pubsub3 )
- in house unrelaiable ip-multicast system (i named it blue
ray because i thought that sounded cool)
- 3rd party implementation of a PGM derivative that we licensed for god knows how much
the engines subscribed to various topics on the system, processed market ticks (sent by marketdata publishers over blue ray) and fired orders at whatever instrument they were handling
the risk engines modeled risk scenarios inresponse to trading shifts, powering through the risk models our quant trader wrote in c. we had a 'calcparser' system that constructed a DAG rerpresenting the AST of the c code and distributed the DAG across several machines.
The complexity is also in having to deal with limited resources, and factors outside of your control. Have you seen many bootstrapped HFT implementations? The budgets are just... different.
When you spend 10x or 100x hardware, get massively redundant backbone connectivity (connecting to a tier 1 core router directly, and not through an oversubscribed shared switch port that a typical bootstrapped startup would get), you end up with a much better controlled and somewhat predictable environment.
With a typical "web" (non-HFT) distributed system, you've got millions of "subscribers" aka users - and I don't think people outside of IT fully realize just how bad a typical user's [Windows] machine is messed up - and that was before they installed a couple of toolbars, adware, firewall, and AV products... And all of these subs are tying up your publishers in their own unique screwed up way, keeping sessions open, dropping packets, blocking ports...
I've designed global CDNs, and operated one for a decade - and the kind of randomness our distributed systems have to deal with are not always technical, even though we don't typically count microseconds.
On the other hand, web development is solving many many more problems for many many more people.
HFT firms are solving the single problem of making their owners rich with the authoritatively, hand-wringingly repeated justification about how critically important that liquidity is to everyone and their grandmother.
i was just trying (inarticulately) to make the argument that even if the "only" cost of HFT is the massive amount of talent going into that field, it's still a huge cost.
> if you completely ignore HFT, you'd think there would be way more programmers and mathematicians available for hire.
Can you clarify this? My interpretation is that if you discount mathematicians / programmers working in HFT, you would expect more of them to be in the market for jobs?
if you completely ignore HFT, you will ignore that there are many very qualified people working in that field, and therefore overestimate the number of those people available for you to hire.
Any idea how much Knight ended up losing?
I remember interviewing with this company, Allston Trading in college. They were the only ones who showed up at the Career fair with the salary printed out on brochures. $100k for a fresh java developer. During the interview in their plush offices in the Chicago stock exchange, they were casually discussing how they ended up losing $5million in forex last night.
All of them were young, brash and looked like they had little regard for investors money. One trading guy was telling us how he flies to vegas just for the weekend to get with hookers (during a lunch interview!).
The managers seemed like your typical alpha-male..bullish, dominating and little regard for morals. Although some computer scientists that they had seemed brilliant and were genuinely interested in building safer, better algos. So i wouldn't be surprised if it's these wall st. types that are pushing the limits further even if deemed unsafe.
Consensus number seems to be $440M in losses on the bulk liquidation of $4.5B in unwanted positions to Goldman Sachs. This represents about a 10% loss.
GS does bulk liquidations like this and typically takes a haircut of 2-5%, presumably based on a risk and liquidity assessment.
>i wouldn't be surprised if it's these wall st. types that are pushing the limits further even if deemed unsafe.
It's human greed. Those 'wall st. types' and those brilliant computer scientists are all in it to make as much money as possible. Those you looked up to as hero's were not corrupted by those you wish to look down on, they all went into it wanting the same thing. The environment changed allowing them to try and get it faster.
I know HN's frontpage has had a few posts dealing with what financial industry coders do and the validity of it. The ones who claim to be HFT-bot coders argue that they're highly-skilled coders...but it was not always clear if their skill is in algorithm-implementation, or in engineering. That is, they may be great at working with the traders in coming up with a highly performant bot, but they have less ability in exception-handling and design, which may be things that their bosses don't really think about. (I honestly don't know, was just wondering if someone could competently recap the relevant discussions.)
Most everybody I know (in the field) is in disbelief concerning the length of time the Knight disaster unfolded over. I've had systems "loop" in BAD ways, but never for more than 1-3 seconds.
Safeties are paramount in the field. There's many layers of protections to avoid *ing yourself. Most of these are deployed in ways that are not terribly expensive on the 'critical path' -- that said, there are a LOT of people doing REALLY stupid things in the field. (As there are in any field, it's just more 'scandalous' when an error in a financial application occurs)
Algorithm implementation is largely trivial, and the high pricetags paid developers in the industry are usually for specific engineering skills along with the semi-requirement that developers be 'all-hands-on-deck' when the situation calls for it. (There's a bug in the new strategy...well you need to fix it before tomorrow's open etc) At least, that's how I see it.
i worked in electronic trading for a few years. we weren't high frequency, though - it took us a few milliseconds to respond to price movements, which is why i left for the startup scene in 2010.
exception-handling is something we took very seriously, because the consequeneces of a fuckup are so bad. in late 2008, our firm lost around 300k in ~2 seconds. one of the exchanges sent a signal that the interest rate was now 0, which, according to their spec, meant that something wasn't right. our system didnt' handle that properly, so the algos went nuts because hey money is free! the fuse system caught it a few seconds later, but not before we shot orders at everything in sight.
i wrote a system that handled federation and service location, and as part of the design, i constructed four state machines which were used to handle the four possible cases in the formal model we developed. i went over the state machines many times by hand, and coded unit tests for each possible (State, Input) transition, followed by a battery of stress tests.
these guys are good - but nobody's good enough to never make mistakes.
If the chart is showing frequency of trades it's just showing that there are more trades right?
I find it a little misleading that the graph is presented as somehow being connected to market instability. If they are correlated, they haven't demonstrated the relationship clearly to me. Am I misreading this?
Also I fail to see how someone else's algorithm going nuts could impact regular Joe traders as the article also suggests. If a stock price starts plummeting because of a bug in a trading algo goes haywire, a savvy trader would look to their news sources and realise nothing has happened. Or if they are passive investors probably by the time they mobilise to panic sell it's already corrected itself.
> in 2008, a pattern starts to emerge: a big spike right at the close, at 4pm, which is soon mirrored by another spike at the open. This is the era of traders going off to play golf in the middle of the day, because nothing interesting happens except at the beginning and the end of the trading day. But it doesn’t last long.
Most of comments here are missing the big picture.
Most of the general public and individual investors do not have clear insight into how HFTs work, or if they are beneficial or not. Most are inclined to be scared of crashes and erratic behavior traced back to HFTs. Trading off liquidity for a loss of trust in capital markets is a bad deal.
When investors lose trust in the system, it all falls apart.
Do HFT firms have contracts with larger banks to take their positions with a discount in situations like this, so the stocks are not imidiatly dumped on the market causing havok? Or to insure them against loss?
Sounds like we could segment, rate, and pacakge these contracts up into insured "credit-default-swap" type-of vehicles and make money making/selling/buying/trading them.
This should be illegal long time ago -- think Captcha for buying/selling stocks.
The real issue here is that -- whoever was in charge of those stock exchanges hasn't properly addressed this issue as we all can see now its been growing slowly but constantly since 2007.
I'm imagining this would mean an auction (something like the current opening/closing auctions) every 100ms ... I like it. It seems more scalable than the current system.
As has been pointed out a few times in connection with this incident, Knight is largely a market-maker (not an HFT firm): Knight keeps open quotes on both buy and sell sides of transactions at all times and provides liquidity. Their strategy to make money is the "spread" -- they price the buy a couple of cents lower than the sell and on average they end up buying a couple cents lower than selling.
They are part of the infrastructure of the major markets; a company listing on NASDAQ or NYSE is required by the market to choose at least one market-maker who will be designated to always keep open buy and sell orders on the books.
As it happens, Knight does its market-making through computers; this replaces sweaty men who would stand in the "pit" of the trading floor and trade by waving fingers at each other (including, occasionally, the middle one). It looks like HFT in that computers trade quickly, but it is intended to be a liquidity-providing service to the market: there is always somebody willing to buy or sell anytime the market is open.
(In contrast, HFT does not require always-open buy and sell quotes.)
Yeah, "liquidity" is the standard bullshit distraction trotted out every time HFT is discussed here. Strangely enough, we've got "yummyfajitas" defending it again, which maybe, just maybe, might have something to do with the fact that he works in HFT.
As for liquidity, since the bots only make trades with the goal of shaving off tiny little slivers of profit on millions of trades, why and how would one of those sweaty guys ever buy anything from them? Even if one wanted to, the opportunity to buy might pass in a couple of microseconds.
All in all, working in HFT or even the financial "industry" at large, is complete bullshit. No self-respecting, decent human being should do it. No matter how you look at it, your mission there is to help world-raping scumbag bankers and the like make more money. They certainly don't need it.
Finance still, though? I bet it's difficult to let go of the fat paychecks.
> As for your other questions, I suggest you go read my blog posts on the topic. They answer all your factual questions.
Ah yes. I was just giddy with delight when you used Futurama character names in those examples. How could I not love everything HFT represents after that?
> Your guesses as to my personality type and employment situation are just as wrong as your speculation about the nature of HFT.
That's easy for you to say. We're both human though. Humans are above all selfish, and have a hard time letting go of easy money.
So who knows, if I were some kind of math wizard living in the US, I might have ended up working for a bunch of sociopaths too. It's possible I'd be here on HN, rationalizing and defending HFT and blowing smoke up people's asses with cutesy link-bait-titled posts about it.
I didn't actually say anything about your personality type though.
We shouldn't even bother, but perhaps you'd care to point ouf my mistakes in describing "the nature" of HFT?
It creates liquidity, which makes it easier to buy and sell shares when you want to buy and sell them.
Unfortunately, the only time you really want liquidity is during a crisis, which is when all the HFT firms exit the market, since they're in it to make money, of course, not to act as a regulated and guaranteed market maker.
Regulated and guaranteed market makers are both (1) required to post a bid and ask even in crisis situations (2) high-frequency and high volume traders because they need to deal with a great amount of market order flow. These things are not necessarily exclusive.
I think the implication was, is the liquidity level of 2007 inadequate and we need the liquidity level of end of 2011 (which is about an order of magnitude higher).
In the end maybe I don't care if these firms just transferring money between themselves but when people's retirement accounts and the fate of whole companies in their hands it seems there should be something preventing this (another meta-algo that punishes aggressive and pushy algorithm...?)
I think the concern is not that they lose someones money, but that they break the whole system. Knight did that the other day, on a small scale, nearly destroying the company.
If that isn't enough to get you to test your software, nothing is. And if they're that crazy, who knows what damage they can cause.
They either need to figure out how to do it safely, or slow the heck down before they break the whole damn thing.
If computers are running the markets, and the markets have such a large influence over society, does that mean the computers have already quietly started their takeover?
I don't know stocks but this strikes me to be the same mindset as domain tasting and should be ended entirely, not regulated, just killed, regardless of any claims of industries being built around it - it's just wrong.
That's rather simplistic, not to mention completely wrong. US markets are highly visible (you just saw a free visualization of HFT activity since 2007), very tightly regulated and well controlled. Things go wrong, as they do in all complex systems, but rarely do they go wrong to the extent that ordinary investors are harmed.
Read this, before jerking your knee any further. Think of it as turning on the bedroom light, showing the monster in the corner to be nothing more than a pile of clothes:
Financial markets have been moving towards zero latency trading since their inception. First, men gathered under trees or in coffee houses, within shouting distance of each other. They fought over the best spots. Then, traders became some of the earliest adopters of telegraph technology, running private lines from remote cities to the exchanges. Ticker tapes appeared in distant offices. Computerized price dissemination followed, together with electronic trading connections to the exchanges themselves. All in the name of getting information and acting on it before the next guy. It's the reason you, as an individual, can trade global markets cheaply and instantly. It helps you take control over your retirement and savings, rather than paying through the nose for some managed scheme where only a privileged few can access the markets.
Where is your arbitrary line where progress stops?
I think you are arguing something different than I. I am open to being wrong, but I think I am making a different point.
You are saying that by having a zero latency trading capability, financial markets are the sign of progress and it is an inevitability that, not only is desirable, but preferred.
That's fine, but not what I am am taking issue with is fully automated system of extremely high volume, low margin, trades that are conducted by bots and have no human interaction other than those that are profiting from them.
I admit I am not savvy enough in this area to argue about whether or not HFT provides more equilibrium to the price of a stock, but I don't see that as a wholly convincing argument as to why HFT is important and adds value.
I am skeptical of that claim, mostly due to my ignorance, so please educate me on the following: The claim that HFT rarely goes wrong such that "ordinary investors" are harmed; who then is the non-ordinary investor who is benefiting from the HFT, and what value are they providing in the markets, other than profiting from HFT.
It appears to be making the claim that HFT is a meta market that the ordinary investor shouldn't even be concerned with except in cases of extreme rarity where a flaw causes them some financial harm/risk.
Again, if this is true - what true concrete value to the world does this meta system provide? I simply cannot see it, and again, I claim awareness of my ignorance, so please explain like I am five.
> ...and what value are they providing in the markets...
It used to be that if you wanted to buy a book you would have to wander around town until you found a place that had chosen to stock it. If it was an obscure or specialist book you might have to drive to a different town. You might just be shit out of luck. Nowadays you search for the title on Amazon and it shows up at your door the next morning. When you are done you can sell it again on Amazon with minimal hassle. Amazon improved the efficiency and liquidity of the book market which is why they are rich, even though they are charging less overhead than the bookstores.
Your local corner store doesn't create anything. All they do is buy from manufacturers and sell to you at a markup. But you don't care about spending an extra few pennies when its 6am and you're out of milk. You just want breakfast now.
Liquidity providers make it easier to buy and sell financial instruments whenever you want. HFT enables firms to provide liquidity more efficiently and with less risk. Just like Amazon, they get rich by saving you money.
Hav you not noticed the banking scandals in the past few years? Have you not noticed that NOTHING has been done about them?
Also, if you look lower down the thread - I stated that a moratorium on bot based HFT should be put in place until we can fully understand how to regulate these properly.
Equating me with dictatorial banning of cyptographics is ridiculous.
Yes, if the spread narrows enough [1], then the 'price' becomes something like a stochastic process that bumps around in real time depending on the instantaneous balance of supply and demand. This is probably better for the average market participant, compared to the old situation of demand piling up at quarters or eighths, and larger trades taking place at more infrequent intervals (with wider spreads and larger commissions for brokers and dealers). Price improvement and faster order fulfillment is the average case, but it doesn't make headlines because it's not dramatic and politicians and regulators can't drum up popular support against it.
Certainly, brokers' commissions have come down significantly through price competition (empowered by automation of trading). This is generally a good thing for a free market. Some broker-dealers may disagree because it's grown harder and harder for them to be the middleman and collect fees.
[*] $0.01 is the minimum because sub-penny pricing is disallowed for 'most stocks' on 'most trading venues'.
Maybe we need much tighter rules around how the trades can be made.
I have no clue at this point what the model should be - but HFT is something that is completely undoable for a human, though it can have impact on humans at large scale and should be thought of more critically before they can be deployed into production.
I say impose a moratorium on all HFT until a determination of their value, impact, risk etc can be done and published and debated widely.
without it, investors would not be easily able to change their minds about how much other investors think wildly complex financial instruments are worth!
What I've heard, and this is almost worse than anecdotal evidence, is that HFT helps stocks converge on their true 'value' or 'equilibrium' much faster.
This could, of course, be utter bullocks, and I have no way of tracking down the source, but it's at least an interesting idea.
Let’s piece apart some of the language here:
* “High-Speed Trading Bots Go Berserk” — doesn’t mean anything, they’re processes, they’re just following their algorithm.
* “…unnerving the financial markets.” — doesn’t mean anything, ‘markets’ don’t have nerves. In fact, I once wrote about the dangers of reification: http://blog.zacharyvoase.com/2012/03/31/reification/
* “activity becomes much, much more frenetic and erratic” — to a human observer. These algorithms don’t have concepts of ‘frenetic’ or ‘erratic’; neither does the universe.
* “…so complex that its potential systemic repercussions are literally unknowable” — suggests that systemic repercussions of anything were ever ‘knowable’. If they were, we wouldn’t have markets. c.f.: http://mises.org/daily/2608/
Don’t just read. Think. Thanks.