I worked in HFT for a while. I find the whole space fascinating. I'm glad OP found similar thrills.
"This aspect, the platform itself, seems to be often overlooked in most discussions. Many conversations revolve around strategies (mean reversion, trend following, linear regression, etc.), and backtesting, without fully addressing the practical mechanics or logistics of strategy implementation, particularly in the context of live, intraday trading."
I'm glad you had fun, OP, but also I think I can shed some light on why most people discuss strategy.
Trading is a perfect storm of ridiculously high tech, ridiculously complicated, ridiculously regulated (Not over-regulated, mind you, this isn't a value judgement. But the amount of regulation is extremely high.), and ridiculously competitive.
But that said, it's the last bit that drives it all. Since it's so competitive, even though building an order entry system, and a risk system, and a position-tracking system, etc is a huge accomplishment (again, congrats OP!), it's table-stakes to even dip your toes in the pool here. Trading shops can attract top talent and robust, bespoke trading systems are basically cost of entry.
So people talk about strategy because everyone already has the table-stakes stuff and are now trying to make money with it.
It doesn't help, too, that lots of market participants aren't even playing the same game. In HFT, we operated on trades with alphas that lasted a few seconds, where races to entry/exit were battled in shaving nanoseconds off FPGAs being able to shoot out orders and microseconds off wireless networks flying market data around new jersey. Meanwhile, banks are more concerned with elections and geopolitics than they are about the weather in Carteret. (Rain = no microwave network for the day). And then there's a million strategies in the middle with alphas that last from hours to weeks.
So it makes it really hard to even speak the same language to each other when talking in common forums.
Not only are the platforms table stakes they are the more straightforward part to build. Even at the bleeding edge of latency you can usually work your way to the limits of your platform budget without having to find anything novel.
The strategies though are where the discovery is. There are a few strategies that are well known and still profitable but those are largely consolidated to the biggest firms. For everything else it’s a discovery process. And done strategies are only profitable for very short regimes.
I miss it sometimes too, but so much has been consolidated it’s largely a big firm world now.
More straightforward, heh, sure. But still damnably complicated. Which just goes to show how much money and how much engineering talent is invested in this world that these things are so taken for granted.
Essentially yes. When I worked in algo trading, it never bothered me that we were extracting profits from the markets, nor that we served little social good. It felt like a step up from where I’d been before (being told that we were making the world a better place, when every engineer knew otherwise.) At least we weren’t making things obviously worse.
What did bother me, and was acknowledged by my coworkers, was how much top talent was being pulled away from productive tasks to essentially wank around. Not just technologists either, but all sorts of mathematicians and scientists were drawn to the flame.
We as a society have managed to allocate so many of the “best and brightest” to either fintech wankery or placing ads in front of eyeballs. It’s enough to make one want to give up on capitalism, except that everything else appears to be worse.
Ads are tremendously useful, just ask any business owner.
The problem with ads is not that they're useless, it's that it is an industry prone to scams and grift. (Because doing advertising right requires all sorts of actual science, and ain't nobody got time for that when there's money to be made.)
Seems like every person I've ever met in M&A has ended up hating life because of their career choice. They seem to burn out in 5 years or so after salting away a million bucks or 3. Was that your feeling doing the HFT thing?
The jobs in algo trading are very interesting for technically - mathéamtically inclined people. It’s really one of those fields where you have a direct impact on the results of your - measurable in additional dollars made.
There is 0 social impact. That’s the downside of course - but hey, how many jobs out there are really having any kind of positive social impact ? Not 0, but close to it.
That impact is not made by HFT - finding the right risk premia for different investments is very valuable but that is a signal measured over days/weeks/months because actual capital investment decisions take that long.
Intraday financial games are zero-sum. What HFTs gain, they leech away from mutual funds and pension funds and retail investors and market makers who operate over a longer horizon.
I work on very interesting technical problems with very smart colleagues for excellent pay. All my career progression is in compensation, so I can remain an IC forever and no one thinks that's a negative. I'm subject to no politics whatsoever, and there's very little politics in the company as a whole. The work I do every day has a direct material impact on the company and I'm rewarded proportionally to my impact. My WLB is so-so, but it's better than it was in grad school so I'll take it.
Regarding social impact, the world does have some demand for liquidity and price discovery. Providing those services is both essential and extremely difficult. It's definitely not the most social good I could be doing with my talents, but I think it's weakly positive.
What would rising levels of being lifted out of poverty suggest? The equality gap doesn’t count for much if you starve or freeze to death, and poverty is declining world wide.
The worst way to measure it is inequality. How the top person is doing relative to the bottom person isn't important. How the bottom person is doing now compared to the bottom person 100 years ago is.
sorry, i don't believe inequality is rising, at least in ways that matter.
in the last 50 years, a substantial number of people were pulled out of poverty.
Obviously this would mean that countries that previously relied on subsidised labour from those third world countries would have to start paying up. I'm not suggesting that it is a zero sum game however.
>We as a society have managed to allocate so many of the “best and brightest” to either fintech wankery or placing ads in front of eyeballs
It's nothing to do with "we as a society". I'm a quant trader and know many others, and the vast majority are in the industry because we care about making money not some leftist save the world crap. Even if socialists managed to completely destroy the financial market, we'd just find another way to make money without trying to save the world (e.g. like the mostly corrupt officials in Russia and China when they were communist). "Society" can't change human nature; even mass brainwashing on the scale attempted by Maoist China failed.
I.e. the best and brightest will always work where they want to work, not where leftists like you want to "allocate" them.
I’m a lot less leftist than you apparently assume. I’m a landlord, among other things. My problem with our system is that the things which “pay” are not things which have much benefit to society. I’ve got zero fault with you for choosing to make money. Didn’t I say I did the same thing and loved it? I just question the system that drives the behavior.
I think you just illustrated the problem. You have complete awareness of what you do, and you somehow manage to see it as a battle against "leftists" rather than selfish greed. I'm cool with that, but it precludes you from having valid opinions on anything related to improving the world for the majority of people. Please accept this and refrain from any social commentary. Your comment above demonstrates that you don't actually operate in society.
> Please accept this and refrain from any social commentary. Your comment above demonstrates that you don't actually operate in society.
Cut the fascist bullshit. Everyone can comment about how society should be run, even people that are very successful financially.
If you try to exclude greed from the design of your societal system, it will immediately fail. In large numbers, economics shows us that altruistic people wash out of the model and everyone operates in their own self-interest (greed).
Self interests have externalities. The pleasure you gain from improving someone's life is in no way comparable to the pleasure you gain from making other people suffer. It's a fallacy to say "everyone operates in their own self-interest". It's usually parroted by anti-social people in order to justify anti-social behaviour.
> Cut the fascist bullshit. Everyone can comment about how society should be run, even people that are very successful financially.
It's not financial success that precludes someone from commenting on how society is run, it's blatant, deliberate anti-social actions. This is the foundation of a social contract. It's the same reason you'll get locked up if you go around punching people in the face.
Also, I don't think you know what "fascist" means.
You’re suggesting that people can’t participate in societal planning because they disagree with your causes. You just dress it up in overly dramatic phrases like “anti-social” and “selfish greed” to make inhumanely oppressing them palatable. This was bog standard behavior of fascism in Italy.
It doesn’t seem like fascism to you because you’ve done some mental gymnastics to pretend you’re not just crushing intellectual opposition, but that’s all it is in the end. Op is not punching anyone in the face. He/she is just blatantly unsupportive of socialism.
Focusing solely on getting rich to the detriment of other people is the antithesis of "societal planning". I'm not sure why you're defending this behavior. Do you really believe that people who strive for profit over the well-being of other people are in any way being "social"? Do you see them as the necessary negative force for positive action to be possible? Do you really believe that humans require adversity to succeed, and that those who choose to be adverse agents are engaging in some self-sacrificial martyrdom so society can benefit from their negative influence?
Please, I want to know; what benefits do the actions of the op have for "societal planning"? The person he was replying to was at least honest in admitting that this behavior "served little social good".
Once again, I think you'd do well to actually read what fascism actually stood for. The propaganda of fascism had a lot more to say about nationalism, "shared history" and the good of corporations than it did to say about things being "anti-social" or even mentioning "selfish greed".
Largely, though I receive 1 or 2 job specs every week for start ups with the keywords 'hft' and 'low latency'. Admittedly there's going to be duplication there if you read them closely.
I think it's a bit of a myth that (ignoring FPGAs) that writing a low-latency software trading system is a time/cost expensive process. Anecdata = I worked at two firms where we did a rewrite from scratch with teams of 5-6 people and traded in the market within 3 months. I'd argue a senior dev that's been around the block a few times could achieve similar when you remove corporate politics, and bikeshedding over design.
The big firm part is paying for multiple quants at $200k++ to come up with strategies and historic market data access for trading models. Small firms are getting backing as long as the co-founders are 'ex-CxO from MegaCorp'.
> I think it's a bit of a myth that (ignoring FPGAs) that writing a low-latency software trading system is a time/cost expensive process.
This depends a lot on the complexity of the trading system and the trading venue specifics. A system to trade single stocks or futures can be built, certified and running in 3 months. A system for options market making will take a lot longer.
Yes, stocks and futures was exactly what I was talking about. Though we also hit the market with options models within another couple of months.
The big costs for small firms are historic data (if you don't have any), colo, distance to exchange, and number of connections.
From the number of job specs I see, it feels like the HFT/low latency market place is healthy enough that there are always new firms appearing. It's competitive, so it's hardly surprising that if someone has new ideas they'll find a backer.
HFT provides a great deal of liquidity and efficient pricing in markets that are adapted to it. It provides a real service, allowing people to transact without using a large bank or broker efficiently on an open market. The fact you can click buy and it buys on almost any stock is likely due to a HFT on the other side. That may not mean much to you directly but it does provide a lot of utility in markets. The biggest gripe people bring about HFT is it during high volatility HFTs usually pull out of the market at a time when people really value the increased liquidity. I think some of the more advanced HFT firms though have moved into longer time range trades which helps provide more liquidity in those markets.
Depends on their strategy but a lot of strategies depend on some sort of price discovery which requires having a probable estimate of what the current price should be and some form of market making around the level. In a very dislocated market the price is unknowable and it becomes gambling, and generally market making strategies are explicitly not about gambling but about facilitating trades around the “true” price.
It may seem so at the first glance, but such trading if carried out by many independent providers actually increases the liquidity if the market for everyone else.
Likewise with crypto. Personally I don't think that if you're calling an API over an Internet it matters if your trading bot is written in go or python (mine was in python). Use the language you're most comfortable in. The network and trade submission/execution at your broker will be 10x slower than your bot anyway. Unless the size of your operation approaches the size where you can get direct market access which seems to be reserved for big forms only.
"increases the liquidity if the market for everyone else"
That's just it. The whole premise is pretty absurd. The market, the actors, everything. It's so far removed from literally anything remotely human. It's the financial equivalent of an infinite sea of AI bots producing CVs and research papers which are only being evaluated and read by other bots.
If you step away from it all for a second, what the hell is the endgame of this whole hustle.
It was always the only end game. AI on quantum computers is the ultimate computing technology — basically the final research technology in our made up little game of Civilization.
Anytime you create technology that sufficiently replicates the creators, you end up with the spirit of the creator embodied in the technology. So, of course digital brains are going to do weird things like crossword puzzles and sodoku at scale, because it's the same kind of useless shit we do to entertain ourselves.
You get into this conversation whenever you dive hard into cyberpunk, which is so many, many things. Even the Internet itself started out that way. The endgame started as a game, and it will end up being a game, played by our creations as odd mirrors of their creators.
I think there's a lot of people who subscribe to doing the same thing to save the planet, of which I have a keen interest. Solarpunk is the name of that movement, and it also has similar crazy ideas.
We're inventing digital brains. It's literally an architecture designed to be removed from being anything remotely human as it's a mimic or replacement technology for intellectual capacity.
If you really take a step back, the endgame is crazier than just bots producing content merely for other bots to consume (which basically describes the vast majority of scientific papers these days too, ironically). When you take that absurdity and multiply it by tens of thousands in terms of efficiency, the whole system we're building looks WILD and almost inconceivably strange to the way we do things now.
I know this is only tangentially related, so I appreciate your understanding that I already understood that and wrote this anyways. :)
At any point in time yes, this is an accurate description. When you integrate over time it’s not. At the end of the line it’s people betting with advanced tools for execution of their bets. Behind the mathematical beauty of risk free and arbitrage, there is a reality of someone, somewhere making a bet at end of a line of contracts. And there is also someone making the opposite bet. Markets are not disseminating information but beliefs as a proxy for information, and as such they are much more human than you would think. The end game is a network of beliefs, livelihoods, politics and petty or grand aspirations, the same mess as in any organisation. The only reason you question the futility of this one specifically is that it’s a clean room abstraction of the futility embedded everywhere. I would call it futility as a service.
Your analogy isn't applicable here. What the OP was trying to get at is that even an individual who doesn't know anything about markets, HFT, liquidity, etc can still benefit from high liquidity from HFT (since it allows for transactions to occur sooner and quicker). In the AI example, the implication is that the final product isn't benefiting consumers.
I mean, efficient allocation of capital is the overarching goal. HFT is either a facilitator of that or a perverse byproduct, depending on who you ask.
It's quite a statement. You're almost saying capitalism and efficient markets are pointless. Maybe they are, but I think it's nothing like crypto.
In the old days before HFT, you weren't sure you'd get the best price. You'd have to rely on a broker to make sure that happens, but as a retail trader you generally got a worse price/out of date price.
Nowadays with HFT you can get pretty much the best price anywhere. Those <1ms HFT traders make that happen. It's the efficient market hypothesis in effect, made possible by HFT.
Now is it pointless to shave off even more ns in the all out war to grab a piece of the order flow? For retail and institutional investors, at a certain point, yes it's completely pointless.
But it's also just pure capitalism at work. HFT firms compete against each other, and the competition is about speed to provide the best price and volume. If you try to regulate with something like enforced delays, then what do you compete on instead?
> You're almost saying capitalism and efficient markets are pointless.
Capitalism necessitates efficient markets (and efficient markets necessitate HFTs) so any criticism of HFTs is a direct criticism of capitalism as well. I mean this isn't really a problem that's specific to HFTs -- there are just a lot of jobs that we can perceive as providing no value or even negative value (jobs that are possible specifically due to capitalism e.g. payday loans, 2008 style trading, certain scams or predatory practices etc.)
I think we can all agree that this is a flaw, and we're not criticizing capitalism to replace it with something else, but rather just recognizing this as a problem. At the end of the day, a useless job is a useless job even if it exists solely because of capitalism.
You are profiting off workers as a middle man in the economy by doing HFT, and trying the justify it by some vague concept of the "correct price". You are producing nothing of value, merely taking away value before someone else notices it is there.
>You're almost saying capitalism and efficient markets are pointless. Maybe they are, but I think it's nothing like crypto.
People are saying this because, HFT sounds similar to 'crypto mining'.
That's people with best infrastructure, the 'big-guys' -- win.
While leaving out the retail investors as broiler chicken, pumped with 'drugs' (by influencers) to spend more on imaginary assets, so that they can be used for 'food' by these 'big-guys'.
There are different influencers for retail investors vs crypto.
In retail investing there are promises of 'retirement paradise', actual tax deductions, the Jim Cramer-like people (at least what I heard in US)
For crypto investing the influencer are different, the geography is wider.
A promise to participate in markets if you do not live the country that has adopted US/UK-based financial services.
- - -
By the way, I think the markets will still have liquidity if there is a rule to wait, for say, 30 min before a stock that was just recently bought -- can be sold (unless by a clear fat finger mistake)
This rule will cause the HFTs to stop existing in the current form.
So are retail investors truly no different from the people gambling their money on the lottery at the 7/11 every night?
Do any regular pedestrian middle class retail investors EVER actually earn enough to retire on? Or do anything meaningful with? If so.. what is THEIR secret sauce, since it's not HFT...
a)Members of political elite that get insider trading stock tips.
(illegal of course). The number of folks in usa congress and senate that become
'very lucky' investors after they join the rank, has to be amazing
b)Lucky
c) everybody else -- that looses.
Overtime, I would say last 30 years, the amount of 'influencing' retail investors to trap them into unreasonable actions had gradually increased.
So the percentage of folks going becoming the victims of the charade, will become higher.
Certainly if you concentrate on ( b ) you create the plausible deniability defense for the manipulators
Nonsense - what could be more beneficial to society than providing imaginary "liquidity" by interposing yourself between legitimate buyers and sellers?
It's a bit like stock brokers - and why wouldn't we want stock brokers to operate at drastically faster-than-human timescales, because we all know the value of a company changes every nanosecond! And "flash crashes" create opportunities for investors to make huge amounts of money!
And just as crypto has poured money into GPU companies (providing opportunities for enterprising secondary market resellers of same) HFT has poured money into networking companies.
Short of creating the great firewall or helping governments slurp up all the traffic on the internet, what could be a more beneficial application of network technology?
HFT (like finance in general) has also gobbled up lots of tech grads, making it easier to find jobs at tech companies.
> Nonsense - what could be more beneficial to society than providing imaginary "liquidity" by interposing yourself between legitimate buyers and sellers?
Now when I say the same thing about index funds people get all huffy
It's similar, though in theory you are paying the index fund to act as your agent, and the fee structure is observable.
Regarding liquidity, I imagine it's possible that the index fund might allow you to withdraw money faster than you could sell on your own, but realistically I kind of doubt it.
The classic theory of longer term investing is that you are providing capital to businesses and rewarded for that. That is a little bit different from being a middle man. My personal opinion though is that lending money/investing capital and capitalism in general is power. Money quantifies the potential of making other humans do things for you, it allows you to excercise power over fellow humans. Inheritance means you can inherit that power from your parents. It's not all that different from feudalism.
YES, 100%! I honestly, cannot agree more. I've read pretty much every book that I could get my hands on and no one talks about the platform. This makes so much sense. Thank you for your perspective. It seems like so much of this stuff is silod off into each company and no one is talking about it.
HFTs are definitely playing a completely different game. I was reading about the exchange architectures and how things are actually wired. I'm getting my data from SIPs while HTFs are directly connected to the exchanges [1]. I'm transacting in seconds and they, like you said, are transacting in microseconds, so there is no comparison. Which, in a way is actually nice in that I'm not really competing with them. Or, maybe I am but I can still make some money. haha.
I'm in the medium frequency trading world. (calculate lots of market conditions, send at most a few orders a minute) I'd like to understand the HFT world a little better.
Is it basically "do what MFT does, but faster", or is there any specific advantage like getting into an order queue with priority?
There’s a very nice book about this subject called “Dark Pools”. I found it very enjoyable, you certainly seem to have the knowledge to enjoy it even more :-)
Happy to answer any questions about this. It's been a side project that turned into a full blown obsession. There is nothing too secret about the system since it's more about having a solid platform that you can plug your strategies into. I'd probably even open source it but I'd have to clean up all my hacks :)
"Tech Trader is a fully autonomous trading system live with no human intervention or updates, now for over 10 years. It is unique from conventional algorithmic systems, not only because it actually is fully automated, but because it takes a "human" approach to markets. It is not quant. It is not stat-arb. It is not high frequency. It is a program that looks at stocks the same way a person does but with the cold discipline and infinite attention span of a machine. It is analogous to having a thousand independent traders each focusing on a single stock, as opposed to a single quant manager trying to make sense of a thousand datapoints. A person doesn't think through stats, correlations, or complex math models when trading, and neither does Tech Trader. Tech Trader leverages technology to do what human traders do at scale rather than approach markets from the point of view of an academic, mathematician, or scientist.
Since its launch in Dec 2012, Tech Trader has been trading live capital completely on its own, fully automated in the truest sense with no human input, no tweaking, no updates. It is, for all intents and purposes, an autonomous hedge fund, one of the first to truly trade unsupervised for years on end. Whereas many "automated" or "AI" funds may have a hundred scientists providing the actual intelligence behind the curtain, the creator of Tech Trader consists of just one person - a self-taught individual going by the gaming moniker pftq, who created the system at age 21 and has long moved on to other interests. "
Awesome write up. I have a similar project in Go myself, although I just use minute bar data instead of realtime ticks.
Can you share your approach for plugging in various strategies? I quickly learned that having a pluggable strategy system is tricky as it could span across multiple layers of the system.
Also, with backtesting, are you storing and replaying all the quote/tick data? or just using the historical aggregates?
I wish this was better, but honestly, I'm just hard coding them right into the BUY loop. So, I need to restart the app anytime I want to change something. That's why I build this logic to dump and reload the state into a gob file (go memory dump essentially). Ideally, you'd have some type of format to write out your algorithm, and then have a way of hot loading it or something. But, I don't change these enough to really do that yet.
For back testing, I download all raw trades and quotes, and put them into 1 file sorted by time (1 file per day). Then, I compress them using lz4. This allows me to sort of replay the entire market and build up all my intraday backtesting from the source. This took me a long time to figure out and build but has been so worth it. So, I have an off-line script that basically, loops through these files, and replays the market, and makes simulated trades, and then spits out what would have happened. There is a GUI for that too so you can go in an explore the trades and see what triggered the buy and sell. I have seen nothing that goes backtesting for intraday like this.
This is super inefficient but I'm just building the aggregates on the fly. I could probably cache them somewhere but it takes maybe 4-5 minutes to replay a days worth of trades/quotes and build all this so I haven't bothered yet.
Count me in the group of solo algo trading developers (Go and Python and Tradingview Pinescript and other sruff). Planning to send you an email but maybe we can form a group/discord or something online.
I've been algo trading as a side project for a few years now, but all through TD Ameritrade. I get all my quotes and whatnot with them, and it's free with my accounts. I also get other data elsewhere, which is free as well.
I was curious in getting a websocket setup with Polygon after seeing your post, but I noticed it's $29 for first package with the websocket feature. Then I noticed that the package with real-time data is the Advanced package at $200 a month. I'll admit, real-time with unlimited API calls sounds pretty sweet, as my strats rely on to the second data. However, I don't know if I can justify $200 as my algos run no more than 78% win rates with a normal bankroll funding it all. I'm looking to save wherever I can while I build these things and get a passive income stream rolling in.
I would love to know which package you were using, as I didn't see it in the time I quickly read your post? Also, any pro/cons to that specific package? Any and all other details are welcome, and if you would rather respond to my email, it's in my profile. Thanks!
You connect to IB's TWS API... If you execute your trade there only, wouldn't it be an option to fetch the price in real-time from the IB API and not having to use polygon.io at all?
I have no idea: does IB send the price feed in real time? (they certainly send the data in real-time to TWS as it constantly updates right? But is the order book available through their API?)
Basically and even though I know this was published on polygon.io's blog, would that work by only using IB / TWS's API?
Yeah, I initially tried that actually. They have very low resolution data and do not provide raw trades/quotes (something like a message every 250ms), you cannot watch the entire market so you're stuck with like 100 tickers [1] vs 5500 tickers, and fetching historical data is very cumbersome. You can basically pull down pre-aggregated candlestick data and I wanted to base things off the raw trades/quotes. I actually probably spend a month or two trying to make this work. So, that's why I'm using polygon.io in that you can stream trades/quotes for the entire stock market in real-time, without any caps, which is amazing!
I've been going through this journey myself. I started learning on Tradingview, then bought Build Alpha to discover how to test strategies. I chose Portfolio 123 for my automated factor trading but had been working towards creating a program/basket trading system like yours that can act on intraday data.
I moved to long-term investment until I could build a simulator capable of verifying the correctness of my investment strategies using fuzzy testing ideas stolen from TiggerBettle.
I have almost two years of polygon quotes and trades for the whole market captured with a monotonic timestamp to be able to replay the data –and test the handling of polygon socket glitches.
I'm focusing initially on capturing the data in a way that allows fast replay and aggregation, similar to what Kafka can do with topics but in-process using zig and custom memory-mapped data structures. My idea is to be able to generate signals like VIX (once I add options data), ETFs, and indexes and hopefully be faster at doing so than others :), please HN folks, call me out here if I'm being too naive.
This has been a three-year learning process for me. I have been a retail investor for +10 years, but over the last three years, I've gone deep into learning algo-trading, drank del Prado Kool-aid, and read numerous trading and investment books.
I'm now focusing on my technical chops to build the engine to build order books for individual stocks, baskets, and indexes with realistic market prices. I aim to develop a system that can get as close to the market price in the next dollar bar as possible.
This has been a very lonely journey, and after reading the responses to this post, I'd love to connect with others on a similar path. Sending you an email!
I've been really wanting to use Go, but as you say, much of the community is Python due to the data analysis strengths. To the detriment of the other things Python does do poorly.
Can you give some thoughts with your experimentation on the following from a Go perspective.
1. Supported TA libraries in Go. I'm familiar with TAlib (python), bloom, etc.
- certain forks tailored to real time rather than historical (eg: no re-compute on ticks)
2. Data storage (article mentioned you're all in memory). I've been using S3 & ArticDB
3. If your in-data memory is treating you well for multiple TA calculations (example: in Python, you can compute & save pickled dataframes - and re-read those over longer time periods)
> Supported TA libraries in Go. I'm familiar with TAlib (python), bloom, etc. - certain forks tailored to real time rather than historical (eg: no re-compute on ticks)
I've been basically, just manually coding the algorithms from python into Go. ChatGPT is amazing at this. I really only just about 4 so it was a one time thing.
> Data storage (article mentioned you're all in memory). I've been using S3 & ArticDB
Yeah, I ran into issues and then was like what would be the fastest, then just went in-memory. I download all raw trades/quotes each night and store then into gob+lz4 compressed files. Then for backtesting and stuff I can load these in and build the aggrogate bars on the fly.
> If your in-data memory is treating you well for multiple TA calculations (example: in Python, you can compute & save pickled dataframes - and re-read those over longer time periods)
Yeah, I have a historical lookup table that I build nightly too. This gives me a reference point when I'm doing % change calculations and stuff. I should probably have mentioned that.
I'd be careful using ChatGPT for this. I tried the same recently to bootstrap spot yields from par, which is a fairly common code problem with hundreds of examples online. ChatGPT couldn't do it. It produced code that looked right, but would fail my tests abysmally. I ended up writing it by hand. I hope you are validating their code with known data.
Yeah, I'm heavily testing everything it writes. So, I'm very sure it's correct or get it close enough and then code it. I've seen ChatGPT add imports, functions, etc that don't even exist in Go. What I've found so useful though, is not even coding, but asking it how to solve problems, and then having it code things up. With Google, you need to pretty much already know how to solve the problem then go looking for answers. With ChatGPT you can ask it how it would solve this problem. For algorithms and stuff it have been unreal. Thanks for the warning though. You're totally right.
I had not looked at Mojo... thanks for the pointer.
In the past, I've had issues with library compatibility with compiled derivative languages of interpreted ones (eg: Crystal of Ruby, etc, etc).
Know if Mojo directly uses existing Python ecosystem?
I'm excited for Mojo, but it'll be years before it's ready for prime time as a general purpose language. It's not even available to download yet AFAIK...
I've used ATR bots for years, and would love to hear your thoughts on how what you're building delineates itself as a distinct product strategy beyond just the programming language GO.
I've used wonderbit, zigz, and some of the others.
No way. Not even close. That would be the dream though. Taxes are insane in Canada. Like 50% insane. Housing and kids just eat money. Maybe if I was renting and didn't have a family but that's not happening.
One of the most misunderstood concepts about algorithmic trading is that speed is not a critical factor for the majority of systems. My systems like https://grizzlybulls.com/models/vix-ta-macro-mp-extreme have been crushing the market with live trading for 3+ years and yet trade on a frequency of only once ever 18 trading days on average (some a little more often, some even less), and are set up to only generate signals around hourly breakpoints.
The last 18 months have been weaker than the first given the enormous structural shift in the market in this high inflation and rapidly rising interest rates environment, but we've still managed to deliver a return of +14.11% since the site launched in Jan 2022 compared to -7.83% for the SPX. We've managed to do it without any use of leverage and also with lower drawdowns as well of -16.48% vs. -27.57% for the SPX over that time frame.
Your chart including what looks like backtesting data starting 2009 looks a bit disingenuous. It’s a totally random starting point. It’s obviously not your actual performance, if you say you started 3 years ago. It’s made to look like your model actually outperformed the market significantly.
Kudos to you if you really sit on an untapped gold mine, but imho, there are some red flags that makes me not buy in.
Thanks for the feedback. I try to call out in several places in the app that 2009 - 2020 is backtest only, but perhaps I need to make it more clear. No one should ever expect a model to trade as well in live trading as it backtests, and that's been true of nearly all our models (save the anomaly of TA-MR-Basic). However, there's more than enough room in the returns and drawdowns to underperform the backtest while still producing significant alpha, which is what we've seen with 4/7 models and especially with the top two.
However, the April 2009 start date is not actually random--it's the first start date for which intraday futures data is available for more than just front month contract. Several derivative indicators of the VIX futures curve are the most foundational to all the VIX-based models, and they simply cannot be processed without it. The VIX futures were only created in 2004, and I've scoured the internet for intraday data for more than just front month (can't create the curve if you only have front month data), and the earliest it can be found is April 2009.
The models vary greatly in the number of external data sources they pull from to compute signals. The simplest ones like https://grizzlybulls.com/models/ta-mr-basic and https://grizzlybulls.com/models/ta-trend-basic only use a technical analysis calculations on the raw price and volume data of SPX on a few different timeframes. The most complex pulls data from dozens of external sources as it incorporates macroeconomics, monetary policy, various market sentiment sources, yield curve, valuation data, fundamentals, earnings estimates, etc.
It's also worth noting that every additional data source adds some risk of that data source being down or publishing inaccurate data during real-time signal calculation which can cause inaccurate signals, so in order to justify that risk, the external source must meaningfully contribute to alpha or better risk-adjusted returns.
None of them involve a human element in real-time. However, they are occasionally updated as new data comes in, but any updates only apply going forward so as to preserve the live trading history accurately (live trading start date varies by model from mid 2020 to jan 2022 with 2009 - 2020 being purely backtest for all models).
For those that are interested in algorithmic trading - take a look at Collective2. Where engineers make available their buy/sell signals for a subscription fee.
It's like the minor leagues for algorithmic trading. It's fascinating.
The system keeps track of the gains/losses, so no cheating on the reporting.
You can authorize Collective2 to access your Interactive Brokers account, so that the trade signals are managed on your behalf.
It's been around for at least a decade, so you can see some longstanding performance numbers, but most systems just don't last that long.
When developing an automatic trading system the following aspects are important:
- Data feeds and data ingestion. It can be a fairly independent component which collects data from different sources (might be even discussion forums) making it available to other components in some uniform format
- Feature generation. The source data is rarely used in its original form for decision making and having good (informative) features is frequently the primary factor of success. Moving averages is an example but nowadays this will hardly work
- Signal generation. Here some logic should be applied in order to emit discrete decisions and such models are heavily parameterized with thresholds.
- Real trading and order management as well as coordination of all activities.
The article sheds some light on the technological aspects and the general pipeline used to process the data and manage orders. Although it might be interesting indeed, I would expect more details about how to scale the solution and how to implement it asynchronously. Especially if it uses Go which has a special construct for that purpose - channels.
I understand that it is not the focus of the article, but having some general information about its trading logic and how to plug new and parameterize existing strategies would help. Some links at the end are quite interesting for me because I am developing an intelligent trading bot based on ML and feature engineering (https://github.com/asavinov/intelligent-trading-bot) for which such articles might be quite important
Hey, thanks. Yeah, I agree with you. That's an oversight on my end. I'll tell you here though.
I'm just using go routines and channels to talk between them and then a giant mutex for locking. That's basically it. So, as new data comes in, it builds aggregates (tick based candlesticks) as needed, this then triggers the the BUY logic loop on that new data, if something is detected, that triggers a IB API order. It is dead simple and nothing complex in here. I've had upwards of 100 positions being tracked at anyone time and seems to just work. So, I haven't messed around with complex async logic too much.
I'm actually just hard coding the parameters right into the BUY loop. This probably sounds crazy but for a small setup like this they don't change that much. So, I can run some trades, tweak things, restart, and then test some more. I imagine if you were doing that in an enterprise setting you've have some formal language and hot loading and stuff. But, for me hard coding seems to work well enough.
1. Do you have anything that limits the size of a single trade or position?
2. How do you measure and manage overall volatility/risk/VAR to your portfolio?
3. What kind of safeguards do you have to avoid catastrophic bugs? (For reference, see Knight capital and how a single bug brought down the entire company: https://www.henricodolfing.com/2019/06/project-failure-case-...) Given how fast your system trades, I imagine it must be difficult to visually spot these errors. (You did mention paper trading, but I wonder if you have anything else you want to mention)
Thanks so much for sharing your knowledge publicly. It's very much appreciated!
Here's some simple rules. I basically just read about these and then stole the ideas. I wish I had data to back this up and it seems to be working.
- No single bet can be more than 5% of all money. The small bet sizes are what really saves your bacon in that even if a few lose 10+% you're still fine overall.
- I'm also limiting the amount of shares I bet and try to keep it in the low hundreds so that I get really fast fills.
- I have something that stops everything if I lose more than 1k in a day.
I don't do anything to measure overall risk or volatility. Almost everything I'm in is highly volatile. I'm basically betting as things go up and then try to cash out. Sometimes you hit the top.
Yeah, I inspect all the trades at the end of the day, well and during the day, and try to feed anything new back into the system. This is 100% manual. But, like if a single trade loses $100 or something definitely I'm in there looking at what happened.
Around 45-50% win rate but I try and keep the runners going for as long as possible and exit out of the loser trades quickly (this basically just costs you the commission on losers but sometimes more). You can tell within a minute or two if what you expect to happen is going to happen. So, even with a 50/50 win/loss rate you can still make money. I make around 100-200 trades per day. Sell everything before market close.
Does it work synchronously or asynchronously? For example, if you integrate WebSockets then you could act as soon as you get new data. If it works synchronously then you regularly request data (say, every second) and then act.
If you process many positions and then choose top 5 candidates then do you choose one of them for buying or you can allocate resources between them (depending on some score)?
Both? I'm getting a constant stream of data via the websocket. As data comes in it gets added to the large in-memory object, and then once I have X number of trades, I'll go and build the candlestick bar. As soon as that bar builds, it triggers the BUY loop to inspect it, and see if it matches what I'm looking for. If it does, it send a buy order out to the IB API. I'm not ranking anything. I'm just looking to see if that matches and then buy. So, I have something that tracks the total money I have, and if I have any free money (and we don't already own that stock), it makes a bet. That's the logic. I wish I had some ranking logic but that's what it's doing.
That logic is happening for 5500+ stocks all in real-time which is pretty insane. But, it works really really well. Go is amazing. At market open and close there is like 60k+ events per seconds across trades and quotes. So, that loop is processing like 60k events at times, and building each stock out, and then looking to see if we should buy/sell.
Go itself, or any other language won't give you too much of an advantage. What gives you advantage is trading algo, which is always hard to find. I've spent months on figuring out the best parameters for trading. Ended up this working only on historical data, while in reality it was totally different.
I could use Visual Basic, and it would be better than Go, Rust, or whatever it is out there, given the algo and strategy are flawless. Language is just a tool. It's great you used Go, but I think the title is a bit misleading - people think of it as some kind of advantage. It isn't.
And for HFT trading a language with Garbage Collector is not a great choice IMO.
> What gives you advantage is trading algo, which is always hard to find.
At the end it is necessary to make a decision whether to buy or sell (and how much), which will compete with other decisions made based on some logic. Developing such a logic (strategy) manually is of course quite difficult. I developed an intelligent trading bot which derives its trading strategy from historic data:
> I've spent months on figuring out the best parameters for trading. Ended up this working only on historical data, while in reality it was totally different.
It is a typical situation. The whole problem is to develop a strategy which works for future (unseen) data. Even backtesting algorithms should be designed in such a way that future experience (data) does not leak to the past.
Hey, I write this. I did program this in Go. The reason is that it's my go to programming language. Using it is actually a disadvantage in that the industry uses C++ and Python but that's just what I know how to program in. I wasn't trying to be misleading. Go does work really well for taking in data, doing something with it, and then calling a remote API. So, it actually works really well but if you wanted to get a job based off this it probably wouldn't help you.
Go is great for network heavy apps like algo trading!
I used Go to write trading algos that would find small windows of triangle arbitrages in crypto exchanges. Made me some money but the risk of a big loss made me stop pursing crypto trade and it required too much time and attention. It's a full time job from my experience. Reasons I could lose big at any given time if I scaled up the stakes:
- Exchanges temporarily pausing some specific crypto trading for N reasons (happened very often) while I'm in the middle of the arbitrage.
- Getting caught in the middle of a pump and dump event (also frequent)
- Any algo mistake that would perform excessives trades in succession would incur huge losses because crypto exchanges charge %.
Also, most cryptos have too low volatility. Every time I tried to scale my bot would start interferring oo much with the market. And it wasn't even much money.
It was my (admittedly ignorant) impression that C++ was preferred over garbage-collected languages because you'd be more likely to avoid GC pauses. Yes, of course I know that Go's is super efficient, but if you're doing a million trades a day even a tiny percentage of slowdowns that would be otherwise preventable could be a career-limiting move. Am I just behind the times?
I think you're right! Go will never beat C++ in this aspect.
I used Go because that's what I knew and for the non-professional trading I was aiming, C++ wouldn't have made much difference. My bottlenecks were network (1s+ per trade roundtrip) and chaotic unreliable crypto markets.
Just note that Java is used in HTF, but it's a different beast than our average CRUD Java. For example this article states:
"Essentially, we use a contrived form of Java that avoids all the Java constructs that make things go slow. We only use the constructs that are fast and efficient, and we avoid all the garbage."
Yep, you can with GOGC=off but then I would have to learn about how to generate less garbage in Go for long running processes. Which is good knowledge even for non-trading applications. I just didn't have the energy and time to try it.
Maybe? Every language one would have to work at to make performant, and generally pre allocation is what often happens to do so. allocation is probably more expensive with a gc’d language like go than C++, but both would want to avoid it.
I read like 5 or so books about trading, mostly the classic ones, and tried to toy around with backtracking some self-made strategies using MetaTrader: https://www.metatrader4.com/en/trading-platform
First week was funny. I kept thinking I found goldmines with fine-tunned stochastic lagging indicators, only to realize that I overfitted and an algo that made me rich when tested against a certain period of time would make me go broke when applied to another period.
I tried applying some defense against local maximum, but to no avail.
Best I could do is tie after paying exchange fees. And that took me endless weekends toying around with code, reading books and articles.
My hunch is that algos that don't rely on external signals (i.e. news) need extreme technical edge like High Frequency Trading have with their ultra low latency and premium exchange data feeds.
And I wasn't ready to dive into news sentiment analysis and other external indicators.
>And for HFT trading a language with Garbage Collector is not a great choice IMO.
Depends. I've heard of places that use Java by creating a huge heap, minimising allocations to almost zero, and just restarting the app periodically before it can fill up. You can achieve very good performance doing this and you don't have to worry about memory bugs.
You overfitted. That can happen due to human optimisation as well as computer optimisation. After you have experimented on some historical data for a while it becomes less useful.
I've been building a bot with Typescript because it is what I know and Python because of the tools available.
It is an overwhelming lonely endeavor. With all my other projects, I've always worked on teams, although they have always been very small teams and most of my work was autonomous -- still there was the occasional meeting and stand-ups. I've been working on this for six months and thought about bringing a friend onboard for no other reason than to not be alone.
Myself included. One issue I've run into with the public Discords is there are just too many people and therefore too much noise. A smaller non-public group would be ideal.
This article is well-timed as I was about to use options monitoring as an excuse to get into some other technologies. Probably not going to make a ton of progress as a side project, but I'm always up for a chat!
How can you hope to compete with the pros? You're not going to be faster than them and unless you have several PhDs in maths it's unlikely you'll come up with a better algorithm.
The only way I can think of is to get data that they can't.
Not everything has to be super fast. I was using quantopian to trade based mostly off fundamentals data on a monthly rebalance basis successfully. At least enough to beat the sp500 for a few years- I stopped because I went to a hedge fund that prevented me from trading that frequently and in most small caps. If you want an idea take a look at the piotroski f score and factor based investing. Aqr for example is still largely factor based.
There are plenty of strategies that can be profitable on a small scale but which just don't work as you scale up the capital or leverage. Such strategies can be simultaneously profitable for a small operation and not worth the bother for most larger trading shops.
This is just one example, but at a small scale you might be placing buy orders with let’s say $1k or $10k. This is insignificant compared to the total amount of money being traded, and will not affect the stock price.
However if you start increasing scale to $1mm or $10mm, your buy or sell orders begin to actually move the stock price itself. You might not be able to successfully sell $10mm of stock without dropping the price, signaling others to sell, further dropping the price, cutting into your own profits.
For the point I'm trying to make, the particulars of the strategy don't matter. What matters is the fact that the larger and/or more numerous your orders are, the more likely you are to move the price, and any such price movements will necessarily be disadvantageous to you.
I.e., if you're buying, the larger your buy order is (well, assuming a visible buy order) the more likely it is that liquidity-adding sellers will increase the price of their sell orders. Also makes it less likely that liquidity-taking sellers will want to trade against your large buy order, because (like everyone else) they'll tend to interpret your large buy order as a sign that the price is likely to increase, so not as good a time to sell.
You could of course use hidden orders to avoid some of those disadvantages, but hidden orders have their own set of tradeoffs too.
I find this article to be absolutely pointless and clickbaity. It can be boiled down to: "you can use poligon.io for market data but algotrading is difficult anyway so there is not much to share yet".
Hey, I'm sorry you feel that way and I wrote it. Personally, I wanted to share the high-level structure of how you'd build your own system. I would have loved to have seen something like this when I went down this rabbit hole in that I needed to figure it all out myself. What do you think would have made it better? I'd be happy to roll that back in.
I love finding other people who enjoy the pure feedback loop of code, strategy, and money that trading provides for me. I’ve been working on timing and correlations between stocks and indexes as a source of alpha and having some success. Now trying to automate the process and absolutely loving the work along the way. Not trying to play market maker and not jumping between stocks. Instead honing in on an accurate model that monetizes a few really well. Some tools I find useful: TradingView and Pinescript indicators and strategies, excel models with exported data for backtesting, Python and Go for the backend machine learning, ChatGPT for faster iteration on new code. Would love to talk shop if you guys are interested: trading @ dianazink.com
Nice writeup. I appreciate the concrete discussion and especially the screenshots. I understand not discussing strategies, but can you elaborate on whether your trading is entirely based on technical analysis, or do you use external (or "alt") data feeds? In other words, is this a mostly closed system with inputs from Polygon and outputs to the IB API, or do you have a more extensive setup of custom data feeds of news sites, twitter, reddit, etc? If so, I'd be curious what challenges you face balancing those with historical backtesting, when some data sources might only have partial historical coverage.
All my data is from polygon.io and nothing external. I have been exploring using lookup tables of pre-computed historical values to do things like anomaly detection (ie, is this normal activity for this stock). A good example would be BXRX today [1]. I have been looking at options trading activity too and trying to use that as a signal.
Regarding open sourcing: don't be afraid to show your flaws; it doesn't have to be lonely if it is open source.
Obviously whatever trading bot you're running separate from the actual trading engine itself is somewhat proprietary, but it would be great for the community to get more of this type of software in the hands of other hackers.
Quantopian / Robinhood tried and failed, and the numerous clones since then have been somewhat sub par.
You might find https://www.quantconnect.com/ interesting! They offer a platform for quickly developing and backtesting trading strategies. They have a good community and overall do a great job. Give it a shot!
Eh, honestly only thing horizontal is good for is media, any documents/code is better in narrower vertical columns.
Honestly my perfect screen for work would probably had something like 1:1 ratio, so I can have 2 nice columns of code that are tall enough that can be split horizontally if needed while still being useful.
Yeah, personally, it's been a game changer just being able to load so much more into your field of view. Having two side-by-side makes it really easy to look at other parts of your code and make changes. I'm loving it! I'll give the H method a try for a few days and see how it goes.
>I went from not knowing how to solve a problem, blindly googling around and reading books, to just telling ChatGPT the problem, and then asking how it would solve it, then asking it to code the solution. This is absolutely insane and has easily 3x my productivity.
Very interesting. We can facetiously say that ChatGPT is using you as a medium between setting up algorithmic trading in Go!
While I was laid off and looking for work, I connected with a cryptocurrency market making firm that had access to a private broker feed which is not attached to any matching, so would occasionally go crossed and offer a pure arb opportunity for the same pairs. I had done some algo trading with Java in Kospi 200 options in 2011-2012 so I decided to put together a simple bot for them to try and grab the crossed markets when they occur. Even an incredibly simple trade like this requires quite a lot of work to get the risk management in place. I also took it as a chance to catch up on new Java features since I had been out of that ecosystem for awhile.
It did successfully grab the arbs but there wasn't enough juice to justify more work on it and I got a job in the meantime, so I open sourced the whole thing: https://github.com/abissell/cempaka
I mostly went with Java because it's what I knew best, and I thought was likely to be "fast enough" for the market I was in. I'm sure Go would be similar in this case but didn't want to have to take the time to learn it.
I do think that if Java can deliver on the combination of the foreign function/memory interface and value types, it might really start to look competitive for certain strategies which are just a bit too complex for the "do everything in the network card" approach. When the Aeron guys first implemented their protocols in Java, C#, and C++, C# was actually the fastest, which they attributed to the presence of both the optimizing runtime and value types.
I've also just started a blog series for getting started with algorithmic trading in NodeJS for complete beginners: https://grizzlybulls.com/blog/building-an-algorithmic-tradin.... A lot of folks would raise an eyebrow at the thought of using NodeJS for algotrading, but I've found it to be a great fit for my non-HFT, more swing-trading style, and I've used it successfully in production for over 3 years.
I caught the algotrading bug recently and I've been wanting to backtest a couple of strategies involving ETFs. Right now I'm only looking at the open, close, daily high/low. The problem however is that you have to account for dividends and splits, which I'm not sure how to do (Admittedly, I haven't looked too deeply into it). Any advice on how to do that? Let's say I want to backtest some strategy involving UPRO; would I be fine just using yahoo finance historical prices as is, or would I have to manipulate it in some way? Also, what would be the ideal backtesting software/service to use?
Harsh answer - if you have to ask this question here, then stop it now.
Thing is, you will be fooled by bad data - you will find strategies that works in a backtest but not in production. And the reason will be because you missed some important finance concept (like taxes, dividends, stock splits etc).
In this field 99% of success in my opinion is knowing what you don’t know. And only model that small part that you know you know and are fairly sure about it.
Ps. By stop it now - I mean stop algo simulations and learn about those concepts, make sure you understand perfectly what data you are putting into your backtest.
Fair point. Right now I'm just doing it as a hobby. I'm not going to be making any real trades for a while and even when I do I don't plan on making money from it (at least not for some time).
Depending on your provider, but there could be an "Adjusted Close" (Yahoo has it) which includes stock splits and dividends. I tried a few backtesting libraries, thought backtesting.py[1] was easy to work with and straight to the point.
Thanks. I've heard that yahoo finance historical price data is usually pretty accurate, but can get it wrong sometimes. Is there a better source for price data, or is yahoo finance good enough in your experience?
Yeah, I've been messing around with this for 4+ years so. Got to see COVID nose dive, meme stocks, massive build up, market taking a dump, bank collapses, and now a bull run again. I have no idea if it was always like this but it has been in insane.
This is really interesting. Have your strategies out performed buy and holding index funds, or are you mainly just doing this with a small amount of capital to learn how the markets work? I have always wanted to try algorithmic trading to learn about it, but I have always read it is a fools errand to think you will beat just buy and hold.
This is going to sound crazy given all the scams out there. But I was interested in testing the idea of small compounding returns. Like, could you get a daily 0.5% compounding return. Sure, you could go all in on TSLA for example and get a 1% daily return. But, could you do that with automation, using lots of small bets, across the entire market. You can, but there is a scale issue here. In that you need to make exponentially large bets as your bank roll increases. So, it's capped. Well, that's what I've seen. So, yes you can beat it but only with smaller amounts of money.
That makes sense to me and lines up with what buffet says about there are alot of people on wall street who can average 50% returns with 100k but once it gets into the millions it is much harder to find alpha.
32. It’s easy to make 50% on a million, but much
more difficult on larger amounts
WARREN BUFFETT: Station 9. We’re just about — yeah, we’ve got time
for a couple more.
AUDIENCE MEMBER: My name is John Dorso (phonetic), and I’m from New York. Mr. Buffett, you’ve said that you could return 50 percent per annum if you were managing a one-million-dollar portfolio. What type of strategy would you use? Would you invest in cigar butts, i.e., average businesses at very cheap prices? Or would it be some type of arbitrage strategy? Thank you.
WARREN BUFFETT: It might well be the arbitrage strategy, but in a very different, perhaps, way than customary arbitrages, a lot of it. One way or another, I can assure you, if Charlie was working with a million, or I was working with a million, we would find a way to make that with essentially no risk, not using a lot of leverage or anything of the sort. But you change the one million to a hundred million and that 50 goes down like a rock. There are little fringe inefficiencies that people don’t spot and you do get opportunities occasionally to do, but they don’t really have any applicability to Berkshire. Charlie?
CHARLIE MUNGER: Well, I agree totally. It’s just you used to say that large amounts of money, they develop their own anchors. It gets harder and harder. I’ve just seen genius after genius with a great record and pretty soon they’ve got 30 billion and two floors of young men and away goes the good record. That’s just the way it works. It’s hard as the money goes up.
WARREN BUFFETT: When Charlie was a lawyer, initially, I mean, you were developing a couple of real estate projects. I mean, if you really want to make a million dollars — or 50 percent on a million — and you’re willing to work at it — that’s doable. But it just has no applicability to managing huge sums. Wish it did, but it doesn’t.
CHARLIE MUNGER: Yeah. Lee Louley (phonetic), using nothing but the float on his student loans, had a million dollars, practically, shortly after he graduated as a total scholarship student. He found just a few things to do and did them.
Yeah, I'm not saying this is low risk. It's more about trying to build a system to be in the right place at the right time. It is massively risky. I was sort of hesitant to even write that comment because you constantly see all these youtube day traders selling courses on making 1% daily. If you try that you'll lose money extremely quickly.
This comment made me think of the tax advantage of buy-and-hold versus lots of small bets. Buffett also talks about this in his Berkshire shareholder letters.[0]
Yes, 100%. Like I said in the article there. I once lost like 40% in a few minutes in the pre-market because my app didn't have the correct sell logic. Honestly, I don't even know if I had the correct logic that would have even saved me, because the market was falling so quickly I probably could not have even for the fills I wanted. So, I've moved to breaking things into tons of small bets and really trying to manage risk.
Yeah, you could look at it like that too. But you need to have a platform before you can even do something like this. So, that's why I framed it like that.
1. I think buying and holding is more probable to have a higher return
2. Sometimes you get a cool api and think wow this would be fun, and next thing you know you've lost thousands on boneheaded trades.
I did something similar during the pandemic with Rust with the Polygon API (and instead of interactive brokers, I used tradier). Eventually I learned I actually had more fun building the thing than actually trying to beat the market.
I have won and lots thousands for sure. Haha. When stocks were on a rip the bot was making lots of money just because everything was going way up. Then, in 2022 when everything went way down, like tons of tech stocks, my bot sucked. So, I really need to add shorting or something. I'm still exploring things on the strategy side.
Yes, 100%. Everything looks amazing while the markets are going up. Sometimes, I've just shut the entire thing down when there is Fed news or the markets are taking a dump. That's a legit strategy too. Only run it while the markets are going up.
Yeah, right now I'm only set up to buy stocks. I haven't tried to short anything yet. I want to get to this eventually since it would be nice to make money when the market goes down too.
I'm sure you know this, but for others reading this who are novices at finance / trading, like I am - the gotcha here is that while the strategy may be symmetric, the risk is not - when buying a stock, there is a floor to how much money you can lose (the price you paid for the stock), while with short-selling, there is no such floor, since the price can rise to any amount and increase your losses to infinite. I believe that traders will use hedges to account for this, however, these hedges will eat into your profits if they are not exercised (but may save your bacon if they are!).
Yeah, I haven't explored this yet. The unlimited risk thing bothers me thought. Which probably sounds funny since this whole things is risky as hell. But, I was more looking into options or something to hedge but honestly everything takes so much time to learn about, test, and then do. So, I just wanted to focus on this and then expand.
Fascinating application of the language and a terrific write-up. I would presume a GC language would normally be a disqualifying factor in real-time trading, but I think I'm coughing up some premature optimization, especially with what looks like a pretty beefy rig. Congratulations though, this is spectacular.
Yeah, I never really looked a GC. Most of my trades take at least 500ms+ round trip. It's more in the latency to the broker and getting confirmations that I've been trying to tighten up. At least that's what I'm seeing right now.
I guess my presumption was that algorithmic trading was a very tight feedback loop, with as many controlled variables (i.e., GC) as possible, so I think it just subverted some of my misplaced expectations.
Excellent write-up! Is there somewhere where I can read up on all the trading jargon?
Also I wonder how can there be changes in the price of a stock after market if the exchange has closed? Isn’t the whole point that trades need to happen for stocks to get a certain value?
There's after market trading on venues. But because most of the liquidity has "gone home" you can't transact as much at those prices and those prices are based on relatively few/small transactions.
Yeah, this is where a system like this shines. In that you can build a tool, like top, but for the stock market. You have all the real-time data. So, you can look at where the party is happening at. Just by looking at the trade flows. This is where building your own system absolutely kills anything else.
https://www.investopedia.com/ is probably the best resource and chatgpt. I honestly, just google stock related terms and all roads lead to investopedia.
This is professional gambling. Embrace (weak form) market efficiency. Why do you think mutual funds have never generated profit in the history of their existence?
QQ for everybody since OP is using IB API which is notoriously bad that many wrappers have been written for it (ib-insync).
This is a general question, I'm wondering is there any good framework/wrappers out there that one can learn from to code up a complex trading application?
Like dealing with all the asynchronous nature of process/submitting trading and quotes messages.
Yeah, I ended up taking https://github.com/gofinance/ib and rewrote my own wrapper. This took a long time but has been stable since. I'm basically only doing buy lmt, sell lmt, cancel, and updates orders though. So, the logic is pretty simple. Catching all the return messages and structuring them correctly took tons of debugging, trail, and error. Basically, mapping the messages into the correct orders for state tracking.
I use ib-insync with asyncio and it is surprisingly easy, the most difficult is order management. Jason Brownlee’s SuperFastPython was the most approachable asyncio explanation I found.
Are there any examples out there ofactual algotrading strategies that made money in a real market, say within the past 10 years? I know these are closely guarded secrets, but figured someone may have published a strategy that ceased being profitable.
Here's a recent piece on a change going on in systematic trading engineering:
THE GIG ECONOMY COMES FOR HEDGE FUNDS
Platforms that offer money managers the freedom to build a business and maximize their return on performance while removing the hurdles of launching independently could change financial markets.
If the trend continues it could have a big effect on financial markets by making it easier for a wider assortment of unconventional managers to rise in the industry and offering investors better and cheaper access to them. …
Well-received start-up ClearAlpha Technologies has moved closest to the gig model. Its first offering is a commingled fund apportioned among its managers, but it has the platform to act as an exchange, matching investors to individual managers or customized portfolios of managers, cutting out all the expense of intermediaries.
You no longer have to have graduated as finance into finance. (In fact, we prefer if you graduated with some other math modeling heavy emphasis, think turbulence and flow, or actuarial modeling.)
I'm working on an algo crypto trading platform. I have pre-defined rules set up in NoCode fashion, but it seems like many people will want to code their own algos. I'm thinking of supporting a scripting option.
As a layman boglehead, what are the returns of doing trades like this as opposed to an index fund? I assume it either works or there enough money to hope it works, but really curious what the situation is.
This whole project started when I was looking for an investment adviser and didn't know anything. I was looking at all these funds that had like 4% return per year and then looked at stocks like Tesla that were up like 45% return and was like this doesn't make sense. When you dig in you see that these index funds are super super low risk. However, I like extremely high risk so I was willing to explore this a little. I don't really want to say you can get better returns, because you could end up losing all your money, extremely quickly betting on single stocks, it's basically a sure thing. So, that's why I turned to building a tool that basically bets for you. So, it's more about risk. If you put everything you had into TSLA you'd be up like 50% just in the past month but that's risky as hell. So, it's more of a risk question than a return question.
A couple of pointers. One on data, one on that RAM usage.
First I’ll go with the RAM usage because this is hackernews and everyone loves algorithms.
—-
There are a lot of libraries out there that do technical analysis, and most of them are designed for batch processing. TALib is an example - it works on large data sets but is not appropriate for live trading because it repeats calculations over and over and over again. If you have 10000 datapoints and calculate indicators, it’ll calculate 10000 of them. Add one more bar, now you have a dataset of 10001 items, which TALib will calculate the indicators on from scratch. Or maybe you just feed the last 10000, and still perform that calculation over all of those, but save that 10001st oldest one. Either way, it’s bad. Same goes for every library I’ve seen, presumably because no one would open source a production grade indicator generation system.
The production approach to this is somewhat different. Turn your features into state engines. Most features are just running calculations that are very easy to perform once per bar.
Moving averages are a perfect example - for a moving average of N bars, store N items in a fixed size array on the stack and keep track of the latest index to be written to. When a new value (X) comes in, increment that index, and grab the value (Y). Modify your old mean by adding (X-Y)/N. Then write X over the old value Y (It’s a ring buffer).
For EMAs, it’s even easier, because you just need to keep track of a numerator and a denominator - nothing else is needed. On a new value X with the scaling factor A (such that A^halflife = 0.5), the numerator N becomes NA+X and the denominator D becomes DA+1. Divide the two and you’ve got your new value.
If an indicator depends on another, don’t recalculate it. Break the indicators down into fundamental calculations and you’ll often find a lot of redundant calculations being done. Rearrange it all so it fits. Automate that process if you enjoy that kind of thing like I do.
Most indicators can be handled this way. The indicators that can’t - are rarely useful. After all, what you’re tracking is the evolving state of the market, and if you’re doing gymnastics over an indefinite number of bars, it probably doesn’t mean much.
The end result is that you don’t end up accumulating memory throughout the day. You receive a bar, you throw it through your indicator generators (all of which using a fixed size of memory), then you discard the bar and wait for the next. Save state at the end of the day and load that on the following market day.
The result will be a speed up like you couldn’t imagine. I promise. I run an indicator generation engine in a container capped at 40MB ram on one CPU, and it generates hundreds in much less than a millisecond after the bar arrives.
—-
Now, onto data. I recommend cleaning your data. You have a screenshot of a Tesla chart in there with some funky highs/lows every so often. It has been a long time since I’ve worked with US equities (and gladly so, it’s a mess of a system!) but the following is the best of my recollection.
The trades you receive will come from several sources. For US stocks, there are several different venues that operate their own order books. These will operate as typical markets between the open and close of the day. By typical I mean that the bid and ask represent what you’d get if you market order instantly (which you can’t), and the market trades on them have to take from the bid and ask side of the order book (formed by people placing limit orders).
However. There’s also the ADF - the Alternative Display Facility. This is the DIY of trade reporting (and quote reporting, but no one does). If someone sells some shares to their grandmother for a low price in exchange for the recipe to her famous apple pie, the ADF is where they can tell other participants about that trade, manually, subject to fat finger errors, prices of weird fractions of cents, and very relaxed constraints on timing.
It’s also where dark pools post trades.
The problem with this is that this data has no direct relationship to the rest of the market. This is why, every so often, you’ll see those blips.
If you don’t clean them, they’ll play havoc with any indicator that uses highs and lows.
The other problem is that ADF trades - at least when I last analysed this very issue - are not rare. They make up a large fraction of trades. So my approach was to clean ADF trades more rigorously than the venues by matching them against prior prices from an ADF-free background.
Are you using number of trades or sum of trade sizes?
Because if the former, there is little distinction between me firing off two market buys of 100 shares each in quick succession vs me firing off one market buy of 200 shares, so your sampling shouldn’t be impacted by the difference.
I’m not altogether convinced by volume sampling. It’s an idea popularised by De Prado, but I’ve never seen it actually work in practice. It makes you trade more when the market is going through turmoil, it makes you trade more over time (as volume per day generally increases), and I haven’t seen any evidence of the importance of information content.
If you’re trading based on patterns in the market, it’s easy to lead yourself to believe that you’re predicting the market. That isn’t the case, though - the market is formed of many many independent people making guesses.
The thing you’re actually doing is predicting what other people are going to predict. If there’s an established pattern, like some moving average crossing another or a wedge or anything else like that, the reason it tends to complete is not mystical - it completes because other people see the pattern, think it’s going to go up (or down), buy (or sell), and then that has the effect of pushing the market in that direction (it also means that anyone late to the game can’t benefit from the movement).
As such, the strongest strategy when trying to use technical analysis to determine possible market moves is to use the resolution that everyone else is using. This is overwhelmingly time-based. There are people trading in the 1s regime, the 1m regime, the 15m regime, etc, and they’ll often stick to that and execute trades with a proportional rollout time and aim for a proportional profit.
If you pick just a random number of trades or amount of volume that suits you, you can easily find that you are out of sync, competing against no one in particular, and you’ll see that it’s impossible to find a pattern.
Many other people use price levels. So they’ll have their limits and stops at round numbers, or at percentage changes on the day, week, month, etc. So there is an argument for price bars too.
Do you mind sending me an email and maybe we could chat?
Thank you for your amazing comments. I really like your idea about indicators and saving state. I'll give that a try! Yeah, Marcos López de Prado is actually where I read about the tick bars and sampling at higher rates. You need like 2 phd's to read his books though. haha. I am doing this based on X number of ticks and not even looking at volume. I was using tick count as an indicator in that you can really see patterns when activity picks up. This sync issue is really really interesting and I'll explore this.
"The thing you’re actually doing is predicting what other people are going to predict." I think I've actually seen this in the data. In that you can see this mini-cycles almost when you really zoom into a fast moving stock. I'll check out. Both your comments are amazing and it 100% shows you know what you're talking about.
Yes, both. Basically, I learned early that I needed a backup power supply. I have a couple that power my computer, monitors, and the internet router. For the network, that one is really tricky. In that, you're screwed. I've only had this happen once and then I needed to manually sell all my positions. Luckily, I didn't lose any money.
What's cool about this is that you can look at the IB TWS client and see things happening in real-time. So, it acts as sort of a sanity check. I know they have that gateway too but personally I like to look at the client all the time too. My workflow is to run the bot and the TWS client side-by-side and watch it make trades, see how things are moving, etc.
Really funny coincidence that I am seeing Interactive Brokers mentioned in this good article. Story + rant time, feel free to skip if you are not interested how one guy gambled and lost. It's also a tentative call for partnership if somebody is interested. And a call for chat if anyone has any interest in the topic.
(It's also kind of off-topic, my apologies for that. To me it seems semi-related but would agree with mods' assessment if it doesn't match mine.)
---
I recently "broke up" with some extremely toxic "investors" that wanted me to do a fully parallel trading demo -- meaning it receives ticks for N instruments (I successfully got to little less than 300) and trades with each of them depending on strategy. All in real-time.
I got very far but the open-source libraries for Interactive Brokers are quite low quality in general and it was very hard and slow to progress (one of them couldn't even post orders, another used Mutex-es for "parallelism" which was of course not parallel at all, another one seemed to work well but only worked on servers of older versions compared to those I have access to, etc). I also had to gather code from separate places and assemble my own Frankenstein as I went along.
Eventually I muscled through but by that time I have drained all my savings, my tax fund and even got a new loan. And the liaison + the investors of course refused to acknowledge the demo was basically 90% done (couldn't do full parallel trading due to defects of the IBKR libraries I have used and I used like 5 of them, and was in the process of repairing 2 of them to unlock the said full parallelism). They refused to send a small pre-funding wire (we're talking something small like $30k - $50k, not millions; for the work that was done, namely months of professional Rust programming work, that's a -75% discount if we look at market rates).
They had all the proof and paper trail they needed to see that I was very close but I had to stop because I was literally about to be unable to pay rent and bills. They still did not concede. Obviously I picked a job and dumped them but I still have some regrets because the door is technically still open (they have not cut my access to the IB Gateway servers that they own), but other factors like now-ruined health are seriously getting in the way as well. Not to mention completely shattered trust.
They insist "if you just finish the demo we'll give you money" and used every gaslighting technique I knew about (and many I didn't know about, so I learned a lot about gaslighting from them, lol) to try and coerce me to keep working for free with zero guarantees of funding -- but I no longer trust such rich investors to fulfill a promise without a binding and legally enforceable contract; I am in Eastern Europe, they are in the USA, even if they sign contract and violate it I practically cannot do anything to them i.e. I can't afford to travel and sue. Also they insist to get access to the source code but swear to everything that's holy that they will not run away with it and never give me a penny. Which is exactly what I think would happen.
I had to draw the line at one point. To me it was red flags all around.
---
I made many concessions and I will not make another one until they do. I'll be finding a new job soon again since the previous contract was agreed upon to be for several months, I helped a team accelerate bootstrapping a business-critical product (and we did that successfully). And then maybe, just maybe, after I settle a bit, I might work an hour or two on this again during some evenings. Maybe. And that won't be used to provide the demo to these toxic investors -- I'll use it to have a proven implementation that I can pitch to other people. These guys don't deserve the time of day from me.
I have quite a lot of good code (mostly Rust, but also some Elixir and Golang -- I experimented a lot) that interfaces with Interactive Brokers and I have many building blocks of a good trading bot framework. That kind of creative and thorough work needs funding and peace of mind to be finished properly, of course. And I can't afford to invest so much energy on this without a stable income so I am shifting focus to that for however long it might take for me to feel comfortable to invest time and energy in this again.
If any business person wants to partner up -- give me a shout. Mail is in profile. I also wouldn't refuse fellow techies (or anyone else really) stopping by and telling me how stupid and naive I was -- I'll agree right away and say that they are right. Because truth is truth.
I admit I still can't get over the fact how close I got but I can't afford to count the stars while my livelihood is being endangered the longer I coast on savings. Dreams don't pay bills. And some dreams need more than one person to be fulfilled.
Rant + story over. Thanks for reading if you stuck this far.
How did you get into this kind of shady arrangement? This is not how it works if you go to a reputable prop firm. Also sometimes they already have infra setup for you to build on.
A liaison of these "investors" found me somehow. I can only hypothesize it was through HN.
And yes I was tempted. I also don't have any trading credentials. I'm simply a senior programmer with an eye for details who always wanted to try his hand at algorithmic trading. So yeah, I got hooked. :(
It all boils down to what do you know in terms of languages and how good a quality of libraries you can find.
IBKR is kind of an elitistic VIP club, not just anyone can gain access. Thus you won't find a lot of libraries. There's a good number of them but the quality is not great.
I always wanted to learn OCaml by the way but in my current life and career phase I still can't justify the time and energy expenditure, and I know it will be significant.
also having worked in the space: HFT should not exist. Break up the day into segments and have a single crossing. Do it every five minutes or whatever for sufficient timeliness. the millisecond race does not make anything better for anybody except the people doing the trading.
I think you would still end up with races, they'd just move around. IEX's "speed bump" was just marketing and didn't really matter to actual HFTs.
Plus, some of it is unavoidable if you don't have a single unified exchange. Where there's latency, there's inefficiencies, and where there's inefficiency, there's profit to be made. And competition among exchanges is healthy for the ecosystem, so I don't think we'd want to consolidate.
And lastly, HFT has consolidated so much that I don't think it's worth worrying about. Virtu literally switched sides and make most of their money on order execution. Industry-wide, HFT revenues are down like 80% over the last 5-8 years. Between wholesaling/PFOF taking non-toxic order flow off the lit exchanges, and banks finally wising up on not being pants-on-head about their order execution, it's literally just sharks in the pool now, there's not even any water.
Also, "exchange competition" is a thing in US stocks, but not, for example, in the futures' market. And there's HFTs in futures too, so having a single exchange wouldn't "remove" HFTs (not that you'd want to).
Single point of failure is a simple justification right off the top of my head. What happens if you have a critical failure that brings down an exchange for several days?
As a counter I had some VW shares I couldn’t access for 6 months because they were moved to another exchange. 20 calls to the broker to find out what had even happened. Eventually I was able to sell them.
Well, your first problem was investing in VW (KIDDING!). But seriously, I usually stick to self-serve stock purchases. Was this a situation where you had to contact a human to make the buy in the first place, or was it that you bought it yourself, then VW changed up their infrastructure and inadvertently locked you out?
Self serve, one one of those trading sites. No human contact, until the site wouldn’t let me trade the shares. I think the exchange changed up things and I am buying from another country so communication issues etc.
I checked my email history on it. There isn't much but from what I can tell the exchange was changed that they were traded on but there was also a change in back-end broker and there was a delay of them moving the shares to the new back-end broker. So maybe the change of exchange wasn't the actual reason, just a coincidence.
I assumed that some exchange provides enough volume that without it there would be worries that you aren't getting true price discovery. Isn't that a concern with low trading volume? Not that that would necessarily be true, but there would be enough concern to cause some people not to trade. And others to want to trade specifically with those kind of nervous people wouldn't because the people they expected to profit by were staying put. And then it would cascade.
Running the matching engine at discrete times doesn't eliminate the speed race, as that's mainly about jockeying for queue position. You could put in an auction instead, but that's just changing the rules of the game rather than eliminating the game.
But really, the key thing is that liquidity provision actually is a service -- market makers are basically selling insurance. There's money to be made doing his, so people will compete to do so. That's what the millisecond race is about. Faster trading -> less risk for the MM -> less capital needed -> lower profits acceptable. Contra what you're claiming, the race is cutting into profits, not raising them.
If you want to reduce market-maker profits, crack down on payment for order flow, and let everyone compete for a chance to trade against it.
If you know more than others, that's great—please share some of what you know, so the rest of us can learn—or else don't post. Sneers and putdowns only make everything worse.
He’s referring to the fact that the HFT “provides liquidity” as in getting a share off your hands and flipping it to someone else that is not you a moment later. This way you didn’t have to wait 3 seconds or 1 minute or whatever to sell your shares because someone knew they could flip them (alpha) on a short notice so they participated in a transaction with you. He’s assuming that without the HFT, the transactions wouldn’t be as frequent and you’d have to either a) wait for the fill on your trade , or b) lower your ask price if selling under time deadline. Analogously for buy orders.
The only parties with semi-global visibility are prime brokers by definition; they see every position of everyone who custodies with them.
The parties with global visibility of the US market are TRF (trade reporting facility) and those are the only parties who can sort of evaluate the HFT claims without bias or vested interest. Most of the studies in the field have some sort of an angle or vested interest so it’s hard to evaluate the veracity of the claims one way or another.
A counterpoint to HFT is that stock markets existed before the the advent of computers and they had runs, panics and blow ups just like regular markets do now.
I don’t have time to reply properly until a few hours from now.
In the meantime what I can say very simply in the hope that someone else knowledgeable can contribute earlier.
It’s simply an empirical fact that the costs of intermediation to the system are the lowest they have ever been. The US and other global electronic markets are incredibly efficient and deliver unmatched liquidity, information efficiency, and the lowest costs to the entire capital markets than at anytime in history. That march forward is continuous and brutally competitive
There are many many way to see this and measure it, any serious quantitative analysis, by professionals, for instance trained in econometrics and with access to the raw data, like those at say the Fed, or those operating exchanges, as examples.
Your dismissals seem to be very biased and you’re not allowing the possibility of being wrong.
As a practitioner who worked on the systems you’re citing I can tell you’re wrong on many fronts.
As far as costs are concerned, you’re not answering the most relevant question that applies to most users of this forum: costs to retail traders, and are only tackling the institutional side of things. Nobody from retail concerns themselves with costs of intermediation because those costs are irrelevant to retail. Not at the order volumes that don’t even incur slippage.
Plus, fees are only part of the game. I can give you an NBBO improvement now to be compliant with the regulation, but I am not guaranteeing anything in terms of price in the next second. See where this is going?
Edit: by the way, what you’re repeating is what I jokingly call a “party line”. Especially if you look at the hard cold data. For example people often cite narrower spreads etc. but even with Interactive Brokers you can get MPID displayed on the NMS aggregated full depth order book and see who quotes how much and at what levels and the spreads don’t add up to the half of the myths people keep repeating. It’s easy to hide the real numbers in the _averages_ and various other statistics though.
I’m a practitioner and no, the costs to retail traders is included and analyzed.
The most vocal critics of HFT are very often previously practitioners who are upset when their strategies and models becomes obsolete are are outwitted by even more efficient operators.
Give any example of where you are going?
I will say RegNMS and NBBO regulations are actually preventing even further efficiencies. Dark pools and off exchange matching or internalization are complex topic that are easy to misunderstand. There are absolutely bad actors to be found in the system. This is true in any system. But on aggregate the system is continuously reducing costs and improving efficiency.
Intermediation costs are a friction on the real economy and capital markets and they will always exist, but on aggregate they are dropping for all users, institutional and retail.
In regards to party line, it’s absurd, you can simply take the aggregate income, not profits, of all the top HFT operators, their income is their counterparties costs, and when attributed per market, this number is continuously dropping on aggregate. If you look at an individual firm, you can see it’s income growing, however that will come from 2 dimensions, either expanding their operations to other markets, or taking share from a competitor. However if you sum all profits across all HFT operators on a single market complex, say US equities, on longer economic timescales, this number is continuously dropping. Obviously during periods of market volatility this number can increase, but the trend over years, will be always downward.
HFT is a absolutely brutally competitive industry.
I’d be interested in what you saw working on such systems seem “wrong” or “unfair”. My guess is you don’t understand that all is fair in war and that includes HFT, as long as it is legal.
By the way, your description of profits dropping for market participants reminds of the concept you may find interesting.
Here’s an example of P&L dynamics:
At time t, your throw an unbiased coin. If it comes out heads your wealth multiplies by 0.6 with probability 0.5, or if it comes up tails, your wealth multiplies by 1.5 with probability 0.5.
Now you could simulate this process and take two averages. One is an ensemble average, averaging over many trajectories (of many participants) at a pre-defined time step t.
The other average is a time average (what happens to a a single trajectory picked at random over time). You may find the result interesting and close to what you’ve just described. This result is because of the process being non-ergodic; but it exhibits a few “winner takes all” and has nothing to do with the properties of the winners. It is a purely random property. An illuminating exercise. There are quite a few other results like this one from the field of stochastic processes that relate volatility bounds to your expected P&L, etc.
I am not disagreeing with you that the costs are dropping and that participants benefit from that. In fact I agree. Your projection that I find the field unfair is also unfounded. I love the field and find it immensely interesting. I just don’t take mythology surrounding it at face value because I often found it to be a) outdated b) full of mythology but no hard data.
What I am saying is that the thesis that the costs are dropping due to HFT style strategies has not been proven. Majority of the HFT tend to be market makers which tend to help with liquidity but not all are. I agree that liquidity helps offload or acquire large stakes.
As for NBBO and the effect a regulation has on markets you seem to be extremely US centric but if you go across the ocean and find out that Europe has no concept of NBBO at all, and that a retail person trading experience is equivalent for getting a different price on Amazon depending on which web browser they use you could imagine how that would make an average retail person feel.
Another example of the regulation is trying to move a large stake outside regular hours when NMS is suspended. Why do you think OTC block trades are pre-arranged at a fixed price? You can even look them up in relevant reporting facilities.
As for dark pools, ATS and internalizers there’s nothing difficult about them. I don’t think you’re doing anyone any favors by obfuscating an extremely simple concepts. Those market participants with their specific mechanics that are learnable. What’s complex is how to devise strategies and how to rely on the liquidity sources to get your desired fills and desired rates. You’re describing bread and butter of trading at an institutional level and the fact it seems difficult can maybe be attributed to the fact that it is opaque, doesn’t enjoy public communities, e.g. very little blog posts exist on the subject and the knowledge is sort of centralized to a specialized corners of the industry. But it is learnable without great difficulty if you have access to the resources.
What’s surprising to me is that a lot of people can talk about abstract concepts in computer science and then don’t connect facts that bridge into a separate discipline. As an example I could refer to consistency that people love to rave about but somehow forget the concept the moment capital market is introduced.
And yes, I’m familiar with the no trade theorem and relevant academic concepts but they are all models that often don’t translate to real life due to extremely limiting assumptions that don’t enjoy any connection to reality. I think it’s a problem with the community in this particular industry to take academic market models at face value. Often you will find that looking at a model carefully with assumptions that mirror real world, your performance gets better.
It’s true I’m US centric. What I do know about European equities market structure actually points to one specific problem that actually we also have some of in the US. I’d suggest you look at clearing house collateral regulations, especially around ETF transactions, creation and redemption, and posting of OTC positions. That part of the system, which btw isn’t HFT, is very very shady.
I agree it’s all quite understandable if people had the resources available. I would argue part of the reason it is opaque is the fears of operators that too much public attention can bring a lynch mob. This business can literally be one of the most profitable activities on the planet, but contrary to populist sentiment this is not evidence of wrong doing, it’s simply because it’s so automated and scalable and all electronic.
Back to ATS and dark liquidity, this isn’t as trivial to explain as you suggest, it requires understanding of flow toxicity and information theory to be understood completely. You must know that in markets, transactions can be beneficial to both parties as they have different objective functions and timescales.
What evidence can be presented that supports the idea HFT operators are extracting increasing costs on the system? I’m not aware of any.
Yes, toxic flows and adverse selection are the magical buzzwords that get thrown around a lot. I’m familiar with glosten-milgrom model and private information and all related paradoxes.
What you’re failing to disclose is that there is an easy way out of solving the adverse selection problem. For example you could buy “uninformed flow”. I think that you could agree that for the touted sophistication of the field you would expect something … more sophisticated?
Again, don’t get me wrong. I love the field but I think it’s stagnant in certain aspects and I like to have a sober view of it.
EDIT: As a thought experiment, envision a setup where the trading strategies compete on the basis of the strategy itself, with a single global market with a single API that takes bids and offers in rounds and anyone who wants to is allowed to participate for free with no fees whatsoever. For the sake of example, suppose it’s a government owned and operated project just like the GPS (for which you don’t have to pay a subscription) in your phone. No market access fees, no market data fees, no preferential latency treatment. A perfect coding competition playground.
How many current market participants do you think would survive in such an environment and if your answer is different than the current number, why?
While I fully believe you are an experienced practitioner however I feel your thought experiment actually shows you don’t understand the underlying nature of capital markets in capitalism, the relationship between risk and liquidity, and the fractal nature of timescales of participants. I’m fortunate enough to have both a background in HFT but prior to that a background in many other aspects of capital markets and the broader financial system, which gives be a better perspective than the typical HFT strategist or engineer.
It’s too late in the day for me today to attempt to write a coherent and accurate explanation of exactly why I say that. However I will say you don’t understand this isn’t a zero sum game and it’s an incorrect argument to claim HFT is an arms race in a game like a war, which is a negative sum game.
You start sounding a bit like chatgpt with the “it is I who have the sole power of understanding” and self contradicting arguments. I never mentioned zero sum games or anything of the sort. Please don’t ascribe your thoughts to mine. I did helpfully screenshot your comment invoking war and claiming HFT is like it. I can jog your memory if you’d like. Now you’re claiming the opposite. Your arguments start becoming internally inconsistent.
Your risk argument doesn’t hold water because liquidity is not the only risk. There are at least 20+ I can think of from the top of my head without even trying.
HFT is war but the system as a whole isn’t. My impression of our interactions on HN is you have a valuable insight that perhaps I can even agree with. However unfortunately I don’t know what it is yet as you haven’t explained properly what your hypothesis of negative effects or externalities HFT operators have.
HN isn’t the best forum for this type of ongoing discussion. Engage me on reddit if you like. u/alchemist1e9
"This aspect, the platform itself, seems to be often overlooked in most discussions. Many conversations revolve around strategies (mean reversion, trend following, linear regression, etc.), and backtesting, without fully addressing the practical mechanics or logistics of strategy implementation, particularly in the context of live, intraday trading."
I'm glad you had fun, OP, but also I think I can shed some light on why most people discuss strategy.
Trading is a perfect storm of ridiculously high tech, ridiculously complicated, ridiculously regulated (Not over-regulated, mind you, this isn't a value judgement. But the amount of regulation is extremely high.), and ridiculously competitive.
But that said, it's the last bit that drives it all. Since it's so competitive, even though building an order entry system, and a risk system, and a position-tracking system, etc is a huge accomplishment (again, congrats OP!), it's table-stakes to even dip your toes in the pool here. Trading shops can attract top talent and robust, bespoke trading systems are basically cost of entry.
So people talk about strategy because everyone already has the table-stakes stuff and are now trying to make money with it.
It doesn't help, too, that lots of market participants aren't even playing the same game. In HFT, we operated on trades with alphas that lasted a few seconds, where races to entry/exit were battled in shaving nanoseconds off FPGAs being able to shoot out orders and microseconds off wireless networks flying market data around new jersey. Meanwhile, banks are more concerned with elections and geopolitics than they are about the weather in Carteret. (Rain = no microwave network for the day). And then there's a million strategies in the middle with alphas that last from hours to weeks.
So it makes it really hard to even speak the same language to each other when talking in common forums.
It's a fun world. I miss it sometimes.