While working at a Big Bank, a new policy was instituted where we had to fill out time sheets. We also had to allocate our "budgeted time" into these timesheets (Regardless of what we actually did).
It ended up being a bin packing problem that no one actually did till I wrote a script to do it for me. This in turn led to some interesting conversations with my manager and the head of Capital Markets and Banking's CFO.
Anytime these discussions pop up, it's good to link to the documentary "Everything is a Remix" (here is the 2023 updated version: https://www.youtube.com/watch?v=X9RYuvPCQUA" )
It highlights how essentially everything in culture is a remix of some previous version of the culture (or some aspect of it). E.g. even language is largely a remix of earlier languages with some occasional creation of new words.
Also, while true that there is definitely an oligopoly, it's also clear that the Long Tail is both huge AND easy to access. In the past, you could only see movies by going to the movie theater and there were only a limited number of theaters. Now, you can both create and consume media in niches as small (or combined) as you want.
> I’ll happily order from somewhere I know, but not some unknown restaurant
Exactly this.
Part of why I go to (or order from) a particular restaurant is that I've been there before and I have an expectation of the taste, quality, service etc. I could also, in theory, go look into the kitchen and see if there are safety issues etc (or outsource this to an inspector). Feels a lot tougher to do this if the "restaurant" is just a label on top of the dark kitchen product.
On a side note, this is why chain restaurants were so successful: you could go into one in any part of the country and have a predictable experience with some base level of quality. The mom and pop restaurant in the town you've never been to wouldn't have that same offering to you (it might for locals though).
I imagine this correlates somewhat with having children.
It's both a "traumatic" single event (especially for the mother) and also traumatic over time in that your sleep schedule is affected, big changes to your life etc.
People joke that evolution made it so we don't remember how hard it is to have small children but maybe there is some truth to that statement.
Good old saying - if you work hard manually, rest by doing something mentally challenging, and vice versa. For most of us vice versa applies hard.
I do something every day or evening, even if its just 1.5-2h evening walks. Days I don't do anything are definitely harder to fall asleep, to say the least, and some small achievement is missing.
I take a stimulant for my adhd. If i didn't have physical exhaustion (usually a 10km 2h walk before sleep) I don't think I could sleep at all. It still affects the quality of sleep (not much deep sleep) though.
Now, sleep quality really affects everything. I think in general, mental health/sanity is hardly taken seriously in the tech industry. If I had employees I'd rather they had enough physical activity and sleep over a long and sustainable period than them sleeping under their desk hacking through the night.
- has worked in finance for both hedge funds and banks
- has managed a project where KDB was mandated to be used by mgmt.
- on the above project, tasked one of the smartest developers I've ever worked with as the person to learn KDB and use it in the application
- I'm a SRE (and former Operations) so that colors my perspective.
Given the above, I list out the pros and cons:
PROS
- KDB is pretty fast (on some metrics of fast)
CONS
- VERY few people can write/read good Q (compared to say people who know Pandas/SKlearn etc)
- The learning curve is INCREDIBLY steep. Even the most cited documentation and tutorials have something like this in the intro "are you really sure you need KDB? B/c Q is REALLY hard to learn"
- As you mention, open source industry standards have come a LONG way since it made sense to have KDB (e.g. in the late 2000s/early 2010s)
Conclusion:
If you have a lot of in house expertise, then sure, it probably makes sense. If you are starting from scratch, I would not recommend it.
On that note, this point stood out:
> People have built their livelihoods around it and use it as a hammer for all sorts of nails.
If you work in the industry long enough, you will find a lot of complexity added to systems for three reasons:
1. Some things in finance really do need to be complex due to the math etc
2. Smart people with quant backgrounds tend to LOVE complex things.
3. Smart, rational people realize that adding complexity is one way to build a fortress around their job. This is particularly true in high paying firms where people realize that it's their knowledge of the complex systems that is keeps them in that high paying job.
Give that, if you are looking to make a name for yourself at your firm: making things run faster, with fewer issues etc is a good way to stand out. Just be careful that you don't eliminate so much complexity that people get mad at you.
"The governments of the world provides these big weather companies (weather.com (cough IBM), Accuweather (cough IBM cough), etc) a metric shit ton of their data completely for free (by law) including data transfer. These are things like radar, satellite, ground station data, forecasts, composite models, etc. These companies profit substantially on it, as in billions of dollars"
Michael Lewis' book The Fifth Risk goes IN DEPTH into how Accuweather/Weather.com and the government were interacting (particularly during the Trump administration).
I highly recommend the book in general and for this particular story in particular.
I should really write up a thread on some of the stories from those days.
reply