> The bank, which has a $10 billion annual tech budget
Am I the only one completely befuddled by this number? What the fuck are they doing with these money and 200k servers? These are facebook numbers. For a bank. What?
I'm not sure why you find this surprising, a bank has a much more complicated problem space than Facebook.
- They have a reasonably similar number of users (a fraction, but a large one).
- Mistakes cost a lot, so they have to be a lot more careful. It's a lot easier to make money hacking a bank than hacking facebook.
- They have to comply with all sorts of regulations.
- They probably don't trust their own employees to not be trying to commit fraud.
- They have to parse data on a scale that is likely similar or greater than facebook's. To detect fraud/lost credit cards/.... To decide who to give loans to. To price insurance. To decide how to trade stocks. ...
- They have to run a physical fleet of devices in the field, outside of their control, that have to give people the right amount of money ~100% of the time.
At a glance I see that Facebook has something like 300 petabytes of data [0]. I've worked at a bank, my team had something more like 10, but I don't think much of it was things like video that are just naturally huge. BOA is also approximately an order of magnitude bigger than the bank I was at.
One rumor I heard while there was that there had been a bug in one of our mobile apps that had been costing us a million dollars a day in server time.
How do you spend that much on software and have such an abysmal usability and security story? I don't think there is anything technically difficult about the consumer software they offer, namely https://www.bankofamerica.com/.
I think you would find that their security is better than you think. Otherwise they'd be hemorrhaging money left right and center to North Korea and the likes.
As for usability, probably a degree of incompetence, mixed with design-by-committee and legacy. Edit: It's worth pointing out that banks usually don't gain or lose customers based on their UX, so it's not something that the business optimizes much.
I don't think there is anything technically difficult about almost any of the consumer software facebook offers, except scale. The same applies here but exchange "scale" for "scale, reliability, security, and regulatory compliance".
> I think you would find that their security is better than you think.
You can get in to my account by verbally relaying my grandfather's first name over the phone. You can open a bank account with a SSN and no photo id. What security? Their "security" is a fraud department, much like our credit card industry.
> Otherwise they'd be hemorrhaging money left right and center to North Korea and the likes.
This is not how transactions work.
> Edit: It's worth pointing out that banks usually don't gain or lose customers based on their UX, so it's not something that the business optimizes much.
All banks offer the same shitty experience. What does differentiate them if not their software? They offer literally nothing my local credit union doesn't offer.
> I don't think there is anything technically difficult about almost any of the consumer software facebook offers, except scale.
No argument here, but facebook at least manages to hire designers and not impose weird non-sensical patterns of auth, like "Look for this image when you log in".
Their security questions are a massive security hole. Credit cards don't require a PIN. I don't see much indication that it is difficult to steal from people.
>What the fuck are they doing with these money and 200k servers?
Gluing together layer upon layer of legacy systems that are so old & opaque they are essentially black boxes nobody dares remove or replace.
oh and that small regional bank that was acquired 4 years ago? Yep...they've got an entirely separate stack of legacy systems. And that other jurisdiction with different data laws? Everything is different there too.
It's insane looking under the hood of these things. How banks manage to not lose half the money daily is a complete mystery to me.
The sell-side investment bank alone is a huge consumer of this tech budget.
13 exchanges x 3000+ securities x 2 quotes for each security x 23,400 seconds in the trading day (with the very very very generous assumption that quotes only change once per second)
That’s the inbound data alone, now consider all the algorithms they have to run on it, the models, the strategies, etc.
And then to each of their several hundred institutional clients, they have to give a unique and dynamic price.
Again, this is only the sell-side investment bank, not even looking at their absolutely gargantuan consumer bank business
Pen-and-paper banking doesn't exist. Everything is done electronically, and BofA is a massive behemoth of a bank.
Tech companies also aren't subject to the same level of scrutiny / regulatory requirements as big banks, plus the need to support old processes and software, etc
I could turn this around and ask, why would Facebook ever need infrastructure on par with BofA? I'm not an expert on infrastructure, but given their size, revenue, market position, number of customers etc, there's nothing baffling about these numbers.
When I wrote software for telcos, we had levels of logging that cloud users are only starting to reach ten, fifteen years later. From what I hear banks are even worse.
When you are collecting that much of an audit trail, that data transitions from being a problem to manage to the problem to manage.
I wonder how many bespoke Kafka work-alikes there are out there that are older than the authors of kafka.
any software the bank is licensing can run into the 10's of millions per year, for anything they use: loan orignation, underwriting, etc. they have a lot of users also so just think about their basic workspace cost per user for all the software necessary.
- some banks are spending huge on internal infra in order to tie together various arms of the bank, e.g. consumer lending with small business lending with personal banking, quantitative finance getting better feature vectors from other parts of the bank, ... . banks understand that they have mondo data, so want to make it accessible to everyone. sort of like the API craze, where people build cool stuff with APIs once they are exposed; you can't quite imagine what but know value added stuff will bubble up
- the instantaneous nature of things in today's world is bleeding into finance, where banks want to advertise/offer near instant access to credit, loans, etc. instead of turn around time of days. in order to do that, it helps a lot to basically unify data across tons of previous disparate orgs, shared infra, etc.
- cybersecurity/auditing/compliance can be very expensive to license or contract for and occasionally has to be run on company hardware due to legal issues
> " What the fuck are they doing with these money and 200k servers? These are facebook numbers. For a bank. What?"
Are you surprised that a an organisation handling billions of real money, millions or billions of financial transactions, and decades worth of legacy systems, can make do with only as little in servers and software development as a social network?
Am I the only one completely befuddled by this number? What the fuck are they doing with these money and 200k servers? These are facebook numbers. For a bank. What?