> A commit was made ~9 hours ago replacing usage of load_instruction_at with load_instruction_at_checked, which actually confirms that the program being executed is the system program. It's interesting that this commit was made ~9 hours ago and the exploit happened a few hours after that. Possible that an attacker was keeping an eye on the repository and looking out for suspicious commits. Could be that the Wormhole team spotted the bug, patched it, but the attacker got to it before the patch could be rolled out. Super important to keep these sort of patches lowkey and to try to stuff them into larger commits.
> It looks like maybe Wormhole tried to do this by including the change in a much larger and unassuming commit called "Update Solana to 1.9.4". Not sure exactly what happened here, but a clear lesson to try to deploy before making any patch details public, if you can afford to do that. Of course this ends up being at odds with Web3 ideals, so not always clear how to best handle these sort of things.
One thing that would be healthy for the larger ecosystem would be for chains to build in regular "maintenance windows" where trades are halted by contract, at which time sensitive security patches can be rolled out to the codebase and then to the network by the maintaining team. Of course, this requires a lot of foresight. But the alternative is something like this.
Also - why would you ever set up a system where the majority of its assets can be drained by a single transaction, whether legitimate or not? Just because it's not centralized doesn't mean every transaction is made equal; one could require timeout periods for transactions above a certain amount, or any size of transactions could trigger a halt once a certain amount has been bled in aggregate, that requires supermajority consensus to "unlock" the chain. That this wasn't built in, in an ecosystem where hacking is rampant, by a team focused on creating a cross-chain transmission utility, is surprising, to say the least.
There are far, far worse things than a halt on trading. There are many domains where unsupervised 100% uptime on systems with access to a substantial portion of an organization's assets is ideal; finance, whether centralized or decentralized, is rarely one of them.
> Also - why would you ever set up a system where the majority of its assets can be drained by a single transaction, whether legitimate or not? Just because it's not centralized doesn't mean every transaction is made equal; one could require timeout periods for transactions above a certain amount, or any size of transactions could trigger a halt once a certain amount has been bled in aggregate, that requires supermajority consensus to "unlock" the chain. That this wasn't built in, in an ecosystem where hacking is rampant, by a team focused on creating a cross-chain transmission utility, is surprising, to say the least.
Easier said than done. This is obviously vulnerable to sybil attack- the attacker just make a lot of different transactions. You could build in a circuit breaker that would slow down transactions to only allow a certain rate of increase in total dollar volume going through the bridge. Hopefully this would slow the attacker down enough.
Now you're basically trying to devise a heuristic of what every single attack might look like in terms of transaction volume increase. If you're wrong, the bridge will freeze right when a lot of people want to use it. Plus, you also need to build a price oracle now.
All of this is possible, but maybe it's better to spend that effort on just making sure the code is secure in the first place.
I'm not trying to be mean here, but this is your chance to learn now so I say seize it if you find the passion! It might be daunting because the parent commenter is referencing several vast and technical fields, but it's very doable and my preferred way of learning is to start with what is immediately in front of me so I can gain somewhat of an understanding and then dig in every time I don't understand something again (depth first search).
The parent commenter is referencing several fields:
This is a type of peer-to-peer strategic attack. So you could postulate that this broadly falls under the category of "common p2p network attacks and vulnerabilities". Google that!
This is a type of software pattern for making sure a signal (usually bad) doesn't keep recurring and its blast radius is contained.
> Now you're basically trying to devise a heuristic of what every single attack might look like in terms of transaction volume increase. If you're wrong, the bridge will freeze right when a lot of people want to use it. Plus, you also need to build a price oracle now.
This is blockchain specific, but you can definitely deep dive into how blockchains work to get to a point where you understand the concept of a "bridge", "price oracle", etc.
All this said, some people are more breadth-first-search learners, so, if you find yourself in a situation where this is overwhelming but you still want to know, you can always start teaching yourself fundamentals that lead to these areas. I'm roughly breaking down the stack for you here:
First stack:
* intro to programming (and getting deep into programming in general)
* understanding the basics of an CRUD database app (client-server systems)
* understanding multiple services (microservices and systems architecture)
* distributed systems (consensus algos, time drift + vector clocks, etc.)
* specialization in understanding the blockchain data structure
* specialization in understanding p2p systems and their faults
> This is blockchain specific, but you can definitely deep dive into how blockchains work to get to a point where you understand the concept of a "bridge", "price oracle", etc.
Circuit breakers based on price/volume/volatility is standard finance tech.
Win/lose/draw on blockchain long-term, in between the generally scamminess some of us are trying to figure out if we can create lasting utility/value in the space. Honest answer: we don’t know!
With that said smart-contract programming is (at least for now) an in-demand skill set.
The company I work for is interested in developing that skill set because such people are in short supply. A background in conventional distributed systems is very helpful, but anyone with a solid grounding in CS fundamentals can pick it up. It’s a different execution cost model and it’s an adversarial environment, so one needs to keep their wits about them, but it’s learnable like anything.
If you want to know more: ben@rocinanteresearch.net
Sybil attacks are common problems in many blockchain related problems. Especially for governance and “sources of truth” such as price oracles. Just google it.
You don’t need to worry about throttling dollar volume. Instead, you can just use a system like Optimistic Rollups use - build your system so withdrawals are delayed to give others time to submit fraud proofs. A successful fraud proof submission gives you a bounty and stops an invalid withdraw.
Yea, again, very complex and experimental technology. Now the entire blockchain you are bridging to needs to be written in an experimental VM technology which is being developed real time by dudes on twitter. And this Solana hack was a VM issue on Solana. That vm issue would presumably be present in whatever was running the fraud proofs on the other side. Rollups are only useful if the scenario you are worried about is a 51% attack
> Could be that the Wormhole team spotted the bug, patched it, but the attacker got to it before the patch could be rolled out.
It's a dark forest. Many predators who are smarter, faster and better connected than you are silently watching, ready to exploit your mistakes and ignorance for their profit.
At this point I'm convinced Satoshi Nakamoto was actually a public administration professor trying to teach kids why financial institutions have the rules in place that they do
given enough time the entire crypto space will have reinvented every regulation they tried to get rid of and understood why they existed in the first place
> given enough time the entire crypto space will have reinvented every regulation they tried to get rid of and understood why they existed in the first place
And the hilarious thing is that they see the absence of this regulation as a feature and don't even recognize just to what extent they are reinventing the wheel.
My favorite example is kleros.io, a "decentralized arbitration service". There's an app "Kleros Court". Can't wait for the "Kleros Supreme Court" version, for appeal processes and such.
> given enough time the entire crypto space will have reinvented every regulation they tried to get rid of and understood why they existed in the first place
Right, but they have to get through the robber baron and feudalism phases first.
I wasn't aware of the name of this. But often during code review have asked why some code is being removed, and asked the developer why that code was there in the first place.
Some googling found this article on the origin of the phrase "Chesterton's fence".
Yeah but the regulations would be opt-in, for people who want a safe avenue. Anyone else will still be able to interact with the underlying permissionless blockchain network.
Yup, but the current system is not a decentralized, immutable database.
At least you kind-of-sort-of have a chance at reverting a transaction.
Heck, my bank completely blocks my transactions over a certain limit (I need to ask for temporary limit increases through a separate process) and frequently delays transactions larger than another limit and they call me to verify that I want to do them.
Sure, blockchains could do the same, through intermediaries, but then... they've reinvented banks.
But this time they get to be the ones who make money off the regulations. Modern crypto is about becoming the new banks, not getting rid of the concept of banks and such.
> given enough time the entire crypto space will have reinvented every regulation they tried to get rid of and understood why they existed in the first place
Which is funny because you always hear how fast crypto transactions are. What everybody seems to leave out is that fully settling on the blockchain is slow and costly.
> you always hear how fast crypto transactions are
Where do you hear that? One of the big criticisms of the biggest cryptocurrencies as currencies is that transactions are glacially slow for most practical uses (small transactions). Improving on that seems to be the main selling point of Solana.
So if transferring cryptocurrency isn’t free, then how is it ever going to replace cash, theoretically? Right now I can buy a coffee for cash and the exchange of currency is free if I hand over the bills right then and there.
I guess the alternative would be to actually transact on the exchange instead of the blockchain? But then we’re just back to centralized banking?
What’s the ideal scenario here? Transactions on chain eventually become free and then we truly have a useful distributed currency?
The exchange of cash in your coffee example is definitely not free. Vendors spend a lot of money in cash management. Compliance costs, loss prevention, storage & movement etc. They price that in to the $5.
But frankly that’s besides the point. There are very few physical transactions in the “cash” world. Crypto can become very successful and never touch those transactions if it can work to standardize & disintermediate the electronic movement of money it could be a massive change.
I don’t know if that will happen and it’s not a goal of the crypto diehards I’ve met, but that’s a path to success that allows for fees along the way.
> The exchange of cash in your coffee example is definitely not free. Vendors spend a lot of money in cash management. Compliance costs, loss prevention, storage & movement etc. They price that in to the $5.
Fair point. But there are still many more informal cash transactions that don't have that issue, but maybe that's not what cryptocurrency will solve. That's fair.
> But frankly that’s besides the point. There are very few physical transactions in the “cash” world. Crypto can become very successful and never touch those transactions if it can work to standardize & disintermediate the electronic movement of money it could be a massive change.
Definitely agree traditional electronic money transfer today has many issues. But I don't yet understand why cryptocurrency is the necessary solution. Sure it's a solution, but surely it's possible to fix the issues without cryptocurrency.
I think I understand the core principle and appeal of crytpocurrency as a decentralized currency, underground currency among those wishing to avoid traditional financial institutions. But everywhere I look I see cryptocurrency turning into traditional banking and I don't really understand the overall benefit. It seems the few benefits it does have over traditional banking (speed, ease of transfer, univeral currency) are outweighed by the drawbacks (power hungry, surprisingly insecure, easy to simply lose everything you have with no recourse).
For sufficiently large transfers, there’s the good old “I’ll track you down and break your kneecaps”. Which often results in most, but not all, of the funds returned.
> the fundamental problem of slow worldwide, globalized, decentralized consensus?
That's not necessarily a problem. Slow is ok if it's highly secure, reliable, and inexpensive relative to the amounts being moved or compared to other alternatives.
Not if the chain can scale -- ie not "fast finality" but being able to throw more resources at the problem as adoption increases (unlike Bitcoin, Ethereum and most single-chain POS systems however). Gas' true function is to stop griefing, the fact that it sends tx costs through the roof is because immature tech.
There are applications that would really not perform well under a whole chain gets locked situation, e.g. anything that requires collateral ratios, things protected by timelocks, bridges.
You can build in things that limit velocity of transactions, send limits etc in the smart contract it's self, no need for it to be at the chain layer, it's just that many of the largest protocols haven't done that (also you either enable some form of DoS or you are vulnerable to sybil attacks)
Why not? Porn drove much of the storage technology and video codec technology forward we use today. Electronic trading drives much of the networking technology the world runs on today. As much as it is mostly scamming and haxx, there is some genuinely interesting computer science research happening around Byzantine fault tolerance and distributed systems in the blockchain space. Just because you don’t like it doesn’t mean it isn’t legitimately moving the needle forward in tech we will all use eventually.
Whether it is bananas or not, it's the same principle Microsoft uses for patch Tuesday (ie slow down patch analysis by bundling everything into a big monthly release)
Wow! This was the most surprising to me in the thread:
> It's interesting that this commit was made ~9 hours ago and the exploit happened a few hours after that. Possible that an attacker was keeping an eye on the repository and looking out for suspicious commits.
> Could be that the Wormhole team spotted the bug, patched it, but the attacker got to it before the patch could be rolled out. Super important to keep these sort of patches lowkey and to try to stuff them into larger commits.
Is this something inherent in the cryptocurrency space? Where monitoring for security patches and exploiting them before they are rolled out results in an instant multi-million dollar payday? Is this a common risk? If so, that seems crazy. Vendors already struggle rolling patches in closed source environments.
Yes, this is a risk. Exploiters will be running legit nodes with software analyzing the mempool for profitable transactions to abuse, they will be running nodes using the GitHub branches looking for changes in advance of merges so they can run exploits, they will be looking at framework changes and automatically running tools like fuzzers to find errors in downstream chains and projects that use the framework. It's a really really hard space for this reason. I work as a test engineer in DeFi and have been in the blockchain space for years but releases still give me stress like no other domain I've been in.
The nice thing is that engineering isn't usually financially bound so you can use all the toys and hire all the consultants to build really extreme testing environments that most domains won't use. It's also fun to flex tools like TLA+, property based testing tools, fuzzers, etc because it's financially sensible to be as sure as possible that you won't release a bug.
In the 'code is law' world of crypto, so are/should be these hacks. How are they any different than lawyers pouring over the fine print of a law or contract to find a loophole to exploit?
This bug was on the Solana blockchain. Solana has built in support for updating code.
Even on Ethereum, with it's mostly immutable contracts/programs, there is a super common pattern of a tiny shim contract that forwards all calls on to another contract that does all the work. By changing a storage variable which changes the contract the shim points to, you can effectively upgrade the code.
And therein lies my biggest frustration with crypto evangelism. Almost every theoretical benefit that gets trotted out is such a detriment in practice that the ecosystem tosses it aside. While still proclaiming it as a virtue.
I would wager most contracts deployed these days are immutable. Upgradable contracts are frowned upon in the space, moreso every day.
But given that humans are prone to making mistakes, when code is fresh and unproven, proxies are a practical way to go. Once you've ironed out all issues, you blow the fuse so that the contract can't be changed anymore.
Yeah this is something that famously happens with software development, you just iron out all the issue and then never have to change anything ever again.
Correct, but what they can do is hard fork and encourage users to switch to the new version. Users will vote with their feet and choose the version that benefits them most. In a sense it's still upgradable, but only through direct democracy instead of delegated democracy / oligarchy. (and for the record I don't necessarily think that's a good thing for a financial system, but we'll see how it plays out in the next few years)
A common pattern is to deploy a proxy contract, which simply forwards all function calls to an implementation contract, then an upgraded deployment generates a new address, the proxy contract is changed to point to the new address, and all end users only interact with the proxy contract. This is a little bit frowned upon in the community because it's a point of centralisation and control.
For those not following the space, solana has a smart contract language (a DSL in rust) that is new. Ethereum's primary smart contract language is solidity which has iterated many times & has linters and auditors that have learned their lessons through plenty of bugs in the past. (e.g. the DAO hack that split ETH into Ethereum and Ethereum classic) was a bug of the type "re-entry", it still crops up.
The new language & platform will likely need to learn the classes of bugs that can crop up through a similarly painful process, although this one was using something that was unknown to be unsafe :shrug:
As a security researcher, I absolutely assure you, the pattern of "checking signatures by checking that the last instruction executed was, in fact, a correct signature check over the right data with the right results" sets off ALL my alarm bells. This is so backwards I don't even know what to say. Just reading through the linked code I'm getting "this might be exploitable" feelings, and not just the aforementioned bug, other parts of it too.
I can't believe others trust these people with their money. They have no idea what they're doing. They're building a ridiculously overcomplicated system layered upon layers of systems where any single mistake can cost them all the funds with no recourse, and aren't even doing it in ways that make sense. It's insane.
Yeah I hadn't looked into Solana's contract language much and this sounded like a nightmare. Another good one is that Cardano follows a UTXO model for everything, including contracts. So the address of a contract changes every time someone interacts with it
I don't have any specific pointers handy, but it might be a good idea to look through historical security vulnerabilities in widely-used protocols like TLS (and x.509 certificates). The cryptography/security community has slowly understood over the past couple of decades that complexity is the enemy of security, and it is much better to build stupid simple systems that you can validate (and ideally prove are secure) over complex systems which are almost certain to contain exploitable corner cases.
This also ties in with general security hygiene and understanding; you need to know what is trusted, what is untrusted, and how to make them interact. Ideally you don't validate untrusted data; instead you build your system so that is not necessary. Every validation that needs to be performed is one more place where something can go wrong. If you need to validate a signature, you go and validate it; you don't ask the user to do it and then validate that they really did it properly and the validation happened. That's what happened here, as far as I can tell. The extra, avoidable validation went wrong.
It's not just that Solana is fairly new; the entire way the chain works is “torment the programmer on behalf of chain speed”, and if one of those decisions causes trouble down the line, instead of taking a step back and going with the simple and elegant approach, they choose an ad-hoc patch that causes more trouble down the line.
One example of this is rent. You can store data in accounts. But if the balance of the account is lower than some amount (that depends on the size of the data), the entire account might disappear. Depending on the amount it might disappear immediately and you would find out, but if it is just below the threshold, the account could survive for years. So in any kind of transfer that involves an account that holds data, you need to be careful to check that its balance does not drop under the threshold. If you forget to check in just one place, your state may disappear.
Another example: program calls take a list of accounts. You need to manually serialize and deserialize your data into accounts. It's like writing a program in C where every function can only take an array of void* as arguments, and it is up to the caller to cast (serialize) all arguments to void* and pass them in the right order, and up to the callee to unpack the array again, cast back the pointers, and check that they are valid before dereferencing ... That's fine for a low-level target if a compiler could generate the tricky code for you. But on Solana it's your responsibility to do it manually, in Rust. Much of Rust's safety is useless here. (There exist eDSLs that alleviate much of this, but if you don't understand the underlying model, it is still easy to make a fatal mistake.)
They need a new name for these things that better communicates the risk level. Right now calling these "smart contracts" is like calling dynamite a "lovely candle". There's nothing smart about something that lets you screw up this badly.
If they didn’t formally verify the smart contract in a way that covers all edge cases including known “exploits” they kinda deserve to have their ethereum transferred to its new holder. Code is law.
> If they didn’t formally verify the smart contract in a way that covers all edge cases including known “exploits” they kinda deserve to have their ethereum transferred to its new holder. Code is law.
It boggles my mind that anyone would want to participate in anything like that, let alone believe it's the future of anything. It's like volunteering to live in a hole in the middle of a WWI no-mans-land, when there are nice homes available in town where you don't have to worry about being killed all the time. It's one thing to be forced to live in a dark forest, it's quite another to choose to live in one (as anything other than the apex predator).
According to contract theory, it may not be possible, in practice, to write "complete contracts", that is contracts that specify what is to be done in every possible contingency. See https://en.wikipedia.org/wiki/Complete_contract
Apparently not, as the hard fork into Ethereum and Eth. Classic proves. Once every big stakeholder decides the game isn't in their favor anymore, they change the rules.
Thanks for sharing but I immediately drop stories that are sliced into a million twits instead of being put in a single blog post. I hate modern internet.
*edit: another good thread https://twitter.com/samczsun/status/1489044939732406275