Not really a question about the article, but just a thought/recommendation.
Every now and then I check the Mullvad blog to see if any new servers have been added, or features I should know about, etc etc. In essence, things that pertain to me as an end user. Stories like this, or new servers, WireGuard being added, etc.
I always thought it would be cool to see some sort of article or blog post with a 2019/2020 road map, however. Just so I can see things that are upcoming or know what’s being worked towards. With VPN services, sometimes it feels kind of stagnant, as the only appreciable updates are new servers or new features. Plus, since I don’t use the Mullvad app itself, updates to it don’t really pertain to me all that much.
Idk, I’m rambling, just food for thought. A roadmap blog post would be cool imo.
I meant to ask this by mail but I might as well ask here:
Many other VPNs have opted for not yet adopting Wireguard since they claim it's currently difficult or impossible to offer the same privacy guarantees, and also due to the unaudited/new nature of the code base.
Mullvad has offered Wireguard for a while though, and without any warnings about this topic.
Has Mullvad found a way to provide similar guarantess?
Or is there another reason why Mullvad does not see this as a problem while other providers do?
Hi! Thanks for the question. It's an important one. Both the security of WireGuard as a protocol as well as its Linux kernel implementation are most likely superior to all alternatives. Code audits and age of the project function as signals for decision makers, but there are stronger signals, if you look deeper. The simplicity of the protocol state machine, the fact that it can be implemented without dynamic memory allocation, the cryptographic primitives used, and the lines of code required, are arguably equally or more useful. WireGuard is 4000 LOC, IPSec is 400,000 LOC and OpenVPN is of similar complexity. I don't expect Jason Donenfeld who has a background in kernel exploit development to write code that contains 100x more vulnerabilities than IPSec and OpenVPN. We have noted the prevalence of those arguments. Let's wait a while and see who was right.
As for the privacy issues with a "standard" WireGuard deployment we are well aware of them and are working with others to develop a standard which will hopefully be adopted by privacy-focused operators. Unfortunately we only highlighted them to others in January this year. Note that no criticism of WireGuard's privacy seems to be older than this email: https://lists.zx2c4.com/pipermail/wireguard/2019-January/003...
We have deployed solutions for some of WireGuard's privacy issues while others are in the pipeline. It should however be noted that some of the attack vectors are somewhat esoteric and quite unlikely, so we don't consider them critical. The devil is in the details, as you can read in my email above. On the whole we think WireGuard is where the industry should be heading, fast, for many reasons, and that's why we are spearheading it: https://mullvad.net/en/blog/2017/9/27/wireguard-future/
At the moment IVPN is slightly ahead of us in terms of privacy features in WireGuard, but they are the provider we probably have most in common with, and we frequently discuss projects like this one, so they are definitely the one we most expect competition and collaboration from: https://www.ivpn.net/knowledgebase/253/Using-WireGuard-for-P...
While wireguard is probably safer and easier to setup, it has its own idiosyncrasies. Jason is stubborn enough to push through on creating a vpn, but also stubborn enough to see things his own way. I'm still wary of using it, not sure what will become of it.
It has idiosyncrasies like:
* Jason refuses to implement or see point of binding to an ip/address. I do not know of a serious network service that cannot bind. I guess roaming to any interface is always the right thing to do and sharing a port number for separate services on separate ips should be done via NAT.
* If you assign an ip/route to a peer, that is already assigned to another peer, previous assignment is simply removed. Somehow, this is a feature and not a blunder >95% of the time. On a security product, it's better to keep going and load a kind of obviously wrong config, discarding conflicting assignments, than to raise an error. Or even a warning (a warning is/was planned).
* on Windows Jason invented tying network id to hash of entire key+address configuration. Cool idea, but I guess you're supposed to fiddle with firewall settings each time you change any peer key/ip, because it's now a new network.
* wireguard is marketed as a modern secure VPN. It has proofs and what not, but I haven't seen a specification of what are it's security guarantees. It (obviously) leaks packet size and timing, also ecn bits, maybe server time? Does it leak public keys, dscp markings? I guess you're supposed to read the whitepaper, maybe source code and keep an eye for any changes.
You're comparing apples and oranges. Most critics implicitly count IKE and other PKI-related bits as part of IPSec. But Wireguard doesn't offer anything comparable. So if you want to compare apples & apples, you can only compare the actual transport layer. Wireguard is still more elegant, but of course Wireguard has the benefit of hindsight and IPSec has 20+ years of operational fixes.
What's happening is that because Wireguard doesn't include PKI people end up gluing together poor man's PKI with shell scripts and shoe strings. PKI is the hard part of key management, and because Wireguard doesn't attempt to solve it, it doesn't incur any of those costs. And the people who do try to solve it and get it horrible wrong, well I guess that's just not Wireguard's fault, right? How convenient.
Hey kfreds, great job first and foremost! You’re definitely working on some great stuff. In the world of VPN, it’s great to see another being innovative too.
We (Private Internet Access) are working on something similar called zero access. In response to your call to collaborate with other VPN providers, I would love to get on the phone with you to trade notes.
My contact info is on my profile here! Hopefully looking forward to it!
Hi Andrew! Thank you. I definitely took notice when PIA followed our lead and decided to donate to WireGuard as well. I suspect that was your initiative?
If you are interested in it you are more than welcome to join the implementation effort when we have put some basic things in place, or do your own thing right away. Above all it's important that we start moving in that direction as an industry. We don't really care who gets there first. It is also a more interesting game than the currently dominating one of affiliate marketing using hyperbolical and unverifiable security claims.
In a few weeks I will do a talk at the Open Source Firmware Conference together with our technology partner 9eSec ( https://9esec.io/ ), which will coincide with publishing a website as well as more planning of the implementation effort. As far as collaboration goes I have discussed System Transparency at length with IVPN many times, whose values and vision I find very well aligned with Mullvad's. We have an ongoing conversation with them about this and other things.
Thanks for the invite to chat, I agree it doesn't hurt to have a one-on-one talk. Even if Mullvad and PIA doesn't turn out to be a good fit for collaboration we should definitely share ideas on our vision for the VPN services market. Our perspective is pretty much all in the blog post about System Transparency. Looking forward to reading more about your idea "zero access". I'll reach out soon!
Amazing! Definitely, we think WireGuard is something that needs to happen, and we're happy to support it.
We are absolutely interested in joining in the implementation effort and likely have a lot of research/code to bring forth. I agree, that security theater is a joke xD
I like to cheer on any company working towards transparent systems and privacy-first technology, but Mullvad has a special place in my heart for always being on the front line. Thanks for the article.
Architectural question: why are people fighting to drag x86 unwillingly into the open space rather than taking something open (ARM, MIPS, anything else) and bringing it up the security curve?
Crypto and VPN seem like ideal points to basically put a fork in x86 since they are not dependent upon raw CPU performance. It would seem that fracturing the x86 world by picking off individual use cases until x86 has very little left would be the best option (ie. exactly what x86 did to mainframes).
> This is the first time a modern off-the-shelf server platform gains coreboot support, and it is an integral part of realizing our vision of transparent and independently auditable VPN servers.
Important work. Could you share more about your envisioned OSS toolchain for independent auditing of servers? What were the factors which lead to choosing this board, e.g. could other VPN providers contribute resources for extending this coreboot port to a newer chipset?
Sure! coreboot + LinuxBoot (with u-root) in the integrity-protected SPI flash memory, which then netboots (download and kexec) a Linux system. All artifacts need to be reproducible. For the CT Log probably trillian.
As for choosing this board, I alluded to it in a post down below. Note that I'm not a firmware expert though. Support for x86 was obviously already there, as well as for the microarch if I recall correctly, so "all" our technology partner 9eSec ( https://9esec.io ) had to do was support the specific board. Had we chosen another microarch or something with multiple CPUs we could have spent months on it. Just another example of how important it is to work with experts.
This port took roughly two-three weeks of full time work by an experienced firmware developer.
I'm not a firmware expert so take the following with a grain of salt. In general the more your target platform has in common with currently supported mainboards the easier it will be. First of all there obviously needs to be support for the instruction set architecture, then the CPU microarchitecture, and finally for the components on the specific mainboard you are targeting.
Given what has now been done by 9elements and others for SuperMicro boards based on Kabylake I expect more boards to be ported in the coming year.
As kfreds correctly states, there are important questions other than who the OEM for a mainboard is. With the exception of notebooks (where OEM specific components like the embedded controller require a fair bit of work that can be transferred from one device to the next), I'd even say that the OEM is among the least interesting bits about a mainboard on the technology side.
However, if the OEM supports the coreboot port (e.g. by providing schematics and generally answering questions that come up) that's very helpful.
So, question to kfreds: how was the cooperation with Supermicro? Are they aware of your effort, were they helpful?
(disclosure: doing coreboot development for 12 years now, some of that professionally, now for Google's Chrome OS devices)
Absolutely - nobody should accept anything less than a fully owner controlled and fully auditable system - especially when you're providing secure networking services to customers. Good on them for porting coreboot to this new, modern supermicro platform - it's a gargantuan amount of hard work and I'm glad they did it and published it.
However, I have to take issue with this statement:
> This is the first time a modern off-the-shelf server platform gains coreboot support, and it is an integral part of realizing our vision of transparent and independently auditable VPN servers.
While this is the first modern coreboot system - it is not, by the implication, the first modern server system with fully open source firmware. In fact, I contend that the firmware isn't even fully open source, nor is it fully auditable - the ME taint is still present, as the article clearly states.
(EDIT: I did not read the original article terribly clearly - it is extremely clear in this regard - and it doesn't position the system as being "fully" open source. My mistake and I'm sorry. See comments below for response/apology in response to author's correction.)
In contrast, the Talos II server system, a POWER9 (powerpc64/LE) system, by Raptor Computing Systems has been shipping for well over a year now - and it has fully open source firmware - and by that I mean everything, supported and developed by both the community, Raptor Computing, other OpenPOWER Foundation members and IBM.
The only remaining blob on the base Talos II system is the blob firmware for the onboard NICs - and that's been fully documented and clean-room reverse engineered and the replacement open source project is just about completed (project ortega). It's already usable in beta form, but if it's an issue for you until them - disable them and use an add-in card.
However, unlike the Xeon platform referenced as being the first modern coreboot server platform, the Talos II, or any other OpenPOWER system, does not have an invasive management engine in every CPU you have zero access to with zero auditability.
It's not enough for the firmware to be open source, I want my systems documentation to look like this: https://wiki.raptorcs.com/wiki/Category:Documentation - so that I can write that firmware if I have to. I seriously doubt the supermicro board would include schematics, like the Talos II or Blackbird does, either.
That said, if instead the existing open source firmware, you really want coreboot - which the Talos II does not yet support - for a specific reason - it's being actively ported to the platform, but not out of necessity, instead it's an exercise in open source firmware diversity - as it should be.
If you want a fully auditable, owner controlled, open source platform for your VPNs, or for any other application, x86 based systems aren't it - because, by Intel's ME and AMD's PSP designs, they can't be.
Please note that you might have misunderstood the article though. Specifically you seem to have missed these two passages:
> Modern Intel and AMD processors unfortunately still require some closed-source (and encrypted!) firmware in order to boot. This insanity is unfortunately very much a part of the technological foundations on which we rely daily.
> Today marks the day when (mostly) open-source firmware for off-the-shelf modern servers becomes an option.
I did miss that, and I'm sorry. You weren't positing the system as "fully" open source at all, so I certainly apologise for not making that clear in my writing. Quite the opposite, I misunderstood and/or did not read your writing closely enough nor pay enough attention to it - and so I mischaracterised your post. My bad.
Thank you for your transparency and forthrightness - and hard work in your involvement in porting coreboot to the new platform. The world needs more people like you.
In the future, I'll read more carefully - take some time to digest what I've read - and write in a tone which conveys a bit more charity than I have done.
I'm with you on your point, but to be fair, it seems like the Talos II motherboard is roughly 7.5x the price of the Supermicro motherboard ($350 vs $2600). I imagine that when they say "off-the-shelf" they really mean "commoditised". Still... I've got to say,though, that the Talos II is tempting if I ever got hold of that much money...
I'm extremely satisfied with the performance of my system and that's with good availability of software packages but relatively little Power-specific optimizations. As these start getting added, I expect it to get even better. As it is I don't distinguish much, if any, performance delta between this and a conventional x86_64 workstation and it has been my daily driver almost since the day it arrived.
Using Talos for a major global deployment of VPN servers is fantasy.
Having open source coreboot on those machines and what it allows you do on top (linuxboot for example) can massively improve your operational security and its viable for a global VPN network.
Yes, it absolutely can. However, the first part of good opsec is not lying to yourself about its shortfalls. Fantasy is thinking you can have a "fully open source firmware" for x86 systems, it just isn't - unless it's leaving out major parts of system initialisation and monitoring. Of course you can remove bits and pieces from the ME, but you can't guarantee that'll always be the case - Intel may not let you do so in the future when the next major microcode update comes out. That said, if you're going for cost cutting and as such you're willing to compromise on opsec, you won't beat major established VPN services on cost. That said, cost is a balance, but costs change - as a VPN provider, for your users - opsec is forever.
You can in theory. In practice buying hardware that is 10x more expensive is not really viable. Mullvad is already fairly specialized mostly for privacy nerds and there are just not enough people who car about the potential security of Intel ME.
Improving the state-of-the-art step by step on the platform that dominates the market will lead to more security gain overall then using what is essentially specialized hardware.
You and I could probably debate at length the merits of the trade-offs involved and the effect such trade-offs would have on fulfilling any given threat model. The point I was making is that it's not 10x as expensive now, it's probably 3-4x, and that cost is likely to become lower in the future anyhow. It's not specialised in the sense that it's single purpose - it's niche, so, the implication of "special purpose" in the computing sense, as in limited in functionaly for the purpose of greatly improved performance for specific tasks, doesn't really apply.
Why do you think it is fantasy to use an existing platform that has the desired properties today, but also think it is viable to go build a new platform yourself? (I like this effort, but given their goals it's not obvious to me that it was the simplest solution)
I really hope the author is correct. This would make many things easier. What too much proprietary hardware can do, you can see on the phone market. I doubt many people are happy with what they see here.
Intel's SMM doesn't actually reset the platform after 30 minutes. Intel's ME does if the firmware is missing on ME 11 and above. However, you can still strip the ME heavily and have it "disabled" via the HAP bit.
[1] https://www.fwupd.org/