Microsoft is certainly within the rights afforded to them by the curl license to do this, but:
> The provided functionality is certainly a very stripped down and limited version of the libcurl API. A fun detail is that the quite bluntly just link to the libcurl API documentation to describe how xCurl works.
is IMO a major dick move. It seems inevitable that doing this will cause users with questions about xCurl to seek out curl maintainers/forums for support with xCurl (as with the email that prompted this article), even though they have no insight at all into how xCurl works. And some of those people might not be very happy with the response "we have no idea - try contacting Microsoft".
Now, that might not have been Microsoft's deliberate intent, but I don't see how it's anything but completely forseeable, and should have been avoided.
(Yes, Microsoft need to credit curl with the copyright info and license, but the place for that is the "Credits" and/or "License" parts of the documentation, not the "Usage"/"API reference" parts.)
Oh that's a huge dick move. And this behaviour you mention, "people seeking help from libcurl for xcurl" is an already documented and detrimental problem. You can see why Sqlite temp files has an extension of "etilqs". That's a grandiose example of this. That was not even a documentation link!
Another common example is people seeking help at upstream projects for packages provided by their linux distro that have subtle changes which upstream knows nothing about.
Honestly though, that's life. I don't think its that big a deal, if you've ever done open source user support you'll know that people ask about all sorts of crazy things. You're lucky if they are even asking about software and not how to wash a car.
> It seems inevitable that doing this will cause users with questions about xCurl to seek out curl maintainers/forums for support with xCurl
I doubt this will be a large number of people. If xCurl is only available via the GDK, then you get access to it you need to be working for a company that has developer contract signed with Microsoft. If you are such a dev, you will also have access to the Xbox dev forums which _should_ be your first stop for GDK-related questions.
> you can basically deploy stuff to your own Xbox by just signing few things online
Is this true? I've been wanting to be able to develop my own console stuff for a long time, but the hoops have seemed insurmountable so far. Can I really deploy my own stuff to Xbox without having to pay huge fees and have calls with a sales team?
For publishing to some store, that's fine, but just for development I just wanna be able to get a build running without too much red tape, otherwise it's not fun and therefore not worth it.
Ah, this is specifically for Xbox apps, not games. You're not getting access to a bunch of things with this method (especially w.r.t. graphics APIs.) As far as I can tell, the full GDK is still behind a contract and NDA.
> How do I run a GDK game built for the Xbox App on Windows 10 or Xbox Game Pass for PC on Xbox Consoles?
> Xbox console development requires the “Microsoft Game Development Kit with Xbox Extensions (GDKX)”. Games will need to retarget and rebuild for Xbox One or Xbox Series X|S with the GDKX installed.
> You need to download, install, and retarget your project for Xbox consoles using the GDKX
> The GDKX is currently only available under confidential license within an NDA Xbox program (e.g. ID@Xbox).
Yeah, its a somewhat unfortunate reality that if you want to get full access to the consoles, you need to be an actual gamedev and sign some contracts. Microsoft does not suffer poor fools just trying to learn how to get the most out of the Xbox. Sony is much the same in this regard, and nevermind Nintendo.
I do wish they made Xbox more available to students in gamedev/cs degrees. When I was in school, Microsoft actually sponsored the program I was in, and I had a chance to work on Xbox stuff that as a student I would not have had any access to otherwise; That ended up being really useful to my early career -- not a lot of college grads have experience with developing and publishing for real on Xbox.
I think that's the case for 'Apps' but not games. Like I said in another post, my info is several years dated at this point, so things may have changed, but when I was in this field, if you wanted to publish a game vs a non-game app, there were many more requirements. Apps had pretty limited amounts of scrutiny and certification hoisted on them, while games had several hundred certification requirements for a ton of different things they had to meet. Apps also did not get access to the full capabilities of the console, whereas games did. Under the hood there were quite a few differences between how apps and games were provisioned, published and run -- games are almost like a container image rather than a regular windows app.
Oh if you want to _publish_ I'm sure it's a whole different ballgame, but if you're just playing around and/or developing a game (and don't need a full on DevKit), it's relatively straightforward.
AFAIK this isn't really true. Microsoft made the GDK public, but only for targeting Windows 10/11. The portion for targeting the Xbox is still under NDA. You can get access to developer mode for your Xbox by signing a few things online and paying a nominal fee, but that level of access only lets you deploy UWP apps to your Xbox, not do native development.
I don't know what it's like on Xbox, but I recently had to set up an identity provider in AWS that delegates to Azure AD, and Microsoft's documentation claims the integration is single-click with IAM Identity Center to generate the SAML config using an app tile available in their app catalog in Azure. This may be true in general, but it's not true for their FedRamp environment, and the devs with access to Azure, which I did not have, couldn't figure out how to do it manually. This customer obviously had a paid account in Azure, along with plenty of other enterprise IT services purchased from Microsoft, and it still took four weeks for Microsoft to finally get them a scheduled meeting with a solutions architect who knew how to do what they needed to do.
I agree it is. A fitting return volley would be to insert a note in the documentation along the lines of "derivative implementations of curl may have important differences from the documented implementation. Please refer to the derivative implementation's documentation for any behaviors that don't appear to conform to curl's implementation".
Yes, they should make it clear that xCurl has nothing to do with real curl.
But if it intends to implement the same interface, I don't see any problem with saying look at the curl docs for what its supposed to do. I don't see how that would be much different then if you implemented your own web browser and was like, look at MDN for how its supposed to work.
"Hey, your docs say that curl should be doing this, but it's not working for me?"
<2 pages of back-and-forth posts later>
"Oh you're using xCurl, yeah you need to talk to Microsoft."
It probably depends on the reach this has within the intended audience (people wanting to deploy libcurl on Xbox?) as to how much of a support burden it will end up, but I suspect more than Mozilla has to deal with from similar situations.
bawolff says it wouldn't be a problem to just link the curl docs.
Cogito says it would, since users would just ask Daniel for help, thinking xCurl was the same as curl, despite them having different implementations. This would place a large burden on Daniel.
The point I'm supporting is that users will not realise that the documentation that is linked to is from a different project than the one being used.
Hopefully, by focusing on an example of how a support interaction could go (clueless user, lots of back and forth before the key issue is identified), it shows how this practice is good for neither users nor for the curl team who will end up dealing with the fallout.
I also address the MDN example by stating that I suspect the audience and reach of xCurl will create a larger support burden for curl than Mozilla would see in similar situations (aside: even though it wouldn't be reasonable in that case either!)
Unless it's done very carefully, pointing users to the parent project's documentation will cause a support burden on that project, and that's just not reasonable for Microsoft to do in this situation.
If this was an Alpha software that was created a few months ago and is rapidly changing then sure, link back to the original.
But we’re talking about curl here. MS could have an intern copy paste the curl document, delete the stuff that doesn’t apply to XCurl, and host it on a shared OneDrive Word document all within a day, and it would be orders of magnitude a significantly better experience.
Instead curl maintainers will end up spending man years supporting XCurl issues instead.
What if you called your browser xFirefox, and suggested it would be equivalent to "Firefox" on the MDN "Browser compatibility" tables. And you have 1000x the resources of Mozilla at your disposal.
Based on the limited implementation surface and marriage to WinHTTP I would've imagined simply writing their own API distinct from curl's would've been a smarter choice, albeit with worse SEO to leech from curl.
The mentioned security requirements likely have to do with certification requirements for games released for the Xbox. You couldn't just make network connections willy-nilly, there was some amount of having to declare ahead of time what ports you would be using and for what. The normal socket API is also severely limited, and largely cannot even be used without obtaining a special exception from Microsoft for your games.
I was actually at one point (years ago) tasked with determining if we could use curl in a project I was working on. There was indeed much surgery needed to ensure that the code even compiled, much less worked correctly. We ended up just using WinHTTP directly instead.
As for getting access to the source code for xCurl, that will likely not be possible without actually signing up as a dev with Microsoft and going through the whole process. And even then, you'd be under a NDA.
(My information is several years old though, so it might be somewhat outdated)
That is highly likely to be the case. I no longer have access to take a look, but from my experience, it would be silly to do otherwise. There were a handful of other xSomething libraries that they had that were effectively wrappers or simplified versions of libSomething.
> xCurl differs from libCurl in that xCurl is implemented on top of WinHttp and automatically follows all the extra Microsoft Game Development Kit (GDK) requirements and best practices. While libCurl itself doesn’t meet the security requirements for the Microsoft Game Development Kit (GDK) console platform, use xCurl to maintain your same libCurl HTTP implementation across all platforms while changing only one header include and library linkage.
Yet the author of the article (and curl itself) seems to think it's a fork:
"With large invasive changes of this kind we can certainly hope that the team making it has invested time and spent serious effort on additional testing, both before release and ongoing."
"...since I can’t find the source code I cannot really get a grip of exactly how much and how invasive Microsoft has patched this."
It's not a patch of libcurl if they just implement a mostly compatible API from scratch.
The presence of the license strongly suggests that it's a fork.
"Fork" and "API compatible wrapper" are also not two mutually exclusive things. Often, one of the easiest ways to build an API compatible wrapper is to start by forking the original project, scoop out the internals and replace them with what you are trying to wrap, and then discard all the dangling code (e.g. features that you couldn't wire up because your custom internals don't support it).
This was my assumption reading it; it very nearly has to be exactly that to work as described by Microsoft. Which IMO makes all this a complete nothing burger.
This sort of reason is also why gRPC isn’t usable in games that also release on consoles. It comes with its own HTTP implementation which you can’t use and even if you wanted to reimplement it, it requires HTTP/2 which isn’t supported by the “blessed” console libraries.
I doubt it. If you've ever worked on a console game, you'll know that these requirements are no joke - they're pretty hard to consistently comply with, and using APIs from the manufacturer is often much much easier than to do your own thing. It will also be much cheaper, because failing certification is unimaginably expensive - in the worst case, it can even lead to cancellation.
A lot of studios that ship on multiple platforms have abstraction layers that can internally be switched out to every platform specific API.
> A lot of studios that ship on multiple platforms have abstraction layers that can internally be switched out to every platform specific API.
Exactly this. At one point, I was the man responsible for that abstraction layer. Even most of our engine devs never even saw those low layers of the console APIs -- the single exception being the graphics peeps who by necessity had to work at the same layer I was. I basically wrote a Xbox/Playstation/Nintendo/PC libc/runtime that the rest of our engine was based on. I Imagine a lot of other studios that shipped cross-platform had some level of this as well.
The voice of the battle-weary maintainer comes through clearly in this one. I can't imagine the burden of being responsible for something as widely used as curl. Ok, I can kind of imagine it, or I wouldn't be writing this comment. That feeling of "oh no, someone has just created a support nightmare for me" is familiar, even if I've only experienced it at a minuscule scale in comparison.
As a cynical Enterprise person, my take is that some team within M$ wanted to force some other team to no longer use Curl/Libcurl for "reasons", so they had to gut it and replace pieces with some other team's cruft. There is no interest in upstreaming, supporting, extending, documenting, releasing, etc this monstrosity as it was probably an ugly kludge to meet an internal bureaucratic requirement.
This is why the MIT license is great. If it was GPL'd or something, they would have to rip out curl entirely and develop a completely in-house library, which takes more time, costs more, adds bugs, etc. On top of adapting any programs that normally use libcurl to use their custom thing, or creating a shim library (either way more work). MIT-licensed code allows corporations to still build on top of open source, and monkey-patch their own shittiness in the process. Many of us OSS devs choose MIT for this reason: we just want more people to be able to use it.
It is purely due to fact that developers on Xbox (where GDK is mostly used) do not have free access to sockets but instead have to clear every network connection with Microsoft. This is much easier to do if every http call goes through WinHttp which they can control/monitor easily on the console.
Obvious question, why didn't they just implement the security features necessary? They could have contacted Daniel to get his cooperation.
Edit: Answer is in the comments. So they should have just implemented a wrapper ontop of libcurl. But I guess the real effort here is to promote WinHttp as a replacement for libcurl. And since the developer community at large is so used to libcurl they made a wrapper for WinHttp instead. Oh lord.
The purpose is to restrict HTTP features and where it can connect to and proxy usage to restrict "wrong" doing. (for some definition of wrong)
WinHttp itself is relatively high level, comparable to curl. Most likely this is using curl headers, but a complete custom implementation, nothing upstreamable. (Especially as Daniel probably won't like the burden of supporting a proprietary platform he doesn't have access to)
So in summary, this seems like it could be a security issue waiting to happen for games that use that xcurl lib from Microsoft, since it seems they don't keep up with upstream and will miss important patches.
How that improves "Microsoft security requirements" is beyond me, really.
My interpretation is this is an implementation of the interface, basically wrapping WinHttp for existing code that uses curl.
So security vulnerabilities in curl are not necessarily present here, since the two implementations would have matching forward declarations but little in common in terms of implementation.
> this seems like it could be a security issue waiting to happen for games that use that xcurl lib from Microsoft, since it seems they don't keep up with upstream and will miss important patches
And at the same time, missing out on the continual stream of bugs that are being put in. (I found it interesting how Daniel was able to give such a positive spin to the comment, "We merge bugfixes at a rate of around three bugfixes per day.")
This was my thought as well. Software that is as mature and well-defined as curl, I would be shocked to see three bug fixes per day. That means it is getting lots of new features / new code. That's not a bad thing necessarily, but most of the people that I know who use curl have been using it the exact same way for many years. A version that is a few years old would definitely not need three bug fixes a day
Speaking of strange Microsoft CURL variants, have you ever faced problem with HTTP2 in practically any network client (for example Eclipse strangely failing all attempts to fetch from new HTTP2-enabled Maven) that could be traced to
C:\Windows\System32\curl.exe
failing handshakes with such HTTP2 servers, whilst any other CURL (from e.g. git-for-windows, even with lover version) could connect to the same server correctly?
From what I recall from aforementioned incident, I was not able to reliably prove that client app (Eclipse) really used Windows' curl (I would have guessed that such software would have it's own internal stack for network connections, but my knowledge in this area is non-existent), only that it exhibited suspiciously similar problems.
So I came to conclusion that it either uses that CURL or CURL that is similarly broken as the Windows' one. I was unable (or more precisely haven't even tried) to swap Windows curl with the "better" one to see if it fixes Eclipse Maven. Result back then was only that HTTP2 was disabled on the Maven server and all (windows) clients (both of them) were happy. (Other might become less happy since it presumably slowed something down.)
If you excuse some speculation:
(clarifications appreciated of course.)
If i remember correctly, ages ago (Windows Server 2003?) Microsoft introduced in-kernel HTTP-handling for the IIS web-server. I think it was for performance improvements with less copying between kernel- and user-space memory.
I suspect the for me unknown WinHTTP apis mentioned here could use these optimizations? Maybe that's why they mention security requirements? (Which would obviously be needed when doing parsing in the kernel.)
The security requirements are more due to trying to keep nasty things from being downloaded in the context of a game running on the Xbox. Games tend to run at fairly high privilege levels on Xbox, so Microsoft is probably worried mainly about jailbreaks.
The thing you're thinking about is http.sys and it was introduced in Windows 2008/IIS7 to support the new "Integrated Mode" request pipeline.
WinHTTP doesn't have anything to do with http.sys it just listens for HTTP requests and then hands them off to the right bits inside IIS.
WinHTTP is essentially a HTTP stack for client services running on Windows Server to allow them to make HTTP requests. It has a sibling API named WinINet which is aimed at use in desktop environments. I think the threading models are main differentiator (I been a while since I looked at this).
Their position on a previous name collision involving an overseas trademark sounds like "we are bigger, nobody ever confuses us for you, but we'll be sure to redirect anyone who we notice is looking for your project!"
Not sure what that says about the inverse, but since the post mentions nothing about trademark concerns, I imagine the author isn't too worried now either
The tone of this article is really strange. It somehow sounds like the author is upset that someone forked his project. (There are several indicators of that throughout the post.) That's the whole point of free software. If you didn't want that, you shouldn't have chosen a free software license.
Also, as explained in other comments, it's unlikely this is actually a fork of curl. It's more likely it's just a wrapper around winhttp with a curl-like API.
He got an email requesting support for software with the same name as his, that points to his documentation, while having zero access to the software itself. I'd say being mildly annoyed at that is warranted, especially since we're talking about a gigantic corporation on the other side.
This isn't even the first time[1] Microsoft has done something like this with the curl name. And, then as now, people bother Daniel with support requests.
Yeah those corporate kindlings deeming themselves "senior developers" or whatnot have those ideas all the time. In this case the schizophrenia is that on one hand Microsoft is using unmodified curl in Windows (used by "billions"), and it's okay. But on the other hand for their game blahblah SDK the "requirements" are higher so they do need to have to maintain separate fork. That's not curable I think.
Given that their own implementation of curl itself answers with "curl : The remote name could not be resolved: '--help'" just for trying to figure out how to use the damn thing, I wouldn't count on XCurl to be implemented with any degree of competence either.
Windows' Powershell is in fact an old version that they still ship because it's the last one compatible with .NET Framework. Newer versions must be installed separately (so they've ironically had more adoption in Linux).
But curl.exe should still bring the actual curl, I think.
Is that a reference to EEE? I feel like it is, but the capitalisation is confusing me, and now I'm wondering if I'm missing the actual intended meaning.
He's hardly the first person to do this. Often when people do this it causes real problems, as most people aren't lawyers and sometimes their edits have unintended consequences. It also tends to mean that reusers have to re-evaluate if the trivial change affects them, possibly get separate legal approval if they are in a company, etc. Tends to just cause a huge headache. Anyways, I'm sure Daniel meant no harm with this, but i still think people should be strongly discouraged from doing that sort of thing.
Because now, see, you need an army of very expensive lawyers to very carefully analyze the changes to determine if you can still be using the thing. The answer is always tricky, boils down to “it depends, now pls give us more of those sweet hourly fees and your own project description in detail so we may determine if maybe probably it’s ok in your particular case. Not guaranteed, of course, gotta have a court case to guarantee anything.”
So legally tricky, much legal busywork. Very courageous.
> The provided functionality is certainly a very stripped down and limited version of the libcurl API. A fun detail is that the quite bluntly just link to the libcurl API documentation to describe how xCurl works.
is IMO a major dick move. It seems inevitable that doing this will cause users with questions about xCurl to seek out curl maintainers/forums for support with xCurl (as with the email that prompted this article), even though they have no insight at all into how xCurl works. And some of those people might not be very happy with the response "we have no idea - try contacting Microsoft".
Now, that might not have been Microsoft's deliberate intent, but I don't see how it's anything but completely forseeable, and should have been avoided.
(Yes, Microsoft need to credit curl with the copyright info and license, but the place for that is the "Credits" and/or "License" parts of the documentation, not the "Usage"/"API reference" parts.)