Valve recently announced partner access to their network backbone which addresses a lot of the problems mentioned here:
* Access to our network, giving your players protection from attack, 100% reliable NAT traversal, and improved connectivity.
* Tools for instantly estimating the ping between two arbitrary hosts without sending any packets.
* A high quality end-to-end encrypted reliable-over-UDP protocol.
>> Second, clients can select a route that gets off of the public Internet and onto our dedicated links as early as possible. On our backbone we can ensure that the routing is optimal, since we have peered with over 2,500 ISPs. We also prioritize the latency-sensitive game traffic over HTTP content downloads, which we can afford to do because game traffic makes up a relatively small percentage of our overall bandwidth utilization. And on our backbone, a sudden surge of traffic unrelated to gaming won’t degrade the experience.
Which is great if Steam is your only platform and you want to be tied to Valve forever, but as soon as you want your game on mobile, console or even another PC store like Epic or GOG you need another solution.
Microsoft and Apple are far worse IMO by pushing their proprietary graphics APIs. I see Valve more as their competitor than as a threatening monopoly. In the future, things can change though, nothing is fixed.
More "graphics API is the worst lock in possible"... With an online game it's way easier to escape graphics "lock in", especially considering almost everyone is using a game engine (Unreal or Unity) that hides most implementation details from you. There are challenges with porting graphics apis are different shader languages but with glslang and dxc supporting different languages it's basically a solved problem. The main issue with service lock-in, and why it's worse than proprietary graphics APIs is that it blocks cross play. So even if you wrap the API for different platforms Steam users cannot play with Xbox users, etc. The idea that graphics API is the biggest bottleneck to game portability (especially across systems that also have proprietary windowing systems, threads, audio, input, programming languages, network libraries, etc) because John Carmack said we should use OpenGL 20 years ago needs to die.
That's not really a lock-on. The "tube" level of multiplayer stack can usually be abstracted away even easier than graphics API (and almost no modern game engine supports less than at least two of those). Just compose a packet of a reasonable amount of bytes, configure a connection overall profile (reliable/unreliable, which usually, but not necessary, maps to TCP/UDP), and you're done.
I'd be much more afraid of vendor lock-in that actually runs throughought your code base and shapes it.
Well, in defense of Network Next, their system seems to fall back to public routes if there's no benefit. That doesn't seem too much like lock in, since you have to have public servers working anyway.
Our APIs are just simple replacements for UDP sendto/recvfrom and they’re open source. While they won’t work unless you work with our marketplace, if they don’t work, they just fall back to regular UDP packets. How is this lock in again?
The Internet is optimized for throughput at lowest cost.
No amount of good netcode that you write can compensate for this.
The problem is the internet itself.
The internet doesn’t care about your game.
The Internet did change, such that streaming movies became viable. Given that games are now bigger than movies, could it be argued that that the network lag we're seeing is the industry lagging behind the culture? (Which it generally does, at least a little.)
A part of this might well involve special deals between large companies involving specialized infrastructure. There are such things now involved with movie streaming. Google's plans involve the infrastructure part of that already. What happens to net neutrality?
Once game streaming becomes mature, then it's only a matter of time for App streaming to follow. We'll be back in the mainframe days, just with much larger companies with much larger multi-box cloud "mainframes", commanding the attention of a much larger portion of the population and the culture. "The computer is your friend. Trust the computer!"https://en.wikipedia.org/wiki/Paranoia_(role-playing_game)
Counterintuitively, the market that Network Next is making shifts us closer to actual net neutrality than relying on the benevolence of FCC or Comcast by putting power into the hands of content providers where previously they were at the mercy of last-mile ISPs.
If businesses can collectively bid for performance from private networks to offset actions by last-mile ISPs to cut costs and reduce service, then they also have the power to organize a general strike by refusing to bid for performance from traffic from specific ISPs until they work out better rates throughout the value chain.
So I see this as a mechanism to restore the spirit of net neutrality despite the best wishes of the FCC.
Network neutrality does not apply to private networks. But even so, we care deeply about network neutrality. We are creating a truly neutral marketplace where suppliers cannot act in an anti-competitive manner towards buyers, and can only compete on performance and price.
To the poster who said that we are trying to restore the true spirit of network neutrality. Bravo, you get it. This is exactly what we intend to do.
Bandwidth has followed an improvement curve since the early days of the internet, though it has slowed down a lot lately.
Streaming movies became viable despite the slowing rate of bandwidth improvement. We had 30 kbps modems in the 90s, then 8 Mbit ADSL1 in 1998, 24 MBit ADSL 2+ in 2003s, 40 MBit DOCSIS around the same times. If the speeds had continued to improve at the same exponential curve, consumers would have 100+ gig broadband by now.
The same is true about datacenters, the bandwidth per compute capacity available to servers has decreased for a while.
Bandwidth is an easier problem than latency, because it's not limited by speed of light.
And pay an obscenely high amount of money for bandwidth? Those providers all charge at least $0.05/GB; you can get bandwidth for a tenth of that elsewhere. Games use a lot of bandwidth at scale (agar.io would use a couple of PB a month). For F2P games, server costs eat a lot of revenue if you use those premium providers.
Not to the datacenters, just to the POPs or edge nodes, after that, it is all private networks. And, afaik, the distance to the closest pop or edge node is a small share of the total distance to the datacenters.
We are measuring significant latency reductions for real players to google data centers right now. You seem to be overestimating how perfectly anycast finds the best PoP to get on the private network. You also assume that google’s private network is optimal from every point on the USA to their own data centers. This is not true. What’s better that google’s network? Everybody else’s, and google’s network, evaluated once every 10 seconds.
You didn't, you just hand-waved and threw some claims around without any data to back it up, and you didn't address the key issue: you don't have access to datacenters, only to POPs.
> We are measuring significant latency reductions for real players to google data centers right now.
Again, you only have visibility to the POPs, so you're only seeing improvements on the path to the POP.
For example: there is no Google POP anywhere near its Oregon datacenter[1]. You literally can't just push your traffic to the Dalles and plug into the datacenter.
So, no matter where your customer is, you can only improve the traffic to the Seattle or the Bay Area POPs, where the traffic will move over to Google's private network.
As I understood it, they are measuring the latency to game servers located in googles datacenter. This is why they are talking about that. They cant measure game latency to a PoP, because the game server isn't at the PoP.
Instead of routing to the nearest PoP, they route to the PoP that provides the lowest latency to their game server. I don't understand the confusion. They are simply measuring latency over a number of different routes and choosing the optimal one (for their use case).
Maybe you are assuming that there is no latency between the PoPs and the datacentres? I can assure you thats not the case.
Great reading this, I just want to drop a line to say that I have been working to solve precisely this problem -- for gaming but also other applications -- at the root. I have been working on a paper on this, here is the pre-proposal I published a whiled ago: https://docs.google.com/document/d/1xWaSB-3VSOMaiyxfjDqTalDC...
-- a lot of progress has been made since and I am looking for funding/grant before moving to publishing the actual paper and moving to implementation.
The root of the issue is that the IP protocol is incomplete, in that is relies on private agreement and ISP servicing that cannot react to a real-time, real-world demand. IPv4 had some attempts at public QoS features that were mostly dropped in IPv6 because people realized it didn't work in practice.
The only way to fix this for real is to merge ISP servicing and peering into a per-packet peer-to-peer network market. This solves a great many issues at the same time and essentially achieves the promise of mesh networking and more.
If you would like to help, please drop me a line: tarik@graphnet.org (sorry no website at the moment).
This was an awesome write up. The most interesting thing for me was learning that bottlenecks are not on the edge, but across the backbone because T1 carriers are optimizing for low cost, not low latency.
The thing I'm still a bit confused about is: is this a limitation with how their networks fundamentally work, or an artefact of my product choice when I buy a relatively cheap home Internet service? i.e. if I want a specialist low-latency and low-jitter link, could I not just pay for those benefits by purchasing a business-grade service? Or is there no distinction once the data gets onto a peered-network?
Or is the idea here that I don't have to cough up for all the benefits of that superior product when I only want some of them and only some of the time?
And this whole thing does rely on the bottleneck not being at then edge, which I don't see any proof of or reason for. EDIT: is there any proof/data to support this?
Modern games have a problem that I believe is a direct consequence of consoles. The lack of a server browser.
Just let me choose a server to which I have a good connection and I will be much happier. Instead of dropping me in games with random people all the time with random connection issues.
One of the main reasons I don't play fps on the consoles is missing this. (And there are social advantages of playing with the same 'random' people as well).
Having a server browser does not solve all issues mentioned in his presentation, but it does at least inform the player of the connection before they join a game.
Random players can work with proper servers too, and many games just do that.
The problem with server browsers, or user hosted servers are/were arbitrary rules enforced by the admin of that server. It's what killed Black Ops 1 on the PC.
The benefit of the server browsers is that there were tons of positive gaming communities that competed around making the best and friendliest server possible in the early 2000's. I fondly remember playing Call of Duty as a tween with other community members that were adults and learned how to have fun, manage disagreements, and deal with problematic people without screaming profanities and raging about other peoples' mother.
The death of self-hosted servers means the death of admin powers which means the death of building a gaming community one wants to see come to fruition. Those admin powers are key to protecting vulnerable people from general assholes. And the "modern" online "report a person" is woefully laughable in comparison.
Often times admins are the assholes here (kick for using weapon X, perk Y, attachment Z) and you have no way prior to connecting to know which stupid rules are now in place.
(that and the fact that dedicated servers in BO1 meant you could rent them just from one or two places, no files for you)
Hard to support all possible mods and changes I guess. The best they might be able to say is "modded: yes/no". And that is another great thing that is dying out. Mods. :(
This is correct. We work with game developers to sponsor the best route for their players. If we can’t significantly beat the public internet for that player (according to what the game developer specifies in their route shader), then we take the public internet and that player is free.
This is epic. Great to see that someone who's been such a good support to the gaming community with his excellent learning resources is also being successful in his business.
It's also interesting to see these "private internet" endeavours have popped up to solve centralization issues on the internet. Streaming sites like Youtube and Netflix have solved it in similar way, striking deals with ISPs, as have the big CDN's, simply putting their machines at the ISP's datacenters.
Network.Next is solving this for the gaming domain, but there is a more general preferential routing on the internet problem this idea could be expanded to.
As someone who works in the games space, I find this to be an interesting proposition. My primary concern revolves around similar 'net neutrality' aspects - more and more of the internet is becoming pay to play, this just further entrenches that theory if you want the best networking for your games as a studio.
That's a good observation. Given the alternative between bidding for performance from Network Next vs paying some flat fee to Comcast, et al. directly, I'd go with NN and let them deal with Comcast.
... and then maybe add "faster gameplay" power ups to our loot boxes (kidding).
When we work with game developers, they find this aspect of the system incredibly compelling. Does it really make sense for every game company to negotiate with every transit provider and ISP to make sure traffic is delivered properly for their game? We think not.
Plus, how could they get a fair price if they’re the small guy. Suppliers on Network Next cannot identify buyers and cannot have one price for one buyer and one for another. So now the small guys get the same price the big guys do and competition should drive the price down over time.
Games need some low latency data, but not much. Player actions, character movement, and maybe bullets need low latency. Asset downloading does not. If you had a dial up modem running over a non-packet phone network, and didn't overload the buffering, you'd have better latency than the Internet.
Many games make this distinction already. The latency-critical stuff goes over UDP, and it's limited to data where a missing packet is superseded by the next packet. Nothing is retransmitted. Bulk data goes over TCP, with reliable retransmission but more delay.
It's too bad ISDN died off. 64Kb/s end to end, not packetization delay, and no jitter. The ideal gamer network would be an ISDN connection and a IP connection in parallel.
If we had QoS systems where 1% of your maximum data rate could be at high priority, this, and the VoIP problem, would be mostly solved.
This is correct. Most FPS game protocols are 90%+ unreliable UDP acting as a carrier wave for a very small subset of reliable/ordered data. The key here is to get this unreliable time series data across with the lowest latency and the greatest consistency (least jitter), and packet loss. That’s what we do at Network Next.
And also, we only accelerate the part of the game that needs the low latency. Game downloads and everything else... ehh, that’s a solved problem for a while now. We accelerate only the game traffic between client and server!
Glenn, if you're still hanging around -- can the shader be written to account for the flow of P2P network traffic? Let's say I'm using an old-school configuration of a centralized lobby server that simply connects players to each other and the players connect directly over UDP?
Clever concept. I wonder if they'll publish any more information on how their service works. Things like what this 'route shader' is that people can read without having to go through Email-Our-Sales-Team shit.
It's a custom bidding script that tells their real-time auction platform how much you want to pay for route improvements for a particular user session.
Typically, shaders are used to move vertices in 3d meshes or color pixels in the GPU. Network Next is just using the term in a creative way to help connect to their audience, but it's nothing different than a bidding model.
Route shaders select the best route out of multiple ones available. Right now they could be used to for example, specify basic things like: acceptable latency (eg. I don’t want to pay to optimize a player’s session from 20ms down to 19ms, 20 is good enough), minimum improvement vs. public internet before a session takes network next (eg. 5ms improvement or greater), and of course maximum price for the bid. If we don’t find a suitable route at the price specified, the session takes the public internet and is free. Right now it’s fixed function like OpenGL 1.0 (eg. Parameters), but long term, it could even be LUA bytecode. Hence, route shader is not just a marketing term, it’s a real thing :)
The really interesting thing I see for this is a possible other route for game revenue in the future. Instead of paying up front (traditional) pricing or advertising, or cosmetic (or not so cosmetic) in game purchases, you could have two tiers, free and optimized latency for a cost.
Or some combination. Cosmetic in-game purchases aren't necessarily bad, but the amount my son has spent and tries to continue to spend (I strictly limit what of his money he can waste on it now) on fortnite stuff is ridiculous and concerning. A sane middle ground would be helpful.
Our customer is the game developer not the player. They are free to pass the cost on to the player if they wish, but I think it’s much more enlightened to simply sponsor your player’s connection and help them have as much fun as possible. When players are having fun, they like your game more, stick around, tell their friends, and spend more in your store.
> Our customer is the game developer not the player. They are free to pass the cost on to the player if they wish
That's exactly what I was referring to. In this situation, you would be sort of like a transaction processor.
> I think it’s much more enlightened to simply sponsor your player’s connection and help them have as much fun as possible.
I'm not sure I would use "enlightened" to describe anything to do with the majority of games that use microtransactions, at least when it's a constant barrage of "special deals that last for a limited time and you'll only ever be able to get this digital item right now and if you miss out it will never be back."
When aimed at a very young audience, this is anything but enlightened. I don't mind my kids paying for the content they use. I do mind them investing in digital content locked to a single platform that they both don't own, and which becomes useless when they stop playing. At least if they collected cards for a collectible card game in real life, they could look at a collection, pass it on, or sell it.
Basically, I'm advocating for revenue streams that don't require hiring specialists in promoting addictive behavior linked to spending for games where a large part of the audience includes adolescents.
Why? If it's up front, and it allows two classes of players and clearly defines them (and allows people to sort by them for games), what'st the problems that would make you think twice?
I can see some problems with the model, where it might not allow for a more evenly distributed set of revenue from players and certain players would get the benefit without the cost by having a very good and well connected and routed ISP that happens to line up with most servers, but what's the problem? If the accounts were designated "competitive" tier, that seems to clearly signal what the user gets. Additionally, at least it's then more information the service can expose about a player advantage, rather than the user paying for it outside the service and having a similar advantage but without the same signalling.
If it's about pay to win, I understand the sentiment, but I think there's a clear difference between pay nothing (or a token amount) and pay something for a "regular" account without a bunch of microtransactions and extra tiers, and what we see with games where you literally can buy your character into an elite level of capability that changes the game at a fundamental level (either from hard to easier or from nearly impossible to possible). I think making it so your players can easily separate based on these levels is good as well (which is another point towards very few differentiating levels). As long as the price is low enough that anyone could easily afford it for a while, it's not a real problem. Close to unbounded capability to pay for advantage is though.
Pay to win is a serious concern. There’s also the fact that not everybody gets improvement. For some players, some of the time, their default route is just fine. Would be a shame to charge those players $5 a month for something that gives them no benefit.
Well, there's plenty of payment options for games that include multiple things, some of which many people never find a use for. What use is a novelty model/skin that's part of a paid package that is never utilized?
As part of a paid account I think it at least has utility for a large percentage of people, and may provide more or less useful as their play mode changes or they play on different servers or with different people over time.
I think a $5 a month plan that included your service, plus a visual indicator or something on the name or player that's customizable, plus early access to some content such as maps, would be a good sell. Some people might not take advantage of the player customization, some might never play the maps early because their group includes people that aren't paying, and some people might not have a use for the better networking, but all together, it's an interesting package.
Explicitly passing on fees to the customer for a capability that may (or may not) help them see lower latency erodes trust and makes NN look like an insurance scam. As mentioned, not all users will definitely see performance improvements, so your proposed $5/mo is more of an insurance policy than a defined capability -- it could even be pure profit (who would know?)!
The more honest way to do it would be to convert funds to "latency improvement" tokens that trickle out of customers' accounts when used to improve performance, and provide the user with analytics of that improved performance. (NN could take note here and manage this token bucket feature themselves in a 3-way arrangement between customer, vendor, and NN that maintains NN's B2B focus. Happy to explain more if you want to spitball the idea).
I really was joking when I said that players could find latency improvement tokens in lootboxes.
> Explicitly passing on fees to the customer for a capability that may (or may not) help them see lower latency erodes trust and makes NN look like an insurance scam. As mentioned, not all users will definitely see performance improvements, so your proposed $5/mo is more of an insurance policy than a defined capability -- it could even be pure profit (who would know?)!
Plenty of things are offered as value-added services for products all the time. Sometimes you get a free skin you might never use, or a coupon on something else you'll never buy. Players are free to buy a similar service themselves through other providers, but as part of multiple incentives for pay, I don't think it's that bad, and really, when it comes to the current status quo of paying for products with advertising or with cosmetic enhancements that are designed to have no upper limit in purchasing amount and advertised in a way to make children go crazy over them, I'm happy if the payment alternatives include more options that don't include that, even if they aren't ideal in some other aspects.
It would be reasonable position if we were talking about a free market, and not a market captured by last mile ISPs operating a cartel on bandwidth supported and enabled by their regulators.
The near future involves last-mile ISPs openly selling $40/mo bundles for Facebook, Amazon, and Google. In that environment, flat fees for sometimes potentially lower latency seems exploitative (albeit no doubt wildly profitable).
If NN objects to their service being used to further entrench that future, then being opposed to working with customers who want to do flat-rate passthrough pricing seems defensible and consistent.
(Regarding your comment on the status quo, I completely agree.)
This reads like a traffic targeting system ala FB's cartographer (c10r) bolted onto an ad exchange (continuous bidding etc.)
Interesting to see if they can keep the 10-second update intervals while scaling to more users.
Curious to see if the market actually wants to pay for marginal improvements in gaming latency. I would have expected something like this targeting VoIP to be more lucrative.
This sort of stuff has been around for a while, it helps some, but realistically your game needs to be designed from the ground up with latency and stutter in mind.
If you do this, you wont need the complexity and expense of this sort of thing.
First you need client side prediction and smoothing. Next, you must design your game so it follows certain very strict rules about player interaction. These rules allow the players to feel the have real time synchronization when in fact synchronization is lazy.
The specific rules required are different for each game, and must be nailed down before game design begins.
Are you teaching game networking to Glenn Fiedler? You probably learnt it from him, or someone who read his articles!
Now seriously: client side prediction is not magic and it always introduces its own problems, like getting killed after getting into cover (especially in low TTK shooters) or the opposite: bullets not registering when if they hit in your screen.
Client side prediction is sleight of hand. It will never replace actually perfect network conditions and will still feel better the lower your latency is.
Sure you can client side predict, but since the real game lives on the server, you want the minimal latency to that server so you see the most up to date version of what’s really happening, and your actions take effect as quickly as possible. You still want the lowest latency here.
This could be useful for long-distance remote desktop and SSH too. Curious how to build on top of it for diverse apps though (use it as a VPN somehow?)
Right now we are an SDK that embeds in the client, server and backend for applications. It is open source too (BSD license), but not yet publicly released. Will be released soon, I look forward to seeing what people do with it, outside of games.
This is true, but without a marketplace and somebody paying, how can we tell whether the QoS requests are legitimate? I mean, “yeah, my traffic is always real-time priority”. People just will cheat it... so Network Next is a way of doing this, while saying, OK, for the truly real-time traffic, applications are willing to sponsor it. Since it’s paid, it won’t be exploited.
* A national policy that actually encourages competition
> Also one that classifies 'the last mile' as public infrastructure to compete on top of.
* Fiber to the home/MDU (Ethernet the rest of the way)
* Where cable Internet must be used, the anti-buffer bloat parts of DOCSIS 3.1 (DOCSIS-PIE).
* Actually having peer to peer focused ISPs, not content consumption ones.
Upload should be no less than 25% of download; ideally symmetric.
* Better visibility in to peering between users
and actual effort applied to removing the bottlenecks.
> A national policy that actually encourages competition
I'm all for improving the Internet and dunking on large ISPs, but what does this mean? Specifically what kind of national policy would foster competition (regulation tends to favor monopolies, right?)? Also, don't these private networks and marketplaces constitute competition? Or are you talking specifically about the network edge while the article is talking about the backbone?
This article has me thinking that there is potentially a lot of economic efficiency to be unlocked by these sorts of marketplaces--basically routing based on economics, but maybe this is antithetical to net neutrality ("pay to play" internet routing)?
Ideally competition in all areas, but different areas might require different policy and this is a question that should actually be studied by experts, or at least in academia summarizing the results of policy in different nations (as well as how closely they might match similar housing density for actually lived in areas).
Some places that I think should be at least initially studied / considered for LAST MILE* competition include:
* LAST MILE (connectivity) being the connection between citizen's houses and peering/point-of-presence locations: connecting citizens to a marketplace of different service providers.
* Remove legal barriers that prevent municipalities and utility entities from providing last mile connectivity
(often laws lobbied (paid) for by existing monopoly or near monopoly companies).
* Things similar to "Local Loop Un-bundling"
> Require the existing providers break out last mile connectivity from their back-haul package.
> Require they sell access to their private PoP at a cost which they also pay
at a similar rate (to the PoP on the balance sheet).
* Things equivalent to 'right of way' grants (poles on public land, trenches, etc) must be owned by the citizens.
> Also leased at FRAND rates to any entities
> Serviceable with 'one touch' by any entity licensed with the owners (citizens via the City/County/State/etc that owns it)
Possibly other things I can't think of in a short break.
> but maybe this is antithetical to net neutrality
By separating the bulk transit last mile provider from the private network direct routing solutions, I think we actually guard against a lot of the problems net neutrality is meant to fight (as long as the ISP doesn't classify the private network provider as higher tier). Net neutrality is only really needed because there's not enough competition in the ISP space for large swaths of the country. If we each had 4-5 ISP choices, net neutrality actually starts being a negative, because it impacts types of offerings that might require special priority routing to work well and that are willing to pay for it. The market would probably settle on specific plans where consumers may or may not pay for better streaming/gaming capability, and that's a good thing. There would be no chance for an ISP to strong-arm a content provider because no ISP would have enough market share to do so (customers that care would just jump ship to another ISP if youtube/facebook/whetever ran like crap and another ISP advertised better speeds for them).
Anti-trust action against the major telecoms would not favour monopolies, certainly :)
There are other regulatory levers here, though. Prevent municipal governments from awarding monopolies to providers (Boston and Comcast, I'm looking at you). Require last-mile fiber owners to lease to multiple competing companies. And so on...
https://steamcommunity.com/groups/steamworks#announcements/d...
They started using it in some of their games a couple of years ago (Counterstrike Global Offensive and Dota2 if I recall correctly).