One of the reasons why the massive OVH datacenter is in its particular location in Quebec is the extremely cheap power from Quebec Hydro. At one time I checked and the large customer rate was 5.6 cents (Canadian!) per kWh. I am curious how much they are going to pay in VA and why they didn't choose to build a datacenter somewhere in OR or WA near the Columbia River hydroelectric systems instead.
Probably because they're aiming to provide a private government cloud and they want to be geographically closer to key clients - Fauquier county is a 20 min drive from DC and big co's in West Virginia. Also because the incentives Virginia is providing ($1.25m in cash, cuts on sales and use tax, cash for employee training) make up for a higher energy cost - especially the tax cuts.
It's also on a major trunk which helps considerably with connectivity, and ping times from there to the east coast of the US are comparable to Chicago.
That cheap power is another reason why Québec is a big manufacturer of aluminum which is an extremely energy intensive process.
Because I used to live there and know the electric company was, prior to being acquired by a bigger one, well run by engineers (a good idea if you have nukes...), I expected their power rates at (eventual) massive data-center scales to be good.
While I'm not sure I believe the "We're Great!" first thing I found with Google indicating the possibility of a rate even lower by a few tenths of a cent, it looks good ... except of course per Google the exchange rate is running 1-0.75. Still, their power should be pretty affordable, I'd more worry about tradeoffs in more expensive real estate vs. easier hiring of talent and the other advantages of being in a major telecom metro area.
And their are other advantages as noted by others in this sub-thread, and doesn't AWS have its biggest region in NoVA?
Data Centers are great for the tax base, multi-million dollar businesses that bring in loads of tax revenue, but not lots of people, which reduces infrastructure costs. The additional tax dollars thus goes out to hire people, etc. etc.
Virginia is on a huge high-tech growth track due to the increased tech thrust in the Fed Gov space. The state is even reviving towns with tax benefits for computing centers in remote parts.
While this isn't along the famed "Tech Corridor" in Virginia, it could be the beginning of a "Tech Triangle" that could quickly grow to a second place tech hub to SV.
While this isn't in Ashburn, it's between Gainesville and Warrenton at a site that was formerly an NSA signals intelligence facility [1] and is now a fledgling industrial park that this just-outside-of-exurban-DC county is trying to push. It's adjacent to major FAA air traffic control facility.
That being said, I wouldn't count this as "new node" in the Tech Corridor, it's simply a good site in a lower-tax neighboring county, but still in DC's exurban western periphery. When other parts of Virginia away from a 20-mile radius of Dulles are seeing this kind of investment, then we can talk about a Virginia Tech Triangle.
The place where I colocate our servers (the company is probably in the top five in the industry in terms of size, with data centers in about a dozen cities) runs with a tech staff of ~six during the day and two overnight. They've obviously got other staff for all of the other business-related roles, but data centers just don't require a lot of staff in the general case.
That's pretty typical. You need maybe a couple of technicians to run the water and air handling systems, a handful of people on skateboards swapping out dead machines, and some security guards.
Here's a recent article that says Google employees 400 people in their Oklahoma facility, which is enormous.
not really, no, a large scale datacenter operator does as much as possible to reduce the number of people involved. Ideally the only staff you need on site are relatively low wage, low skilled persons who know how to rack a server, connect a power cable, connect some network cables according to a master plan. Everything else happens through automated provisioning software and neteng/sysadmin staff who work off site, could be anywhere in the world.
I am actually surprised it is as high as 54, if you visit some of the really huge datacenters in Quincy, WA or The Dalles, OR you might walk through the entire facility and see 3 people working on a "busy" day.
Two people around the clock with no provision for vacation or absence is a minimum of 10 people (if you keep all shifts 40h and below.) Add a couple for redundancy, some sort of manager, cleaning staff and a surge of day time staff and you're at 20. You can probably use a few more people, some tradespersons, and security (this is near DC, they have to be thinking about Gov work) and pretty soon 54 is not too crazy.
OVH provides lots of leased servers so you need larger staff than a pure colo. They're also calling this US HQ so add some sales, support and operations staff that would/will be remote at the second US datacenter.
Apple and Facebook are both establishing datacenters in Denmark. For the Facebook one they estimate 1.200 jobs building it, and 1.000 new jobs both working there and around the datacenter.
I think they are going to be very disappointed if that is the amount of jobs they expect.
> Over the course of the project, more than 1,300 people have worked on the site – an average of 250 workers a day. When completed, Facebook has committed to hire 35 full-time employees to run and maintain the data center.
As per the 1k jobs, sometimes those numbers are calculated based on how many new jobs can the local government (directly or not) create with the tax surplus they'll be getting from the new business coming into town.
In the same way that some companies have negative externalities like pollution, you could think of a data-centre more as having a positive externality of job creation.
Data-centers don't do many things themselves, using manual labor; instead, they're just large machines that consume purified resources (power, live fiber, provisioned equipment) and turn it into APIs. But purifying those resources for the DC's consumption does require people—and DCs need a lot of each of the resources they consume. For every N data-centres that get built in a state, that state probably employs a good few more power-plant workers and linemen, for just one effect.
Not really. That infrastructure scales up. You pick up a few construction jobs and maybe get some reusable utility infrastructure. That's it.
A lot of the other roles are field service type things that
don't use local workers.
That's not a bad thing, just not the end-all, be-all. if the building of data centers or similar facilities are a trend, you will build up good local capability to build these facilities and may start winning other business.
I'm just surprised as the absolute magnitude of the number. Virginia has hundreds of thousands of unemployed right now; 54 is probably within measurement error. Even if they're off by orders of magnitude, 500 is a drop in the bucket, and 5000 is barely noticeable. It just seems weird to me that that number is big enough to be worth trumpeting from the hills.
This amount of job growth seems typical at first blush. A haphazard sampling [1] of job creation announcements in Virginia as a result of state grants reveals:
April 2013: 25 jobs in secure document and card printing
March 2014: 40 jobs in new lumber mill
August 2014: 25 jobs in manufacturing coating spray for aircraft turbine parts
August 2014: 25 jobs in new lumber mill
August 2014: 500 jobs in parts for automotive engines
October 2014: 75 new jobs in parts of automatic transmissions
December 2014: 97 new jobs in HQ consolidation for medical device company
April 2015: 160 new jobs in food processing for fruits
May 2015: 100 jobs in lighting fixture manufacturing
January 2016: 50 jobs in tamper-evident label manufacturing
February 2016: 50 jobs in ingredients for cosmetic and pharmaceutical industries
March 2016: 15 jobs in new lumber mill
March 2016: 1800 new jobs in client support and consulting
September 2016: 50 jobs in paper towel manufacturing
Not sure that all municipalities charge a tax on the servers. However they get taxes on power and utilities like water. Also a larger base of utilities means the capital costs are spread across more billing.
They may not all do so, but the largest data center concentrations in the US are in Loudoun County, VA and in the Dallas/Fort Worth areas. You will pay local tax on your equipment in both locations.
You see similar things with manufacturing coming back to the US. Plenty of SKUs, not many jobs. This is because it's the most easily automated SKUs that get brought back to the US.
i'll speak to what i know -- an equinix datacenter probably employs about 20 people, most of them security staff.
most of the people who work on the actual facility are vendor staff, i.e. hvac specialists, power system specialists, the people from caterpillar that maintain the diesel gens, etc. for that matter, even security is outsourced (but they are still full time @ the facility).
everyone else is a customer, coming in and out to do internet-y things in the cloud aka working on computers in datacenters, same as it ever was.
OVH does mostly dedicated server so they may find it economical to have slightly more operations staff. The job count also includes the US HQ so sales/management type positions.
I would love OVH if they would finally implement automatic recurring credit card billing. Ive been asking them for years.
What a horrible single point of failure that is.
I believe this is due to them being a French entity and the French laws or banking systems requiring this. But I could be wrong, I'm basing it on the fact that Gandi had the same thing unit they created a US Entity which could do the billing. I'm guessing this is not far away for OVH either now.
They do have automatic recurring billing for the Public Cloud servers now, but on the dedicated servers it is still the old payment system for some reason.
So does this make my data with them fall under the category of American companies and is not subject to warrant less searches? Or only their VA data center?
If it's out of the US, the US don't have jurisdiction and I don't see them breaking into OVH's datacenters in France anytime soon to get to your data.
If they really want to access it and are doing covert operations I don't think being in the US will stop them.
I might be naive, but as far as I know there's no question that data is always safer if it stays in datacenters in Europe rather than in the US (which is why OVH is going to great lengths to separate their US entity as much as possible from the French one, and also why their first step in North America was Canada and not the US).
I suppose my communication with the server would be encrypted, but wouldn't the NSA try to store all the back and forth transfers between my US-based laptop and my OVH server in France?
I'm definitely questioning this decision because I believe a lot of their business comes from people doing some pretty questionable things with copyright. Might open up some of the larger video hosts that work with them to the scrutiny of US authorities.
I wouldn't worry about a personal seed box, but some pretty big video hosts work with them closely.
Yeah, I guess this means that their plans to build a data center in New York in 2016 fell through. [1] I love OVH, but I've been waiting for a while for a second data center in North America. I hope they build this one.
I believe reasonably priced dedicated servers (from companies like OVH, Hetzner) will be big thing in future. Cloud based virtual machines are useful for many tasks, but it is also pretty nice to get 256GB/6 core/4TB machine for 140€ per month.
I agree, using dedicated hardware is certainly nice and has some advantages over VMs. It's not only a big thing in the future but also at present, companies like OVH or Hetzner seem to do well despite the rising of the "Cloud".
You exactly know what you get performance wise, e.g. no bad influence from some random neighboring VM. Security is also a concern with virtualization, isolation between VMs turned out to be not that perfect as desired. You can also roll your own virtualization exactly tailored to your needs. "Cloud" VMs also tend to be rather expensive if you need them continuously, you pay for the flexibility they provide.
It should be a big thing, but their marketing/hype machine is seriously lacking. As a first step, building a data center presence in SV/west coast would help.
Their ability to do automated recurring billing, amazingly, is also lacking. I really wish they would fix that already. It's kindof a deal breaker for most people.
I had an even more annoying issue with their billing system.
I had 5 VPSs and whenever it was time for renewal, one of the VPSs would turn off ~5 days prematurely (before the renewal date). I contacted support multiple times about the error in their billing system until I got too frustrated and migrated last month.
And there were offers that were even lower. I have a dedicated server from them for about 42EUR a year (yes, a year, 3.5EUR/mo). It's just an Atom box with a single 500GB HDD but still amazing.
The Kimsufi line is a bunch of low-end boxes not meant for serious work, they technically allocate every server a /64 but you can't add RDNS records for anything but the "/128" you've been allocated.
If you want a proper /64 block and support for it you need to get a server from their SoYouStart or OVH branded offerings.
Good question. I don't think it has that much of an effect, as none of the above compete with businesses like OVH on price at all.
Companies like OVH that sell low priced are basically real estate investors (and, in fact, a number of real estate investment firms are large data centre owners) with various degrees of value-adds, but where the way to grow your profit margins is to identify lower priced property and cheap power relative to your competition and wait for your capital investment to appreciate.
AWS/Azure/GCE on the other hand are tech play where they are getting their huge margins on value adds or perceived value adds and are much less directly affected
> Governor McAuliffe approved a $1.25 million grant from the Commonwealth’s Opportunity Fund to assist Fauquier County with the project. The company will also be eligible to receive sales and use tax exemptions on equipment.
Was this worth 54 (unspecified types) jobs? I guess It's not that bad since it doesn't mention property tax exemption which usually goes along with new developments.
I'm not sure what the average salaries look like for data center employees, but assuming a reasonable, round figure of $50k/year, that's about $2.7mm/year in additional area income. Which has a multiplicative effect as it gets spent in the local economy and supports other area jobs. It also costs a lot of money to build out a data center and get all the infrastructure in place to support it, and that money gets spent (idealistically) on local contractors.
Plus, once the area infrastructure, local zoning/regulations, and talent pool gets built out to support this type of project, it becomes a lot more attractive for other companies to bring similar projects.
Tax exemptions to attract business can get real shady and dubious, but it's not just the permanent, direct jobs you have to factor in. The potential positive externalities in this case seem like a solid case for tax exemptions. And the grant helps absorb some of the cost of getting local infrastructure in place that'll be needed to support it; infrastructure that'll benefit more than just OVH.
Great. If they figure out the ordering system, it could be interesting. Last time (3 months ago) to install 200 servers (that they claim to have up in 120sec) would take 9 weeks, even if with our long term commitment.
If you're installing 200 servers that's not a dedicated order. That's a colocation. Why on earth would you order 200 from OVH? That's what, 4 whole cabinets if you squeeze? Do people really order whole cabinets through a sales system clearly designed for 1U to 3U at a time? You really expected four cabs in 9 weeks with no involvement on your part?
Of course they need 9 weeks and that's actually pretty good, considering what my procurement timetables look like for facilities my company owns. Shit, I can't even get 52U to the loading dock in 9 weeks.
> So I really don't see any problem with 200 boxes from OVH.
Its not cost effective compared to hiring ops and colo'ing it yourself. Once you're large enough, you hit tipping points:
* When to move from cloud to dedicated equipment
* When to move from dedicated equipment to someone else's colo (usually Equinix, but lots of providers in this space with varying levels of "warm fuzzies", which would cover on site techs, power and network redundancy, diesel commitments, and so on)
* When to move from someone else's colo to your own datacenter
(or in the other direction, depending on business requirements)
It also helps that US tax code (Sec 179) provides gracious depreciation schedules for physical compute/network/etc, which means that profit spread between cloud providers and you running your own gear goes back into your business or into your pocket
Disclaimer: 15 years of ops experience, including selling hosting to Fortune 500 companies as well as helping companies move from cloud to on-prem as well as the other way around. I've had to run cost/benefit analysis for this most of my career.
The OVH costs are often not a huge premium over self-leased equipment, especially when you factor in power, co-location fees, and other overhead.
There's a sweet spot between 1 server and some number, say into the hundreds, where OVH is still cost effective. Beyond that you'll absolutely want to roll with your own gear because you can negotiate for a whole cage instead of partial racks.
I think what he's saying is that 200 boxes from OVH is dedicated and that 200 boxes is well past the point where you should switch from dedicated to colo.
The cloud part was just an example of a tipping point and not the main point of the comment.
Depends where are you based. OPS in the Bay Area is above 100k/year which need to be added to the price of servers. Also one person has a huge bus factor so you essentially need two people (if one is sick or on vacation). Now you're at 200k+ annually without having any server installed.
200 servers (at least the one we were interested in) would cost us around $22k on OVH each month. That means if I remove the personal that I would have to hire, the cost of servers is now down to $6k/month (200/12). For that money you can't really find a better option.
In our case, where we are running thousands of severs on any given time, the flexibility is much more important than price. So we built our service around pre-emptible instances on GCE (the same as spot on AWS). You can't beat dedicated server in performance, but it's close enough and they're making for it by having a great infrastructure.
As an operations employee, I'm shocked you've run thousands of servers in some kind of service while talking yourself out of any operations employees. An operations hire is not a prerequisite to moving past dedicated; even with nothing but 200 dedicated servers you are way past the point of needing at least minimal operations. Contract this out if you have to.
We are not a direct cost center that can be discussed in those terms. Our insight will reduce capital and operational expenditure beyond our salary, because that operations hire would have told you how insane of an idea paying $22,000/mo for four cabinets of gear is and why a capital tradeoff with depreciation is a fiduciary responsibility to your investors and shareholders. You can buy at least a dozen U for that each month and then pay for nothing but where it lives with a dash of break-fix to taste.
I can put four cabinets of gear in a colocation for a quarter or less than that if you'd swing a little capital. You are wasting money on poor operations architecture and design and you don't have anybody to really tell you.
Even beyond that operations is a skill, much like marketing. I know a lot of people think they can fake it for a while (and they usually can), but after a point it's time to act like a grown up company and bring someone who does nothing but think about this shit on board. Security, performance, remediation, all the system level grunt work you shouldn't be concerning yourself with as engineers. Or you can keep throwing multiple operations salaries at your four cabinet OVH deal and keep getting ripped off.
We have no dedicated servers as we're running everything on GCE. Our use case is highly specific as we've a huge ingress network requirements, which we're getting free of charge.
> "You can buy at least a dozen U for that each month and then pay for nothing but where it lives."
We don't need dozen a month. We need hundreds now. We may not need them in 6 months though and then what? Will I rent them out?
That's the sort of capacity planning and management an operations chief would do for you several quarters out, based on experience with shifting business needs that they have acquired over a career of dealing with highly specific use cases.
Think of it as putting an intelligent layer between your demand metrics and your server fleet. I live for utilization, just like you live for your product. Hire an operations nerd who does and your company will be much better off for it; based on description it sounds like you or the other engineers are already involved in operations anyway, so you probably won't need two. Hire one and let them tell you.
Would you denigrate a software engineer by saying you had to rely on them as a magic wizard for their knowledge the same way you're referring to someone with ops/capacity planning experience?
Experience and knowledge isn't wizardry or magic, but as a business owner you're free to burn up your cash on pride if that's what you'd like to do. Not every problem's solution is a google search or API call away.
I am a SRE, I've done software engineering for years, and I do ops and lots of capacity planning at the moment. I have no intention to denigrate anyone.
The word you should have focused on was 'magic'. In the same way that magic does not exist, it is not possible to plan future capacity with accuracy. It can even get worse if the system is of limited size because the time spent on analysing and planning can quickly cover the savings.
The only way to have good predictions is to already have the systems running in production [for months], with a [mostly] static user base, running applications that don't evolve, and never add any additional service. Given these circumstances, we could have good metrics on current usage and make good prediction on the future... except that to be at this point the hardware has had to be bought already.
> In the same way that magic does not exist, it is not possible to plan future capacity with accuracy.
I...what? Capacity Planning for Unpredictable Growth is literally SRE 103. It's hard and will have a margin of error, yes, but it's not bloody magic. The "magic" is in identifying and collecting the correct metrics that somewhat model the abstracted utilizations of the property, because almost everyone picks the wrong ones; this is situational so there is no blanket advice to offer except that load average is almost certainly wrong, as well as consulting only one metric. If you're working capacity from five or six key application metrics you're probably on the right path.
SRE is, quite specifically, application operations engineering. If you can't model your application's growth I'd be more inclined to call you an SA. (There's absolutely nothing wrong with that, to be clear, even speaking as an SRE. And I am aware several valley companies are diluting the term.)
We have one server with them as VIP and very happy about it, they react fast on support with a special team dedicated... we asked a quote for this server and they proposed several solutions.
We couldn't get any number of any servers. I asked for 10 servers that they are boosting to be up in 120s. Nope. I asked for 20 random servers in same range. Nope. After 4 days of back and forth we ended up nowhere. They stop responding after lunch time PST so the communication was delayed by 24 hours in every single instance.
That's why the big cloud providers charge what they charge - you're basically paying for them to have a huge excess capacity ready and waiting.
If you want to take advantage of the prices of providers like OVH, use colo or dedicated servers from those kind of providers for "base load" for their cost, and tie in cloud resources for traffic spikes or batch jobs.
EDIT: As pointed out elsewhere, though, 200 servers is way above the point where hiring - or contracting with someone - to rack and maintain your own equipment in a colo should be at the very least considered, with some caveats (e.g. location - rack space rental is largely correlated with local land prices, so if you're in a property hotspot dedicated servers maybe remain cost-effective vs. colo facilities within short enough travel up to larger numbers of servers.
As a french guy, Azure is pretty much unknown here (except for MS shops, i guess) and Amazon is already used by startups. The difference between Amazon datacenters in France instead of Ireland / Germany will not change a thing.
