Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: Advice on bringing community broadband to Los Alamos?
171 points by fraserphysics on Sept 16, 2021 | hide | past | favorite | 67 comments
For a decade I have been advocating for the government of Los Alamos County in New Mexico to help with county wide broadband. The coincidence of my renewed effort, the COVID-19 pandemic, and a sympathetic County Council is yielding progress. At the direction of the County Council, the County staff has a job posting for a Broadband Manager. (See the posting at https://selfservice.losalamosnm.us/ess/employmentopportuniti...) The staff is posting the opening for a second time because the first time, only one applicant met even the minimum qualifications.

My questions for Hacker News are:

1. Does the posting look realistic?

2. Does it describe someone who could get a community broadband network built?

3. How can the County get qualified people to apply?

I've been pushing on this issue since 2011. I went so far as serving on the County Board of Public Utilities and becoming its Chair in 2015. Finding that the Board had no authority over the issue, I turned to political organizing and set up the website blabnow.blog. On that site, you can see what I think we need a Broadband Manager to do at https://www.blabnow.blog/los-alamos-broadband-manager-positi...

Beyond asking for thoughts on the Broadband Manager position, I would like to read general comments on:

4. How to get local governments to take responsibility for modern communication utility monopolies?




Former network engineer that went into management. My email is in profile if you would like more candid guidance than I can give on a public forum.

1. Not realistic, but not outside of normal for an management infrastructure/ops position. What you are writing is what you hope for, but sometimes it sends a red flag to potential applicants. The worry many high level people have is that they will be turned into the 'do everything' person. Trim a few of the less directly related requirements off of the description (e.g. Microsoft certification).

2. Yes... but also No. There are a lot of people who push through their careers collecting credentials and turning it another rung on the ladder. Chances are you will get a rosy candidate at some point, who will put in two years of aggressively spending your budget to inflate their resume and then move on. Not that a majority of people are this way, but the filters are set in such a way that this is what you are likely to end up with.

3. Other comments will say 'pay more', but it will be difficult to meet market rates for this skillset when working with local governments. If you can't get approval to raise the comp, instead try to split this role up into a team. Governments won't pay one highly skilled person 250k, but they will pay four people 80k, and one manager 120k.

4. Politically? Find someone with pull, and make sure that it's 'their idea'. Something that they can put on their win list.

Procedurally? Don't boil the ocean. Handle it iteratively. Start with commercial areas and new housing developments. White-glove your initial smaller install and it will create the broad demand from the community to expand it.

Love what you are doing. Hope this helps.


Pay seems reasonable to me. Los Alamos is very inexpensive by California standards, and it's an absolutely beautiful place to live. As long as you're not planning to commute from Santa Fe, house prices seem pretty good.

https://www.zillow.com/homes/for_sale/1-_beds/?searchQuerySt...


> Governments won't pay one highly skilled person 250k

How can we change this? As a taxpayer, I'd be perfectly happy allocating some of my tax dollars to pay efficiency wages in situations like the one in this post, where hiring someone with specialized skills can save the community millions of dollars.


For the federal government, at least, there are highly paid scientific positions. It’s called the “ST” (scientific and technical) pay scale, it can go over $200k and has nice bonus opportunities too ($25k at the discretion of OPM and 35% at the discretion of the president).

However, there are only 470 of these positions in the whole government. Congress allocated the positions through the budget process, so call your representative and suggest they approve more (you could also mention SL, the non-STEM version of ST). Agencies are always asking for more senior-level headcount, the limiting factor is what Congress will approve.


Governments can pay $$$ to contractors and companies without much fuss. Just don't get hired as an employee.


As a complete outsider, I would think of this as follows. The current status quo is a Comcast/CenturyLink duopoly. Let's imagine I wanted to start a local ISP. What would kill my company the fastest?

The thing that would pop up as #1 for me is access to poles to run fiber. If this killed Google Fiber it can definitely kill a smaller upstart.

There are gatekeepers who limit this access and they work for the county, in the permit office. The incumbents have a relationship with these people, and if they don't you better believe that their folks will start building one as soon as your project is announced. All your local contractors currently supporting Comcast are also not on your side.

It is in these people's best interest to see your project die. To make things worse, they don't have to work very hard to make it happen. All they have to do is sit on a permit application long enough to cast doubt on the project. Whereas the broadband manager has a huge hill to climb in comparison: they would have to build a relationship with all the people blocking their progress or figure out how to sidestep them.

So the person you are looking for should have high-caliber government sales skills. Good luck finding someone like that who also has a masters degree in CS and will work for 140k. This person can make millions in commissions in enterprise IT sales and they know it.


Los Alamos County owns the electric, gas, water, and wastewater utilities. So they own the poles and have rights of way. I think we are in a better position in that regard than most communities. However, you may be correct that the pay scale is not adequate.


Even if you strung fiber to every home, you still need to get the internet to your area.

I looked at some major carriers maps, and it looks like you are going to need to trench at least 25-30 miles of fiber along 502 and 84 to reach the Zayo fiber running along 25 south of Santa Fe.

THe Nuclear Security Administration is in the process of laying a second fiber path to the lab up there, and while you have zero chance of piggy backing on a DoE fiber line the planning documents might be helpful in figuring out what is viable and what isn't. https://www.energy.gov/sites/default/files/2020/04/f74/final...

(Feel free to email me if you need further help, it is in my profile)


I'm sure you're aware of that, but just in case other people are reading: getting uplinks to well-connected places is the easy part. You just need a little money (think 1-3k€/km) and a few authorizations to use public utilities or cross private land here and there. Laying the local loop is the hardest, most expensive part of the project because there's a lot more cable to lay, and you usually have to gather a lot more authorizations from different people/orgs.


The average here in the US is $15,000 for 24 strand and $27,000 for burial per mile. However the area in question is full of historical areas and protected land that limits the fiber run to the most indirect path possible following a highway.


Wow, that's a lot of money. I must admit i don't know about commercial fiber deployment so these numbers blow my mind. But from what i saw for the costs in big french cities (where there are established underground networks to pass cables) and french countryside (where you just use a tractor to dig "for free" on the border of a peasant's field) that sounds very expensive.

To my knowledge, all french highways and major railways are also already populated with fiber so you can just rent some point-to-point transit there from established operators without having to lay your own.

Good luck with this project keep us updated


Factor in Indian Country. Not that simple.


Do you mean indigenous sovereign nation territories? Or Indian country as in India one of the most populated countries on Earth?

I assume the former because the conversation was about USA in the first place. It is my understanding that US industry and law enforcement has exactly 0 respect for peace treaties with indigenous nations and their territorial sovereignty, especially when it comes to extracting natural resources.

However, it's good if local communities care for respecting other people's lands. If your fiber needs to cross private property or indigenous reservations, i'd recommend having an honest discussion with their residents. Maybe some would actually like to have good connection as a public service (as the Rhizomatica project has been building mobile phone and internet infrastructure for some communities in another part of the world) but would be hostile to private for-profits ignoring their needs.


I think there is something called REDInet that runs up 84 to Taos. And I've heard that Los Alamos County has some kind of in with that new path. Also, I think Comcast and others piggy back on the existing link that DOE uses. Finally, I want the County to hire someone to stay on top of such issues.


Not sure the chance is zero. Since Los Alamos is a company town, you might be able to convince NNSA that it's in their best interest to create a community network for LANL enployees. Jill Hruby (who runs NNSA) is a reasonable person and she's from New Mexico. Can't hurt to ask.


Asking for help from NNSA is exactly the kind of thing I hope the broadband manager will do. We need to find and hire the broadband manager first.


If the county owns the poles and last mile electrical grid you should be looking at some of the fiber networks run by county sized PUDs in eastern WA as an example. Go visit some.


You could also just relax your requirements. You need a great convincer. You need a technical person. There is no strict requirement they have to be the same person though. What you have here is two jobs mashed into one job req.


My reflex is to approach a problem/issue from a technical perspective. However, you are correct, we need political and managerial perspectives. I am not in charge, I am only a citizen advocate, but I will advocate for something like what you suggest.


> However, you may be correct that the pay scale is not adequate.

It depends on the cost of life in your area. Here in western europe, 140k€/year is almost unheard of for a public servant (except for a few specific, very corrupt branches of public service).

But i believe having a single person responsible for both the infrastructure and the human networking/sales is a dead end. As long as you have support from other local public utilities a less-skilled "liaison officer" (with some understanding of the technical issues) could be a separate job, or even integrated into another job posting (eg. infrastructure commission).

The hardest part for starting a local ISP is getting permission to use infrastructure to lay fiber and put antennas. If you have cooperation from local public services, you're already 80% of the way and the rest is just throwing a little money at public servant who are convinced about public service and the utility of public infrastructure.

If you just employ any commercial sysadmin, you'll get the same problems all commercial ISPs face: an inadequate network disconnected from the actual needs of the people below. Network architecture should be achieved with consent and feedback from the local communities. For example that's what the Scani.fr cooperative has been doing here in France in partnership with local city halls and smaller communities.

One thing to keep in mind is an architecture for a public service is pretty different from an architecture for a private commercial project. Commercial ISPs optimize for selling as many subscriptions as possible, making it artificially harder/crappier to share connections. When you're running a public project, you're trying to break even (plus save some money for hardware renewal and future investments), but trying to make profits is antithetical to your goals. Here's a few pieces of advice, feel free to take or ignore:

- router freedom: you should enable all clients to run their own hardware if they wish

- ouf-of-the-box router acquisition: for clients who don't understand (and don't want to) network settings, still have a commercial "box" offering for 2€/mo or paying the hardware cost upfront

- public IPs and reverse DNS: since some of your clients will want to selfhost their infra, you need (free or really cheap) options for that

- your routers should support multiple multiple client connections on a single uplink, so that people in more remote communities who share uplink (eg. wireless over >10km) can still benefit from advanced features ; as much as a i despise legal responsibility for the person on the lease, this enables people to share their connections without becoming liable for what other people do with it (not sure if that's a concern in USA, but definitely a concern in France)

- you should setup a "public access" program so that local cities/communities can request shared internet access for librairies and other public places; setting up public wifi networks is a "hard" problem for them and contractors ask for crazy amounts (dozens of thousands of euros for just a few access points)

- your network should serve as educational tool: partner with local associations, high schools, universities and companies so you can take trainees to learn all about network infrastructure... Internet is a public good but lack of access to infra/knowledge is often a barrier to get people to maintain that infrastructure

- two kinds of people may be very interested in early access to your network: people who don't have broadband yet in their communities (because commercial ISPs can't bother to serve them), and organizations with broadband but bad service (usually due to bad routers, or lack of advanced network features) ; the former is easy to get connected for cheap using wireless links (given permissions to lay antennas on high points), while the latter has more money upfront to develop your fiber local loop, which in turn helps develop the network to reach more users elsewhere

- human proximity/support is the most important part of public service, no matter how wrong things can go, having actual humans facing you to tell you all about what went wrong and how they're fixing it is very valuable... commercial ISPs are well known to crap on their customers, local communities and regulators. if you plan to open a public service internet access, there should be 7/7 support (not 24/24, except maybe if some commercial clients are willing to pay big bucks for that) and an actual physical location you can visit in person in case your phone service is down, or you need to bring damaged equipment, or you'd like an actual human being to face you to answer your questions

- social justice is an important part of public service: there should not be a huge wage gap between the different workers ; ideally, there would be equal pay for all, but that can be a tough sell to your local government (who can sometimes be as corrupt and greedy as national governments) ; in any case, noone should have more than twice the pay of the lowest-paid workers (including cleaning staff), because that's good for morale and cohesion of the team reducing competition and envy and fostering cooperation for the benefit of the entire project ; support staff in particular should have technical background/formation (unlike commercial ISP support staff who doesn't know shit and just gets in the way) and decent working conditions, they are the primary interface between the service and the customers, and having happy/knowledgeable support staff is one of the top criteria to ensure happy clients and good service

There would be plenty more to say about economical models and political concerns, but i think that's already a lot to go through. Good luck with your project, it's a pretty cool one! And don't forget to partner with local non-profits/coops/schools, they can be an invaluable source of insight/feedback into how to do things well given a local context.


I was surprised (and mostly shocked) by the ISP oligopoly in the US. When I heard that it killed Google Fiber and even such a money giant as Google can't properly fight it, I finally understood how bad it is.

It there any perspectives of game-changing in that field? And what can be done to change it?

I'm genuinely curious about it.


It’s pretty much got to be regulatory changes. The regulations we have currently are what makes it legal to exclude competitors.


Broadband is a natural monopoly. Most natural monopolies are either owned by the government or regulated. At the national level government in the US has failed with respect to broadband. I hope we can address that failure locally by building a government owned broadband utility.


> When I heard that it killed Google Fiber and even such a money giant as Google can't properly fight it, I finally understood how bad it is.

The thing is, Google refused to pay bribes. Had they spent to the tune of a couple million in "campaign donations", the opposition would have collapsed.


Random point of Los Alamos networking history: around 1995-1996 the school system (5 elementary schools, 1 mid, 1 high) were apparently able to get dark fiber strung between them. (I may be mis-remembering, it might have been dedicated service like T1 lines, but I thought it was a level up from that. I was the student representative on the committee setting up the first internet service to all the schools.)


It might be worth talking to these folks: https://ruralinnovation.us/

This won't answer your immediate questions, but they've been heavily involved in getting startups to move to small communities. Municipal broadband is a big part of that, and I believe some of them worked on those efforts in Vermont.


I just bought land in Vermont and the state subsidized fiber network there is really impressive. It's not municipal/public-owned, but state-subsidized, but it still looks like the service is really good and comprehensively covers lots and lots of the state.


Thanks for the link. I sent them a note.


As a former Los Alamos resident I wish you luck in this endeavor, but my guess is Los Alamos is probably too small to run a cost effective ISP operation. My recommendation is to approach this from a "how do we bargain with ISPs as a single unit" perspective, not a "how do we build and operate our own network" perspective. My apartment building in Seattle just converted from cable to fiber. We're only 200 units but it was still enough of a prize to get some competition and concessions out of the vendors, and collectively LA has 50x those numbers. If you can get even 2 vendors to bid on a contract for the whole city then you may be able to get a pretty decent deal. Conceivable they'll refuse to play ball because they're scared of the community broadband concept, but at some level they are still businesses and so if the deal makes sense they may take it. Probably still need to hire a manager, but the expertise to look for is how to manage vendors and negotiate contracts more than "Cisco advanced certification", and you may be able to hire them initially as a consultant instead of a permanent employee.


We may be too small, but I think we have a good utility department that provides electricity, gas, water and waste water service. We certainly need a manager as a first step. I agree that "Cisco advanced certification" may not describe what we need.


What would you consider more of a win here: everybody in town has internet that they pay for through the city (either via taxes or fees), or Los Alamos owns and operates its own network? The model used for the other utilities may not work, for example the DOE puts up the money the lab uses to buy electricity which effectively subsidizes the town's costs, but they need power and don't really have much of an alternative (unless they fire up the reactors again...). The lab already has their own backbone connection so I doubt they're going to be interested in buying internet services from the town.


The best win is that the County owns the last mile fiber and customers buy service that they choose. Part of the reason I prefer that has to do with issues like net-neutrality.

You are correct about the other utilities. The Feds actually built some of the infrastrucure and simply gave it to the County when they gave up running it.

You are sort of correct about the lab's backbone connection. However that connection is owned and operated by the phone company, now called Lumen. Lumen sells capacity on that link to Comcast.

The lab is not going to be interested in buying service from the County, but as a consequence of the pandemic they have said that they are interested in supporting employees working from home. Working from home with the available Internet connections is difficult.


Seems like that last item is the thread to pull on--get the DOE to cough up some dough to help ensure that lab employees have reliable home access. If you dangle the promise of federal money you can probably get Santa Fe and Rio Arriba counties interested in the idea too. So now I'm thinking your manager needs political experience above anything else.


I'd guess the layout of the town is a benefit? It's mostly dense-ish between the mesas of Los Alamos and the blob of White Rock. Other towns fade into low density and get a lot sparser, but you only get that in one neighborhood of White Rock.


You might want to contact the people who went through this in Princeton, Massachusetts:

https://madned.substack.com/p/thin-pipe-part-i

https://madned.substack.com/p/thin-pipe-part-ii

They probably have lots of contacts.


Thanks for the link. I'm part way through part-ii. It's a good read.


I don’t really have any constructive critiques beyond just saying that I support what you’re doing.

I think the long game that local politicians miss (or are financially motivated to not see) is that broadband is becoming as important to communities as roads. It facilitates lowering the costs of so many things, benefits companies+individuals+government, improves equality of access+opportunity, on and on.


I mostly agree with you. For the first time in a decade, we have a County Council in Los Alamos that understands that broadband is an essential utility. We are a small rich county with a population less than 20K. I don't believe financial motivations have affected any Councilor's judgement in the 16 years I've lived here.

If you know people who might be interested in the Broadband Manager position, please tell them about it.


I cannot comment on the domain but having lived and worked in New Mexico I'd posit that there is a talent shortage in the state. I believe defense contractors and the National Labs attract and retain all the talent. My anecdotal data is I once interviewed, got a rejection, then 6 months later they called back telling me "no one else in NM was able to do as well as you did on the technical interview." During that position I saw a few LANL people hire on only to go back 8-18 months later. So, I'm not saying hire them but that applicant who met the minimum qualifications just might be the best in the state who is available.


My last post was June 12, 2012, but I did a blog for over 4 years about exactly what I thought. The county has already done REDInet (Regional Economic Development Iniative) which thought it would sell little chunks of bandwidth for big prices after it was done, and somehow that last chunk of fiber through the San Ildefonso pueblo's overhead power poles never happened, and fiber between the townsite and White Rock never happened. Comcast had the only fiber between the hill and White Rock, and consumed it with TV and the relatively slow DOCSIS technology. If the county builds anything, it will be very expensive and unpopular with the voters.

Try talking with Allan and Mariela Saenz at Los Alamos Network, losalamosnetwork.com.

My blog is still there at fiberlanm.blogspot.com.


I just looked at your blog. You quit shortly after I started.

My home Internet connection is through Los Alamos Network.

I believe that the Carson Electric Co-Op in Taos uses REDInet to provide an Internet option to their customers.

I heard for years that the San Ildefonso prevented us from connecting to REDInet. Then Alan told me that he and Comcast and everyone else buys capacity on the link that Lumen operates for LANL.

We need a someone employed by Los Alamos County who knows what's going on.


Maybe there is a way to wake up REDInet to reality. But New Mexico tends toward bureaucracy that sucks up money and really doesn't accomplish much, so I don't hold out much hope, either regionally or locally.

I moved to Camp Verde, Arizona. Things were even worse here, but the small cable company sold out to Suddenlink out of Round Rock, Texas, and that sold out to Altice, from France, and I think Suddenlink here is similar to Comcast in Los Alamos. There's nobody like Allan, and cable's fatal flaw is that upload speeds are capped at 10 Mbits/sec. I paid over $300 a month for a couple of years for less, and now pay about $130 for 50 down, 8 up. Sigh.

Trouble is, if you offer gigabit, or even 200 megabit, symmetric, for a few hundred customers, you need 40 gig or 100 gig upstream, and that's a big-city thing. Citylink in Albuquerque does have that, and I suspect LANL does, too, though I'm not in touch with them the past 8 or 9 years.

Also, for WFH (work from home) for LANL, within the county or from Taos through Belen, there's still the fact DOE (Department of Energy) really, really wants all LANL traffic to go through ESnet (Energy Sciences) so latency that could be microseconds inside town becomes at least 70 milliseconds. That makes a huge difference in how useful it is.

Other nearby resources are Jane Hill and Cybermesa in Santa Fe, Richard Loewenberg and his first mile mailing list, also in Santa Fe and John Brown at Citylink in Albuquerque. The subscription web page for 1stmile is:

http://mailman.dcn.org/mailman/listinfo/1st-mile-nm


You are correct about ESnet. Before I quit LANL last year, I ran traceroute from my LANL desktop to my house. I found that the packets went through Sunnyvale. If LANL is serious about WFH, they will address that.


Terry Wallace was amenable to a direct connection in town, but the determination of whether fiber losses were excessive was left to that now-Lumen contractor, and somehow, yes those losses just didn't come up to snuff. It might have been a problem for 10 gig, but we were only trying to set up 1 gig, and the county was planning to tear down the "annex" near the Smith's gas station and destroy the route for the fiber anyway. Which the county did.


It sounds like we're the same person. I'm on the utilities board for my city and am pushing for the same thing. I meet monthly with a group organized under the Fiber Broadband Association that is focused on municipal broadband. We have quite a few members and most either have some form of municipal broadband or are in the process of getting it. If you shoot me an email (see my bio) I'd be happy to connect you. Everyone is eager to help and many of them just went through this process in the past couple years. I think that's the best route forward as they've already gone through this and can share what they know.


Please note I am not a telecommunications or networking expert. Verizon 5G Home Internet[1] or T-Mobile's[2] version appears to be the perfect fit for New Mexico.

>> "Verizon 5G Home Internet, meanwhile, is a fixed wireless access service (FWA) powered by 5G Ultra Wideband that provides ultra-fast Wi-Fi connections to the home."

[1] https://www.verizon.com/5g/home/ [2] https://www.t-mobile.com/isp


Consider talking to chris hacken of nepa fiber: https://news.ycombinator.com/item?id=13688595 , http://chrishacken.com/starting-an-internet-service-provider... .

Also, consider looking at https://godigitalmarin.org/ on what they've done.


There's currently a test going on in Belgium about a low cost alternative ( wireless / mesh / 60 ghz)

https://pharrowtech.com/news/pharrowtech-telenet-and-unitron...

The trial is expected next year. Follow up on it and consider it later on. It could bring down the cost to a fraction and has an expected bandwith of 1Gbit/s per person.


I'll have to see the results to believe it. Current 60Ghz options are several hundred dollars for a link (at least), and the link is ~1Gbps. You can't put that in an actual mesh network and get 1Gbps per customer.

That article you linked is marketing speak.


The source of how i found it ( dutch): https://tweakers.net/nieuws/186942/telenet-test-inzet-60ghz-...

You should check about where it came from. It's a spinoff from Imec. Which is one of the most unknown R&D companies in the semiconductor area for the public ( Eg. ASML is similar, but ASML is a for-profit)


I would look to Westfield, Massachusetts as an example:

https://www.whipcityfiber.com/

They got municipal fiber to their town and then started providing services to other towns in Massachusetts to do the same. All the tiny towns around my slightly larger town have gigabit fiber now for $79/month while I'm stuck at 25% of that for the same price from Comcast (my town is "fully serviced" by Comcast and thus not eligible).


Step 0) Have at least a few dozen million dollars and solid guaranteed access to right of way for aerial or underground fiber

Step 1) Hire somebody who has experience running a facilities based last mile gigabit class ftth isp. Sorry to say your job posting requirements and duties doesn't look anything like that right now.

Step 2) Have that person recruit the rest of the team


I may not have much experience in broadband but i’ve spent most of my career in municipal/county software. My email is in my profile if you want to reach out, I’d like to help any way i can.


Thanks, but I don't see your email in your profile.


Would it be cheaper to just get everyone that would use it StarLink?

I’m just weary from trying to fight horrible broadband providers blocking every step of the way. Maybe the answer is to go over their heads.


One of the big drivers in Los Alamos is working from home. I think that StarLink's latency would be a problem there.


As a LEO network Starlink should have latency comparable to terrestrial connections. I've signed up and they say they'll service my area in "Fall of 2021." And I can see LANL from my house, so it's worth checking.


Please tell us more about StarLink’s latency.


My starlink terminal is 23-30ms to downtown Seattle. Not as good as well implemented terrestrial connections but it's amazingly better than anything else satellite based.


I don't have any personal knowledge about it. I just did a Google search on starlink latency, and as Trump would say "people are saying ..." Seriously, I just checked the search results again and find that https://www.speedtest.net/insights/blog/starlink-hughesnet-v... says starlink's latency varies with location. EG. 45 ms in the US and 101 ms in New Zealand. Better than I would have guessed, but still not great.


Others are reporting 23-30 ms in US.

What latency number do you require?


I don't know. I want to run an X-11 server at home and use applications across town.


If it’s good enough for gaming it should be good enough for X-11 and, I don’t know, CFS.


Maybe also worth looking into tge history of Longmont, CO's NextLight (https://mynextlight.com/).


Corporate democrats in the state legislature will be happy to trade away your local rights for the promise of union jobs in purple counties.


Stupid question: could it be useful to set up an associations where all broadband users are [voting] members?


I helped a similar network get off the ground in IL. I’d love to help - email is in bio.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: