160 employees remote is impressive and commendable. Zapier is fully remote as well (but half the size in employee count). I'd say "write everything down" is a great shortcut to the sorts of practices you need to cultivate.
We've also noticed that over-communicating is critical but hard - it is surprising the things that are "yeah yeah, we know" to some but are "oh we're doing that?" to others. This is only natural - organizations become complex as they grow, and individuals are busy doing their thing. You often have to bring the important data to them.
On another note, working remote is awesome. I recommend everyone give it a spin once in their careers - but try to find a team that embraces it. I've heard mixed experiences from those who were the single remote person on a team.
Honestly, I haven't seen a company that does the hybrid remote/on-site thing well. I'm sure they exist, but every time I've worked at one the people who were remote were out of the loop on almost everything.
Hybrid remote/on-site requires some great discipline. If you have a watercooler chat with someone about a feature, it's easy to forget that a remote colleague wasn't there for the discussion/not document what was said.
+1 for working remotely on a team that makes an effort to make it work.
MY company does hybrid remote/on-site, and one of the best ways to mitigate issues with off-site communication is part-time work from home. Since on-site workers spend some of their time as remote workers, everyone has a good idea of what's required to empower remote developers.
It's not always roses and rainbows, but it definitely helps.
The company I work for does hybrid remote/onsite as well. I'm remote and I actually think that having onsite people take some remote days is almost required to get things working. After people take a couple of days remote they suddenly understand the issues a lot better.
Also for me, not always roses and rainbows. But that mutual understanding is really helpful.
WHY?! What parent poster describes above seems like "heaven" to me. Some people can be "true nomads", others can spend 50% time in office, 50% working from home, everybody can be happy and find "what works for him/her".
I hate both working in-office full-time AND working fully remote. But a mix sounds awesome, I always feel energized when working one day from one place, another day from another, and I also enjoy having the meetings in-person...
We do it here are DigitalOcean and it's just awesome. We have people in NYC HQ and remote all over the world (for example I'm working from Turkey). The key is not only over communication, also to care about your remote employees, let them feel like they're really a part of the company, bring them to the HQ, not per year, multiple times, etc.. There are many things but so far DigitalOcean is one of the best companies I've worked remotely (and I'm working remotely for almost 5 years for different companies).
I'll be contrarian here and say that I actually hate that most remote companies make frequent on-site travel part of the job and pretend like it's a perk. Travel, including on-site visits and "group retreats", are not perks for me.
I have a family and a life where I'm at and part of the reason I like to be remote is because it allows me to be productive with minimal impact on other things (e.g., don't have to waste 2 hours in commute time each day). Places that require quarterly visits to the home office are kind of just aggregating all of that disruption into a single week. Still better than doing it every day, but not a thing to tout as a benefit.
It is contrarian. As someone who has worked remotely to a significant degree for quite a while, I also find face to face get togethers (whether in an office or elsewhere) incredibly valuable.
You're welcome to your own preferences of course but, as someone on a largely remote team, I consider regular get-togethers in various forms pretty much essential.
Just curious, what specific value do you derive from these events that makes them "essential"? While I understand that a lot of people who may not have a lot going on at home may find them enjoyable, I really don't think it can be characterized as essential for accomplishing the job tasks.
If face-to-face communication is "pretty much essential", what's the argument for continuing to run the company as a remote/distributed enterprise? Isn't the whole concept of telecommuting predicated on the fact that face time is not essential to performing the job function?
Leaving aside the personal interruption represented by such events, I don't understand the philosophical underpinnings. To me, it seems like someone said "There's not enough chance for politics and cliques to emerge over IRC [which is incorrect btw], let's make sure we all get together physically at least once every 3 months so we can find new things to take petty offense at."
Again, I think that well-run open-source projects provide some really good examples of the right way to do a distributed company. They usually have one big pow-wow per year, and it's a convention with planned talks, networking events, etc., and of course, attendance is voluntary and the meetings are transcribed and broadcast. Debate and consideration occurs online where everyone can participate.
If a massive project like the Linux kernel can get by without having such events every 3 months, why can't $Random_Startup?
I've been a full-time remote worker at the same company for the last few years. I'm the only remote worker on my team and one of a small handful dispersed throughout the company. I've been back to the head office once. It's absolutely true that there are negatives and downsides to being remote, but nothing that jeopardizes my ability to perform my duties.
Beyond that, I did several years fully remote as a full-time contractor. Some clients were local and I would meet with them face-to-face up to a couple of hours a week, but the majority were non-local. I only met a few of these people in person and I don't feel that impacted my ability to do my job.
Like so much else in contemporary tech culture, my opinion is that it traces back to putting single 24-year-olds at the helm and making everything subservient to their whims. They think it's cool to rent out a vacation home on a beach in Belize and fly 10 people out to set up camp in there for 2 weeks. Grown-ups with spouses and kids or other significant non-work obligations are likely to be less enthusiastic.
Some people like me dislike working fully remote just as much as they do full-time in-office... a hybrid approach is only thing that can work for me. "Constant consistent variety" is what keeps me ticking and productive. Also, just as you want to take time off from work, don't you also feel the need to take some time off from family too ? ;)
I value variety as well. I mostly work from home when I'm not traveling but I like to go into the office at least one day a week even when I don't really need to. I'm similar with traveling. There are times when it really wears me down but then I'm doing the routine day-to-day in the office/home for three weeks and I'm ready for a change.
>If face-to-face communication is "pretty much essential", what's the argument for continuing to run the company as a remote/distributed enterprise? Isn't the whole concept of telecommuting predicated on the fact that face time is not essential to performing the job function?
I think it's predicated on face time being not necessary on a day to day basis given the tradeoffs associated with requiring it.
A lot obviously depends on the task and individual preferences. I probably shouldn't have used the word "essential." However, I've found some level of F2F very useful and I'd probably encourage teams that weren't too distributed to get together physically on a semi-regular basis (as most want to do anyway).
It's funny that you mention the Linux kernel. I've heard GKH on a panel at a LinuxCon explicitly say that he thought one of the reasons that Linux development was so robust was because there was enough money in the ecosystem to get people to events like he was attending.
Could we quantify this a bit more? A largely remote team for me is a 3-7 person teams, all but project lead are remote. The regular get together are 4-8 hours every month plus some annual dinners. The remote distance is just 20km - 500km
I think our face time is abit excessive but it's ok, even nice, but that's because it fits into my workday and family life, and it's always paid travel time.
It probably depends a lot on the nature of the work and working styles but having a F2F sync every month or two seems pretty reasonable for a not too geographically distributed team.
I also have a son and I understand you. You don't have to go btw and it's not mandatory. However it's encouraged and you're welcome to go and visit. So there is a big difference there!
Can confirm—DigitalOcean is great at remote work. I'm consistently impressed by how well connected I feel, despite being across the ocean from our HQ in NYC. People really care and put in the effort to make it work. It also definitely helps that both "sides" are well represented—we're almost exactly 50% remote at the moment.
And of course, I'd be remiss if I didn't say that we're hiring. =)
I applied to Digital Ocean and the recruiter before even speaking to me sent me a link to an online test that tested exclusively for knowledge of how to automate things using Chef. Nowhere is Chef mentioned on my CV as I have not worked with that config management tool, although I have with worked with others Puppet/Ansible. Needless to say I deleted the email. Talk about ridiculous. What a massive fail.
This was the second such miserable experience I had with D.O. I assumed that my first bad experience may have been due to a lack of maturity at the time with the interview process and so applied again a couple years later. Wow, was I was wrong.
Last year I had my resume submitted by a third-party recruiter for a role there. My resume was rejected out-of-hand by the hiring manager because the companies/customers I had worked for were "too enterprisey" (i.e. I had only worked for government contractors at that point). The "too enterprisey" quote came directly from the recruiter. I suppose he could have been blowing smoke up my ass, but I don't think he would have said that if he was.
Ah, okay. Sounds like they were trying to focus on people with a specific type of experience and not blacklisting people based on a specific employer or anything.
Yep, this is absolutely correct in my experiences. I'll chime in, since I'm a remote employee at a company that is hybrid remote/on-site but with the vast majority of employees being on-site.
Long story short: I actually started on-site but negotiated part-time remote. Then after about a year of that my spouse accepted a dream job offer on the other side of the state and so it was no question that we were moving. They wanted to keep me on and so I transitioned to full-time remote.
It has it's ups and downs. From my experience, it depends on the team and ultimately the lead and/or the manager. If they get it, it's good. If they don't, don't bother.
My last team, it was beautiful. Everything worked great. It became obvious to me after transitioning away from that team just how much extra work my manager was doing to keep his remote employees in the loop. With the team I'm on now, it's not working at all. Why? Because, as you so eloquently put it:
> the people who were remote were out of the loop on almost everything... If you have a watercooler chat with someone about a feature, it's easy to forget that a remote colleague wasn't there for the discussion/not document what was said.
This is exactly the problem I am facing down. About a month ago, I even tried talking to my manager about it. Even going so far as using this exact phrasing. His response was that I should come to office for 1 or 2 day trips! Completely missed the point. I'm not surprised, I guess I just expected some effort.
I'm trying to remain positive but it's tiring. I can do good work and make progress but I'm with a lead who is insulating me from meetings and it feels deliberate. The few times I am invited in to a meeting, it's easy to see that he accepts and assumes credit for _the teams_ efforts. He was hired recently as a Senior, our manager wanted him to transition to a Lead, and he's actually landed somewhere around trying to be an architect. Ranting aside, when I bring my concerns up to him it's obvious that there's no attempt to better understand the issues/challenges of the hybrid approach. It doesn't help that he's never worked with remote employees before, either.
>His response was that I should come to office for 1 or 2 day trips!
I understand he missed the point in this case. However, if a company is willing to pick up the travel costs associated with a partly distributed team getting together on a semi-regular basis, that's often a pretty good approach.
Yes, processes should be such that ongoing communication is good. However one of the costs of having a more distributed workforce (which has various benefits to the company, including financial) should be a bigger T&E budget.
It seems that having great speech to text would make remote teams workable. Meeting notes are automatically taken and organized into a searchable journal
If nothing else, that would at least be an improvement over "let's take a photo of the whiteboard after this one hour ad-hoc meeting and attach it to the jira ticket".
I do make trips up every quarter, and they do reimburse expenses, but they are unarguably disruptive and occasionally burdensome to my life and my family.
I think there are a couple of challenges. You are right that it is hard for people to understand how easy it is to get out of the loop. On the other hand too, if you are in a team with people who are politically motivated, it's pretty easy for them to actively exclude you if they think you are danger to their goals.
For me (being remote and even 9 time zones away), I've tried to scale back my own ambitions. It's hard because I've worked as an agile coach for a long time, so I'm used to having quite a lot of influence. I think trying to reinvent yourself to work for other people's success can be useful.
It can be a different style of political game. You need to show how they are better off with you than without you. It's not nearly as depressing as I'm sure this sounds. Just a different way of making a contribution.
I don't see why your manager missed the point. If the "vast majority" of employees are onsite, then it seems reasonable to expect the few who aren't to come visit, or otherwise change their working style, rather than the majority to change their working practices for the few. Particularly if it's the same state in the same country and not, say, a 20-hour flight from Bangalore to San Francisco.
Yes, I believe you must be. I'm referring to where the OP said that as a remote employee I feel out of the loop on a lot of things. So, I tell my manager that I'm missing important information about the projects I'm working on because people have these infrequent, but important, impromptu meetings and those details don't exist outside of those spoken words and their heads and his response is to tell me to come to the office more often. To do what, exactly? This is strikes me as missing the forest for the trees.
The fact is, this entire team is all on Slack and we all have email and we all use zoom. We have internal Wiki's for documentation. We have JIRA. And on and on. The point is, there are plenty of established avenues available for people to dissiminate this sort of information but it doesn't happen. I don't think that I'm asking them to modify their behavior by asking that they include their team members or to remember that they exist when they have those impromptu meetings.
You ARE asking them to modify their behavior when you say that they should use Slack/email/whatever other tool for your benefit, or that they shouldn't have an impromptu chat around the water cooler because you're not around.
Since you're in the minority, you can't demand the majority change their working style for you. It's up to you to fit in to how the majority works, say by showing up in the office. Or you'll miss out important information, which will be to your detriment, not theirs.
> To do what, exactly?
To communicate with others the way they apparently prefer to communicate — in person.
And to build relationships, which lead to working smoother, and prevent misunderstandings from snowballing. It's easier to get pissed off with someone over email than in person.
There are many advantages to facetime. You can decide to forego those, but you can't tell others to, just because you want to.
> You ARE asking them to modify their behavior when you say that they should use Slack/email/whatever other tool for your benefit, or that they shouldn't have an impromptu chat around the water cooler because you're not around.
No, he was asking that already when he asked if he could work remotely. And the company said yes.
> Since you're in the minority, you can't demand the majority change their working style for you. It's up to you to fit in to how the majority works, say by showing up in the office. Or you'll miss out important information, which will be to your detriment, not theirs.
Then they shouldn't have agreed to let him work remotely. Majority or minority doesn't matter after that decision has already been made.
Maybe they should revisit that agreement if apparently they can't make remote workers fit in the team.
"No, we believe we can't make it fit with the way our teams are currently communicating" (and do not want to change that) is a perfectly legit answer to the question if remote work is an option.
However, once the decision is made, of course effort is required from all sides.
Otherwise, why even bother asking about it.
It's like applying for a job on the condition that you can't work Wednesdays because of allergies. The boss is perfectly fine with this, you can work the other 3.5 days, you're hired. And then it turns out the most important crucial meeting of the week, of which no minutes are kept, is on Wednesday. Now this is your problem, somehow. Who messed up here?
This all seems to boil down to unstated expectations from both sides, which is a no-win situation. My last boss's attitude to people who wanted to work remotely was "You can do whatever you want as long as your work doesn't suffer or you don't end up disconnected", making it explicit than the onus is on the remote worker to fit in to the team.
scruple's boss should have told him that right at the interview stage, before a job offer is made. Maybe say that he expects scruple to turn up in the office for a few days every month. Or whenever a major decision needs to be made and scruple is out of the loop. Or have regular one-on-one meetings over video-conference with everyone in the team to remain connected, making up for his physical absence.
Whatever the details are, scruple's boss should have told me he'll need to make an extra effort to make up for his remote work, and both sides can then decide whether to go ahead with the job.
In the absence of that, both sides end up holding the other responsible for the problems, which is a no-win situation.
> scruple's boss should have told him that right at the interview stage, before a job offer is made.
Well, this is awkward. I don't think you even read my original post. I literally said that I transitioned from on-site to part-time remote to full-time remote. I've been with this company for many years, it's gone exceptionally well until very recently.
> Maybe say that he expects scruple to turn up in the office for a few days every month.
We've had the working agreement ever since I went full-time remote, back when I was on a different team, that I am unequivocally full-time remote with no obligations to turn up in the office. *editing this to add: If that needs to change because of new team dynamics, that's fine, but it's going to be awfully hard on my colleague in Mexico.
> Or whenever a major decision needs to be made and scruple is out of the loop.
Major decisions should NEVER be made in isolation or with team members who aren't even sure what is being tabled. Why do we need to play "catch up the other 3 team members" when we're having a discussion about a major decision? Oh, because another team member keeps having side conversations with product people or customers and not relaying the information back to the rest of their team? Hmm, sounds like a personnel problem to me -- and not with the "out of the loop" team members, either.
> Or have regular one-on-one meetings over video-conference with everyone in the team to remain connected, making up for his physical absence.
Myself, my other full-time remote colleague, and my other 2 team members who are on-site meet every single working day. We scrum, we go our agile rituals, we pair remotely, etc... We meet with our product people over video conference twice a week. We meet with our managers infrequently but I personally talk to my manager over the phone at least once a week.
> Whatever the details are, scruple's boss should have told me he'll need to make an extra effort to make up for his remote work, and both sides can then decide whether to go ahead with the job.
Sorry for getting that detail wrong. I re-read your original post to make sure I didn't miss anything else.
Since you now say that you agreed ahead of time that you had no obligation to turn up in the office (that information wasn't there in the original post), I now agree with you that your manager was wrong. (This is something I've seen happen over and over again with managers — they say yes without thinking things through.)
If people are deliberately withholding information from you, as you said you suspected in your original post, that's obviously a bad thing.
Regarding my suggestion to have regular meetings, you're already doing it, so that's good.
Speaking as a manager here, what am I supposed to do when a valuable on-site employee approaches me and asks if they can transition to full-time remote?
If I say yes, I get skewered later when people on both sides aren't adapting to the new situation.
If I say no, them I'm a do-nothing windbag who can't feel like anything is getting done unless I can physically see the people typing and clicking.
If remote work is the minority situation, you're going to see challenges. Even if it's the norm, there are decisions that are time-sensitive and you're going to sometimes be dropped (same as if you were on PTO that week; you're just more exposed to it as a remote worker in a predominantly co-located team).
I have teams all around the world; even with that, we still bother to move desks to put squads all together in the same location. If you agree with the latter practice, I think you have to give some nod that co-location matters, even if only a little bit.
I think you still missed the point: If he is there every now and then, then he still has missed all the water cooler discussions that happened while he was not there. It's not as if they will see him and say "ah, right, now that I see you, we discussed this and this last week when you are not here.".
The only way to fix this is with some concious effort to reproduce the results of water cooler discussions explictly -- and if you spend that effort anyway, you might just as well do it via Skype or Slack... He doesn't have to be on-site.
It's not all or nothing. If he shows up every once in a while, he's better connected than if he never showed up. People tend to forget people who they don't see.
I'm not for one second suggesting they don't talk to each other and have impromptu meetings. That is absolutely ridiculous. I'm just asking that important details from those discussions find their way to the rest of the team members. I don't think that's asking a lot.
And, going to the office, those conversations will only happen to include me if they happen (they're impromptu, somewhat random) and when I'm there for 1 or 2 days at a time. When I go back home and to my co-working space I'll continue missing things. That's the point.
If the project suffers as a result of all of this, which is what is actually starting to happen, then I suppose by what you're saying here I shouldn't actually care? Heaven forbid I ask _individual people_ to keep _the team_ in the loop about the thing we're all trying to _collectively_ build.
Usually the modifying of behaviour in these cases just means writing down stuff more, which is usually beneficial to everyone.
On another note, having remote employees without supporting that type of working company wide is just silly squandering of resources. Sadly that is fairly common anyways.
I agree that writing down stuff more is usually beneficial to everyone, but at times, it has felt like rehashing things that everyone who was present in person already knows, just for the benefit of remote people. That's a cost at times.
Exactly, it requires an active investment in over communication. Hopefully someday the tools will evolve to help reduce the need for this to be so manual.
Bouncing an idea off of someone remote gets difficult. I feel bad for interrupting them and they feel annoyed at being interrupted. Culture also plays a big role in that as well.
I'm not a developer but just last week I was visiting one of our locations where a lot of the people I work with are actually located. I was down there for a specific meeting but I bumped into enough people and had serendipitous conversations with them that I left saying to myself that I should really go down there more often.
To your other point, communication tools are still pretty limited. IM/Slack/IRC are good for some things though a lot of people I work with don't use them. I've yet to see anything that remotely compares to being together in a conference room for high bandwidth interactions.
Unfortunately, while most/all of the tech folks hang out on IRC, it's much less used by a lot of the marketing/business people I deal with. I should use it more routinely myself but the fact is that it's not that widely used by some groups.
I think what helps is to keep everybody in the team in a permanent video call during working hours. This requires suitable hardware and software (I would love to build a company producing these tools one day.), but also a considerable amount of discipline. A wordier version of my thoughts is here: https://www.konstantinschubert.com/2017/02/02/improving-remo...
That sounds just awful. Maybe I'm spoiled, but even when I'm at work, I go in my office and close the door when I actually have work to do. Trying to work in that kind of a panopticon style environment sounds nerve-wracking.
That sounds like a modernized version of the guy with the lash monitoring if the slaves work constantly. Like really, there are people who find that managerial technique desirable? Wow.
I've tried this on a team that was mostly remote but on certain days located in there regional offices. It didn't really help with anything. The only conversations that took place were conversations that otherwise would have happened anyway but would have been initiated on chat and then taken to video. The only difference was that now someone walked up to an iPad and started asking loudly if the person they want to talk to is available.
It works at our company because the only remote guy is the only one in his "domain" (iOS). He probably doesn't care about the frontend watercooler chat and anything important about the backend will be mentioned in standup.
That definitely helps. I find there are other little cultural things you miss like inside jokes/shenanigans. They're not work-related but they're related to work and can make you feel like part of the team.
God, who cares of this work jokes shit? They're never funny, it's always the boss that makes them and everyone laughs way too much.
FFS, We're not friends, we don't have to be, and it's OK!
I like what I do, take pride on it and everything, but at the end of the day, my private life (where I get to choose my friends) is OUTSIDE work hours. And I'm committed to close the laptop as soon as I'm done.
I guess this kind of "school feeling", "we're on the same boat" feeling, where groups of people linger together in long coffee breaks tends to belong to corporate, especially in USA where they work stupid hours to "show off" their commitment. Utter nonsense.
One more thing that I really feel I "broke free" from (after becoming full time remote) is the ALCOHOL CULTURE. Peer pressure into drinking: company credit card on the tab and unlimited drinks (no food, otherwise it's cheating). So degrading.
No offense, but this works both ways. I mean, this is not a secret that there is this sort of people who feel that way, and if I can see that you are one of them — I wouldn't want you to be a part of the team. I mean it strongly: if I don't need you for some very specific kind of job, no way you are hired. Even if you are great professional (but replaceable) and a nice guy otherwise.
I don't like myself this weird culture where it's supposed that you have to be always a nice guy, politely nod and smile and say "Completely agree! Except..." when there isn't a single point you actually agree on, so I often dismiss such labels as "toxic attitude". But your attitude is just that — toxic. I'm okay with people being somewhat rude and arrogant and whiny and always complaining about "why should we", but this attitude of "I have a real life outside, and here — I just work here, so fuck you all" — that makes you both absolutely unreliable and very unpleasant to have around. Unfortunately, the word "toxic" fits really well.
I've actually never had anybody to work extra-hours, but knowing that they would is significant part of what makes me care that they don't have to. Technology is worthless, people are what makes companies to succeed, and absence of meaningful relationships with your team basically renders you to be a piece of quite primitive carbon-based technology.
You are entitled to your sentiment, but calling people and their approach to life "toxic" is a good way to alienate them and dismiss what you have to say. There are many ways to run a business, and I've run teams that have shipped multi-million dollar software entirely remotely, with minimal face time and non-work related socializing.
There are plenty of companies that manage their work in a way that minimizes the human component and maximizes the technical utility of their engineers (judged on productive output alone), almost in a way that makes the human component and all it brings with it (unreliability, politicking, emotional outbursts, backstabbing, you name it) irrelevant. In fact, my ideal company reduces face time to close to zero, because that's the only reliable way to eliminate politicians, bad actors and people who try to game the system using social skills and emotional manipulation. A nice side effect of remote work is that it automatically creates that sort of environment, unless you go out of your way to change that (but then the remote work model is probably not for you).
>renders you to be a piece of quite primitive carbon-based technology.
The fact that you don't know how to manage engineers in an entirely meritocratic, non-political environment, says more about your management abilities than anything else.
I'm in no way suggesting that people spend all of their time at work or have significant overlap between their work friends and after-work friends. However, it's nice to work with people and not robots. Some people can tolerate sterile work environments devoid of anything but neutrality, and some people can't. I'm not even saying you have to be friends with the people you work with, but being friendly goes a long way. You spend a large portion of your day with these people, so it's not a huge ask for someone to give a crap about them as people.
I think most people have a better office experience than you. I have to be comfortable actually expressing the situation I'm dealing with when I run into troubles, be that cursing about some code or explaining why something will take longer or shorter than expected to my boss. I feel much more comfortable in a less "sterile" work environment in that way. If I'm in a work environment that's too sterile I'll feel like every error I make, every little delay detracts from my value as an employee, and that's immensely stressful to me.
That said, no one here puts in extra hours to "show off their commitment", everyone shows up when they want and dumps at 8 hours later. I've stayed late when I was buried in debugging once or twice and got told to go home by numerous colleagues. Just things like that, telling you to put it down is a positive influence sometimes. I came in the next day and solved the issue that I had spent 3 hours on the previous day in about 20 minutes.
I think the importance of that varies from person to person... I don't dislike inside jokes but also don't care if I miss them—I'm just there to do my best work.
That said, I completely understand why someone would feel differently.
Agreed, especially RE shenanigans. I'm remote most of the time, but in the office this week, and small things like telling stories at lunch make a difference.
Red Hat has 10000 employees, a large proportion being remote (including me - about half of all developers work remotely). I can assure you that remote workers at Red Hat are not out of the loop on anything.
The thing is that even if I went into the local office, I would still be "working remotely", because the development teams are distributed all around the world. Therefore everything has to be done on mailing lists, IRC, bug trackers and occasional conference calls.
You have to be diligent about researching a potential employer with multiple locations, too. My last job on "on site". I learned on Day 1 that all of my teammates were all remote (or at least not in my office).
It's definitely doable, but it has to be remote-focused. At Wildbit, we have 12 in Philadelphia and another 14 in 14 different cities around the world. They keys are in the small things.
When we do video chats, the folks in Philly call in just like they were on a home computer. They don't all get together in a big room and call in. We also make a point to run all of our conversations through group chat and project management software. And plenty of the Philly folks work from home regularly as well. Remote work isn't simply tolerated, it's put first.
The remote folks (myself included) miss out on some of the office benefits like family-style lunches and get togethers, but we all make regular trips there and get to partake. It's not purely about location either since we're spread across numerous time zones. It just takes more deliberate communication.
There's always going to be water cooler chats, but those can just as easily happen via email where nobody would see them. We have to make a point to capture and share things to the right people.
I suspect if you want to do a partially remote team the best way would be to make an entire team of remote employees and then treat that team like a satellite office. We know those work for most companies, so as long as the team can communicate well amongst themselves I don't think them being remote will ultimately be any different for the in office workers than if they all went to an office in a different city.
I work for a hybrid remote/on-site company - we have made it work well. The most important thing I found is to make sure all of the stakeholders are present online for a serious discussion on whatever it is of concern. As soon as one is cut out of the loop, trouble starts.
Anecdote but our team does it brilliantly. Working from home? Just let the team know via slack and go for it.
Management is onboard with it and has taken the openly stated position of "we don't care as long as you're reachable during the hours it's reasonable for someone to be online" (and the obvious: "deliver your deliverables when they're due"). In my case as the ops lead, I have additional hours but we are so well baked into slack that if someone needs me, I get pinged via mobile, reply from where I am or delegate it. Heck just last week I did a couple of deployments from 4 timezones away, tested it, and left a PR note in the channel with a link to the JIRA ticket. It got worked, updated, fixed and closed while I slept.
I wouldn't say our team made a concerted effort, we just set the expectation that if you're going to be away, let us know, and during production hours the absolute bare minimum is that you'll be reachable if absolutely needed. Heck, even if you need to leave early to take care of home stuff or what have you: go to team chat "Hey I'm taking off early to take care of thing" and go. You'll get replies like "Take it easy" and "Cya tomorrow".
Every team meeting has a join.me session for remote people to join, they speak up and contribute if asked or needed, even our stand-up comes with an auto generated Google Hangouts link that gets published to slack every morning at 10am. If you miss the stand-up or can't connect, people have been really great about just typing up what they're working on when they're available again.
Otherwise, no one really cares. I've absolutely loved it. This is one of the better teams I've worked on just by how well everyone communicates without bombarding each other. People stay out of each other's way, but will ask when they need help and there's never any fear that you wont get someone willing to at least look at a problem with you; even if they don't know the answer-I've had devs pipe up and offer ideas and a second pair of eyes. Problem gets fixed, I shoot over a thumbs up emoji and a "thank you" and we go back to our own little worlds. Read-only Friday comes along, and we go drink.
Contrasted with my last job where I negotiated the right to work remotely because the distance constituted a total four hour commute. I was consistently turning in deliverables on-time or early, conducting client calls via an expensed voip phone and getting several orders more work done from home because I wasn't waking up hours earlier than normal to make the 2 hour drive in.
That privilege was arbitrarily lost because of a few critical management slip ups (owner took a vacation with more than a couple projects open in the lurch, and left no instruction behind for me in his wake, so things failed in a big way when). Shortly after becoming an 'office worker' again and putting in 12 hour days between commuting and sitting at my desk, I was out the door.
"Write everything down" is very important for any org hitting 150 people. It's when an organization hits Dunbar's #. [0] It's even tougher when you pass it and everyone is remote.
Yes - that's my point. And most startups act as one org (like 3 year olds chasing a soccer ball) and if they haven't subdivided by 150, that's when they need to.
The problem with remote, at least for me, is the visa. Being remote-only is nice for traveling around but the sad reality is that you will always be a non-citizen no matter where you go. Even if you really like a place, you will be working from there mostly illegally and can never fully integrate into local things that require bank accounts, IDs and friends. That was maybe the thing I disliked the most when I went completely remote, but for a year or two it's for sure awesome.
In my particular case, I want a company that is remotely open, but sponsors a visa for city/country X which could hold a tiny cheap apartment and a space to call home. But in my case these countries I want to call home are Japan and Korea and now try to find a remote company there.
I didn't know Zapier was a fully remote company, that's awesome :)
I pretty much agree with your thoughts about over-communicating, it's super important and we've run into exactly the same things, where people aren't always caught up on everything going on at the company. In my experience this happens in physical workplaces as well, just to a lesser extent.
We're around 50 and we have about half of our company working remotely at any given time - and we love it. We've introduced a lot of communication rituals to make this work. Starting a weekly all-hands with good AV setup was a huge improvement for us. In addition to saving the space for people to ask questions in a big group, it also got us in the habit of preparing data & updates to present to the entire company (e.g. write everything down). It can be monotonous at times, and we resisted big group meetings for 4 years, but it's pretty critical and works well at this size.
We also schedule quarterly travel to the office, and have a weekly meeting where remotes get together and share what they are currently feeling anxious & excited about. That's a remarkably effective way to learn a lot of things you might otherwise pick up on w/ body language, and more.
MySQL AB was about 400 before being bought by Sun and it was said that about 70% were remote so that gives about 280 persons.
I was impressed how well organized that work was.
I believe we've settled on Quip - we used to be Hackpad. I think Dropbox Paper would be quite nice too (but was too nascent when we were searching ~6 months ago).
>> 2:41 – GitLab values boring solutions: our product should be exceptional
Exceptional products have exceptional UX. Gitlab IMHO has the worst UX of all git based products out there, I much rather take BitBucket over Gitlab. I tried using Gitlab, but no, I would much rather pay the 7$ to GH for my private repos.
I sincerely hope they make an exceptional product. And 'should' better be 'must'!
I think the GitLab interface is pretty good, especially compared to GitHubs. I can't speak for BitBucket since I've only used it once, but I do remember having a hard time finding my way around it.
Isn't his just moving the goal post? Now the subjectivity is choosing what metrics to measure.
Are 10 easy clicks worse than 2 hard clicks? Is a long, easy experience better than a short stressful one? And so on. Couldn't that vary by each person's preferences?
By trained or by untrained people? Domain specific experts, or newbies? .And why "clicks"? For many tasks, the way to get them done realy quick is with keys...
I had to do some IT support for a doctor's office many years ago, and (unrelated to what I was doing), had a chance to look at the system the front desk lady used to manage the patients, appointments etc. and it used a "clunky" old text based system (perhaps even DOS based?). TO me, it looked utterly bizarre and incomprehensible. But man, the front desk lady was lightning fast doing anything with it. Of course she trained on the system for years, and just entered keypressed rapidly that didn't make sense to me, but achieved the desired result... So, zero clicks, low search times, no (for me visible) user errors... so I guess they never should have switched, as this was the "objectively optimal" UI?
Personally, I don't think so. I.e. I think the UI could have been better, and easier to learn. But of course it's also important to see how usable it is once you do learn it... I was at that doctor's office again, not so long ago, this time as a patient... new lady (the old one retired, I assume), new system... flashy graphics... everything done with clicking, I don't even know if there still are shortcuts, the new lady certainly didn't seem to use any. And it did feel slower overall -- though this is of course a subjective evaluation only, with many years between the two observations. Plus it's somewhat unfair to compare somebody who trained a decade on a system with somebody who for all I know started a week ago... but I guess that's exactly my point here...
Yes, performance can be objectively measured but different companies choose different metrics to optimize. The choice stems from subjective preferences. Hyundai optimizes for price and caters to the middle class, whereas Bugatti optimizes for speed and caters to whatever their market is. So subjectivity is inherent in the metrics we choose.
You shouldn't make design decisions from case studies alone, but the subjective experience of each individual user is a perfectly valid measure of UX performance.
No it's not. That is completely against the idea of ux. What someone says is not what they do. Measure. Analyze. Experiment. Tweak. Results. It's all objective data.
GitLab may not be as good as some others in the space, but I've never seen a repo UI that actively repulses me nearly as much as BitBucket's. Even SourceForge doesn't put me off like that.
Honestly, all Atlassian products have terrible UI. JIRA is just as bad.
> Honestly, all Atlassian products have terrible UI. JIRA is just as bad.
The individual UIs are not that bad. The problem is that people tend to use not only e.g. JIRA, but also Confluence, or more. And then the (massive) differences between their products in UX appear.
As a sysadmin, there's one thing that really annoys me: all that Atlassian stuff is written in Java (which means: on every problem one has to dig through pages after pages of Java stacktraces), and it eats resources like nothing else. GitLab got better in the recent versions, but it also had its fair share of interesting problems with memory leaks.
Oh, and both don't support MySQL (Atlassian does, but very limited) - which is bad if nearly everything else is MySQL and you already have a MySQL DBA.
I've still gotta disagree. Jira's UI is horribly inconsistent and haphazard. Some actions are buttons, some are links, some are hidden until you mouse over, everything feels like it was just tacked on wherever there was (or really wasn't) free space. It's probably the worst UI in a modern, commercial web application that I've encountered.
I'm curious what you dislike about the UX. Personally I like the "feel" of the website generally (though it's not as polished as GitHub) and just find it to be slow, which I think they're working on improving.
Personally, not a fan of how a readme-less repository just has an entirely useless main page. I much prefer having the file view a'la github on that page. That's the main UX complaint for me. Still use gitlab bunches.
Main complaint overall is the incredibly sluggish git push speed.
I think it should be the default setting. Even make it not configurable. Why would anyone want to see just Readme or just Files view when it's possible to have them both?
This is something that has been on our minds for a while, and we must do it better. We created an issue for a smart project dashboard that aims to provide a better solution, it would be great to have your input there: https://gitlab.com/gitlab-org/gitlab-ce/issues/27112
I've only spent a few minutes there, having signed up just a moment ago. What jumps out at me about the UX is that source code does not seem to be front and center like it is in Github and Bitbucket. The README is all you see on the first page of a project and I had to hunt a few seconds before I noticed the "files" link.
Hey Deepak, thanks for the feedback. We've built out a UX team and improving that aspect of the product is high on our list of priorities, but please feel free to reach out if you have any specific issues. Happy to document those for you.
Hey Amanda, I would list the following:
1. Intelligent profile: shows my top 6 active repositories, ability to mark work I do as salient so that shows up on the dashboard in summary instead of my recent issues/merge requests etc, the salient work marking should be automated at a future point of time (ML). I really think of this as an automated Developer Story in the long run.
2. Recent area: Where you show the contributions in the last year, recent issues/merge requests etc.
3. Code Reviews: A list view on the left which shows all the files changed and a cart where I can add multiple files (ex: a.py, b.py, c.py) to show how the changes my be inconsistent etc to a peer. A 4 quadrant view will be very handy.
These are the top 3 things I can think of, based on my interaction so far. I wish Gitlab success. Thanks.
The most frustrating thing about working remote was the complete lack of documentation around anything -- meetings, requirements, internal frameworks/libraries, etc. Good for them for emphasizing documentation.
I lost track of the amount of times I was told to "look at the source code" when asking about documentation for the internal framework.
Also fun was being assigned a feature and having it's functionality explained over the phone, with any subsequent follow-up being met with "Didn't we already talk about this"?
I never would have thought that a remote based team would have the worst documentation out of anywhere I've worked, but that's exactly what happened.
You work on product A. Your VP of very important things declares that every product should use library L. Library L has been designed to incestously interact with product D and makes a bunch of assumptions about what the "working environment" should be.
Teams B and C have already made the transition, so L made some kludgy APIs to make B and C interaction possible, though it is maintained with low priority because everybody knows D is the money maker. Both B and C have a bunch of assumptions in common that A does not share, and they both had to compromise and emulate D environment to some exent.
You are given L binaries, a few toy examples of how to use the most basic functionality in L, and access to B codebase. L codebase does not use the same source control system, so you don't even know where to begin looking for their code.
Please keep in mind that your manager does not expect you to learn all the inner workings of L. Your priority task is to use L to support some new feature in A. You still have a number of other tasks to accomplish in A that are not related to L, and some of those might raise in priority with little or no notice.
Do you still think it is a good idea to just browse L code, without knowing how large (or well writen) it is?
The fact that it is your preferred way tells me that the only internal frameworks you have encountered in your career were tidy, tiny, and self contained.
Having struggled with very large tools that were anything-but, I beg to differ... but I guess it is a matter of personal experiences.
Instead of being able to quickly peruse docs to see if a method existed and what it took/returned, I would be directed to dig through source code to see if it exists and what it did. This is basic API doc work. This meant something that should have been fairly quick took significantly longer.
Then there were times when I did dig through to understand why something worked. When I pointed out that something should be done differently (an id-lookup done hierarchal instead of flat) I was told "Oh that's deprecated, you're not supposed to be using that anyways". And how exactly am I supposed to know that?
Ultimately it was a frustrating way to spend a lot of time.
Proper documentation that explains what the code should do, how it does it, and if there are any known bugs is a lot faster (IMO) to read them to open a file to try and parse what the people who worked on it intended and what all its doing.
Add in cases where people go crazy with inheritance and you need it or else you have no way of knowing what all it is this new class can do.
You seem to work at a horrible place, with abrasive characters.
It begins with the fact that the company's code needs documentation. I'm a big proponent of Robert C. Martin's "Clean Code" and the idea that code should read like prose - by reading the code, its purpose and ideas should come across naturally.
It ends with people asking "Didn't we already talk about this?".
Is this your first job? Don't let yourself get treated like this.
> You seem to work at a horrible place, with abrasive characters.
Or a normal place with poor-mediocre leadership.
Documentation doesn't happen by magic, and documentation is a skill. If it's not a priority from management, folks will be heads down getting their own tasks done, not thinking about what the other person/team needs.
I worked at a place with remote and onsite employees, and I think two of major reasons it worked is because we had a habit of documenting all designs and projects in a wiki, and used Skype at all of our meetings (while sending an invite out before hand to anyone tangentially related so they could sit in if they felt the need to). The wiki would inevitably get out of date, but it usually gave you a good big picture and the edit history let you know who to track down for questions.
All of the development teams were also on Slack, and many people tended to have those quick side conversations online instead of in person, so there was a public searchable record and you could jump in more easily.
To be fair, this happens in dysfunctional organizations that are all on-site, too. There's no good excuse for being that disorganized, but it happens all too often.
I think the problem has more to do with their recruiting. To be a successful distributed company, you need a disciplined and highly experienced workforce (note that this does not mean a highly educated workforce, which may actually be a contraindication).
GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living (both of which may modify the base downward; my CoL adjustment is -40%, and I live in the United States!). They explicitly state on their hiring page that they prefer people from cheaper areas. [0] [1]
On top of that, just from their headcount, it sounds like they have way too many people. 160 for a technical company like this is hard to manage when the employees are local; it's even harder to manage when employees are remote. Again, for remote work, you need mature, highly-skilled staff with a strong history of successful self-direction. You're not going to get that when you're chopping their salaries in half based on their location. Bad local wages is a major reason to work remotely in the first place.
[1] https://about.gitlab.com/handbook/people-operations/global-c... ; gems include "you are required to notify us when you move and we may adjust your compensation up or down" and "[a]ll things being equal we will hire people in lower cost markets vs. higher cost markets."
Those salary modifiers are atrocious. Their range in my city for a senior, very experienced engineer is comparable to what most entry level jobs actually offer around here.
I 100% think you are on the right track. The only people from my area that would take a GitLab salary just can't get a job elsewhere.
It makes more sense for me to get a PO box in Washington DC and drive there to get my paycheck. Hell, it's probably cheaper to "rent" a single room out of someone's house just to have them forward the mail.
Agreed, the Washington DC and the Fairfax, VA modifier should be the same. Otherwise your not going to be competitive in the Washington DC area as a whole.
Agreed. Based on their calculator, moving from Dallas to Ft. Worth is a 20% pay cut. That's ridiculous; Dallas and Ft. Worth are essentially a single market. If any difference at all is justified it should be pretty small.
As it is, for me the calculator gives me roughly what I was making in Dallas, a bit less (at the top end) than I make now in New York, and a Lockheed entry-level salary if I were to move to Ft. Worth.
> Agreed. Based on their calculator, moving from Dallas to Ft. Worth is a 20% pay cut. That's ridiculous; Dallas and Ft. Worth are essentially a single market. If any difference at all is justified it should be pretty small.
That's... aggravating. You'd think their system would treat all locations in the same Core Based Statistical Area as the same. Better yet, treat all locations in the same Combined Statistical Area as the same; in fact, the definition of a CSA is a set of adjacent CBSAs with interconnected commuting patterns.
And the data is all freely available in machine-readable form from the Census Bureau's website [0], so there's no excuse to not use it.
In fact, just this past week, as a personal project, I've written a bunch of code to parse a whole ton of census data, combining CBSA/NECTA data, data from Summary File 1 (i.e. population, sorted by a number of demographics), and Gazetteer data (for land area, so I can calculate population density). Over the weekend, I'm going to take a stab at including data from the American Community Survey too.
amyjess, thanks for pointing out the CBSA; I'll admit I had not seen it before. When we set out to develop the calculator we wanted it to work globally, and focused on that. This will be a helpful way to improve it specifically in the US, so I'm adding it to https://gitlab.com/gitlab-com/organization/issues/24
Yeah, I put in some pretty conservative estimates for my skills/experience/etc, and got out a salary estimate that is not good for my area (London, UK). I tried the city I went to university in and there is no way they would be competitive there with other companies.
Would love to get more input from you to make sure the calculator reaches its goal of providing fair market compensation in your location. Can you please send me an email to let me know your city and whatever further data you are willing to share? ernst@gitlab.com
My local city, but the data isn't so much mine as data that the US government shares.
Sacramento, CA.
Mean Salary (per latest BLS data) [0] for Software Developer, Applications: $107,540; for Web Developer: $78,050
Sacramento, CA Gitlab locality pay index: 0.39+0.25=0.51
Vs.
New York City, same BLS figures [1]: $108,770/$81,430
NYC Gitlab locality pay index: 1.00+0.25=1.25
Sac/NYC Salary Ratio (BLS): 0.989/0.958
Sac/NYC Salary Ratio (Gitlab): 0.512
Fundamentally, the rent index based methodology you are using is, I would guess, giving you below market pay almost everywhere that isn't NYC or San Francisco, assuming that you've hit the market wage for the skills you are targeting in your NYC baseline.
I'm not sure what it is about small Midwestern cities but the calculator seems to dislike them. The calculated rates for Cincinnati are roughly half of market rate here. I understand the calculator will not be perfect in every city worldwide but it might be better to have no calculator as the current one is extremely discouraging even though I like GitLab.
It's based on rent. Each city is punished in proportion to how much cheaper its rent is than the rent in NYC. As NYC is one of the top 5 most expensive cities worldwide for real estate [0], any city marginally more reasonable is going to get pummeled.
Perhaps it's that there is lots of cheap housing in the suburbs here, and in the greater metro area but that is outside the city proper. But most of the tech / startup activity ie concentrated in one small gentrifying neighborhood where rent is much higher (+50–100%), so coincidentally people in tech aren't buying those houses because the commute is so far from the core.
I wonder if any of these indexes breaks down a city by zip code. A good analogy for here would be if you averaged all rent over the neighborhoods in Brooklyn then used that number for a bunch of candidates in Williamsburg.
It really penalizes living in a third- / fourth-tier startup hub, for example, vs either a first- / second-tier hub or a non-hub.
Taken from another perspective maybe it's an intentional filter on the candidate pool to specific cities or countries. That wouldn't seem to be consistent with most remote philosophies but it's something to consider.
Interestingly, Buffer's salary calculator is almost the opposite with a relatively small percentage difference between Nashville or Austin vs SF or NYC for instance.
Correction: even though the ratio calculated from it and the calculation for it are correct, I somehow mistyped the Sacramento locality index, which is 0.64, not 0.51.
Didn't notice it until after the edit window was closed.
I sent you a more detailed email, but basically it is preposterous to assume a "very experienced" senior engineer in ANY part of the United States is going to be happy with making $65K.
Exactly this. A very experienced engineer who is capable of working remote will likely be pretty intelligent. They'll realize a company is attempting to steal the surplus they created by living a low-cost lifestyle. I don't really understand why a company feels they have any claim to this.
Remote employees who are good at being remote have some very good options at this point of their career. Why the heck would they take less money simply because they had the foresight to move to a low cost of living location? Half the (stellar) people seeking remote work have built their entire lives around this fact - and have made very conscious decisions regarding their career and lifestyle.
Obviously it's working out for Gitlab, but I can't imagine any of my senior remote talent finding this acceptable in any way. I guess I could see it working for a couple years "converting" an engineer who is super excited to move into remote work vs. on-site. But beyond that, this policy seems extremely dangerous to me.
This is precisely what turned me off applying for a job at Gitlab after initially being hugely enthusiastic. My family and I moved away from London to the city we grew up in to save some money, partly because I knew I could work remote for the same sort of money I had been making while paying considerably less on rent and living expenses. The cost of living adjustments Gitlab do would mean entirely negating that benefit.
I'm not really sure what the justification for CoL adjustments is, beyond "we can get away with it". Apparently I'm worth paying $120,000 working from home in London, but if I move 70 miles south I'm worth half that. Not only does it make little sense logically, there's no way cost of living here is less than half that in London, so its not even calculated correctly.
Just ran my numbers and got about 55% of my current salary assuming a lead position (what I have now, which I would probably not get with them) and "a lot of experience" which I'm on the fence about and they would probably disagree with.
Getting rid of the COL deduction by selecting NYC as my location actually makes it reasonable so something tells me not pinning the US salary floor to 1.0 is saving them money and simultaneously getting them lower quality developers.
By offering people medicore, local salaries, they are losing a massive opportunity to clean up on the great talent in remote areas.
Why would someone want to live in a low-end city (assuming they are mobile) - when they could live the same 'quality of life' in a much cooler place?
Assuming a degree of mobility - the whole point of living in a 2cnd tier place is that you can save a lot of money, which makes up for the '2cnd tierendess' of the place.
Montreal is 1/2 Boston. There are a good batch of great devs in Montreal. I could double my salary and move to Boston? I'd probably just do that. And still save more.
If they paid a 20% premium over other Montrealers, they'd be able to attract the best talent - and still save a lot of money over Boston/SF/NYC.
Paying 'regular market wages' for remote localities is not a winning proposition - it doesn't take advantage of the fact they are great startup, reasonably well financed etc..
Even talking about "Boston" as a single thing is tough. There's Boston/Cambridge urban folks and there are people who live an hour west or an hour north. That's an enormous difference in terms of housing prices. And even that is an oversimplification. School districts and other factors make a huge difference within that same one hour radius.
But what neighbourhood you live in Boston is a choice.
Picking up and leaving a city is not a choice for most people.
Also the 'cost of everything except housing' is pretty consistent - everyone pays roughly the same a gallon of gas, and an identical basket of groceries.
Yeah, same here. If I max out the possible compensation by setting lead w/ a lot of experience, it's roughly 60% of what I make now (as a non-lead with a lot of experience).
The kicker is that I'm employed remotely by by a company based in a city whose CoL hit is even larger than the one I currently live in. The NYC range they give is plausible for me as someone who lives in a place that is much cheaper than NYC, but it's not exciting and wouldn't really motivate me to apply.
GitLab's numbers may look right on paper (and they do explain their methodology in the second link, discussing rent indexes, etc), but they're absolutely not comparable to what good people earn in the real world. Good talent commands good wages everywhere. A skilled remote worker is not going to come to GitLab for an average-for-their-area salary.
Even when working locally, talented people make much more than the average for their area. I know this because I've hired many good people and unless they're entry-level (meaning they're good but haven't had a chance to prove it yet), they don't start coming in the door until you're offering at least a 50% price premium over the reported median. In fact, we had a lot of bad candidates come in asking for around that much. The senior people I hired would make more than double the median for their nominal role. The stats on these government reports don't account for seniority, skill, niche demand, etc., and likely include some people that self-report as holding a title when they're really just aspiring to that position.
Once you're a mid-level dev, you should be pulling in a bare minimum of 80-85k, no matter where you live. I say this not as a bubble-dwelling San Franciscan (who surely finds such numbers laughable even for entry-level), but someone who has lived in various parts of "flyover country" his entire life. When local opportunities for at least this compensation are not forthcoming, go contract or remote.
There's no reason for good people to leave money on the table, and GitLab doesn't appear to get that.
My wife is a compensation consultant and we have discussed this phenomenon a few times. She says that for some jobs "there is a national market" and comparisons to local salaries is therefore invalid.
Set it to "lead" and "a lot of experience" (I'd grade myself intermediate-to-senior and above-average experience IRL) and the top of that range is less than I make now, and I'm not big on self-promotion/networking and could probably make quite a bit more if I were. Plus their benefits are worse than what I get at my tiny company. Their comp calculation sucks for my area.
They're paying talentless Wordpress dev prices for their mid-tier devs, by local standards. This is in a mid-sized midwestern US city. LOLWUT? No one decent in my city would accept a job there.
Having that sort of calculator also encourages developers to embellish their competancy level as well, which is not going to help in finding quality developers. I'd much rather a company figure out where I am on their scale (each company I've worked at seems to have different ideas about what makes a junior/mid-level/senior/lead) than taking my word for it.
It's more likely set up based on an untested (and easily falsified) conjecture about the relationship between local residential rent prices and local market wages in the industry.
"In developing the compensation formula above, we looked at the compensation of our team members which had been set in the past (without the formula), and found out that there was a statistically significant correlation between compensation and the factors that are now in the formula. We purposefully chose to look for correlations with metrics that are probably causal and definitely relevant in people's lives (the rent!). This also has the advantage of letting us work with data that is readily available publicly, as opposed to trying to scour the web for market compensation rates for all roles in all locations. Perhaps surprisingly, there was a stronger correlation between compensation and rent index than with the more general cost of living index available through Numbeo (or the cost of living with rent index, for that matter); and so we moved ahead with the Rent Index."
The immediate problem with the method of developing your formula that is described is that it assumed that the pay of your staff before you had a systematic formula already accurately reflected prevailing market wages in the locality of every employee.
And, for US locations, BLS data is readily available publicly. While that may not have been a suitable source for a worldwide formula, it would have at least been a suitable reality check on the validity of the formula you came up with to see if it reasonably approximated the variations in market rates within the US.
You are right that we used compensation from before we had the systematic formula. However, we then repeated the exercise (of finding linear correlations with various COL indices) when we had a larger cohort but before the formula had been implemented systematically, and it still pointed to the use of the Rent Index instead of another index.
You're also right that we need the calculator to work worldwide, and I suspect that that may be part of the reason here that the calculator works well on that scale and then not always as well as it should on more local scales. We'll continue to explore how to get better on both scales; the challenge is always that we want to keep things as simple as possible also.
I've made two issues based on your comment here and the one where you shared specific data for Sacramento (thanks for that!):
BTW, I know I've been kind of hard (fair, I think, but not gentle) on you guys in this thread, and I want you to know that I really appreciate how well you've taken it and how responsive you've been.
I won't quote it because I don't have the source in front of me but there is a quote elsewhere on this thread about Gitlab requiring that you tell them when you move so they can pay you less.
A) What's to stop me from getting hired in a 0.21 COL area and moving to San Francisco or Washington, DC? Are you going to fire me or pay me $20k a year because I moved?
B) My labor is not worth less and my work is not lower quality because I move to SE Asia or Eastern Europe. This is idiotic.
I applied for a position at Gitlab a week ago, and didn't even get a response. Probably because I am too expensive, too old, and live in an area with strong labour laws and staff protection. It's a shame, because it would have been decent to at least get a "sorry, not interested" reply, but no response at all is just not cool. They write on their hiring page: Always leave feedback, this will help everyone to understand what happened and how you came to your decision.
@mdekkers glad we saw your post. I tried searching for you in our ATS, but could not find any application under that name. Can you email me directly on jobs@gitlab.com? I'd really like to see what happened to your application and why we didn't respond.
Hmm, in Boston they only pay 2/3rds of what Google does for somebody with my job title (not counting stock or bonuses). And as soon as you get outside of the "hot" markets, those multipliers are pretty bad. If I compare their Rochester, NY salaries with what I've actually earned working in rural college towns, well, I've gotten better paychecks in academia.
They seem like nice folks, but as an experienced remote worker (with a history of both remote employment and remote contracting), I wouldn't bother to apply based on what I currently know.
Wow, that is bizarre. Obviously you want to take advantage of wage differences in different markets, but why would you tell people ahead of time you're going to lowball them? Why not try to get the best you can get, and then pay whatever the actual market rate is for them?
I just checked the calculator for a major German city and I can say they won't be recruiting anybody away from the likes of Siemens. Rocket Internet maybe.
They have picked a strategy consisting in hiring large numbers of less experienced developers, rather than a few great ones.
This is very much impossible to reverse, so now, they have to make it work with the people they have.
> GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living
Their base is off of NYC, apparently, and they adjust based on location; from what I infer from that, if they are really middling for NYC, their modifiers are all wrong for the way market wages actually vary by location; in Sacramento, the adjusted range is below public sector pay for similar roles (salary alone, not considering benefits, etc.), much less private sector pay.
The thing is, their basic methodology is arguably somewhat sensible, but they could (at least in the US) use BLS numbers for job categories by location to get modifiers that actually reflect local markets and keep their pay consistent with respect to local markets. By choosing an index that evidently is a poor proxy for variations in market pay, they end up with pay that doesn't make sense.
I would agree with this -- Gitlab seems like a company with a great philosophy and I'm sure a lot of people would love to help them achieve their goals... but I just used the calculator for my own area and my pay would be around half what I'm getting currently. I have to assume that makes it really hard to get serious talent, and makes it more of a pass-through for people to get their foot in the door.
FWIW, I have seen one example of hiring a remote junior person and it worked really well, BUT there was a conscious effort to do mentoring properly, and the guy running it was a mentoring guru/nut.
I've worked mostly from home for a long time now through a few different jobs. (Always relatively nearby an office where I had a space but I haven't been in a job where I went in regularly for 15 years or so.) I'm mostly happy with the tradeoffs and would be fine with going 100% remote.
That said, acknowledging that a lot of things are different from when I was starting out, I can't imagine walking into my first tech industry product management job as a remote worker.
There was some previous discussion about this here: https://news.ycombinator.com/item?id=13302906 . I too had to turn down their offer of $2000 USD per month for a Senior position after the second round. This issue keeps coming up in posts about Gitlab and there doesn't seem to be any progress.
I love the Gitlab product and what they have done to Git hosting. But, their salary structure is really screwed up.
It's definitely not our intent to pay below market in any location. If our calculator shows a large negative adjustment from your current compensation, please send me an email (ernst@gitlab.com) so I can review the data. We don't promise to make overnight changes to the calculator, since changes need to be robust and simple (i.e. no cherry picking city by city numbers), but we do gather data and make edits as we learn.
The philosophy is flawed. Good people don't make average-for-their-area salaries.
Depending on which of the 5-6 applicable job titles listed by the Bureau of Labor Statistics's Occupational Employment Statistics [0] one chooses to apply, my salary is between 40% and 120% higher than the "average mean wage" (itself an inflated statistic) in my area.
This is true not just of myself but of all the good candidates I've hired. Good people don't work for average rates, especially not good people who are capable of doing remote work.
GitLab is doing itself a big disservice by targeting the "average" local wage.
Remote employees' market is the entire world though, that's the part you're not taking into account. Your competition is the other NYC or Boston or SFO or wherever companies that are willing to hire me and pay me a generally competitive rate (very roughly 80-100% of SF pay, in my experience). Sure, you don't have to compete with a GoogFaceAzon offer, but you do have to compete with the companies similar to yours that do pay a more globally competitive rate.
There's a very important difference between: "SFO is very expensive so we have to pay people who live there more" and "we can pay people who don't live in SFO less"
No one makes a salary calculator that robust without intent. I think paying twice as much, and cutting your work force in half would do you a greater service, than playing around with these cost of living calculators.
Every company goes through rough patches, specially one that is growing as fast as they are doing. Very few companies are as open as transparent as they are, even in the fuckups. That is really hard to do and kudos to them
Thanks. I do agree that the timing of this post is not good. The video was recorded a few weeks ago. We are eating humble pie right now and work on fixing things instead of promotion.
Some of the reactions are understandable, but I expect most people on HN to be able to separate unrelated issues like these. Some of the comments here are absolutely unnecessarily hostile.
Not that this excuses anything, but maybe it'll help explain it: This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Now that we're aware, we are working on correcting these things.
But you're right, the timing of this piece was probably not ideal.
When I say process, I mean some of our operational processes like making sure backups work. There was an incident with data loss last week: https://news.ycombinator.com/item?id=13537052
Like any company, we have our own business operations issues too but it's still one of the best jobs I've ever had. We're pretty open about our flaws, to the point where the CEO even has a page that lists his out: https://about.gitlab.com/handbook/people-operations/ceo-pref...
Companies are run by people and consequently they're fallible. Every company makes mistakes occasionally. Gitlab handled theirs incredibly well. A more opaque company would have kept the problem, and the experience they earned handling it, to themselves. Gitlab's openness means we can all learn a little from their event. I think that's great. I wish more companies shared more when these things happen.
> A more opaque company would have kept the problem, and the experience they earned handling it, to themselves.
I liked their transparency a lot too! But it's neither possible nor acceptable to keep six hours of data loss and eighteen hours of continuous downtime "to yourself".
That was not our intention. This piece was recorded before the outage. We should asked to delay publication until after the postmortem was published. That would be the marketing 101 lesson, first show you corrected your mistake instead of talking about something else.
Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)
> didn't do anything particularly more wrong than anyone else?
This is almost completely on their admin staff, maybe other people aren't willing to say it, but I will. Test your backups. Or at least make sure they're non-zero in size. It should really be Operations 101.
Whether you do this automatically or manually by setting a reminder on your calendar once a week or even month, doesn't matter. Something this simple would have solved their entire issue. I do this and we run a much smaller shop than GitLab. Heck, if we were larger I'd have hot spare database servers in another datacenter in case the primary got nuked by disk failure, network outages or mistakes.
As part of our disaster recovery plan when I was working as a sysadmin at a 150p company we had replicated server (database replicated and webfiles rsync) on hot standby, we just switched the front-facing servers manually.
GitLab has a very short blurb on a similar styled HA setup, I'm not sure if and how they have implemented such themselves and if it would have helped in preventing or shortening the recent downtime.
They have probably documented their own setup somewhere.
"Automated failover can be achieved with pacemaker alongside STONITH network management. Keep in mind that application servers need to be prepared for transitioning to the new network addresses.
In this situation you can also opt to synchronize the database via a database specific protocol instead of DRBD. In the documentation for each database you can find out more about the options for MySQL and the options for PostgreSQL."
https://about.gitlab.com/high-availability/#filesystem-stora...
We had a secondary server. The secondary we were using was basically dead.
That's what lead to the problem. We were trying to fix it, but ran the wrong procedure on the primary instead of the secondary
Thus wiping the primary, instead of what might have been left on the secondary.
The secondary was already removed at that point.
Basically the procedure was the following:
1. Secondary falls behind too much, stops replicating. At this point you need to manually re-sync the whole thing
2. A re-sync requires an empty PostgreSQL data directory, so this data was removed
3. Re-sync doesn't work, leading to the other problems
4. At some point team-member-1 thought previous re-sync attempts left data behind, so team-member-1 wiped the data directory again to be sure; except team-member-1 ran this on db1.cluster.gitlab.com (the primary) instead of db2.cluster.gitlab.com (the secondary)
Synchronizing the database isn't quite what you want. It's true that in the case of an errant rm -rf it would almost certainly have helped, but it's approximately as easy to run a "DELETE FROM importantdata" and leave off the "WHERE" clause, which would get replicated. And certainly if you're using DRBD (replicate the volume, not the database), an rm -rf will get replicated.
I'm just genuinely unsure what a better outcome would have been here. (It's certainly a process failure that no backups existed other than the manual 6-hour-old snapshot, but I'm not sure you can do much better than automating that.)
You can easily add snapshotting using on-disk technologies like LVM or ZFS to that and achieve reliability against such an issue though, as well as being able to do a full text backup (ie: to SQL) from your replicated server at higher performance than you would on production.
They had LVM snapshots every 24 hours. They lost 6 hours of data because someone had coincidentally triggered a snapshot 6 hours prior to the deletion event. Otherwise, they would've lost several more hours of data.
The "trigger" of the failure (accidentally deleting the wrong database) is bad luck that happens, but the lack of preparedness for it isn't. If I understood the document right even if the regular backups would have worked, they only happened every 24 hours (same as the staging replicas they recovered from, except one was manually created out of schedule that day).
Maybe being fully remote helps to let stuff like this "slip through the cracks". (Not great phrasing, but I don't think anyone made a conscious decision "loosing X hours of data is ok", and nobody questioned what goal the current practices could (not) achieve.) I don't know, but it seems at least a question one might ask.
What's the best-practice approach for taking backups that are significantly more frequent than every 24 hours, but also robust to things like an rm -rf or a DELETE FROM table;? Something like continuous data protection seems like it would be far too much data for an active database server, no?
(Or are we just saying that they should have been taking backups every 15 minutes or hour or so?)
With modern database servers, people will take checkpoints at some regular interval and automatically stream append-only copies of all intermediate transaction logs to a storage server some place (the same transaction logs that the database servers use for replication).
In the event of accidental deletion, the most recent snapshot will be restored and the transactions up to a certain timestamp will be replayed on top of it. If done correctly, this can prevent all but a few minutes of data loss.
AWS Aurora (and possibly other database-as-a-service providers) has this functionality built-in.
I'm not really qualified to confidently talk about best practices, so these points have question marks attached, since I can't judge what's possible in their setup and what's not. I guess I'm not sure if they did judge it, see last paragraph.
More frequent backups would of course help, if their impact is tolerable. They also need to be tested to actually exist and work.
At least LVM snapshots apparently are cheap enough that they can be done out-of-schedule just to get slightly newer data to staging, so they likely could have been done more often (but they probably weren't thought of as backups, which is why 24 hours seemed enough and nobody had a prepared plan to get back to production from them).
Similarly, Azure-side snapshots are mentioned as not enabled for the database hosts. Maybe that's not viable to do (they had performance issues with Azure, and I don't know how much overhead they case), maybe just something they forgot to set up.
Others have asked "why is there only one replica", which also seems like a good question (but my experience with database replication is close to non-existent, and maybe the higher load of more replicas would have caused other issues. Don't know.).
My point is more that I don't have the impression that the 24 hours are a figure that they arrived on by evaluating what "service level" they could achieve, but more a result of someone at some point setting up a backup with some interval. I don't think 24 hours is a figure where they can say "that's the best we currently can achieve", or even "that's what we planned for", but "that's what we have because that's what we have and nobody has paid closer attention". There is at the very least a cultural angle to it, which is why how they work could be relevant.
One job I worked in had a slave set up with 30 minutes if replication lag intentionally applied to reduce the impact of someone accidentally nuking a table.
Then as someone else mentioned you can stream your replication log to storage elsewhere giving you the ability to do point in time restores.
Use that live database replication along with LVM, Btrfs or ZFS snapshotting every hour and purging the old one. Take a full, proper backup every hour if you can or every day if you can't. If you can afford to, do a SQL dump at reduced priority and compress it instead of a binary copy as it's easier to check a text dump.
Anyone with more experience with large databases have anything to add or any concerns with this sort of scheme?
It's unfair to characterize the entire organization by one failure, but their data loss issue is not attributable to some obscure, convoluted technical issue or freak accident. The simple fact is that they didn't verify that their backup and recovery infrastructure was working correctly. Moreover, they didn't have any automated monitoring confirming that the backups even appeared to be working. Those are things that every moderately-experienced IT person knows needs to be in place and occurring routinely.
If you look up the doc, they have like 6 different places where they looked for backups and each point is like "We should have these backups, checking for them now... woops, it hasn't been working."
The only reason GitLab's data loss wasn't even more disastrous is that an employee just so happened to take a manual snapshot 6 hours before the incident occurred.
Now, I'm sure we've all been in situations like that. I don't think it necessarily reflects on the individual technical employees of GitLab. I think it says more about the organizational and management resources at the top. It was their job to ensure that the appropriate resources were committed to data integrity, and they failed to do so.
90% of us probably work at places with similar problems, but publishing "the secret to [managing technical employees]" fresh on the heels of such a spectacular organizational failure is probably unwise.
Are you kidding? How is not ensuring your backups have EVER worked, or have even been setup, considered "bad luck", or "anything particularly more wrong". We're not even talking testing backups, literally if they have even been setup to run at all. The only reason this wasn't a complete meltdown, was that the tech decided he'd snapshot before he did work six hour earlier.
I think you're confusing sympathy and solidarity towards their engineers (which I was very glad to see) with "wow, this could happen to anyone". Quoting from their post-mortem:
> So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.
It's not true that having zero tested working backup/restore strategies happens to everyone. It's a catastrophic failure in process. Let's not blame any specific engineers, but it seems totally appropriate to blame the CEO (who was the subject of the puff piece!) and decrease your trust in the company appropriately.
This user account is not active or something and his tone is a little offensive but I think he makes a point. Writing everything down does nothing if no one reads it.
writing things down is not the problem... failing to read the things they write down is the problem.
they probably don't have time to read, because they are too busy always writing. they can't find what they need to read in the sea of endless wasteful and pointless drivel that never should have been written down in the first place.
you're all idiots.
Scale is freaking hard. It is absurd to say they figured out the secret after a 7 layer failure on disaster recovery, even more so when one of the main reasons for source control is to have disaster recovery and the ability to rebuild old source.
How is their remote working practices at all related to the database disaster?
For that matter, how are either of those related to source control and rebuilding from old source? What would rebuilding source help with when you just deleted your production DB?
Not talking about this case specifically, but it could be that remote working makes defining reponsibilities harder and that could lead to not making sure that there are proper backup/restoring procedures in place.
I don't understand why hearing some effective management practices for a distributed team would not be welcome here at any time. Their recent mistake dealt with a specific technical process and some failures around it. That incident in no way invalidates the entire company.
As a remote worker myself, I'm always keen to hear about how teams make it work. I'm also a huge advocate for distributed teams (and nobody having a commute), so personally I'm really glad this link was shared today.
We're still committed to transparency even if it means taking a few lumps now and then, it's better in the long run and it's baked into everything we do.
Kudos to the entire team, as another commenter said, please don't be discouraged by the petty comments. It takes balls to admit mistakes in public. You have earned many people's respect.
I think a lot has to do with the reason for the failure being perceived as sloppy and irresponsible. Without the transparency we wouldn't be able to bash them like this. Many of the sarcastic comments here refer to the failed processes they revealed in the postmortem.
Thanks, but the critics are right. This is not a good time to be promoting ourselves. We should be working on our postmortem and figure out how to improve our service. We're actually doing that, this video was recorded before the outage. But impressions count.
I think we earned a reality check. Have such a promotional video go out after the outage was out of touch on our part. We were aware it would be posted and didn't ask to hold it.
While I was at Mozilla, John O'Duinn gave a pretty great presentation about this sort of stuff.
The core message (mixed with my own takeaway) was that you have to consider all of your offices, even HQ, more like just another field office or coworking site--no location is "central" compared to others. As long as you consider accommodating remote people to be a separate task, it's a task that can be deprioritized. Ideally, anyone in your office should be able to work remotely at moment's notice with little-to-no change in procedure.
Much of his presentation was outlining concrete techniques towards making this actually doable. He still has the slide deck online.
In practice, of course, real life was imperfect and there's no question that you take a productivity hit over people in the same room. But if you do want a heavily location-agnostic organization as a core value, his take is a nice start.
Glad that Gitlab is pushing the remote working boundaries even further. Yes, document everything, write easy to understand code, offer your remote coworkers to call you whenever they need help, leave feedback, praise work as if they're local. It all matters and even more when remotely.
I think the worst team I've ever worked on was the one that had a team lead that literally never wrote anything down. Everything was verbal, and what wasn't verbal was private messaged. We would have the same discussion three times because nobody wrote it down the first or second time. Nobody even knew what the changes to the formal design were, because they never had a documented change review.
I always tell people on my team that if it isn't written down, it doesn't exist. When you get into the habit (and learn how to manage all that information) it really saves your bacon.
The other thing that would have saved them from the backup drama was learning how to document and date your to-do list. Suddenly someone notices that you've had a test process for backups on the to-do for over a year and they bring it up at the next meeting.
And another thing: in team meetings, people can write their own meeting notes so everyone becomes responsible for documenting their responsibilities and what affects them. It's easier to do this remote than at a stand-up, because you're already at your keyboard.
People seem to have some subtle insecurities around broadcasting their activities which contributes a lot to the problem of distributed teams. Ideally all communication goes to everyone and everyone can apply their own client-side filters for what they want to care about. The problem is people want to present a certain face up the org and a different one to their peers so they PM each other and create isolated channels of communication (distribution lists, slack channels, etc). Now have some cognitive overhead of whom to loop in when which creates some other complications because it means if you're receiving a message it's explicit rather than implicit. TLDR - broadcast everything to everyone by default and create a culture of transparency.
Last I checked a year ago, GitLab had just 80 employees. So, it's now doubled. That is quite some growth. More often than not growing companies just collect lots of employees and then a startup comes and beats them...
I can't wait for a remote-work company that has a policy like this to get into some kind of legal troubles that warrents subpoenas. This is a prosecutors gold mine. I feel bad for the defense lawers already.
I have to watch a video that then tells me to write everything down?
Sorry I couldn't resist.
I believe and practice that as much as I can because it makes things easier regardless.
I think hindsight will reveal that the things that make a "distributed company" successful are really the same things that make a "localized company" successful. I think it's just that having all employees on-site probably makes it easier for companies to "fake it" and stumble into success through sheer grit and determination. Probably a lot of times even when they have less-than-adequate (but highly motivated) human resources.
When everyone is on site, there's enough implicit communication going on that you can get away with not documenting things in writing for a lot longer than in a remote team.
This is interesting, as I interned at Mozilla, and they have a large number of remote employees. There was only one person (other than me) on my team in my location. Mozilla also records a lot of things, but the intent is more to publish that stuff publicly, in the spirit of openness towards the community. For example, all my team meetings had public notes. I wouldn't be surprised if this tendency to record everything for the sake of the community also has positive benefits inside Mozilla.
More than knowing how they manage remote employees, at this point, with more than a week since the data loss incident, I would be genuinely curious to know if their backups are now functional and what plan they have put in place or planning to, to verify backups work.
Furthermore, I won't be surprised if they have seen more concentrated spam attack, after the news of this data loss surfaced.
160 employees? That seems like a lot support a product that is a wrapper around another product. I know the lede is skeptical, but does anyone have any idea how those numbers break down? i.e. is there a large number of sales/support due to complex installs?
Sometimes software and moderators do the same thing, which doubles the effect. When we notice that, we turn off the moderation and just rely on the software. That's why this one fell and then went back up.
Everybody is prone to making a mistake! As long as you learn something from it, so that it never happens again, it's ok. And I really don't get why people give them bad rep for this post. I mean it's obvious why it's posted and we all know what happened week ago, but consider their size and scale, it's not easy to manage so many people remotely.
"...but consider their size and scale, it's not easy to manage so many people remotely." This is very true but the title of the post is "GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down".
That is the backlash, don't say you've figured out the secret then make a near catastrophic mistake.
I mean, really: after figuring out after the fact that five out of five backups were malfunctioning - go here to read our secret to managing 160 employees remotely?
Nah... no thank you. I think I will get my management advice from a company that is not totally broken. I don't care how transparent you are, or how remote-worker friendly you are, if you can't do the basics right. You are in the data storage business! Shut down the company and refund the money to the investors.
So after losing a ton of user data and repositories about a week ago, because a remote engineer couldn't bother to check to which remote machine he was issuing 'rm -rf' commands as superuser, now GitLab is teaching us its 'secret' to success?
The only reason they are still in existence is due to a chance backup they took for a tangential reason. From the sounds of it, their solution is held together with bubble gum, some tape and lots of hand waving. Being in 160 different locations probably doesn't help much either.
I'm pretty sure most solutions on the internet consist of bubble gum, some tape and lots of handwaving. Gitlab's screwup is hardly unique, even if it was very public.
It's difficult and expensive to build and maintain a solid system, and even if you want to, time and financial pressures often just don't let you, on top of the issue of just communicating the need for solid engineering, as it usually only becomes apparent when the problems start occurring.
Unless I'm misreading things, the reason they only lost 6 hours of data instead of 24 hours was a chance backup, but there was never an existential crisis here.
Downtime happens to pretty much every service out there. In this case the company was incredibly forthright and so we can make fun of their stupid mistakes, but really most mistakes are stupid when you look at them -- when you make thousands of decisions a day, some of them will seem silly in hindsight. We just never learn about most of them.
I had a customer once in the 90s take a 30 hour outage that cost them nearly $6M in fines because some asshole put a budget freeze on anything related to cleaning, including tape drive cleaner carts. The dopey ops guy kept using one tape on multiple drives, making them do nothing.
I could personally rattle off a dozen stories like this at late stage startups, Fortune 10 and .gov.
The only reason many businesses are alive is luck and reliable SAN.
"It's hard to build solid process, particularly in growth phases." Hard is an understatement, it is insanely difficult.
However, the interview seems to suggest that "write everything down" is the way to solve this problem and was what allowed for all their growth and success. So the solid process they have built is to write everything down, which they obviously didn't do or they wouldn't have had a 7 layer disaster recovery failure.
"Write everything down" is a terrible process, for two main reasons.
First, no one has the discipline to write much of anything down.
It's a very boring process and it's always going to be low-resolution. Your most meticulous documentation writers will get fired for failing to get their "real work" done. If you personally recognize the value of the documentation they furnish and thus refuse to fire them, all of their peers will feel that they are dead weight, which is still a ticket out of the company.
Second, after all that work, no has the discipline to read much of anything that you've written!
They skim. They glance. They Ctrl+F. They don't read. When you're in a pickle and you can pick out a life-saving bit of documentation, it's amazing, but that happens quite rarely and requires a lot of energy.
How many times have you pulled up docs, tried to follow them, gotten a really confusing error that you spent hours trying to troubleshoot only to find out that there's a one-sentence explanation tucked away in the third sentence of the fourth paragraph on the page you originally pulled up? This just happened to me _last week_, and frequently the most frustrating problems are small things like that.
People don't read. It's nothing personal, they just don't read. It takes a lot of cognitive energy. People are biologically programmed to conserve as much as energy as possible. Good programmers are both lazy and dumb!
If you want documentation that means something, it needs to be part of the process of actually working. I don't mean you need to add "write docs" to your checklist, I mean meeting the operational standards should be the only way things can get done in the first place.
The operating procedure needs to be married to the actual completion of the task, and that means setting good baseline project standards and setting reliable enforcement on those standards.
Code should be self-documenting to a reasonable extent. Tests should be mandatory. Peer reviews and signoffs should be mandatory. Internal company discussions should be recorded and referenceable. Documentation only works when it's self-generating.
In short, it should be run like a mature open-source project with an open IRC channel, mailing list, bug tracker, commit history, mandatory tests, maintainer signoffs, merge processes, code and style standards, docstrings and good automated documentation generators, and so on.
> First, no one has the discipline to write much of anything down.
This is why it's baked into the company culture. It's very common for someone to ask where something is in the handbook or if an issue has been created for something.
> It's a very boring process and it's always going to be low-resolution. Your most meticulous documentation writers will get fired for failing to get their "real work" done. If you personally recognize the value of the documentation they furnish and thus refuse to fire them, all of their peers will feel that they are dead weight, which is still a ticket out of the company.
Fwiw, everyone is responsible for maintaining the handbook/our process and procedure documentation. The docs team isn't on the hook for it, nor is it the sole responsibility of engineering.
> Second, after all that work, no has the discipline to read much of anything that you've written!
After enough reminders, you'd be amazed at how quickly people learn to RTFM at work.
> They skim. They glance. They Ctrl+F. They don't read. When you're in a pickle and you can pick out a life-saving bit of documentation, it's amazing, but that happens quite rarely and requires a lot of energy.
It's true that people skim the handbook (the guide with all of the "Here's how you get access to Twitter accounts" stuff), but runbooks are actually looked at when stuff hits the fan. I think it's important to differentiate these two things since they have different purposes. Imo runbooks should be as lean as possible for that very reason.
> Good programmers are both lazy and dumb!
This level of documentation is helpful for non-developers who make up a significant part of many organizations. We're not just talking about documenting code here.
> If you want documentation that means something, it needs to be part of the process of actually working.
> Fwiw, everyone is responsible for maintaining the handbook/our process and procedure documentation. The docs team isn't on the hook for it, nor is it the sole responsibility of engineering.
I find that in practice, with non-mission-critical activities, especially documentation, when everyone is responsible for getting it done (And making it useful), no-one is.
These are human problems. Doctors used to bitch about shit like this, because they felt it was beneath them to follow checklists. As a result people have had limbs removed and life threatening, unnecessary surgery. Guess what? The insurance companies demand checklists and controls to prevent fuckups.
There is an easy solution to dealing with people who refuse to read things. (Hint: It doesn't involve recording hours of meetings.) You need standard operating procedures to document what people do at an appropriate level of detail. Period.
Operational procedure is different than code documentation.
Yeah, I'm not disagreeing with checklists and SOPs. I'll edit my post to make sure that's clear as it's absolutely not what I'm trying to say at all. Quite the contrary: I'm trying to say that to the fullest extent possible, those SOPs should be integrated into the process so that skipping them isn't an option. Well-run open-source projects have given us good examples of how this works.
The goal is to marry the SOP and the actual completion of the task, to block off the shortcuts and require a minimum expenditure from external discipline/motivation reserves to get people to follow those SOPs.
"Write everything down" is the ante to function at all!
I agree... but I guess I read that coming from the perspective of someone who isn't a super fan of distributed teams. Many teams think that futzing around in slack or hangouts is enough. "Writing things down" is practically discovering the wheel to some places!
We've also noticed that over-communicating is critical but hard - it is surprising the things that are "yeah yeah, we know" to some but are "oh we're doing that?" to others. This is only natural - organizations become complex as they grow, and individuals are busy doing their thing. You often have to bring the important data to them.
On another note, working remote is awesome. I recommend everyone give it a spin once in their careers - but try to find a team that embraces it. I've heard mixed experiences from those who were the single remote person on a team.