I took a job some time ago (back around when the internet was morphing from uucp to ppp) as a consultant to an "independent agency of the United States government" which seems to enjoy placing big blue boxes all over the landscape.
It was my first real job outside of the university system, so it is safe to say that I was rather green. This job basically involved being an apprentice to someone who oversaw the code supplied by vendors that sorted a tremendous amount of non machine readable mail, i.e. it assisted humans using chord keyboards sorting mail with pneumatic arms and big chains and slots. Back then, a fair amount of mail traveled this way - stuff your grandmother wrote that couldn't be OCR'd, postcards, etc. The whole apparatus was similar in size to a semi trailer [1]. Each station dealt with a piece of mail every second (give or take), with two fifteen minute breaks and a half hour for lunch. This was around the time of the postal shootings [2].
Two weeks into the job, my boss quit. Just never showed up again. Not exactly sure what happened. Apparently, I had become the most knowledgeable person in the organization on this software which was currently being rolled out widely.
Unsurprisingly, there were some serious code quality issues. This was QNX real time sorta unix in C and ASM with RTOS daughter boards. The vendor's code jockeys, apparently, had become unavailable.
Main takeaways: Systems seem to keep working despite everything looking very fragile. Read tons of code. If it's working, don't poke it.
A year later I was able to make some form of industrial dance music by coordinating the openings and closings of the mail slots and the chain with the software. Bad situations can sometimes lead to greater clarity and new opportunities.
It seems like a common situation in the tech world for those with a little capability to deal with some legacy system that you somehow get lumped with or become the "expert". Maybe it's inevitable, people do get bored with doing the same things all the time.
I'd love to hear some of the music that came out of your experience - anywhere to listen to it?
I wish the TC article had done more than merely mention that keltranis is "joining Alexis". Hipmunk now has both reddit cofounders, their first hire in keysersosa, and their most senior programmer in keltranis. I wouldn't be surprised to see jedberg join them before too long, and then hipmunk would have almost all of the old reddit team on board.
Does that mean you'd consider joining them if they asked you to? Also, given the concerns expressed at [1], once you get a healthy dev team would you guys look into maintaining redditOSS a little more thoroughly? I imagine personnel shortage has made that impossible to date, but if you had a sizeable group is that something you'd do?
> once you get a healthy dev team would you guys look into maintaining redditOSS a little more thoroughly? I imagine personnel shortage has made that impossible to date, but if you had a sizeable group is that something you'd do?
Yes, very much. We would like to make the open source software MUCH easier to run, so that it is easier for others to develop on. It will probably be one of the first projects for a new hire, because it will make it easier for THEM to develop too.
Unless they're working on new features which are needed tomorrow, one dev seems fine in the interim. The number of sysadmins is more important to keeping a site running, but of course that's not what's in TC's headline.
Reddit's sysadmins neither have to maintain DC racking (nor travel), maintain hardware lifecycles, nor maintain vendor relationships. They certainly appear up to their teeth in working around AWS (EBS) deficiencies. They're not going to control a lot of their hardware, and most "fixing" of AWS pitfalls are going to come in the form of software.
Now if we are talking devops, I can agree. Engineering teams this small need to wear many hats; I think jedberg has been said to have helped in development efforts, if my memory of past blogs serves right.
Yes, I do dev too. I just haven't done it in a long time.
A lot of our sysadmin time is spent tracking down bugs in the stuff we use (like Cassandra) and doing tuning and automation.
Also, I do a lot of the other business stuff, so at best I'm a part time sysadmin right now.
We do pretty much do "devops" though. A lot of the work of Neil, our programmer, is tracking down bugs due to data inconsistencies due to hardware problems.
Both are important. A lot of work of a reddit programmer is finding bugs that come about because of the unreliable disks and also finding scaling inflection points and fixing them before they take the site down.
That's a bit unfair to jedberg, who actually runs the site with his operations team of two (one of them a new hire). This article and the comments are putting far too much weight on the developer, as if Reddit is a site that's constantly iterating new features. That's not to downplay his contribution, but:
* He was hired in November.
* He works on code, and didn't get woken up during
the last major downtime until the end (from their post
mortem, presumably due to the DB eating itself). The
word 'even' in "we *even* woke up spladug" is telling,
that indicates to me that he doesn't wake up often.
* Reddit is pretty much a done product, and the last
features I can think of were meta (Gold comes to mind)
He has a positive outlook[1], anyway, which the TechCrunch piece didn't seem to notice.
Doing operations for a site the size of Reddit with a staff of 2, on AWS, is far more impressive to me. Keeping an operation like Reddit afloat in front of that many uniques must be a pretty significant challenge. There's not a lot of depth for on-call, so I feel for their sleep.
On top of that, jedberg is the sole voice actively fighting for Reddit's reputation in public, since (let's be honest) the exodus looks a little suspect. Watching his comments[2] in the face of a perceivably sinking ship -- not to mention former employees who are saying quite a bit in front of an impressionable user base[3] -- is interesting, to say the least.
They're definitely undergoing a challenge, and I'm interested to see how they come out of it. (Aside: Today has apparently been discuss-Reddit day, looking at my threads.)
He doesn't run it himself. Reddit has two sysops and Amazon AWS staff. I won't pass judgement on Neil's performance or compensation -- we shouldn't speculate, but it'd be surprising to see a developer of only 4-5 months get a big raise.
Supply and demand is an important factor. If CN demands the services of a developer with experience in the reddit codebase at reddit and the worldwide supply of developers able and willing to supply that need is only Neil, it wouldn't be unreasonable business to give him a raise, short tenure being somewhat irrelevant if that's the situation.
This is the second TC article today I see where they claim news site like digg or reddit are in competition with twitter. I don't think they understand that it's two very different type of stuff that is complementary. Or they are paid to say twitter in every article.
Tech journalism has a hard on for Twitter and it's not so surprising why. Twitter is all about self promoters. I see people using reddit all the time...I have never seen Twitter in the wild.
Ironically, a month ago they were looking for devs and asking them to invest considerable time writing demo apps (front end engineers, for example, were tasked with writing an Javascript Reddit clone using HTML5 storage to keep track of votes, etc.) to prove their worth:
That's how you reach out to inexperienced devs. And frankly, if they're posting on Reddit, they know what they're getting into. They have to set up a pretty serious mediocrity filter.
If your resume identifies you as a badass programmer, you just toss them a copy of your resume. You know how it works, and they know that insisting that you do a programming challenge before setting up an interview hurts their chances of landing you, so they'll give you an in-interview test instead.
It's the same with Facebook and all the other places that have programming challenges -- they're there to qualify leads.
Outage, singular. You are talking about the 6 hour outage, not the 6 hour outages. Reddit is prone to go down more than some other high profile sites, but you are talking about a single occurrence of it going down due to a known issue that is explained in the article this thread links to.
No, it is not a coincidence. Please don't sensationalize a comment about an article that clearly explains what went wrong. Go to Reddit for that.
As someone who hangs around in the reddit dev and downtime IRC channels, several-hour outages do happen on a monthly basis, as well as loss of some features more e.g. commenting. Reading between the lines on each outage, it was due to building the site on shaky hardware foundations. It just wasn't explicitly mentioned until now.
If the filter were genuinely indicative of that, that would be great. However, my guess is that working there is quite different from doing makework puzzles for no pay (I hope, at least!).
I don't think you get a good idea of what a place is like until you sit down and talk to the people involved and see the code. At least that much is possible.
> If the filter were genuinely indicative of that, that would be great. However, my guess is that working there is quite different from doing makework puzzles for no pay (I hope, at least!).
The puzzles are in the links provided, why didn't you just go read them?
Because I have more interesting puzzles to work on: my own projects, many of which are available on the internet for potential clients or employers to look at should they wish to hire me.
* You have the time to post on HN strongly implying that these "puzzles" are likely not that great (they're not even puzzles to start with)
* You have the time to purposefully misread replies (I suggested that you go read them before judging them, not that you go work on them) and reply yourself
* But you don't have the time to click on a link and read 20 lines of text?
I actually had a look before, and they confirmed my impression that they're pretty much a waste of my time.
The point is not whether I clicked through or not, the point is whether I'm going to spend my free time on my own projects, or completely useless projects for some company - they're not even things the company needs!
I'm more than happy to be judged on the merits of what I've produced, both proprietary and open source, rather than fooling around with invented puzzles.
This is the wonderful thing about Homininae. I also have lots of my own work and at this very minute I have a ton of outstanding issues to close on the Faux framework alone.
Nevertheless, programming puzzles are like catnip to me, so I don't mind solving them as part of a job interview process (or at least, I didn't mind the last time I was actually looking).
I would NEVER say that I have a superior strategy to you, or that you have a superior strategy, or that programming puzzles are an excellent filter, or a terrible one. Just that some people will like them, and some not.
You don't, that much is obvious. Or at least, you don't in the context of a job application. Fair enough, and nobody should ever suggest that (a) This is a bad idea on your part, or (b) on the basis of your anecdotal evidence, this is a bad idea on Reddit's part.
That's exactly right. I did a similar quiz for my current job, and now I administer something similar when I look for new hires. It's amazing how many people with solid-looking resumes flub basic questions once you ask to see actual code they wrote.
I think its completely fine to have quizzes as part of the process but asking people to spend several hours on one before they´ve even talked to the company is a bit much IMO.
Ah, thanks. A while ago I was interviewed by the SpiderOak team, who asked me to make a FAQ system with commenting and CAPTCHAs in Django to prove I could write code. This would only take one or two hours, but I've already developed countless web apps in Django so I felt rather bad that they wanted me to pass some arbitrary test before even talking to me (as you say).
I thought it was just me being a primadonna, but I'm glad to find someone else thinking the same way. After all, my resume (which is full of all the things they ask) should be enough for at least an initial impression on their part.
They never did bother to reply after I sent them the code. I actually now see that they have deployed a new FAQ system with commenting and CAPTCHAs, although it was simple enough that I don't think they'd just use my code.
Reddit is definitely not a fun code base to go mucking around in. I wrote about this here: http://www.deserettechnology.com/journal/reddit-the-open-sou... . I think the amount of effort necessary to grok reddit's codebase would be pretty large, and I can't imagine many people doing that without getting paid or having a ton of extra time and a lot of love for reddit. The code needs a lot of work before it's ready for anything besides reddit.com.
My experience is that most people that try to get reddit running in the first place give up because it's such a gargantuan monstrosity. reddit will take any reasonably-sized personal VPS entirely over -- I was running on a 1GB slice and had to login every few days to restart Cassandra and paster (after the Oct. push). Reddit is just not well-suited for anything except reddit.com, and I think new developers would have some difficult maintaining that, and that most people aren't going to do it without getting paid. It's a big, ornery project, certainly not easy to jump into.
This is what happens when your primary demographic is people with no jobs and no money -- no ad revenue because nobody clicks the ads, and no possibility of selling subscriptions. Compare Reddit's target demographic with Bloomberg's target demographic. I bet Bloomberg has more than one developer.
(Incidentally, what a scam. Bloomberg terminals cost some insane monthly fee, and they have ads! And all I've ever seen anyone do with them is check sports scores and read celebrity gossip. Google should have put them out of business years ago.)
It's not a money problem. We've got plenty of money. We just weren't allowed to spend it on developers until recently. Now we have a bunch of open developer positions that we are in the middle of hiring for, and will probably have more soon.
I'm sorry, but if you have money and you're not allowed to spend it on what you do need the most, then actually you don't have money and you've definitely got a money problem.
In theory, if they can get by temporarily with just one engineer then the site was engineered well to begin with. There's some good management lessons from "The Mythical Man-Month" being applied here: http://en.wikipedia.org/wiki/The_Mythical_Man-Month
Conde really doesn't know what to do with reddit. They probably acquired it "knowing" they had to have something in "the social media space" (the acquisition predates the salesman's current buzzword of the hour, "the cloud") so they could compete in the "digital age" and all of that, but since they've acquired it my understanding is they've basically starved it and reddit has been running on fumes until they implemented Reddit Gold last summer. They're keeping it because they think they need something on this new-fangled internet stuff, but they don't what to do with it, and it's not generating much revenue, so they're just trying to leave things alone until they figure out something.
Note that this is just speculation based on what I've read, I don't have any inside knowledge.
Apparently, they have enough money but internal politics prevent them from hiring more people (it's a different budget...) Hence AWS, which isn't that cheap but does reduce personnel costs.
See the big reddit thread on this topic, or HN's thread about that reddit thread.
I know that at least one of the motivations for using AWS was that they didn't have to go through purchasing process of Conde Nast which was a huge roadblock.
I think that's actually one of AWS' biggest strengths. Once you get a corporate card in the account it's easy for the devs to buy what they need.
HN has been struggling to keep up with load recently. Remember the pagination of comments a few weeks back needed to keep the server from falling over? Yeah. I think news.arc/HN could really do with some extra scalability work; it still slows down significantly at peak times.
Rarely a good answer. All of that complexity wasn't added for fun. Making a comment like this really shows ignorance in software development. While we all want to scrap and rewrite some projects, most of us know that it isn't as easy as it looks and you will be spending your time on the wrong thing.
Also, I don't have any idea what to make of your comment "...and if it is, it needs to go.". Are you just trolling or do you really suggest that they shut the site down because it's complex?
It appears that Reddit's days are numbered. Sure they may be able to get by with one developer for a while, the product is fairly mature and content is driven by the user base. But how long can a website remain stagnant, with no innovation until the user base starts to migrate to the next best thing? Also, being owned by a large, old media company doesn't necessarily give one much confidence that Reddit is backed by visionaries who are willing to push the envelope and take risks.
And what is the next best thing in your opinion? It's one of the few places that does nested comments properly and the lightweight up/down voting always keeps it interesting. The only thing I can see replacing it is an outright clone, but no one's stepped up and done it in all the years Reddit has been around.
I was a daily user of Slashdot from 2001 to 2006 and then I found Reddit. I completely abandoned Slashdot less than a year later.
What could beat reddit? Well, if there was a facebook-y site which allowed me to associate with people who had my interests (rather than family, friends, and people I went to college/high school), I'd be down with that. (I love my mom, but I don't think she cares about Minecraft, old-school RPGs, French pop songs, Asa Akira, and Jules Verne novels.)
Twitter is close, but the character limit, url shorteners, and "twitter-speak" (intentional bad spelling/grammar in order to "get under the limit") just make my teeth itch.
It was my first real job outside of the university system, so it is safe to say that I was rather green. This job basically involved being an apprentice to someone who oversaw the code supplied by vendors that sorted a tremendous amount of non machine readable mail, i.e. it assisted humans using chord keyboards sorting mail with pneumatic arms and big chains and slots. Back then, a fair amount of mail traveled this way - stuff your grandmother wrote that couldn't be OCR'd, postcards, etc. The whole apparatus was similar in size to a semi trailer [1]. Each station dealt with a piece of mail every second (give or take), with two fifteen minute breaks and a half hour for lunch. This was around the time of the postal shootings [2].
Two weeks into the job, my boss quit. Just never showed up again. Not exactly sure what happened. Apparently, I had become the most knowledgeable person in the organization on this software which was currently being rolled out widely.
Unsurprisingly, there were some serious code quality issues. This was QNX real time sorta unix in C and ASM with RTOS daughter boards. The vendor's code jockeys, apparently, had become unavailable.
Main takeaways: Systems seem to keep working despite everything looking very fragile. Read tons of code. If it's working, don't poke it.
A year later I was able to make some form of industrial dance music by coordinating the openings and closings of the mail slots and the chain with the software. Bad situations can sometimes lead to greater clarity and new opportunities.
[1] http://www.usps.com/postalhistory/images/photogallery/equipm...
[2] https://secure.wikimedia.org/wikipedia/en/wiki/Going_postal