Ok, how can Rap Genius recover from this? They're one of the best (and least shady) lyrics sites on the net. They acted immorally, yes, but is justifiable to summarily execute their company?
Google has a domineering position in the market, and it's a bit ironic that people were quick to call for Microsoft to be split into multiple companies about a decade ago, yet nobody bats an eye when Google exercises their power. (I personally don't believe either company should be split; just pointing out an interesting parallel.)
So what can Rapgenius do? Are they screwed?
EDIT: Also, how long does the penalty last for? I think people are assuming 30 days, but is that really the case?
Even still, 30 days is a long time. They may not be okay afterwards. Whereas, say, a week-long ban would've had the same deterrent effect.
There is no execution going on. They knowingly engaged in blackhat SEO. Now they are paying the rather well-known penalty. They need to wait it out while that penalty is active (likely slimming down their burn rate) and focus on gaining customers through other avenues.
Or kill their brand and reboot as a different one with a different URL on a different IP.
Right. But the inbound links can still get to the new site. What I got from your comment was concern for other websites who linked to RapGenius, and the users of those websites...
Well, 301ing them is probably going to apply the penalty to the new brand, making the technique useless for getting back in search results. Not 301ing them means you're starting out with PR0 all over again, in which case you might as well just wait out the penalty period. In either situation, the rebrand on a new domain solution sucks.
Exactly. I know the scene here feels bad for RapGenius because they're putting themselves in their shoes and appreciating how devastating a penalty this is. But I don't feel bad for them whatsoever, and neither should anyone else.
This is a penalty that they deserve. They took a risk and they lost. They knew what they were doing and the potential destruction of their traffic, and they did it anyway.
You don't get less time in the penalty box just because you're the favorite or best player on the team. In fact, you have a greater responsibility to play by the rules and I would make the argument that your penalty should be even greater.
Besides, if the refs played favorites, the refs would have no credibility.
A friend of mine ran a long tail content generation service, much like HitTail, and recently had all his clients penalized regardless of the quality of the content they wrote for those long tail keywords.
Apparently they already anticipated something like this because several other companies were being manually penalized over the past year and half following an algorithm that seemed like: "Which big AdWords spenders cut down significantly on their budget? Are they getting it organically now? Let's cut that off."
Several of those companies are suing Google for violating EU regulations, so hopefully something will come out of that.
The first step will probably be that penalizations can be fought through the EU judicial system, but it will be a while before we get jurisprudence on that.
I don't know how this would work in the US legal system but, regardless of what the outcome would be in Rap Genius' case, it's probably time for similar jurisprudence to be established in the US.
It's just a matter of how do you quantify a crime against the almighty Google? I understand that Google have rules and regulations in place to protect themselves and consumers from these "online black hat thugs" but even the justice system have standard jail time for certain crimes. Now maybe there is a "standard jail sentence" in the Google justice system for voliating the rules, but what if there is not one? Do we allow Google to impose a death sentence for whatever they see fit?
I expect them to go back to right where they were once their penalty is lifted. Assuming most of their visits were google searchers clicking on the top link, nearly all of those visits/day will come back once they're the top link again. Remember that a huge majority of people have not heard about this controversy. The hacker news sort of community is not their primary audience.
It's possible they'll be bigger, what with all the free (though bad) press they've gotten.
I think the fact that they did lose as much of their uniques means that there is not a lot of brand-loyalty in the market. Then no people would form loyalty for competitors that are now over RG in the google results and they would be back once the penalty is over or RG works things out with Google.
PR0 would be far better than what they have now. I think they consider it a negative 0 or maybe a negative 1. They could buy an aged domain pretty cheaply. It all depends on what they want to change their brand name to.
The penalty is unknown. They could be looking up to a year in the sandbox. Is it really worth waiting out if it could take that long? Change the logo and get on with it...
Not only is this bad for RapGenius, it's bad for Google, and most importantly bad for Google's users. Yes, RapGenius screwed up, and they've paid the price. But they've publicly recognized what they did and apologized. Google really should reinstate their ranking, because people searching for lyrics right now are being given the huge disservice of using the ultra-spammy lyrics sites from days of yore.
>it's bad for Google, and most importantly bad for Google's users
BS.
What's good for Google is showing people that if you engage in bad practices your shit will get nuked from orbit, even if your site is better than the competitors.
You'd better believe that companies are going to avoid blackhat SEO in the future after this, and that's absolutely in Google's interests.
More specifically, companies are going to avoid entering markets (e.g. online lyrics databases) where Google has demonstrated an unwillingness to appropriately punish established players who use the same black hat practices, since it's impossible to compete in the organic rankings against them.
>an unwillingness to appropriately punish established players
Can you find another example of the other lyrics sites pushing a quid pro quo scheme on Twitter?
The other lyrics sites might be scummy and Google doesn't care -- or they might just cover their tracks enough to avoid giving Google enough proof to hang them.
"it's impossible to compete in the organic rankings against them."
Isn't that precisely what rapgenius had done (affiliate linkscam notwithstanding)? They were already the top result for many lyrics queries, they just got greedy
Yeah, it would only be actually "bad" for google if people were switching search engines en masse because they couldn't find the lyrics they wanted.
I think that anyone who characterizes google as needing one site over needing people to not blackhat will find that google really doesn't need said site. (Maybe Wikipedia is an exception to this rule?)
What's good for Google is doing what made them successful, providing the best search results. We now know one type of search they're no longer doing that for.
They haven't paid the price yet. RapGenius knowingly engaged in blackhat SEO and broke the public rules. The punishment is several months in SERP Siberia. RapGenius hasn't paid that yet, but they will.
> Google really should reinstate their ranking, because people searching for lyrics right now are being given the huge disservice of using the ultra-spammy lyrics sites from days of yore.
A short penalty of a few days or weeks isn't much of a deterrent. It may suck for people searching lyrics now, but making the page rank penalty effectively toothless would suck for everyone.
The hell-banning will probably only be temporary (30 to 60 days).
And RG didn't just "screw up" by trying to game the system; they were blatantly cocky about it, so that's probably why Google felt they needed to step in and teach them a thing or two. Let's hope the lesson sticks, at least.
Google should wait for RapGenius to a) clean up their backlink profile b) Submit a request for review just like everyone else.
I work at an online marketing firm and I've seen dozens of companies come to us for help with the process. RapGenius shouldn't get preferential treatment.
Luckily their revenue has remained unchanged... $0. This could have been a financial disaster if Rap Genius was a real business, but you need to have revenue for that.
Many people would kill to have a site with millions and millions of uniques. If monetization HAD to be the primary focus, they could make money. Growth and product is the top priority. Unfortunately for them one of those is suffering right now.
A site with millions of uniques doesn't = revenue. Nor does it equal success, interest from investors, etc. Source: running a ~2m unique site right now.
I have used your site quite often and was wondering why did you decide to make the downloaded program be wrapped with your executable? Why not just have a .rar/.zip of the actual program.
It shouldn't :) Back with Firefox 1.x I think, we were using ZIP files and always supported unzip over the old one. But we ran into a situation where if a file was there from the previous version, it would remain since the new version no longer had that file. Oddly, the new version would try to use the old file and crash. That was back when I was only doing Firefox and Thunderbird, I think. As we added in apps like OpenOffice and GIMP it became much more of an issue.
So, we have the PA.c Installer able to take directives like whether or not to preserve everything in App, whether to preserve or remove specific files, whether to preserver or remove specific directories, etc. In addition to overwriting the existing files with the new ones, of course. This made life a lot easier on everyone. And users didn't need to worry about manually deleting specific directories but keeping others when updating. You just point the new version to the same X:\PortableApps\AppNamePortable directory and it intelligently upgrades. It even supports custom code for things that fall outside those simple keep/preserve so we can specifically remove certain settings on an upgrade where a given app breaks its own settings and doesn't support the old ones, etc.
For example, I just wrote the package for FreeCommander XE Portable for the publisher to release. The new one doesn't support the old settings and will break if they remain, so I have the installer back them up within Data (just in case the user wants to revert manually later). I then had it read the preferred language from the old settings and write them into the new ones for the user for convenience. I could get more explicit and read/write other settings as well if I had the time.
I'm not trying to equate traffic to success or future income. But the original comment was snarky and implied that Rapgenius doesn't make money so there's no financial loss. This loss of traffic will hurt the future monetization of the site and its products.
What kind of revenue are we talking about? A few ads on each page can bring in a few grand or more a month. I'm not sure this is rapGenius's end-game though.
I think I'm spot on. Maybe even under-estimated. Feel free to prove me wrong. Here's a site with nearly 3 million impressions a month, making $2200. A year or two back when they were "hot", those ad spots were selling out each month: http://buysellads.com/buy/detail/125
To recap. Google's algorithm is terrible, and despite having more PhDs on board than my uni, can be played by anyone with two brain cells and a mildly popular Twitter account.
To rectify this, they crippled my search results by penalizing Rapgenius and instead pushing some AZlyrics junk that not only lacks annotations but is inaccurate.
Seems like an easy fix, for the end user at least, add a keyword search for 'lyrics' to go directly to Rapgenius, right? No. Here's a search for smoke on the water
Google is resting on their laurels. I'm convinced their actual algorithm hasn't changed much in the last few years, despite announcements claiming otherwise.
They changed the handling of search queries (searches are interpreted much less literally than a few years ago). They changed the way that pages are indexed, and changed the intervals to be smarter (faster for news sites, slower for sites that don't change). They now even "render" the pages partially to account for CSS and JS. They have smart stemming and synonym algorithms when building the indexes. But the actual algorithm - "PageRank" and the parts surrounding it - hasn't changed much I think, except for some parameter tuning.
Instead, they probably have a pile of semi-automatic "filters" to deal with spam and blackhat SEO. Engineers write a filter in code, test it against a test index, and then it goes into the new index version. I believe they are going more and more into manual intervention, and that they can now directly penalize/push sites without going down to the indexing level.
I can't prove it right now, but I'm pretty sure that the first SERP for popular search terms is actually manually curated, with the really organic links coming later. (I'm still trying to figure out a test methodology to analyse it :-).)
First off, who cares if the top result now gives you no annotations? No one cared about annotations until rapgenius started adding them so people could know what the more obscure portions of rap lyrics meant and then they gamified the shit out of it so that every single word in a song would be annotated so that you now know the meaning of "I set fire to homes, arsonist" is because it helpfully explains that an arsonist is someone who sets fires. And then put annotations everywhere even on their stupid blog posts, it's like they just discovered HyperCard.
They didn't really push AZLyrics, or any other lyrics site up. Those rankings haven't changed. Their position relative in the list you see based on your search terms has, but that's relative.
Why should Google give special dispensation to RapGenius? Why should putting in something like "lyrics" or related content to someone being penalized get them to show up higher in the results?
Why should Google be RapGenius' search engine? Why shouldn't RapGenius make their search better?
How is their ranking method broke? Most searches you make probably give you what you want within the top five results (excluding lyrics, as you've shown a weird need to have RapGenius be the source of your lyrics).
How can you prove RapGenius is the "best result"? Especially considering that courts have ruled that ranking algorithms are just opinions. You can be of the opinion that RapGenius is better, but that's your opinion. Multiple people can also be of the same opinion, which is a huge factor involved in Google's ranking algorithm. Another huge factor involved in their ranking algorithm is dissuading blackhat SEO techniques, which can make it appear like lots of people are of the opinion something is good.
So, again, while you are of the opinion RapGenius is the best lyrics site, I'm not, even before all this brouhaha started I didn't really like their site, especially the annotations, which to me seemed like a bunch of bored English majors who didn't really have any insights on the annotations they were making, but wanted you to think they were really smart. I also don't listen to rap music, which might also affect my view on the annotations.
What are you going on about? I don't think you have the slightest idea about how Google's ranking algorithm, or for that matter any ranking algorithm, actually works.
It's pretty scary how much Google controls the internet. I wish Bing or Yahoo or DuckDuckGo weren't so much worse than Google, or I'd happily use them.
Give me just 20-30 million dollars, legal protection from any patent claims, and (providing my search engine works well technically) make it the default in a small percentage of all Firefox installs.
What I'm trying to say is, Google doesn't have the monopoly because they're the only ones who can build a good search engine. The have the monopoly because they do have good technology, but also because of path dependence, and because there is no political effort to build up a competitor.
I already posted a couple of days ago, there are a bunch of ways governments could support competitors when one search engine controls a large fraction of the market, say more than 2/3. For example a special monopoly tax, subsidies for competitors, etc.
One thing I think would be nice is to make search engines offer a free/very cheap API (as long as they have > XX % market share). Anybody can come along and build a Google + Bing + Pokemon-Wiki meta search engine. If you tune it to be profitable, you'll have a lot of competing search engine frontends spring up. But you'll also have a lot of search engine backends (and not just Bing, Google, and Yandex), because its suddenly profitable to build verticals (e.g. indexing just tech news or recipies), which require less resources technically - because the vertical backends get traffic through the competing frontends.
And I could choose a frontend that gives me Google-like searches, or one that interprets terms more literal (like when you use +"term" in Google), which would be great for programming related stuff. There could be another one focussing on social media, or local stuff, etc..
It seems somewhat unreasonable to manually penalize a website like this. If you can't figure it out algorithmically then it can't be that bad. Major companies (much bigger than Rap Genius) buy links all the time. How do I know? They have asked me for them. I used to run a financial blog and I know a certain company that bought links from pretty much every personal finance blog that would sell one to them, and I know this for a fact because I was in a network of blogs and talked about this on forums as a way to make money. I also have the emails to prove it.
Why doesn't this company get penalized? I have no idea. I guess nobody decided to write a blog post about it, it didn't get picked up on Hacker News, and Matt Cutts hasn't noticed it yet. Seems like if this is possible there should be a reverse black hat market springing up soon. Do some really bad black hat SEO for your competitors and then get them killed by Google.
It wasn't so genius to be expository to random people via email, about a link-back scam. Sorry RapDingus, it's over; maybe next go around you'll have less hubris and daringness in using unethical practices.
Not many funded startups are dumb enough to publicly email folk asking for links in return for tweets/linkbacks. That's pure stupidity; might as well upload a video on a bad political stance with your brand overlayed, because the worms don't stay in the can with any kind of public communication. They should have been more clock-and-dagger with their scheming -- their hubris was their downfall.
Of course many spammy sites use this practice but they don't have the respect from the tech community that Rap Genius warrants as a Y-Combinator funded entity. They lost that, and it's going to be a steep climb for them to find talented folk that want to be part of a seedy operation.
People rob and steal, does that make it acceptable for you to do it? People are racist, does that make it acceptable for you to do it? People don't give up seats for pregnant and old women, is it acceptable if you don't? Not everything is black and white, but this is. What I'm getting at sscalia is that the small minded way people like you think is something that's wrong with this society. Fix up.
To everyone here who's saying they brought this upon themselves, when rules are unenforced and all your competition breaks them, you've already hamstrung yourself if you don't break them too.
If everyone in the MLB is juicing and a rookie gets caught, you don't say "what an idiot", you look at either 1) enforcing the rules on everyone and broadly changing the culture or 2) question whether the rules make sense in the first place.
It's not that fickle. Even if all their competitors are engaged in blackhat SEO and buying links from other blog sites (using cash, tweets, etc). RapGenius was the one who publicly boasted about it. Which caught the attention of Google. Which earned them the smackdown. That's not fickleness at work. That's... what's the opposite of genius?
Not sure. I'm too busy seeing all the gray lines that would be created by Google giving preferential treatment to those deemed more "important" when they could have stayed black and white.
My point is that punishing RapGenius despite the blatant use of the same tactics by hundreds of other big internet brands IS preferential treatment. The fact that RapGenius' competitors weren't punished is giving THEM preferential treatment.
What should Google do in this situation? I fully understand them murdering Rap Genius in their indexes, but it also hurts Google since Rap Genius actually has the best content.
The Bing result is better (though both are valid).
I would suspect that most sites pulling these sort of SEO scams are going to have lousy content, so the rule makes sense, but perhaps there should be some sort of human element to it as well? That's not scalable, though, so I'm not really sure what the best answer for Google is on this one.
Is the Bing result really "better"? Lots of people would rather not have the annotations...
Additionally, with AdBlock on, AZLyrics makes only 21 requests and takes 1.03 seconds to load completely on my machine. RapGenius (still with AdBlock on) takes 27 seconds to load, with 208 requests. Based solely on that I'd rather use AZLyrics than RapGenius.
Yeah, I suppose it's subjective. I personally think the RapGenius result is better, but you make a valid point about the performance and simplicity of the AZLyrics result.
> What should Google do in this situation? I fully understand them murdering Rap Genius in their indexes, but it also hurts Google since Rap Genius actually has the best content.
It helps them in the long term, as it dissuades sites from using blackhat SEO if they know their company can suffer severely for it
Hypothetical question: if we ("we" being nebulous) were to unveil that every other major lyrics site (azlyrics, metrolyrics, stlyrics, lyricsfreak, lyrics365, etc. etc.) uses the same SEO tactics, would they receive the same attention and punishment?
The punishment is about the same for all websites that engage in blackhat SEO. At least, once they clean up their acts. Until they clean up their acts, the punishment remains in effect. This can be difficult when you pay for nebulous backlinks from spam sites in a network, for instance, as played out with the Anglo Links takedown Google did last month: http://searchenginewatch.com/article/2319130/Did-Anglo-Rank-...
Mind you, this is not the exact behavior RapGenius engaged in. They paid for backlinks, but they paid legitimate sites directly for links in blog posts. Either way, though, it's still blackhat.
It's a really interesting world if you feel like reading up on it at some point. I've always done basic SEO (read: real, legitimate, basic stuff that I'd never even advertise as 'whitehat SEO' since that nearly always means 'blackhat' but we'll pretend it isn't) for my personal sites and client sites over the years. But you're going up against these 'players' who routinely setup networks of spam link sites, backlink payment networks, etc. Granted, most of those are pump and dump. They build them up, make their money, get penalized, then burn it to the ground and move on to another domain. Still, it can be disheartening. And it may tempt some people to take the blackhat approach. But, one look at the penalties should be enough to change their minds.
I think everyone pretty much already assumes that all the other lyrics sites are as shady on their SEO as they are prolific in their on-page advertising. RapGenius was kinda held to a higher bar.
I tend to feel like they wouldn't. Maybe RapGenuis's more refined image and branding yields a harsher judgment in this case as they probably perform well in many other aspects of SEO.
The link selling, not the meta data of their pages, was what caused this. It's likely that these other lyric sites do these sorts of arrangements in a less public way, whereas RapGenuis was using Twitter, of all places, to sell links.
Someone should just do blackhat SEO for a bunch of those sites, and then Google would have to punish them too (because who could proove that they didn't do it themselves?)
Or taking it one step further, use these tactics to promote a controversial institution or politician (without them knowing). Then "leak" it to Google, so they get punished. Then claim Google is censoring them. Finally enjoy the outrage :-)
Google webmaster tools lets you disregard back
links which removes any benefit or penalty of the links. So if someone added your site to a known link farm, you can disregard those back links from affecting your ranking.
Domain and page authority weigh in on how much you can get away with link tactics imo. Sites that have been around for a few more years would likely withstand any blackhat tactics a third-party would shoot at them.
They'll recover. As much as I dislike their PR attitude (whether an act or not), they have a good product. I still listen to rap music, and I still want to know the meaning of the lyrics, so I'll go directly to the site if I have to. Google made it one click easier, but whatever, I have time.
Companies have survived much worse blows. I worked for a music company that was not only being sued by every major record company, they were also taken out of all app stores. They persisted, and today I still listen to music from their site, even almost a decade later.
It's quite painful to watch these guys take this beating. What they've done wasn't nice, but the consequences are extreme. What is most concerning is the monopoly Google Search has over driving traffic/revenue for most sites.
Second, RapGenius cheated the system. They knew they were cheating the system. They violated the public guidelines that tell you not to cheat the system. And now they are paying the price.
In addition, they had no furtiveness; it appeared they thought this was something that required no concealment. I'm guessing they don't have a media relations employee, because any decent quality control would have said direct emails from @rapgenius to random folk about a seedy scheme aren't the best idea. At least a proxy should have been used..
Their emails were wanton risk, the wording had no stealth; they got what they deserved.
>When I google "rapgenius eminem" they don't appear in the results. That's pretty close to censorship.
You can also go to rapgenius.com and search for eminem yourself. Or navigate a few pages in the search results until you find the link.
Think of it this way, if I want to be listed in a phonebook but I'm an obviously shady actor and the phonebook company refuses, is that censorship? You play by the rules if you want to be ranked.
Let's get some perspective here. They were offering to promote people's pages in return for them linking back to them. Far from being evil, this isn't even illegal.
The point isn't that Google was wrong in doing so, at least not in the thread you're responding to. The point was that violating a TOS is not "evil"- no one died, no one was tortured or raped, there weren't death camps or civil liberties being violated.
Wikipedia isnt gaming the system to make wikipedia relevant. Theyre not ramping up efforts to get inbound links from random blog websites. If anything this shows that RapGenius's numbers were hyper-inflated by SEO games rather than user relevance.
Google has a domineering position in the market, and it's a bit ironic that people were quick to call for Microsoft to be split into multiple companies about a decade ago, yet nobody bats an eye when Google exercises their power. (I personally don't believe either company should be split; just pointing out an interesting parallel.)
So what can Rapgenius do? Are they screwed?
EDIT: Also, how long does the penalty last for? I think people are assuming 30 days, but is that really the case?
Even still, 30 days is a long time. They may not be okay afterwards. Whereas, say, a week-long ban would've had the same deterrent effect.