Personally, I find it fascinating that this band of merry pirates is so organized, so rules-based. Out of a purely anarchic environment, they've managed to accept this kind of self-discipline. If only web-standards committees had that sort of power...
It's because there's a huge amount of competition in the "scene" to get first and best copies of a show. In order to declare a "winner" in the competitions, there have to be rules.
The (movie/audio/warez/demo) "scene" is not anarchic at all but rather a very well organised distributed processing machine. You probably know these guys organise into "groups" but there is lots of other stuff you likely do not know. I will describe below aspects of the warez scene, as it was some 10 years ago. I believe most observations continue to hold today and are probably mirrored in the movie-release scene. So:
"The scene" is made up of groups and each group is made up of specialists: suppliers provide material for release, crackers (or rippers, depending on context) remove copy protection and sometimes optional "filler" material, coders write useful helper software for the group and couriers distribute releases around the internet. Not all groups fill every role and some may have other roles I did not describe here (hackers, carders, hardware suppliers to name a few). There are sometimes partnerships between groups with different specialities; usually this occurs when there is an overlap in membership. For example: a release group may have close links with a particular courier group and those guys handle all their releases.
The distribution process is actually quite interesting. Each successful release group aims to have a small list of affiliations with well known and prestigious underground boards/ftp-sites. The quality and prestige of a board depends on the speed of its connection, its capacity, its group affiliations, the standing of its admins in the community and the speed with which new releases are uploaded to it. Each well regarded site is allowed only one particular kind of affiliation: a games group affil, an apps group affil, a courier affil, an ISO affil -- you get the idea. The important thing is that there is only one of each. Affiliations carry prestige for a site and being affiliated with the best sites raises a group's prestige, so there is often heated competition between sites and groups for affils. This is the case at least in the so called "zero second" scene where releases are "traded" (i.e. re-uploaded) by couriers within seconds of being first uploaded. Once a release has been traded among all zero-second sites, they slowly filter down to lesser and lesser sites until eventually making their way to the Internet at large.
So why would you go this all this trouble? The reasons are myriad but usually it comes down to prestige: for example, among couriers there is a longstanding weekly and monthly competition between groups that prove their chops by trying to dominate each other. Similar competitions exist among release groups. There is huge pressure to be the first to release a highly anticipated game or well known application. Additionally, there are strict rules about how to release something -- what to keep, what can (and should) be omitted, how to package everything up, the inclusion of .nfo and .diz files and so on. Failure to comply with these strict standards renders a release invalid and causes it to be "nuked" (deleted) among the top sites. This process amounts to a public shaming of the group responsible and makes it possible for competing groups to snatch the credit by doing a "proper" release.
I could go on and on but I think that's enough for now. Hopefully I've convinced you that pirates aren't an anarchic bunch ;)
Anarchism and "chaos" are linked in the popular imagination, but the "scene" seems truly anarchist to me:
"But there also is no lack of free organisations for nobler pursuits. One of the noblest achievements of our century is undoubtedly the Lifeboat Association. Since its first humble start, it has saved no less than thirty-two thousand human lives. It makes appeal to tho noblest instincts of man; its activity is entirely dependent upon devotion to the common cause, while its internal organisation is entirely based upon the independence of the local committees. The Hospitals Association and hundreds of like organisations, operating on a large scale and covering each a wide field, may also be mentioned under this head. But, while we know everything about governments and their deeds, what do we know about the results achieved by free cooperation? Thousands of volumes have been written to record the acts of governments; the most trifling amelioration due to law has been recorded; its good effects have been exaggerated, its bad effects passed by in silence. But where is the book recording what has been achieved by free cooperation of well-inspired men? At the same time, hundreds of societies are constituted every day for the satisfaction of some of the infinitely varied needs of civilised man. We have societies for all possible kinds of studies—some of them embracing the whole field of natural science, others limited to a small special branch; societies for gymnastics, for shorthand-writing, for the study of a separate author, for games and all kinds of sports, for forwarding the science of maintaining life, and for favouring the art of destroying it; philosophical and industrial, artistic and anti-artistic; for serious work and for mere amusement—in short, there is not a single direction in which men exercise their faculties without combining together for the accomplishment of some common aim. Every day new societies are formed, while every year the old ones aggregate together into larger units, federate across the national frontiers, and cooperate in some common work." -- Anarchist Communism: Its Basis and Principles
by Peter Kropotkin (1887)
I already told you: prestige. There is no profit motive, despite anything to the contrary you might have heard from the FBI or BSA. Occasionally people will enter the scene looking for ways to make money but this kind of motivation is seriously frowned upon and the individuals in question are often publically scorned. Simply: you don't earn any rep from being a conman or petty ripoff artist.
Here is a little more context:
Most release groups consist of individuals who break copy protection for fun. They compete amongst themselves to create clever hacks and one-up each other. It's a very tightly knit tribal culture: me and mine vs you and yours. Cracking releases or moving releases from one top site to another becomes an interesting game and so too are the metagames and scene politics that surround it. For example: a courier might strive to get friendly with site admins or group leaders who are privvy to upcoming "pres" (or pre-release) so they can get a leg-up on the competition. Be among the top 10 traders on a site for a few weeks and suddenly you have a reputation: you get invited to trade on more prestigious sites, join more prestigious groups and generally pal around with community elders. You develop very strong bonds over time and that's what keeps you in: the scene is where your friends are, it's where people know and respect you. Plus, the competition to be the best can be intoxicating.
Topsites can offer a pay to leech scheme. It's often a hushed topic and people don't like to talk about it (because it is insecure). Many topsites are at it.
A lot of scene members react badly to topsites like this, and often people infiltrate such sites and public a "scene notice" about them containing siteop nicknames, hostnames, IRC/FTP details, and screenshots.
Scene notices are like the news system of the scene. They allow people to publish messages that will disseminated to all the other scene topsites.
Let's not forget about hardware donors. Donate a hard drive? Get an account and some credits! Sites such as LSD and LOOP did this.. and many more still do.
I don't have a fancy liberal arts education so perhaps I'm misunderstanding the finer points of anarchism. As I understand it, the concept involves rejecting authority and hierarchy and neither of these is true of the scene.
Certainly things are not organised as rigidly as a government or traditional business, but neither does the scene resemble a collectivist commune. There are authority figures and hierarchies everywhere. The fact that there are quality standards for releases, that all groups (of any note) adhere to, and which are actively enforced throughout the community, should be indicative of that.
Let me spell this out for you a little further: everything from groups, to sites to individuals have their place in a pecking order. Each group for example is its own mini fiefdom; usually ran by a single leader or a council of elders. They decide what releases the group will target, what site affils to chase, what resources are required and so on. You march to their tune or leave. Initial (and often continued) membership is based on performance: if you impress with your chops, you're in. Drop the ball too many times and you're out. Always the objective is to be the best.
That is amazing. But you what would be really awesome? If these people got together and created a nice all in one php + sqlite script that allows me to put my DVD collection and PS2 iso rips onto my server and share it with my friends. I could give friends usernames and passwords and see which of my movies and games they downloaded and they can do the same for me. Like file sharing on a much smaller more personal scale.
Also, the bulk of legitimate releases comes from the "leaders" of the ISO scene.. As of right now there are ~38 users in the IRC channel representing the the majority of the top level groups. To me this is absolutely frightening that a channel like this even exists.
Competition forces rules to be born. Not just competition for prestige, a competition for access.
Topsites are fuelled by racers. These are the couriers, who transfer data between multiple topsites they have access to. All topsites have a multiplier. For every 1MB you upload, you can download ~3MB. If you don't each a quota or are not in the top ~15 racers by the end of the week your access will be revoked.
With access comes prestige, ability to race to other topsites, contacts, and leech access to the topsite. These things are many terabytes big and can have archives in excess of 60TB. A huge repository of constantly refreshing and up to date material.
Without rules this would become a free-for-all, with poor quality. That's no good for anyone.
It's funny because if it weren't for the pirates there'd be no innovation and new distribution means or it would be extremely slow. Hollywood and the music industry had become so stagnant, that it took pirates to get them to change their idea of distribution, I think.
I wonder if the producers of the content they distribute will ever get that such a high level of quality.
DVDs are often full of interlaced content (even for short movies), broken subtitles, letterboxed pictures, jerking menus; sometimes they even resize the video to 4:3. The rare downloadable videos are often over compressed or barely compressed (thank you for the 2GB file to this 20 minute show). Similar things happens with bought music: I have paid for many albums distributed as badly-encoded MP3s in zip files with __MACOSX directories and ._DS* files.
That's because there is no competition for those releases, so there is little care for quality.
If content producers released their raw sources and content monetizers had to compete to produce the best possible downloadable and distributable versions, you'd see much better commercial releases. ;-)
For the longest time, xvid was the standard for these guys. They still have standards requiring that all releases be packaged into multipart RAR files, "in 15, 20, or multiples of 50 MB". In other words, they are kind of inflexible and hidebound.
Anime fansubbing groups, in contrast, have no common standards except for -- to paraphrase the IETF motto -- rough consensus and regular releases. If some group wants to encode with Hi10P to get the same video quality at 2/3 the file size, they don't have to ask permission from anyone. Competition between groups gradually pushes forward the rough consensus, which is how they switched from xvid to x264, from container formats like AVI to the more flexible MKV, from SD to HD, and so on. The freedom to not live up to a community standard of quality gave them the ability to exceed it.
I remember reading years ago (possibly on Wikipedia's Scene article) that the multipart RAR thing was for faster distribution among the network of servers: Other servers can start downloading parts before the current server has finished fetching everything.
There's many layers of hierarchy (from the release groups themselves down to the hoopleheads at the bottom) so it speeds up releases significantly.
Internal scene transfer was (and still is, afaik) done using FTP servers, often very primitive ones because they had to evade detection on hacked servers. When you're racing a large game or movie from one server to another, the last thing you want is for a random disconnect to ruin it 90% of the way through, or even worse, to finish it and find that the CRCs don't match because of random corruption. It also means that you can race from multiple sources for higher speed, and like you said, use a server as a source before it's actually done downloading.
First of all no "pro courier" is going to be on HN bragging about his "weapons of choice." Second of all FXP was the protocol of choice back in the day(the time period the parent post was referring to). Thirdly, I remember when telix and minicom where "the weapons of choice."
Autotrading with pftp was a great way to get banned when I came up in the scene. ssh + standard unix ftp and a screen rotation of ~5-10 sites were the usual trading setup. It's much nicer than FXP, particularly as most top sites only allowed a few logins. You simultaneously mget different files from all affil sites and mput to the rest of your rotation. Chaining with FXP just isn't fast enough.
Autotrading will still get you banned. But yeah what you describe sounds like it's from back in the day. Nowadays there are three modes with the standard pftp: autotrade, autotrade only while at computer (redundant, I know), and manual trade.
But chaining with FXP is just fine now (virtually all courier groups do it with the exception of 0day). When I was a courier our pftp coordinated with the other ones in the group and chained them via country, ranking, and by 3-4 other factors I can get into if you'd like.
Are the site rankings published anywhere? When I came up we had e-zines like the Netmonkey Weekly Report. Damn they were cool! :)
FWIW: I traded 0sec to places like: Falsehood, Hades, VDRLake, Firesite, Enigma, Etirnity (whose dutch admins couldn't spell, but their site owned anyway :p) and, my favourite, Stairway to Heaven.
Anime groups can afford to be much more aggressive about technology because they have special requirements that older tech doesn't allow (styled softsubs, multiple audio/sub tracks, ordered chapters), and because their audience is overwhelming using high-performance PCs because nothing else commonly supports those technologies. In fact, modern anime releases are basically targeted towards one platform (Windows PC with MPC-HC and CCCP[1] without DXVA) with coincidental support for Mplayer on Linux because it uses the same basic open source codecs and splitters. Hi10p is even more aggressive - when they started releasing Hi10p releases, the only decoder that supported it was ffdshow alphas, and even now software support for it is very sparse. Hardware support is non-existent, and will continue to be so unless a mainstream content provider like Apple decides to start releasing Hi10p video content.
On the other hand, people who watch non-anime scene releases are much more likely to use mobile devices, set-top boxes and other A/V equipment that relies on hardware decoding support. One of the major reasons that scene releases are standardized is so that they'll play on a wide variety of hardware, and this results in a much more conservative approach than anime groups. The main reason they're moving forward with this change is because H.264 has acceptable support in hardware (primarily in the MP4 container, MKV support is awful) and XviD is really showing its age compared to modern H.264 encoders like x264.
I hadn't tried it before, but VLC 2.0 is a very impressive improvement over 1.x. I don't have an extensive library of files with crazy features, but it handled everything I threw at it fine. Hopefully they'll keep improving, it's good for the other players to have more competition on features.
While that's true, VLC has had a history of problems with H.264 (quite ironic), ASS/SSA and ordered chapters that makes them shunned in the anime community. Apparently, libass still has problems with skewed and rotated subtitles. CCCP is much more preferred on Windows.
They also don't usually support softsubs or multiple audio tracks, which are completely standard in anime releases. As a result of that, the anime community pretty much gave up on device compatibility when they left XviD behind. People who really want mobile versions tend to transcode, resize and "harden" the subs, but that's a lot of work.
That said, I agree that "annoying" is the right word in this case. The Hi10p switch has caused a lot of problems even for people with PCs - software support took ages to reach acceptable levels, and it requires a lot more CPU power on the decoding side for a very minor quality increase. They could've waited 6-12 months before pushing for wider Hi10p adoption, and I think it would've gone much more smoothly.
I was actually surprised to find my settop box supported mkv, multiple audio tracks and can even display styled subtitles unstyled (it would need specific support for substation alpha to do this).
Software support for 10bit was pretty much there in some form when they majority started using 10bit, it is a lot more wide spread now (mostly a few programs like mplayer2, vlc and MPC:HC). Support for embedded products (like settop boxes) will be a long time, or maybe never.
Yeah. It was the same thing when bittorrent became big. It was the anime groups that were the first big adopters. Mainstream releases took a lot longer.
Thank you. I was a bit surprised by the mp4 container (instead of matroska), as well as DTS not being allowed. Now it all makes sense, and it's not really a big deal at all.
SD content is mostly being used on smartphones and tablets. For iOS users, this would require a lot of extra work to convert the files to the limited container choices.
A part of me would like iOS users to deal with the inconveniences of their more locked down platform, and also show some love for matroska. Conforming to restrictive policies is a good way of allowing restrictive policies.
Internal ffmpeg aac encoder is still experimental and tends to create broken AAC streams at times. The other options:
A) libfaac / libvo_aacenc are pretty bad in terms of quality and have problems with multi-channel audio
B) libaacplus only encodes up to 64kbps in less supported AAC+
So Nero and Apple are pretty much only quality free audio encoders for AAC available right now.
Finally, pirates are using ISO standards instead of gross hacks like DivX. RAR is still lame and using the tag "HDTV" to refer to SD downsamples of HD sources is still misleading, though.
HDTV indicates source, not release quality. It's the same with bluray. The naming standards (and everything else for that matter) are highly regulated which is an impressive example of order by consensus since no one can give out orders.
RARs are an artefact of FTP transfers during which files would get corrupted, and not-so-fast connections of the past which made re-downloading the whole release somewhat inconvenient and wasteful.
It seems problematic to set up and could cause nukes for incomplete releases with the slightest mistake or network failure on the capper's part (with the accompanying displeasure of sites' admins). And he would have to do everything on his own -- the capping, the encoding, the packaging. Also, I don't know how hard the tv groups are pushing for early release but usually the content is uploaded to the group's server and only then pushed to affiliated top sites at the same time.
They're not. Cappers and racers are different people. It used to be that racers weren't even affiliated with the release group, just people moving stuff between sites for upload credit.
> Who would have to X?
The same person would have to do all the work involved in making a release. Usually, there's a division of labour: different person provides the source, someone else encodes it, someone else takes care of coordinating the release and so on.
"They're not. Cappers and racers are different people. It used to be that racers weren't even affiliated with the release group, just people moving stuff between sites for upload credit."
Okay let me rephrase: "why do you think there are people called racers and/or what do you think pretime is all about?"
"The same person would have to do all the work involved in making a release. Usually, there's a division of labour: different person provides the source, someone else encodes it, someone else takes care of coordinating the release and so on."
For TV releases? Not a chance. Any decent private torrent site has the TV releases seeding within three minutes of the credits rolling. The capper is the encoder and the someone else taking care of distribution coordination is a bot...
you seem confused on the distinction between the scene and P2P. pretimes have nothing to do with this. the torrent sites aren't the ones doing the capping, encoding, or releasing.
when you see on your torrent site that a show has a pre-time of three minutes, that's the time it took to grab the show from a scene FTP topsite, download it, make a torrent, and upload the .torrent file to whatever site you are using. when a torrent 'pres' is when the scene releases it. before the pretime, the torrent is encoded, rar'd and FTPd to a scene FTP dump.
the reason that split rar files are used is so that the scene uploader only has to upload the first chunk before they can claim credit and other groups can begin to download. if the release was uploaded all in one piece, it would have to be uploaded completely before anybody could start to download it. with split rars, other groups can have the file almost completely downloaded by the time it is completely uploaded.
Scene groups are known as affils or affiliates
when they release on a FTP site. When they release
something..racers and traders start racing
that release to other FTP sites at Gbit speeds
using auto mated scripts.
When they Release something on a ftp site it is
known as to pred or a pre. pre means the
release is ready to be raced to other ftp sites
NOT torrent sites. Remember all scene groups
hate torrents and do not want ANY of their stuff
on torrent sites. If they see someone on their FTP
sites releasing stuff on torrents, his ip gets
banned and he gets NFOed sending a warning to
other FTP sites to delete purge him and never to
allow him anywhere near scene ever again.
That's a practice called segmenting. Some groups do it, some don't.
There used to be a group named NOsegmenT who were staunchly opposed to the practice and included a little diatribe against it in all of their NFOs.
Also bear in mind that some groups have (in the past, at least) had pre-air sources. LOL/DIMENSION, one of the most well-known TV groups, were called out on the practice of obtaining shows from a pre-air source then manually adding television station graphics a couple of years ago. They would "pre" their releases almost instantaneously after they finished airing.
I believe it's for non-torrent sharing methods such as NNTP (since you can't, I guess, upload hundreds of megabytes of data to a newsgroup in a single post).
This has nothing to do with NNTP, BT, P2P. All FXP to distributed FTP servers. The RARs are split so that if one server in the farm goes down, the whole release isn't corrupted. Same for transferring the data.
"Racers" move the els to multiple servers. Split RARs allow for multiple people to be moving parts of the same els at the same time.
Rar has become one of my hot-buttons, so I'll respond as a consumer of compressed files in general:
Multi-file volumes are good and useful for many things, but last I checked, RAR doesn't offer a compelling advantage in performance, compression, or error correction vs. more readily available encodings, yet is encumbered to boot. (Feel free to correct me, if that is not factual).
Not that pirates probably care very much about that, and backwards-compatibility with old practices is always a consideration, but every time I see someone distributing software in a rar (not even pirated, mind you!) I always sigh deeply. It seldom happens, but...sometimes...
I refer to compressed artifacts in general, here. I do not consume "scene" artifacts. I am merely positing that there is no technical reason for the choice in this era. But having lived through the last one, I can understand why it became common then.
RAR isn't used for compression. The archives offer checksumming and splitting, but they shouldn't be compressed. That subset of the RAR format is unencumbered.
It seems they had previously mandated Xvid, which is very much a conforming implementation of MPEG-4 part 2 (for that matter, DivX has been since 2001.)
I noticed this today on Usenet. I've switched to downloading all my TV from torrent sites to Usenet because it's much more reliable. Everything old is new again. It's nice to see that the animated shows I enjoy like Family Guy are available in HD resolutions at roughly the same download size as their previously available SD counterparts. I assume animation is really efficiently encoded with h264. Anyway, smaller files and higher quality = happy me.
But requiring mp4 as container is a bad idea in my opinion. To playback an mp4 file you need to have it wholly (please correct me if I am wrong) so you cannot start watching before you downloaded it all. MKV would be so much better.
Please do not blindly reply with "But my XYZ device does not support MKV". Scene rules/standards have a lot of impact and might help making vendors support Matroska.
> To playback an mp4 file you need to have it wholly (please correct me if I am wrong)
You are. By default, many tools put the meta data of mp4 files at the end of the file which means you have to have the whole file before playing. However, such mp4 files can be "fixed" using tools like this:
> To playback an mp4 file you need to have it wholly
720p or better videos on youtube are all MP4 now. You just have to move some atoms around within the video file to prepare them for "streaming" playback. There's open source software to do this.
Then again, the ability to play an incomplete file is not really useful if you're fetching a movie via torrent, which is what I would assume most of these guys are using.
If the MP4 is "fast start" the index is at the beginning so you can "stream" it.
I suspect scene standardization on MKV would probably encourage Apple and Sony to deliberately not support MKV (instead of not supporting it out of laziness, as they currently are).
When DivX made their H264 codec, DivX 7, they actually chose the matroska container. Of course, DivX has lost a lot of the brand name recognition they once had (their logo used to be on nearly every DVD player), so the penetration of matroska in this respect isn't that high. There is also WebM, of which the container uses part of matroska, which might help penetration (of course, its people like Apple and Sony who aren't likely to implement WebM).
Other then that, only a few embedded products support mkv (TVs, set top boxes). These are pretty much impossible to update, so it is pretty much impossible to recommend matroska at this point, since a large amount of people would watch these videos on such products.
You don't need the whole file. That would just be silly and it's used for streaming all the time. Also even though I would prefer to see an open standard succeed, the mp4 container does have a lot more support while mkv doesn't have any advantages for this type of use.
What makes it beneficial for ripping/distribution groups to coordinate like this? I've previously seen them agree on a common standard for the number and size of rar files a release is to be broken up into, and other things i can't remember the specifics of. Why do they do it?
From reading many of the curiously well-informed comments, I suspect that the reason for standardization has to do with speed and validation for credit. Since it sounds like "scene" people arbitrage new releases across scene FTP drop sites for upload credit, a standard method of "settling accounts" via scripts/bots would be useful. Arbitrage would be less efficient if files had to be transcoded between FTP sites.
I wondered why I was suddenly seeing x264 instead of AVI for TV files. I thought it had to do with the lockers going down.
Well, in a way this will make things even worse for the CopyNazis. No longer will people have to convert from AVI to watch on their tablets and phones.
I suppose you mean mp4 instead of avi, as avi is only a container. So the discussion is two-fold, the container format (mp4, mkv, avi) and the encoding of audio, subtitles and video, of which the latter, h264 is pretty much the standard, and xviD/Divx are and have been for almost a decade, deprecated.
Although I can just imagine all the pirates whining about "h.264 (or mkv as they call it) doesn't play on my dvd player, bring avi (as they call xvid) back".
I can't believe pirates are still burning crappy SD rips to DVD.
Someone I know stopped doing that when he wanted to watch a movie for the second time and realized it was quicker to leech a 1080p bluray rip than search through his mountains of VCD and SVCD discs for the one he downloaded just a couple of years back.
This pirate friend of mine welcomes the change, for the maybe 1 SD show he watches.
Although he personally wishes people paid more attention to sound codecs, nothing worse than downloading something broadcast in 5.1 and finding out the file only has a stereo downmix.
1) I watch it in the corner of my screen, so I'm fine with 320.
2) Here in Australia, we pay for usage. I'm not going to download something in a high quality if I'm near my limit
My point was that there is a large community that has gathered around the idea of sharing films. Large enough to make them want to issue standards. All I'm saying is with that size of community dedicated to sharing film, they should be producing more films. Especially if they dislike the current paradigm enough to remove themselves from its market.
If the skillsets overlap, that kind of shift towards production does happen. For example, the 80s game-cracking scene also became the 80s demoscene, because the various skills they acquired in cracking copy protection overlapped a lot with the kinds of skills that were useful for producing interesting interactive-art type software. It's not clear that x264 encoding and distribution produces skillsets that a particular helpful for film production, though, so I don't see a strong reason to expect them to start producing films.
