Maybe, if we keep bringing it up on here, startups will finally start putting a couple of lines of boilerplate at the top of their blog pages so we will know what on earth it is that they do when we end up there from HN.
A guy can dream, can't he?
At least this time the logo at the top links back to the main product page. Oh, and I see that the title text of the logo has just such a description! You're almost there guys!
Yes, and by the way they could put video NOT in autostart mode, so that when you open a dozen links from an aggregator like HN, there isn't some guy's voice or worse, cheezy demo music blaring from the PC coming from an unknown tab deep down the stack.
"Since our servers can encode video much faster than most of your users can upload it, this means there is literally no more delay between the end of the upload and the video finishing encoding. In the screencast above this makes a 150x speed difference."
Surely the upper bound is 2x if you could transcode faster than the upload before.
Unless you are just measuring the time between the upload finishing and the transcode being done. But why would a user care about that metric rather than the total elapsed time?
With the exception of video formats (you mention quicktime) that puts the metadata at the end, how about real-time preview as the file is still uploading? Even a series of static thumbnails every 10 seconds would be an interesting new feature.
I demand you address the OP question of why you haven't fixed the physical constraints of crappy upload bandwidth? Seriously, wtf would we pay your for otherwise? </snark>
As funny as this sounds, fixing the upload bandwidth problem is also something we will work on. Getting a good route between your users and servers can make a real difference.
So at some point in the future we'll offer upload servers in all major geographic areas.
It's interesting how difficult this can be. I was recently evaluating Linode locations for a new virtual server, and though I'm geographically much closer to California, the Texas location gave me almost 20x the download speed. To be precise, I was able to download from a California node at 300K/s, while I got close to 6MB/s from Texas.
Comcast used to route my traffic to California through Seattle, Washington, then down to San Jose and then Fremont, but now, it's going to Texas first, then across to San Diego, then up to Fremont, across a saturated link.
You're derping, but a browser plugin seems like it might actually be a good idea for this. Even if JS is too slow, someone who uploads lots of video through a site might well be willing to install a little plugin to make it faster, especially if it comes with other simple things like queuing a directory of videos, etc.
NaCl might be good for doing the transcoding on the client side before uploading - and would be especially helpful if the transcoding is a downsampling (i.e. reducing stream length), and if the client machine is beefy enough to do it without adversely affecting performance when pumping out bits at the rate of the upload connection.
However, it wouldn't be particularly useful for things like queuing uploads from a directory. The point of NaCl is running native machine code in a provably secure sandbox. It's about making possible web app features faster on the client side, rather than adding new capability, per se.
Most users who have uploaded videos are probably aware that it isn't available right away, but may not really understand why as far as encoding goes. If that process is sped up by such a significant factor, they don't have to worry about it. The 150x number is at worst misleading if you take Amdahl's law into consideration, but upload speeds really aren't at the mercy of site owners. Improving any part of the pipeline is going to be a big win, especially for video.
This is often true, but not always - some video formats put header data at the end of the file, not the beginning, so you can't just start encoding as bits come in. Or if you can, you're encoding blind.
Youtube have been doing this for a long time now, it's great watching a video being processed and "read" as it's uploaded, it's the even more user friendly step on from a progress bar, makes Youtube uploading more "fun".
I don't believe YT transcodes during upload, since last I checked they are doing 2-pass encoding (its been awhile though).
However they do examine the buffer during an upload. Try uploading an MOV with bad edit points, they will warn you about A/V sync issues before the upload is complete. I assume they also abort uploads for invalid files.
A video starts playing the moment you navigate to that page. Note to OP: please add a play button to your video. The video should start only if the user has specifically requested it. It took me a minute to hunt down the tab that was the source of noise in my otherwise quiet work environment and, once I found it, I killed the tab without even glancing at the page title.
What have you done about the security implications of pushing user-provided data into ffmpeg straight off the wire? How worried about this are you? Obviously ffmpeg was not built with untrusted input in mind. I have only paranoia as a justification, but I bet I could make it crash pretty hard, and with a bit more work I bet I could design an input that caused ffmpeg to just fill your disk forever.
How much work have you had to do around these issues? Did they prove to be bigger or smaller than you anticipated?
This is really neat, but since you said "anything"... I have a legal question that I haven't been able to find the answer for:
How do you deal with the licensing issues regarding open source encoding software? Do you pay the MPEG-LA a fee directly for use of the software? Is it per file/per minute/flat fee?
Just wondering about the mechanics of this, mainly for an in-house streaming application.
IANAL, but generally, you need a license if you're using open-source encoders or decoders and you're the one who actually compiles them. (Again, not a lawyer, but that changes ffmpeg from "a source code description of an encoder" to "an encoder.")
MPEG video stuff is generally pretty cheap or free for low volume. Audio codecs are fairly expensive, on the other hand. AAC + MP3 + AMR is >$20K in minimum fees.
How many ffmpeg instances are you holding in memory? You have a single ffmpeg process for each upload that is happening, do you? Additionally, if I am uploading files with B-frames and GOPs of 300 (which I assume will lead to a lot of frames being kept in memory, not sure about this), is your RAM requirement not going to be massive if you have several uploads happening at the same time? And considering the length of time a single video takes, it's reasonably likely you'll have a lot running in parallel, is it not?
We have one ffmpeg process for each upload, yes. From our tests with the kind of videos we mostly deal with (user generated content), this seem to work quite well while only requiring reasonable memory allocations.
So sure, there could be problems we don't know about yet, but that's the price for doing something nobody has done before. We will deal with these problems as they come up. In the worst case that might mean falling back to normal encoding if a ffmpeg child process starts to misbehave.
We are feeding raw bytes into a command line ffmpeg process via standard input.
Ffmpeg only reads as much from stdin as it can process. Node.js gives us a 'drain' event when this happend, which means we can pump more data into it. So we are no throttling the speed, just the flow of data.
As a curious outsider, I have to ask why are you giving away the recipe to your secret sauce ? I mean, there are several of your competitors such as zencoder , etc. that could easily copy this idea and implement it . So why would you blog about implementation know-how that made your product cooler and enabled you to command premium pricing ?
I don't really think it matters for a number of reasons. Chief among them being motivation. If you are motivated enough to copy this idea then you would reverse engineer it as soon as you heard what they are doing without knowing exactly how, which is what I did knowing that they were based on nodejs and I know how nodejs handles buffers.
Ultimately, if you are a provider of services, these or any other, you will add functionality or run the risk of being eclipsed.
1. We are infinitely [1] faster at converting video uploads than our competitors now. Speed matters. This is a strong USP.
2. Publicity. There are a bunch of features that would certainly be equally useful to our customers. However, none of them would get much interest from hacker news. We got a lot of signups and page views from this.
3. It took me probably 4 full days [2] to implement this, including the screencast and blog post. That's reasonable for the value added by this feature.
--fg
[1] We are really infinitely faster if the upload speed is slower than our encoding speed. This is probably the case for 99% of all consumer internet connections at this point. In any case we reduced the time required for an encoding from SUM(Upload, encoding) to MAX(Upload, Encoding).
[2] We already made a lot of technology decisions that lowered the cost of implementing this. A competitor would certainly need much more resources to pull this off.
One small question: if a user has their upload stream redirected to your service, and for some reason this upload is unable to finish, are we now forced to have the user try the upload again? It seems one advantage of the two step process would be the ability to try the process again on behalf of the user rather than making them wait and that should be weighed into the convenience formula.
I've never used your service so I'm not sure exactly how the upload stream is redirected to your platform, so this concern might not be totally valid if the upload is running through the client platform anyways.
> if a user has their upload stream redirected to your service, and for some reason this upload is unable to finish, are we now forced to have the user try the upload again?
If the upload doesn't finish the user needs to redo it.
> It seems one advantage of the two step process would be the ability to try the process again
You got me confused here. If the upload never finished, there is nothing you can do to fix this.
That being said, resumable file uploading is the next thing we'll tackle.
> It seems one advantage of the two step process would be the ability to try the process again
I should have been more specific: consider the special case that the encoder on your side encounters a random error whereas just a "plain" upload to the client platform would have finished, so precluding any network transmission errors.
Although, with resumable uploading and a simple go-between service on the client platform between the user and your platform all manner of fanciness could be achieved, I think. Thanks for quick response!
The biggest challenge to me seems to be spinning up instances once you get a spike of realtime jobs in parallel. Keeping the 'realtime' promise is often hard once your systems go into production.
Of course. But since most people upload with insanely slow speeds, we can actually handle a spike much better with this feature than we could before : ).
I was thinking this should be done in realtime on the client machine and streamed up the server. Then you cut down on both transfer and encoding time, and use far less server resources.
It's not for end-users. It's a service for developers to add to their web application. You can certainly try to coax all your users into pre-encoding their video - and understanding how to do that - if you like....
The pricing is a tad expensive, but doesn't look bad at all when you consider the need for an on demand encoder for the entire time that the user is uploading.
We know we're a little expensive. But we did this so we could lower pricing in the future (which we will), and to attract people who are getting more value out of our service initially.
If you have a project and pricing is a deal breaker, just email us and we'll set you up with a discount.
First, I agree that it'd be silly not to use thread-per-CPU non-blocking I/O.
However, I don't see why it'd be tricky in a threaded server. The thread gets an fd to recv() the incoming data from, and it popen()s an ffmpeg process then loops to recv() and write() the data until done.
I'm not saying it can't be done. In fact the multipart parser we have build for transloadit has already been ported to C++ [1] and I imagine Java has decent libraries as well.
But most people just sit on a request/response oriented Python/Ruby/PHP stack with possibly stupid buffering load balancing in between (nginx buffers uploads).
If you build this from ground up, it is certainly possible with a lot of technologies.
The fact that it's not actually faster is the entire beauty of it. While our competitors have been super busy to actually make their encoding faster, we have just taken a huge bite of free launch by making the encoding happen in parallel.
You may want to avoid using the word "sucks" in a professional context because there is a population of people for whom the word evokes the idea of oral sex. Try substituting "is not so good." This will have the added virtue of being super charming in your German accent.
edit: I usually don't bitch about being downmodded. But don't you guys know any old people? And know that old people tend to be in charge of things? In any event, you shouldn't rely upon business leaders of any age being ignorant of the language.
Here's the thing: Sure. "Old people" exist. They are often in charge of things. Many of them read blogs and surf the internet. Do you really, honestly think that somebody who surfs the internet and reads blogs, especially blogs about video encoding, won't be aware of the current meaning of the word?
Honestly?
You see the word "sucks", in the context of something not being very good, everywhere. Literally everywhere. I bet that if you googled any tech-related keyword, somewhere in the first ten results, "sucks" would be employed in the context of "is not so good".
My point is this: People that read that blog post _will_ know what "sucks" means in that context and will not infer anything sexual from it - and even if they did, it's hardly the end of the world. Surely nobody who spends any reasonable amount of time reading blogs about video encoding will be in the slightest offended by fellatio. After all, "old people" are no different to "young people" except they have more experiences under their belt (hyuk hyuk).
I remember the difference happening about 12 years ago, though my neighborhood was probably behind the times somewhat. I remember that some of my fellow 7th graders and I were used to the old meaning, but it wasn't too hard to transition to the new one since we didn't really use it much anyway. Also, I remember a trailer for Bicentennial Man (1999) using the more "evocative" meaning.
A guy can dream, can't he?
At least this time the logo at the top links back to the main product page. Oh, and I see that the title text of the logo has just such a description! You're almost there guys!