I'm not sure, whether this propaganda piece should come with some more warnings. The paragraphs starting with 'AV1 mythbusters' are unbelievable:
1. The MSU report ("performance mentioned by AOM sources") with known methodology has to be verified, but that one demo with hand picked scenes and undocumented codec settings are totally the definitive answer.
2. On the performance side, we don't know the complexity of the codec yet. And on the cost side, HEVC is going to be more expensive by definition, due to being an rent extraction scheme.
3. You can't compare reference WIP codec with optimized one. The premise of the entire paragraph is just wrong.
> You can't compare reference WIP codec with optimized one. The premise of the entire paragraph is just wrong.
AOM dev here.
Currently, aomenc takes ~100s to encode a single 4K frame.
Which means to encode live 4K 60 FPS, you need to multiply the encoding speed by 6000 (!).
Indeed, the codec isn't fully optimized yet, and the required performance improvement is huge.
Of course this freaks people out!
On the other hand, reference video encoders are famous for being ridiculously slow. The JM (H.264 ref encoder) had similar encoding times, and this didn't stop H.264 from being encoded live.
This was exactly my point. I remember when the JM encoder was so slow, that it encoded 3 or 4 frames per hour. That's why 100 sec/frame for the reference says nothing about the final performance and putting out articles creating the "general knowledge of what everybody knows" that AV1 is slow is at best disingenuous, at worst malicious.
MSU measures in SSIM and PSNR. SSIM and PSNR, I think i need to say no more. The one demo was done in IBC with industry expert, but not only that but many other independent test. Needless to say AV1 isn't finished yet so it isn't a fair test anyway.
The latest video show [1] shows AV1 still isn't finished yet. And lots of update and changes were still put into place at this stage of development. So any 2017 made comparison is moot at this point. But with all these improvement it is finally looking like the 30% bitrate savings ( in subject measures ) that they said they HAD from the start.
We dont know the complexity of the Codec? Oh hell yes we do. AV1 is complex, much more so then HEVC. It isn't more cost effective by definition, because AOM has to figure out how do make it much more speedier.
The last report of AV1 is roughly 200 - 350 times slower then VP9. And VP9 is already quite slow. The Reference Encoder for AVC or HEVC were only in the 50 - 100 times slower range. Also worth mentioning the Reference encoder of both AVC or HEVC wasn't even tuned for quality or speed, this is radically different to AV1 reference encoder. Whether AV1 can be optimized to being only 4x slower then HEVC remains to be seen.
So there is absolutely nothing in this pieces that is even propaganda. But lots of valid scepticism, as an industry and business you have to do the maths and calculation. And AOM must work to ensure these concerns are being solved and expectation.
Lastly, just because you like free, royalty free, and hate HEVC payment, doesn't automatically mean AV1 is best in everything. This kind of fanboyism or whatever -ism is spreading like wild fire in every circle.
> 2. On the performance side, we don't know the complexity of the codec yet. And on the cost side, HEVC is going to be more expensive by definition, due to being an rent extraction scheme.
The codec is still years away from being widely adopted. Its encoder is still highly unoptimized, to the point of it taking tens of seconds to encode a single frame.
> The codec is still years away from being widely adopted. Its encoder is still highly unoptimized, to the point of it taking tens of seconds to encode a single frame.
Sure, but that statement is valid for every single codec in development phase. AV1 is not an exception.
To be fair, they would be calling shots either way, either by paying up to be a Founding member or if they couldn't do that, indirectly through their own market power which as the makers of their own browsers and chipsets for most of the hardware they move (and operating systems for all of their hardware), is substantial. Working as a "Founding" member is probably easier for everyone involved.
I'm not saying it's a bad thing that they joined. It will be useful for everyone. But Apple are being Apple. They just can't do the right thing from the start.
AOM terminology sucks here. “Founding member” just means they have a seat on the board of directors (and it’s awesome to see Apple join that), not that they were truly part of the initial founding (Apple joined in late 2017/early 2018, which is well after AOM was publicly announced in 2015).
> As an industry, we hope that the HEVC licensors come up with a reasonable and fair solution to resolving the patent licensing issues.
I don't think anyone cares about that (besides said patent holders). As an industry, we should hope that HEVC and its approach to licensing will go bust for good, and everyone will be using free codecs that will unshackle them from patent protection racket.
It's not just patent holders who care. Harmonic sells products where a large portion of the cost is derived from licensing codecs like AVC and HEVC. One of their contribution encoders[1] has separate software licensing for a variety of subsampling ratios.
They're not licensing codecs, they're licensing different versions of their software. The HEVC fee is only 20 cents per encoder, but I'll guess those video software options are thousands of dollars each.
While HEVC fees are only a tiny fraction of their encoders, they are far greater than 20 cents. Adding two of the pools together already nets up to $1.40, and there are more pools with unannounced pricing yet. https://en.wikipedia.org/wiki/High_Efficiency_Video_Coding#P...
Not among the powerful, no. The public gets great benefit from FLOSS codecs (Opus, VPx, AV1, etc.) and containers anyone is free to implement without patent hassles (such as Matroska and the smaller variant used in WebM). Patent holders usually try to wrangle control of everyone (Apple being a part of a media codec patent-holding group meant Apple pushed those codecs and back when people cared what Apple picked that meant more power for Apple). In this respect, it's a class issue.
In 1990 IBM (in "Think" magazine, #5) said they got considerably more value from cross-licensing than fees (in other words, looking at this in terms of money is wrongheaded) because cross-licensing gives them access to ideas). So, it's the power to control one's competition at play as well. This arrangement means (as Richard Stallman pointed out in essays and talks years ago found online in https://audio-video.gnu.org/video/rms-vub-2011-02-22.ogv ) that the entire temporary monopoly one ostensibly gets from the patent system is a sham, a myth. Organizations that hold a lot of patents can always find ways to economically compel smaller patent holders into cross-licensing, thus eliminating any real value of patenting ideas for most.
The boring answer is that they've already invested in H.264, including hardware decoding support, and it works well enough for them, so they don't need to add support for VPx.
What user benefit would there have been? Any format added has a maintenance cost (ever notice how many security updates are found in AV code) so there would have had to be some compelling benefit versus the MPEG standards they already supported.
That's especially important for battery-powered devices since the primary WebM codec (VP8) didn't offer a reason to switch from H.264 and would had to be implemented in software rather than using the optimized hardware on the device (this is why YouTube playback performance is so much worse on most systems unless you disable WebM support in your browser).
Vastly used? I'm not so sure about that. Vastly used in tandem with H264 when required? Absolutely.
If iOS can hardware decode H264 but not WebM it actually isn't in the user's interest to support WebM, because it'll reduce incentive to provide H264 versions of content, which means iOS users will experience battery drain faster.
It might not be particularly fair, but it does make sense.
Native apps like VLC can play WebM on iOS with software decoding, but as you say it murders the battery.
If Apple supported that in browsers I'd bet that Google or other companies would switch their video hosting over, much to the detriment of anyone who watches videos on an iPhone. It's an open standard so clearly they're doing the pro-consumer good thing!
Again, who benefits from this? You’re talking about duplicating a ton of work simply to have approximately the same quality.
You can contrast this with AV1 which is also a lot of work but which delivers better quality at smaller sizes. It’s not surprising that they picked that option instead.
"Normal" people also don't care about TCP\IP or HTTP or HTML or image formats or any of the detail that makes the web or the internet generally work. It's a pointless argument.
"Normal" people do care about their videos starting fast with high image quality. AV1 will deliver them that and deliver it better than H.264 has done. This is true whether they know it or not, whether they care about it or not.
That's pure misdirection. If Apple didn't support HTTP iPhones would not be able to use the web at all. If Apple doesn't support VP8 nobody notices a thing.
Of course faster load times are nice, they will come, but whether it's through VP8, HEVC, VP1, or something else will take time to shake out. Meanwhile somebody on HN will moan about Apple's monopolistic user hostile refusal to support every single blasted codec released ever.
Good grief, there are even people who still think Apple should have supported Flash, and therefore presumably still support it because the world today would be so much better if Flash was still widely used, or something.
> If Apple doesn't support VP8 nobody notices a thing.
I notice. When I build an application which use video streams and I want to be able to use video without having to worry about the licensing implications. I want video to be royalty-free for all use cases just like all other internet formats and protocols are royalty-free. There is no reason for it not to be.
VP9 gives me that today and it's a shame Apple doesn't have VP9 support. Hopefully they'll announce their timetable for AV1 support soon (and maybe VP9 as a bonus).
I want it out of the box in Safari, like all other browsers offer. Apple will get there eventually with AV1 and hopefully they'll add VP9 support as well. VP9 and AV1 have some features in common which makes implementing support for both easier.
What’s the quality of that support like? Chrome and Firefox technically support webm but since it’s software the actual user experience is noticeably worse: CPU fans on, struggling to maintain 30 FPS – exactly why Flash fell out of favor so quickly when an optimized alternative showed up.
Works fine. There are no CPU fans, no struggling to maintain 30 frames per second. I've played back 1080p30 VP9 video in Firefox on a 12 year old Intel Core 2 Duo desktop with no issues.
VP9 decodes faster than H.264 at same picture quality because the bitrate is lower:
It's too bad Apple hasn't turned it on. Intel has included VP9 decoding since Broadwell so a lot of recent Mac laptops have VP9 acceleration hardware ready to go.
> Works fine. There are no CPU fans, no struggling to maintain 30 frames per second. I've played back 1080p30 VP9 video in Firefox on a 12 year old Intel Core 2 Duo desktop with no issues.
> VP9 decodes faster than H.264 at same picture quality because the bitrate is lower
That's comparing ffmpeg's decode performance on the CPU using a single clip from one video. What I'm talking about is the difference which hardware acceleration makes, especially since older systems also have older CPUs without the newer instructions used by high-performance software decoders. VP9 implementations have been increasingly optimized over the years but it's really challenging to beat hardware performance even before you consider the power budget.
As a simple example, I opened https://www.youtube.com/watch?v=wbSwFU6tY1c on a 2.13 GHz Intel Core 2 Duo (2010 MacBook Air). That's playing a 1280x720 stream. In all browsers, even with ads blocked by /etc/hosts I had to wait ~20 seconds for the 80+% CPU from the YouTube JavaScript to settle down before starting playback. In Safari, that video takes between 3 and 8% CPU usage with no dropped frames. In Chrome, that's 80-120% CPU usage and about a 5% dropped frame rate.
Firefox is interesting: it ran at about 15-20% CPU usage, which is worse than Safari but still much better than Chrome. I thought that was odd but stats for nerds showed why: Firefox was using H.264. After using media.mediasource.webm.enabled to forcibly enable webm, it started using VP9 and that meant ~40% CPU with bursts up to about 80%. While that's clearly much better than the Chrome experience it's still a full order of magnitude more CPU than Safari's hardware path.
Remember, I'm not saying that VP9 is horrible but rather than hardware support is a really big deal and optimized video playback in general has non-trivial costs. Apple made a big investment in HEVC and it doesn't surprise me that they're investing in the next generation rather than spending time on the current generation since the fixed costs for tuning, testing, security, etc. are the same whether 100% or 5% of your customers use it.
> That's not the experience I've seen on newer hardware
Works fine for me at 1080p VP9 on a mid-2014 Macbook Pro. 720p VP9 also works fine in VLC on my iPhone 7. Haven't tried 1080p on the phone. It doesn't have the resolution for it anyway. 720p AV1 will probably work on the phone as well (AV1 decode is about 1.5x the complexity of VP9).
> What I'm talking about is the difference which hardware acceleration makes
So it's time to get Apple to enable the VP9 acceleration present in their latest models (around 2015 and later).
But don't worry about it. AV1 is coming and everyone is finally on-board with royalty-free video. The bad old days are nearly over.
Yes. So because few people are about it, it's irrelevant? I don't think so. Most people also don't care how their device works, or basically anything that's not their area of expertise. Are all those irrelevant too?
Your problem is treating this as an emotional exercise in team loyalty rather than an engineering problem. Even Apple has finite resources so it’s unlikely that they’re going to spend time on something which duplicates but does not improve existing functionality. The number of people with a large collection of videos which are only available in VP8 just isn’t large enough to justify the investment.
They don't know whats best for them if there is a user who wants something different. You might invoke appealing to the mass here, that doesn't make it true however.
Netflix on an iPhone? Probably not. At least not holding it in your hand for 2 hours.
> Netflix on an iPhone? Probably not. At least not holding it in your hand for 2 hours.
Do you have data to support this or are you just assuming that your personal tastes are universal? I see quite a few people watching videos on phones or tablets – ever see parents loading up their kid’s iPad before a flight?
> They don't know whats best for them if there is a user who wants something different.
Obviously. But that's not how Apple works. You do things their way, or you get an Android phone/Windows laptop. They make the choices for users, and judging by market share a very solid number of people are very content with that arrangement.
YouTube is not ‘vastly’. Get back to us when it’s ‘vast’ among commercial content providers.
Contrary to your point about not watching on iOS, I, and everyone I know, watch all content on an iOS device, both in the living room and on the go.
There is certainly a segment of the market that is using misc devices or “casting” to USB sticks plugged into TVs, but the 4K TV owning cord cutters I know tend to also own iPhones and use Apple TVs.
We saw this shift happen even more when iOS (and TvOS) gained SSO across apps with the “TV” app as indexer, and another bump when Amazon Prime released.
Why not? Just go by traffic. Youtube default to VP9 on Chrome, so yes it is vastly used. Netflix can't even display 4K on the desktop properly.
If you watch in the living room (on TV) you are not watching it on the iOS device. By that I mean the screen of the iOS device. Being in the living room, watching on TV, means you have no battery issue. Watching on your device means literally that, watching it on your device with your devices' screen. Not some casting or streaming to other displays.
So what you’re saying is since Google has manipulated the environment so that Google’s preferred codec is the most used on Google‘s website, which happens to be the largest video site on the Internet… that means Google’s codec is better.
If we ignore YouTube, which Google controls to their own benefit, what’s the percentage of WebM/VP9 versus H264/H265?
Why are you using double standards when talking about Google's codec choices vs. Apple codec choices. Do you think Apple collecting royalties for h.264/h.265 has nothing to do with them preventing you from using VP9?
I think we can say with quite a lot of confidence that any royalties Apple receives for h.264 are a rounding error in their income, and they will have ~zero impact on their decision to support other codecs or not.
H264 is also used by Blu-Ray isn’t it? I know my cable system switched to it from MPEG2 a few years ago.
H264 is used in other places in the electronics industry. That’s one of the reasons Apple picked it. Isn’t it the format most digital cameras record video in?
WebM/VP9 is used by... Google. And Wikipedia (who won’t use something with patent licensing). Is there anything else big?
No, browsers and hardware manufacturers have implemented VP9 because it has better licensing terms than H.264 and especially better than HEVC. HEVC was released at around the same time as VP9 and yet today VP9 has double the installed base of HEVC: https://ngcodec.com/news/2017/10/21/why-we-are-supporting-vp...
> In this case, Apple went with the industry standard.
Applenuses the same codec everywhere. They needed H264 support for other things (like video foot recorded from iPhones, or physical cameras). They also support it in Safari.
This isn’t like Mozilla, who only makes a web browser.
I really don't know what you're trying to argue. We're talking about web video here. There's nothing stopping Apple from adding VP9 support. VP9 outperforms H.264 and the other major browsers have added support for it.
Apple will be adding support for AV1. Like VP9, AV1 is royalty-free. The Alliance for Open Media has been so effective (even before AV1's release) and HEVC's licensing has been so terrible that MPEG is starting to question whether it can survive as an organization:
I suspect what’s going on is that they’ve already invested software and hardware time in HEVC and instead of making a corresponding investment in a codec with similar performance they’re focusing that effort on the next generation with better performance. I suspect that if the full story of VP9 performance & HEVC licensing had been known at the time they’d have made a different call; HEVC got a lot more expensive late in the game when a lot of early plans had been set into motion.
Apple went with H264 for the original iPhone in 2007.
WebM was released in 2010. VP9 is from ‘12/‘13.
H264 was already the web video standard when WebM/VP9 came along.
I take issue with Google trying to force everyone over to their format and removing support for higher resolution H264. They crippled what used to work for me to push their agenda. And because it’s license free I’m supposed to be good with it.
This is exactly the kind of stuff people used to get pissed at MS or Apple for.
But it’s Google, and people love Chrome. So this is good and Apple is the one not going with the ‘standard’ that ‘everyone else’ is using.
Minor correction: they started with VP8 and a PR campaign trying to spin it as better than H.264 but it took VP9 to actually deliver better results than H.264. Both VP9 and H.265 offer big improvements but with a corresponding jump in processor requirements.
It’s only vastly used if you mean “YouTube transcodes all of their H.264 into VP8 so you can hear your CPU fan”. It’s exceedingly rare to see a webm file without a corresponding MP4 file.
Because in general they are jerks when it comes to free codecs support. The fact that they joined AOM is a major shift from their previous behavior and it most likely happened because they realized they have little choice and not because they wanted to do the right thing.
Apple traditionally ignored non MPEG codecs[1], VPx is ignored for the same reason, and not due to being developed by Google.
[1] i.e. Vorbis, Theora, Flac, matroska containers, etc. Opus is supported in Safari for WebRTC because they have to, but is completely unavailable in other contexts.
ALAC did show up at the same time as the first AirPort Express in 2004, and it is the format used to transport audio over AirPlay (first called AirTunes). That may have been the original impetus.
I was under the impression it was an AAC project that didn't get adoption outside the Apple ecosystem, not that it was Apple created. Do you know whether Apple was the submitter to ISO?
It sounds like when you say "an AAC project", you're thinking of MPEG-4 ALS [1], which was the codec chosen by MPEG for lossless audio in the mid-2000s and last updated in 2009. Despite ALS being blessed by MPEG, it never achieved much acceptance in the market. Its most direct ancestor was LPAC [2], developed at the Technical University of Berlin; but ALS included improvements from NTT Communication Science Laboratories and RealNetworks [3].
Apple developed Apple Lossless ("ALAC"), a different, unrelated [4] format, which was fairly similar to FLAC, but different in some minor ways (paraphrasing the FLAC dev's own words [4]).
Actually FLAC and Opus support was added to the Core Audio / AudioToolbox / AVFoundation APIs in iOS 11.
Though the Opus decoder only supports raw packet decoding, i.e. you still need something to decode the container (usually ogg).
Also, FLAC file decoding is horribly slow - compared to the FOSS reference decoder -, because for some reason the whole file is scanned through on the first read when using the AudioFile APIs (AudioFileReadPacketData). (I’m gonna make a demonstration project for this and send them a bugreport tomorrow.)
It actually makes sense, because libFLAC has no public API for getting the compressed audio packets as is. Rice/huffmann/whatever decompression always happens inside libFLAC.
With Apple’s AudioFile API you can decide whether you want the decompressed PCM samples, or the underlying compressed packets. You can get better energy efficiency if you offload the decompression step to the dedicated coprocessor. The difference is smaller than with hardware vs software h.264 decoding, but there is a difference.
(For lossy codecs this also makes sense when you want to forward the compressed packets to a wireless speaker in order to avoid lossy recompression on Bluetooth transmission.)
Metal release date: 2014
Vulkan release date: 2016 (announced 2015)
So since somebody made a non-compatible clone a couple of years later, Apple should have ditched their work and lost the power to make decisions as they see fit in the process? That doesn't sound like something that would appeal to them.
Unfortunately, the 2014 Metal release is not comparable to 2016 Vulkan release.
Metal from 2014 had a lot of assumptions (for example, that the GPU shares memory with CPU), so it could be released earlier due to being simpler. That's why it was supported only on iOS.
And that's why there were more Metal releases and refinements. Apple had to make improvements to make it equal in its capabilities to Vulkan 1.0 and DX12.
You know Vulkan often changes or update their spec? It's not a nonsense for Apple to develop their own with easier API also made for Swift language, reliability is important.
An AV codec isn't a weapon to lock developers into your platform while wasting millions of dollars and thousands of man hours on pointless api ports. There are too many implementations of encoders / decoders for the various formats lying around to get away with that. Its not like they didn't try with Quicktime in the past (ish? .mov was just a crippled .mp4), but that never really made them any money I don't think.
They invested heavily in their own API (Metal) while OpenGL committee was sitting on its ass doing nothing, and released it a year before Vulkan was even announced.
1. The MSU report ("performance mentioned by AOM sources") with known methodology has to be verified, but that one demo with hand picked scenes and undocumented codec settings are totally the definitive answer.
2. On the performance side, we don't know the complexity of the codec yet. And on the cost side, HEVC is going to be more expensive by definition, due to being an rent extraction scheme.
3. You can't compare reference WIP codec with optimized one. The premise of the entire paragraph is just wrong.