I was wondering if maybe they preconverted the videos into a sequence of tilemaps and a set of approximate tiles. I notice they are only using the same 5 colors, which is under-utilizing the capabilites of the nes which should be able to display a 13 colors at the same time. (Each individual tile uses one palette of 4 colors, and there can be max of 4 palette used at a time, but one background color is shared by all palettes.) To more fully utilize the nes they could also use a different set of 4 palettes for every frame, which would allow for a much broader range of colors.
Hypothetically, the NES could play streaming video. For memeory limitations, a custom rom cartridge could be used which would grab a new set of video frames (consisting of tilemaps + tilepatterns) via a network controller. The 6502 core does not need to decode video, but could simply swap in chunks of these tilemaps+tilepatterns which the picture processing unit displays.
This is exactly what we did. The video frames were converted to tilesets and stored in the rom image. For playback, the memory mapper (MMC3) is used to swap between the frames without having to rely on too much CPU. Luckily Guy figured out the MMC3, otherwise we'd have only had enough space for a few frames.
Correct as well about the colors. It would have been nice to better utilize them, but we didn't have enough time to do that properly.
Thanks! Alex had been studying getting better compression by repacking the tiles based on a scheme I still haven't quite figured out, but we were getting CPU --> PPU bound in what we could do during vblank. The MMC3 solution let us basically dma our way around the problems and push a much larger screen update far more often. The original version ripped through playback so fast that we had to slow it down to get a decent number of frames into the available 32 CHR ROM banks (it's a mapper 4 NES rom).
The homebrew community was a fantastic resource! Shoutouts to nesdev, Shiru, and the countless articles we googled!
You can get even more than 13 simultaneous colours by writing to the palette mid-scanline. With this method it is possible to display all 410 available colours in one frame.
Neat. I was also wondering if maybe it was possible to switch between two colors based on a certain ratio of frames, faster than the CRT phosphors to fully switch colors (or atleast faster than the eye could detect the swapping), such that it looks like a fractional blend between two colors based on that ratio. I remember doing this trick with the TI-85 calculator to give any degree of greyscale even though the LCD was only black and white.
I may be a sentimental old fool, but 8088 Domination genuinely brings a tear to my eye. The perfect timing of the "I WAS WRONG" reveal (i.e. "Oh look, this IS graphics mode") and the familiar breakdancer appearing - greatest callback in demo history. I really hope the room erupted into applause at that moment - just perfect.
"This is really hard to do on a 256K cart" - so I'm guessing it just plays back a little bit of pre-converted video. At that screen size and frame rate this is something the 6502 is fast enough to do.
(I don't know enough about the NES to say whether there's some complications I'm overlooking.)
Hey, I'm Alex, one of the two developers in the video. You are correct, the short video you see is pre-converted and stored statically.
In fact, the original plan we had was to stick a raspberry pi in the cart to handle networking and video conversion. Due to time and resource constraints we ended up building a standalone rom.
Forgive me, it was a Monday morning when we reshot this. I think the final cart was the original PRGROM + MMC3 / 32 CHRROM banks, so it was probably more like 270k total. The video playback I was commenting on entirely fit into the 256K. In my cursory understanding of the MMC3, I think that's the limit that still fits into a normal mapper 4 and will run on device - I know you can up the PRGROM (add PRGRAM), but that would kill our direct cart --> PPU path for the fast playback.
As for the processor, we were definitely pushing the limit with the design we were coding - we had to back off from glitching quite a bit due to busting the number of cycles we had before PPU. I'm sure we could do better with what we know now, or a lot more assembly tuning, but there was the hard limits on being ready by hackday!
did you consider using the hidden expansion port to do your networking (and potentially video decoding)? this would have made the cart free to do a lot more (like subtitle etc)
The data used there is cached. We were basically building around having a daughter card handle decoding the video and handling the communication with Netflix servers since there just isn't the memory (and probably not the real time horsepower) to do everything on main CPU. The idea was that an rpi could prime the nametables and communicate via MMC3 style interrupts with the main board, but we wanted to make sure we resisted the temptation of other hacks and simply punch the HDMI jack out of the back of the device and run everything through a primed emulator. We set the rules at "real hardware" and trying to keep true to what you could ship in a cart (though possibly a big one) that a user could put in and play.
Sadly, we had to cut the rpi this time due to time constraints. So we set the data up as if that was doing its job and focused on the current rendering issues and understanding the NES side of things. Hardware interfaces aside (I'm no wiz there), the rpi parts and encoding magic should have been fairly trivial. In the end, we had plenty of CPU cycles left during playback which leave me pretty confident on the rest.
That's not to say we wouldn't have run into some sort of problems with keeping the PPU primed, but we can save that for another (hack?) day.
We plan on releasing a technical post later this week with the details. In the meantime, if you are curious about the platform checkout the forums and wiki at http://nesdev.com
There is no way in hell a 1.79Mhz 6502 can play streaming video.
It's a fun party trick though.