Any format that is uncompressed like this is huge. 8bits per channel (RGB/A) times width times height. Many image formats have animated versions of them that allow basic arrays of data.
I've created renderers that do this. You can render out data as fast as your hard drive can write the data but yes these files are often ginormous. RGBA at 1000x1000 at 30FPS is about 7.2Gb per min.
There are raw file formats but because the files are so large they are normally thought of as streaming formats that you would pipe to another process like ffmpeg or gstreamer. If you actually want something saved to disk that other things can read your options are normally MJPEG, APNG, WebM or GIF. Some of the older video formats like AVI or MPEG might also work.
Personally I think animated PNG files are the cleanest simplest format internally, there are arrays of RGBA pixels that are gzipped plus the header data. Because the pixel data is compressed the files are smaller but still huge in comparison to a normal video file.
The answer to your question is also pretty complicated because most of these formats have many different formats that they internally allow, for instance MPEG, AVI and other video formats can be saved with pixel data in different formats. The problem is the specs for these things are often not fully supported by all of the players. To get useable files you will need to be specific about what you are trying to target. The formats that I normally target need to work in a web browser but if you just need it to work on your computer VLC can play almost anything.
I experienced noticeable latency using ffplay so not great for real-time (multiple 100ms, upping the framerate to something like 100-200 reduces the latency but it's stupid). I have much better latency sending the frame data to a browser using websockets and displaying it in a canvas...
While not allowing you to interface arbitrary programs to edit video streams, ffmpeg filter graphs come pretty close to the idea of Unix pipes. You can read more about it here https://ffmpeg.org/ffmpeg-filters.html#Description
Is there a similar format for video, which can be generated on the fly without transcoding or invoking ffmpeg?