Even in 6th Edition Unix, devices were files in the filesystem as well. And in 6th Edition Unix, you could open and read a directory as a file; that was how you found out what its children were. There was no EISDIR. I believe this behavior persisted until at least SunOS 4.
Adding sockets and windows outside the filesystem is one of the things Rob is specifically complaining about here, and in fact complained about at the time as well.
The philosophy is that everything "looks like" a file, and this achieved through the standard file API (open/read/write/close). AFAIK sockets can be used with this API, as well as directories under Linux. The abstraction leaks quite a little bit sometimes, though.
Unix said that, but in fact meant "everything has a unique descriptor associated (an integer)".
Directory is not a file, it's a set of files and other directories. Unix doesn't even tries to camouflage this simple observation and returns EISDIR.