WYSIWHY is a neural network that transforms an input video into an audio sequence in real time. It does so by compressing each image using an autoencoder and interpreting the resulting code as a frequency range.
This could potentially be useful for the visually impaired, helping with indoor navigation. It could also be used to transform an infrared or ultraviolet video to sound and thus enable one to perceive otherwise invisible colors.
In a usual autoencoder, the elements of the code vector are independent of each other. This makes subtle differences between adjacent elements of the code vector hard to perceive for humans. The resulting audio sequence would sound like indistinguishable white noise. For WYSIWYH thus an encoder is used which hierarchically structures the code, meaning that large differences in the input image result in large differences in the output code. This also results in adjacent elements of the code vector being correlated. This makes the autoencoder’s code more friendly to human perception.
