I don't know about user-end software, but all of that image processing would be fairly simple to do in either Matlab or SciPy. It requires image segmentation (which should be fairly easy in pictures like these with strong edges) and then just running a FFT.
There are some edge-detection algorithms around that are tried and tested. The discrete first and second derivatives are very easy to compute and tend to have very high (or very low) values on edges. Canny Edge Detection is more complicated to implement, but easier to use because it returns the lines themselves, while the derivative operators just amplify the edges, to be picked out with later post-processing (like taking a threshold value). I think there's an online version of the Canny algorithm.
Yes, but you need more than an edge detector. The y-values output by the edge detector might not be a function of x - you could have more than 1 y-value per x, or none at all.
You need a segmentation algorithm to draw a line where the mountains stop and the sky starts, that way y is really a function of x.
