There are some edge-detection algorithms around that are tried and tested. The discrete first and second derivatives are very easy to compute and tend to have very high (or very low) values on edges. Canny Edge Detection is more complicated to implement, but easier to use because it returns the lines themselves, while the derivative operators just amplify the edges, to be picked out with later post-processing (like taking a threshold value). I think there's an online version of the Canny algorithm.
Yes, but you need more than an edge detector. The y-values output by the edge detector might not be a function of x - you could have more than 1 y-value per x, or none at all.
You need a segmentation algorithm to draw a line where the mountains stop and the sky starts, that way y is really a function of x.
http://en.wikipedia.org/wiki/Discrete_Laplace_operator http://en.wikipedia.org/wiki/Canny_edge_detector