Have you thought of inverting the function? It's a one-to-one mapping, so it should be invertible. Use the same algorithm, but pick a place in R,G,B space as the seed location and search in concentric circles (instead of spheres/shells) in the X,Y space.
I'm not certain what you're trying to say. When you talk about "the function," I'm assuming you're referring to the nearest unused colour function, which, while it is a one-to-one mapping is also a) a function from colour 3space to colour 3space (input is a targeted RGB point, output is a valid, unused RGB point) and b) it is also dependent on the state of the colour space i.e. the function is of the form (r0, g0, b0, colourspace) -> (r, g, b) + the colour space is modified to indicate that (r, g, b) has been used.
All of the dealings with the X, Y space (the image) are separate from the colour searching algorithm. There is a function to approximate the target colour of an unfilled pixel in X, Y by averaging the values of its neighbours, but that value is then used as input to the above-described search function.
I could simply be misunderstanding what you're trying to say.
