They did try to replace Objective-c with Java and that showed a lot of areas where a framework built for a dynamic language fails on a static language. I think Objective-C is a decent tradeoff of C and Smalltalk. I like the selector syntax as opposed to the C++ syntax.
They didn't actually try to 'replace' objective-c with java, though. It was just an alternative.
In the early days of Mac OS X, many programmers were seduced by it, but they complained loud and long about having to learn this weird new language that had practically no adoption outside Apple. My guess is that Apple decided to give them java to quell the complaining. It wasn't adopted widely, so that was that.
What they should be replacing objective-c with is something that looks more like, say, Python.
Well, Apple has been funding a lot of the development of MacRuby, possibly as an exploration of what it would take to anoint a popular scripting languages as a "nicer" alternative to Objective-C.
The big push for Java was with WebObjects and Apple's half-hearted push into enterprise systems. Initially Apple's version of Java (they wanted tight integration with Cocoa) was always a couple of revisions behind Sun's so that was always a source of frsutration with developers. With the new focus on consumer devices the enterprise space was left to wither and Java with it. Anything that is going to replace Objective-C must be at least as fast - Apples really cares about the customer experience not the developer experience.
