Yep. Clojure is designed around the JVM and libraries written in Java-the-language. If you ported it, by the time you took out the Java-y stuff and added things like TCO and call/cc, it would be a new lisp.
And I think new mid-sized lisps are what we need. If Parrot brings a wave of lisps bigger than Scheme but smaller than CL, designed for Unix and the internet, competing with Clojure and Arc, everyone wins. If they can call each other's libraries, it'll be wonderful.
Tail call optimization is an example of something that Clojure doesn't have because the JVM makes it hard, although it's easy (I'm told) on the .NET system.
My only experience with Clojure is reveiwing Stuart Halloway's books on it, so this is certainly not the most qualified opinion you'll get, but as others have mentioned, tail-call optimization (lack thereof in the JVM, that is) significantly affected the language. It's not possible on the JVM, so Clojure relies heavily on the 'recur' keyword, so that recursive code can be translated into non-recursive bytecode.
Another language feature that could be ignored is the trampoline, which is made easy with the #(foo 'bar %1) notation, but can be ignored if you're not worrying about the JVM.
Aside from that, I would be very interested in seeing a Parrot Scheme based on ideas from Arc and Clojure both.